Magento is slow!
Whatever module you'll use to improve caching, as long as Magento is involved - even if it is only for determining what data to load from its cache - it will be slow. There are some nice fullpage cache solutions around, but still: slow (at least compared to what can be achieved if you can avoid Magento for many pages completely).
When having a closer look to the pages that are delivered by Magento we can easily see that there is a huge percentage of pages that are almost the same for all users.
So let's have a look at a product listing page and spot the areas that my contain some user specific information:
So the dynamic parts in this Magento Demo Store are
- the menu in the top left corner (the third menu item is showing how many cart items are in the cart),
- the cart in the right sidebar
- and the recently viewed products.
Everything else on this page is pretty static and does not differ from user to user. So this page is a candidate for being cached.
The pages (Magento actions) delivering "almost static" informations are
- cms_index_index (the front page),
- cms_page_view (other cms pages),
- catalog_product_view (the product single views) and
- catalog_category_view (the product listings).
Of course these are store specific assumptions and might differ in your store!
A rough estimation is that 90% of all pages delivered from a Magento instance are being processed by one of those actions.
These pages are candidates for being cached by a strong cache mechanism allowing us to deliver these pages way much faster than if they had to be re-generated first or if Magento had to read them from an internal cache.
We could also generate static html files for those pages and deliver them instead (actually this is what I'm doing with this blog using TYPO3 and the extension nc_staticfilecache) but we'll use Varnish for this purpose which comes with some additional handy features and might be even faster then a common webserver delivering static html files.
- a http accelerator
- a load balancer
- a failover system and
- a caching reverse proxy
that claims to "make websites fly".
So we put Varnish in front of Magento (Apache/Mysql/PHP) deciding whether to deliver a previously cached page or whether to ask pass the request to Magento.
(You might consider to put ngnix in front of Varnish to handle compression and encryption. With the upcoming Varnish 3 these issues might be handled directly from Varnish itself.)
Varnish comes with an own configuration language "VCL" that will be translated into C code and then compiled. This makes Varnish extremely fast.
With VCL you can define subroutines that are executed at some defined parts within the request/response process offering you different objects to realize your very custom behaviour.
Here is a simplyfied overwiew of the process and the subroutines. (Check the Varnish documentation for a detailed diagram)
How does it work?
My idea was to replace all dynamic parts of a page by static placeholders.
<layout> <default> <reference name="right"> <block type="core/text" name="ph_cart" before="-"> <action method="setText"><param><![CDATA[ <div id="cart_sidebar" class="placeholder" rel="cart_sidebar">Placeholder Cart</div> ]]></param></action> </block> <action method="unsetChild"><param>cart_sidebar</param></action> </reference> </default> </layout>
By unsetting them instead of removing them they are still available and can be rendered and delivered by a dedicated controller.
So I created a little extension "Aoe_Static". You can configure what actions might deliver cached content and the extension takes care of sending HTTP headers with information on the lifetime and a flag allowing Varnish to cache the delivered page. Future requests to this page won't be passed to Apache but delivered directly from Varnish in much less time. At least for the time period the page's lifetime hasn't expired yet.
Besides that additional information can be passed to the server like the id of the currently viewed product, that is needed for the "recently viewed product" page.
And on the other hand the server can transfer additional user-specific content to the browser like the session id, that is stored in a cookie to maintain the session for future requests.
This is how a dynamic request will be processed. Note that most of the content (the static part) is already visible earlier (see gray bar). Delivering the static part is much faster than asking Magento, because in most cases it should come from Varnish's cache. And even the rendering of the dynamic parts should be faster, because Magento only needs to render these blocks, while ignoring the rest of the layout completely.
There are still some page requests that deliver very user-specific content. The cart, the checkout process and "My Account" are some examples. All cases not explicitly defined as cacheable in the Aoe_Static extension (see System -> Configuration -> System -> Varnish Configuration) Varnish will pass the request to Magento and will not cache any results. As I'm using a whitelist here your Magento Store should not break, even with some custom controllers. An uncached page request would look like this:
I did some speed benchmarks, basically the same way that the Magento Speed Test does it: I've created a Google Sitemap, extracted the urls and tested the speed using Siege. And here are the impressive results:
I have to admit, tests like these create no really significant test result numbers. But still you can see the tendency.
Siege tests only urls found in the sitemap. And all those urls are cacheable ones. Besides that I did a cache warming before running the test. So this test result show only cache hits in the "Magento with Varnish" testcase. But still: this is what you could have if your Varnish is configured correctly after some time. And: Search engines do not go through your checkout process or log into to a customer account and - you have to admit - a big number of real users won't do that neither.
So: Most of your visitors will see most (or even all) pages 200 times fast with Varnish!
Have a look at Firebug and compare times for a page delivered by Varnish's Cache (16ms) and for a page delivered by Magento's Cache (776ms). Even the fastest fullpage cache solution realized within Magento won't be able the bootstrap Magento, pick to correct page from its cache and deliver it to the browser in less than 16 ms!
I've set up a Magento Demo Store with Aoe_Static and Aoe_StaticDemo (which replaces the menu, the cart in the sidebar and the "recently viewed products" by placeholders) and configured Varnish:
Feel free to play around at
Make your store fly
You'll find the Varnish configuration included in Aoe_Static.
Any feedback is welcome! :)
- Magento & Varnish: Ajax vs. ESI
- Meet Magento #5 2011: Performance durch Caching
- Magento Caching Internals
- Magento Automatic Cache Cleaner
- Magento Asynchronous Cache
- Magento Two-Level-Caching