I got a lot of feedback on my Magento & Varnish prototype, both after my last week's talk at the Meet Magento 5.11 in Leipzig (see slides) and today after publishing an article explaining the concept, publishing the modules on GitHub and setting up a demo store.
Most of them are "wow, that's cool", others are "have a look at my (commercial) product/service that does the same, (only better)" and some are "interesting, but why didn't you...?"
As Twitter does not feel to be the right medium to discuss things I'll try to answer some questions or respond to your comments here:
Numbers in the speed benchmark
As stated in the original article, you shouldn't take the results of my (and any other's) speed benchmarks too serious. Feeding siege with the urls of the Magento's Google Sitemap means testing only pages that can be cached. To have better results you should test real user behavior including checkout processes, editing personal data in "My Account" and other dynamic processes. If you have an individual product configurator on your page this might be much more interesting than testing how much pages can be delivered from the Varnish Cache.
Besides that siege of course does not simulate the Ajax requests to the server fetching the dynamic parts, that might take longer than the rest of the page (but still less long, than having Magento generate the complete page).
On the other hand this time is still somehow representative, because the user will get most of the content after the first static request and probably won't notice that the menu and the sidebar might slightly change after some more milliseconds.
Why aren't you using ESI instead of Ajax requests?
Varnish can process Edge Side Includes ("ESI", defined by W3C almost 10 years ago). That means that the will be parsed for
I could insert the individual menu and the dynamic blocks in the right sidebar into the layout using ESI includes. The client would not need to make an Ajax request to the server to fetch those dynamic parts.
While Varnish processes those ESI includes very efficiently and fast this method also has some drawbacks:
- ESI has some problems with compressed content from the backend. (Somebody said this might be fixed with the upcoming version 3 of Varnish).
- Varnish is not able to pass a cookie (including the session information) or any other http header data from a part that is included by ESI to the final output. That makes it difficult to create session, that for example is needed for the "recently viewed products" functionality, the checkout or any other user-specific Magento features.
- Processing ESI includes might be fast, but it is still slower than not processing any includes at all. Additionally the parts that might be included will probably be so individual that they would always be passed to Magento without being cached. That will hold Varnish back from delivering the complete page until all ESI requests (that might be more than one) have returned their response.
- Inserting dynamic content via Ajax has the advantage that you can request all parts with one single request instead of bootstrapping Magento for every part like it would be needed using ESI.
- The Ajax request additionally gives you the opportunity to transfer some more information from the client to the server like the id of the product that is currently visited. This information is needed for the "recently viewed products" function.
- Keep it simple, stupid! I wanted to create a simple and lightweight solution without any side effects. Having Varnish doing much more than what it does best (reliably delivering pages, fast) by assembling the final page in a layer between Magento and the clients somehow feels wrong and increases complexity.
- And last but not least: My approach is to deliver most content of most pages as fast as possible. That's why I'd rather have the page send to the browser containing some placeholders (and you might let them look unobtrusive so that the user might not notice that there is something missing there) very fast. The focus of the user anyways might by the main content part that will be already visible after the first request.
I'd like to have this on my shop, what do I need to do?
Basically it is no big deal and everything can be setup within minutes...
...on a default store. But your store probably is very individual!
First you need to analyze all pages from your store and find out which pages aren't static at all (like the checkout process) and which pages are mostly or completely static (like the product single views or the category listing as long as you don't have stock counters or other data on there that needs to be update at every moment). Then replace those dynamic parts as described in my previous blog post. If you need cache content to be invalidated on special events (like stock data) you need to think of a way to send purge commands to Varnish in order to remove them from the Varnish's cache.
Then setup Varnish (and if you need encryption then also setup nginx in front of Varnish). Take a lot of time for testing and adoptions to your Magento store and to the Varnish configuration (you'll find my configuration in the Aoe_Static project on GitHub) and configure Varnish to listen to port 80 and Apache to listen to another port (or bind both to IP-Addresses). Then decide if you want Varnish to also cache some resources (js, css, images,...) keeping in mind that it costs much more to ask Magento for a page that might have been swapped out of the cache than grabbing a file from the file system.
It sounds easy, but it will probably take some time and involves digging into Magento, Varnish and your Browser. Keep an eye on the request headers at every point in time and use the tools provided by Varnish to check if everything works fine.
Since I published my modules on GitHub I got some code contributions. Thanks a lot for that and feel free to dig into my code, send me patches, pull requests or any other suggestions or questions.
Any feedback and contributions are very welcome!
That's what makes my day after spending the nights writing long blog posts :)