Scaling Magento at Copious

Two Fridays ago I attended a mid-day tech talk at Copious, a fast growing agency here in Portland, OR. The talk was called Scaling Magento, but Copious isn’t a Magento agency. Copious is a full service digital strategy agency, and one of the rare few with an engineering department.

Copious recently took on a high profile client with a large Magento system. The presentation never mentioned this client by name (“SEC concerns”), but if you scan through the slides and look at the sort of order volume Copious scaled for, it’s clear this was a merchant with the sort of sales volume that no system could handle out of the box, and a client whose sales volumes (likely) exceed any cloud provider’s annual revenue.

In other words, the perfect case study for a custom scaling effort.

The presentation was split into three sections. First, Aaron Edmonds talked about scaling Magento’s application code. This was followed by an overview of the data center side of the operation from Kyle Terry. Finally, Reid Parham talked about the black art of total project management and delivery: In other words, scaling your team.

While it’s always interesting to hear how folks scale Magento, it was extra interesting to hear how an agency outside the Magento community of partner agencies tackled these challenges, especially given that two separate (unnamed) Magento partners had tried, and failed, to roll out a system for this particular client.

Scale your Code

The first section of the talk was led by Aaron, and covered scaling your code. Much of this section will be familiar to regular readers of this site. Aaron succinctly summed up much of the tribal knowledge that day-to-day Magento developers have internalized.

Two interesting bits of trivia that came up during Aaron’s section were the size of the Magento project Copious was scaling (820,000+ lines of code), and uncovering Magento’s longest class name

class Enterprise_Reward_Block_Adminhtml_Customer_Edit_Tab_Reward_History_Grid_Column_Renderer_Reason

Trying tweeting that in an @ reply.

Less trivially, Copious found success using Aligent’s CacheObserver to selectively cache blocks they knew didn’t need up-to-the-second data.

Aaron also shared a smart bit of Magento reindexing optimization. It turns out Magento will reindex the entire catalog — including products that aren’t currently visible. I can see how CE and EE ship with this behavior on, but Copious tweaked the _getProducts method on the Mage_Catalog_Model_Resource_Url class such that only visible products are reindexed during a catalog URL reindex.

If that doesn’t make sense, consider a concrete example. A shoe, with 9 possible sizes. One configurable product, linking 9 simple products. In a stock Magento system, that’s 10 reindexed products. Now consider a store with 100 similar shoes. That’s a 1,000 products to reindex. By pulling only visible products, (i.e. the simple products don’t need URLs), that’s only 100 products to reindex. Now consider multiple configurable attributes, and you can see how this one small optimization reduced the indexing time dramatically.

Scale your System

The next section of the talk was “scaling your system”. The was led by Kyle, and focused on hardware, deployment, and databases. Regarding hardware, there’s no silver bullet, but if you melt that silver bullet down, and use it to buy a lot of hardware, you can scale Magento pretty successfully.

The slides contain a lot of specific information on hardware setup (yes, physical hardware), so I won’t repeat that here. It’s a tried and true LAMP stack scaling solution, with front-end web servers, master/slave replication on the database layer (no slave lag checkout bugs seen!), and dedicated redis caching servers. Be sure to pay extra attention to the processors – while Copious’s client had a significant hardware budget, smart spending on crazy processors for the database, less crazy processors for the web servers, and budget processors for the caching servers let Copious spread that big budget further than a novice consulting shop might.

It’s also interesting to note they made some minor kernel tweaks to their server OS. Again, a less experienced consulting shop might run out and buy new hardware when their servers hit a certain ceiling, but knowing which kernel setting are adjustable with a recompile can help spread your hardware budget even further.

Although it was only mentioned in brief, for me the most interesting part of this section was Copious’s database integrity tests. Any developer who’s spent some time with Magento knows that sometimes foreign key contraints go awry, or key contraints that should be there aren’t, or extensions and lack of transactions in important parts of the system lead to incomplete, invalid data in the database.

It sounds like Copious knows this, accepts this, and has written tests to detect problem situations. Rather than try to fix a system they don’t completely control (Magento), they’ve used engineering skills to scan for, detect, and alert system developers of these integrity problems so they can be fixed before they cause real problems.

Your own personal Agent Smith, running down problems in the Matrix.

Scale your Team (Reid in Repose)

The final section of the talk, led by Reid, was less technical, and more focused on the problems of managing teams on a large project that involves both client deliverables and a custom engineering effort.

That I work alone and independently should tell you everything you need to know about my project management strengths and weaknesses, so there was a lot to digest in this portion of the talk.

One part of this section that stood out for me was the number of vendors involved in the project. There were 10 systems level vendors (ERP, tax calculation, Akami, etc.), and over 20 different product/accessory vendors to deal with. This is one of those hidden challenges to online retail work — because the industry relies so heavily on specific vendors for solutions, a certain project complexity threshold is quickly reached.

For Copious, this meant gaining the trust of their client such that Copious could act as the client with these vendors. As a corollary to Copious’s experience, I’d remind solo developers (or even small teams) that it’s important to get the full vendor picture before agreeing to a project, and making sure you’re not taking on the responsibility of full vendor management, but that it is being taken care of.

It’s also encouraging to hear that, despite working for a top secret client with SEC concerns and the sort of secrecy that’s becoming de-rigor in global business, Copious was able to contribute back to the open source ecosystem. Patches were submitted to Colin’s Redit modules (now officially sanctioned by Magento), as well as Redis itself.

Wrap Up

The Web™ has come a long way since 1995. Hosted services like Blogger, Tumblr, Square Space, Shopify, Etsy, etc. means the average web user doesn’t need to think about the technical details of their web presence. While amazing, these steps forward have come at the cost of hiding the incredible technology complexity of handling large scale web systems. When a businesses needs to grow beyond what a cloud provider can offer, the easy solutions are few and far between.

This, more than anything, seems like the future for software service professionals: Learning how to scale systems like cloud providers, and setting client expectations that this is a necessary step. Like everything in business, it’s all about finding partners you can trust, where each party can contribute to the success of the other.