Incremental Migration Scripts in Magento

The short version: To assuage whatever guilt I have about selling commercial software and ebooks in an open source ecosystem, I’ve created and released a new n98-magerun command that allows Magento users to manually and incrementally run Magento’s setup resource migration scripts. Our long international nightmare of opaque Magento upgrades is over.

The next time you need to upgrade your Magento system (to, say, the latest 1.8.1 version), this new command will make the database portion of that upgrade much less scary. Also, if you’ve been looking for a place to get started developing with n98-magerun there’s a number of refinements this command could use, particularly around unit tests. See my comments at the end of the pull request and fork away on GitHub.

As with all my open source work, questions and comments are more than welcome.

Setup Resource Critique

If you’ve read through my original Magento for PHP Developers series, you should be familiar with Magento’s setup resource migration system. In short, these are the scripts that allow module developers to install new database tables whenever their module is installed or upgraded.

While useful, Magento’s implementation of this common pattern is not without its issues. To the surprise of many programmers coming from other frameworks, these setup resource scripts are not wrapped in database transactions. This means if there’s an error state half way though a script, or PHP halts for some other reason, the database could be left in an invalid state preventing future updates from running without direct programmer intervention.

Exacerbating that problem is Magento’s modular system and rich community of third party extensions. In applications worked on by one team, this sort of migration pattern creates a single line of upgrade scripts that are, more or less, easy to follow version to version. However, because each module in Magento has its own setup resource migration scripts, even a simple point release updating of Magento means running multiple scripts in one PHP request. Multiple long running scripts in a single PHP request increases the chances of the request timing out and causing a transactional integrity issue.

Finally, as hinted at above, these scripts don’t run at the prompting of the user. Instead, an uncached Magento request will examine the modules, determine if the scripts need to run, and then run then. The intent behind this is noble — automatic database updates for store owners — but in practice it makes the setup resource system and Magento upgrades opaque, and when things fails no one is quite sure why.

These are more than theoretical concerns — with every new Magento release there’s a spike in questions on the forums and Stack Exchanges about upgrades gone wrong and users unsure how to proceed.

The Solution

Unfortunately, this is a hard problem to completely solve. The weakness of MySQL’s transaction engine and the complicated nature of the problem (how to let thousands of developers safely modify a central database system) means both eBay/Magento Inc. and the community of developers around Magento have chosen to simply deal with the problems rather than solve them.

The idea behind my new sys:setup:incremental command is to make that “dealing with” easier for developers. Developers using n98-magerun can now

1. Analyze a cached Magento system and determine what updates need to run

2. Run each module’s setup resource scripts individually

The main goal of this command was to reduce the opaqueness of the setup resource migration system. By giving you a report of which resources are going to run

+--------------------------------------------------+
Resource Name:             core_setup
For Module:                Mage_Core
Class:                     Mage_Core_Model_Resource_Setup
Current Structure Version: 1.6.0.1
Current Data Version:      1.6.0.1
Configured Version:        1.6.0.4
Structure Files to Run: 
app/code/core/Mage/Core/sql/core_setup/upgrade-1.6.0.1-1.6.0.2.php
app/code/core/Mage/Core/sql/core_setup/upgrade-1.6.0.2-1.6.0.3.php
app/code/core/Mage/Core/sql/core_setup/upgrade-1.6.0.3-1.6.0.4.php

Data Files to Run: 
app/code/core/Mage/Core/sql/core_setup/mysql4-data-upgrade-1.6.0.2-1.6.0.3.php
+--------------------------------------------------+

+--------------------------------------------------+
Resource Name:             eav_setup
For Module:                Mage_Eav
Class:                     Mage_Eav_Model_Entity_Setup
Current Structure Version: 1.6.0.0
Current Data Version:      1.6.0.0
Configured Version:        1.6.0.1
Structure Files to Run: 
app/code/core/Mage/Eav/sql/eav_setup/upgrade-1.6.0.0-1.6.0.1.php

Data Files to Run: 
No files found
+--------------------------------------------------+

and then running though each named resource individually

+--------------------------------------------------+
  Run Structure Updates  
+--------------------------------------------------+                             

All structure updates run before data updates.

The next structure update to run is core_setup
Press Enter to Run this update: 

Structure update core_setup complete.
Ran in 653ms
(1 of 23)

The next structure update to run is eav_setup
Press Enter to Run this update: 

Structure update eav_setup complete.
Ran in 11ms
(2 of 23)

The next structure update to run is admin_setup
Press Enter to Run this update: 

you’ll have a much better idea of what’s going on. Failures may still occur, but you’ll quickly know which resource caused the problem. This means you can confidently move forward with a solution instead of desperately guessing at what’s wrong and playing whack-a-mole with the database layer until things work. Also, by running each script individually, the chances of a script failing due to resource consumption in PHP is greatly reduced.

Wrap Up

While work continues on Magento 2, it’s clear that Magento 1 is still the path forward for merchants who need production ready systems today. Also, if history is any indication, many organizations will stick with Magento 1 for a few years after Magento 2’s release.

With eBay’s focus split and Magento 1 clearly in maintenance mode, it’s up to the community of developers surrounding Magento to improve our own platform experience. Fortunately, despite the growing changes in how businesses approach open source software, we’re still free to alter, extend, and improve the system for ourselves.