Frustrated by Magento? Then you’ll love
One of the big changes for Magento 2 developers is the introduction of a formal, “semantic versioning” system to the platform. If you’re a Magento developer and this comes as news to you, don’t feel bad. Although there’s been plenty of talk about the semantic versioning system and some formal documentation, no one’s really sat down and explained or explored the implications of this system for a working Magento developer.
This article won’t answer all your questions, instead it’s the start of a conversation about
- What semantic versioning is in Magento 2
- Why this system is failing third party developers
If you’re the impatient type, semantic versioning is a system that lets third party developers configure which Magento component versions their third party components need. It’s failing because Magento Inc. has not clearly communicated what things are and aren’t covered under semantic versioning, Magento Inc. has not evangelized the feature properly to developers, Magento Inc. is strongly biased towards you using their component hosting to take advantage of semantic versioning, Magento Inc. has failed to provide a comprehensive API that’s semantically versioned, and Magento Inc.’s semantic versioning only covers PHP code.
All of that probably sounded wah wah wah, so lets get to that rarest of commodities: Context.
What is Semantic Versioning
Semantic versioning, or SemVer, is an informal specification (i.e. no standards bodies involved, as far as I know) that attempts to formalize and give meaning to the common versioning practice that gives software numbers like
2.3.4
4.3.2
5.6.23
5.6.24
5.7.0
etc.
In SemVer’s version of the universe, the first number is the “MAJOR” version, the second number is the MINOR version, and the third number is the “PATCH” version.
Where SemVer provides its main value is in what a change to any of those numbers means. When the major version number changes, it means
We’ve broken backwards compatibility with the API
The the minor number changes, it means
We’ve added new functionality, but the API is still backwards compatible
When the patch version changes, it means
We added no new API functionality, but we fixed some bugs in a way that’s backwards compatible with with the old API
All in all, SemVer is a pretty straight forward, if idealistic, idea of what version numbers in a software product mean.
Magento, Versions, and Composer
This brings us to the first failure of Magento’s semantic versioning. Magento Inc., has not, in a definitive way, let the world wide developer community know if semantic versioning applies to individual components, the Magento product/marketing version (2.0.x, 2.1.x), the product meta-package, or some combination of the three.
The product/marketing version is pretty self explanatory, but component and meta-package versions may require additional explanation for developers coming from Magento 1 or new to the platform.
In Magento 1, the atomic particle of software distribution was called a package. This package format was originally based on the PHP pear format, but later forked into its own thing. In very simple terms, a Magento 1 extension was a collection of files and folders installed into your Magento system’s root folder.
In this regard, Magento 2 improves on Magento 1. In Magento 2, the atomic particle of software distribution is something called a component. A component is a collection of PHP files, in a folder, with a file named registration.php in the root. This registration.php file contains code that identifies the collection of files as a Magento Code Module, a Magento Theme, a Magento Code Library, or a Magento Language/Translation Pack.
Magento 2’s code distribution is heavily biased towards using Composer. There’s a 1 to 1 relationship between a Magento component, and a composer package. Most Magento 2 components live in vendor/[vendor-name] — including Magento 2’s core code. Every composer-distributed component has a version number. For example, you can see the version of the magento/module-catalog component/composer package here
//File: vendor/magento/module-catalog/composer.json
//...
"version": "100.0.5",
//...
The 100 is the major version number, the 0 is the minor version number, and the 5 is the patch version number.
Component versions explained, that leaves “meta-package” versions. A “meta-package” is a composer package that lists every package requirement for a major version of Magento 2. For example, the composer meta-package for Magento CE is magento/product-community-edition. The composer.json for a Magento 2 CE system has a require section that looks something like this
//File: ./composer.json
"require": {
"magento/product-community-edition": "2.1.0",
//...
},
The magento/product-community-edition package contains no code — but it does contain a list of component packages and versions that make up Magento 2.1.
Magento Inc. has never explicitly said which of these is under semantic versioning. By convention, it would appears to be only the component versions. For example, in the jump from Magento 2.0.x to Magento 2.1 (both the marketing version, and meta-package version) the magento/module-catalog version jumped from 100.0.5 to 101.0.0. The change in the major version of the component version (100 to 101) indicates a breaking change. However, the marketing and meta-package version only jumped a minor point release (2.0.x to 2.1.x). This indicates new functionality, but no breaking changes.
Magento not having a clear explanation for how all this works is the first way semantic versioning fails Magento developers.
Composer, SemVer, and Third Party Code Distribution
The second way semantic versioning fails Magento developers is similar. There’s no clear explanation of what semantic versioning can do for them. i.e. SemVer — What is it Good For?
Concentrating only on components (putting aside the marketing/meta-package version), as a third party developer, when you call Magento code, you should list that component’s version as a requirement in your component’s composer.json file.
For example, if I created a Magento module that used Magento\Catalog\Model\Product objects, code that’s part of the magento/module-catalog component, I would list that module as a requirement in my module’s composer.json file.
//File: vendor/pulsestorm/some-module/composer.json
"require": {
"magento/module-catalog": "100.0.5",
//...
}
With the above configuration, I’m saying “this module only works with version 100.0.5 of the magento/module-catalog component. I could indicate more than one version using Composer’s version constraints.
The value of this is, if a user (merchant, integrator) attempts to upgrade their system using Magento’s built in upgrade tools (all based on composer), and that upgrade included version 101.0.0 of the magento/module-catalog component/composer package, the upgrade would fail. However — the upgrade would fail safely, because Magento’s upgrade tools would refuse to upgrade the extension.
Put another way, as an extension developer, instead of getting a support email that says
OMG I JUST UPDATED AND YOUR EXTENSION BROKE MY STORE I CAN’T FEED MY KIDS NOW
you’ll end up with a support email that says
Hey, when are you going to release a version of your extension that’s compatible with Magento 2.1
As a merchant, if you’re using extensions that follow semantic versioning, and using Magento’s built-in upgrade tools (and the built in upgrade tools are working correctly), you can proceed with upgrades much more confidently, knowing that your extension vendors have explicitly stated “Our extension works with the new version of Magento”.
Conceptually, as a feature, this is a huge win for everyone. Unfortunately, Magento Inc. has not communicated this message very well, and most distributed components I’ve seen don’t list their Magento dependencies.
The Composer Assumption
This leads into the third failure of semantic versioning, and that’s the composer assumption. It’s fantastic that Magento Inc. has embraced modern PHP package management via composer. What’s less fantastic is semantic versioning benefits are only available if you’re distributing a component via composer.
While composer is the preferred method of distributing Magento components, Magento 2 scans traditional Magento 1 folders like app/code, app/design, lib/internal, and app/i18n for registration.php files. If Magento finds a registration.php file, it will treat the folder as a component. That’s why you can still drop files in app/code/Package/Module and they still work. It’s great, and very important that Magento still has this functionality, but semantic versioning can’t help you here, because semantic versioning is dependent on composer.
In my mind, this would be an acceptable version of progress if there was a straight forward way for independent developers and integrators to distribute their extensions via composer.
While it’s technically possible for a third party developer or a tech savvy agency to do this, Magento Inc. has spent near zero effort documenting or creating tooling around this process. Instead, developers are encouraged to distribute extensions via Magento’s composer based Magento Marketplace.
This is one of those places where technology and business interests intersect, and even Otto von Bismarck would have trouble navigating the waters. However, if we take a realpolitik look at the world, there are (at the time of this writing) four themes on marketplace that have access to the protections of semantic versioning, and (again, at the time of this writing) nineteen themes on Theme Forrest that do not have the protections of semantic versioning. Also, massively popular Magento 1 vendors like Unirgy are skipping Marketplace and distributing via app/code — again, losing the protections of semantic versioning.
A Magento ecosystem where third party developers have straight forward tools for composer hosting would be better for everyone.
SemVer and APIs
The penultimate failure of semantic versioning is “The Magento API”. Here we’re not referring to the HTTP based method (REST, SOAP, etc.) of doing things with a Magento system. Instead we’re referring to the SemVer concept of an API.
Software using Semantic Versioning MUST declare a public API. This API could be declared in the code itself or exist strictly in documentation. However it is done, it should be precise and comprehensive.
Magento 2 does have a SemVer API. Magento 2 uses the @api annotation of PHP Doc Blocks to tag methods that are part of its API. For example, the following method
#File: vendor/magento/framework/DataObject/Copy.php
namespace Magento\Framework\DataObject;
class Copy
{
//...
/**
* Copy data from object|array to object|array containing fields from fieldset matching an aspect.
*
* Contents of $aspect are a field name in target object or array.
* If targetField attribute is not provided - will be used the same name as in the source object or array.
*
* @param string $fieldset
* @param string $aspect
* @param array|\Magento\Framework\DataObject $source
* @param array|\Magento\Framework\DataObject $target
* @param string $root
* @return array|\Magento\Framework\DataObject|null the value of $target
* @throws \InvalidArgumentException
*
* @api
*/
public function copyFieldsetToTarget($fieldset, $aspect, $source, $target, $root = 'global')
{
//...
}
//...
}
is marked with @api. This means the copyFieldsetToTarget of the Magento\Framework\DataObject\Copy class is part of the Magento SemVer API. This class is part of the magento/framework component. The magento/framework component’s version is (at the time of this writing) 100.1.0
//File: vendor/magento/framework/composer.json
//...
"version": "100.1.0",
//...
If Magento changed the copyFieldsetToTarget method to fix a bug, but the method behaved the same way, the next released version of the module would be 100.1.1. That is, they’d increment the patch version number.
If Magento changed copyFieldsetToTarget to add some new functionality (by adding an optional parameter), or marked a new method in the Magento\Framework\DataObject\Copy class with @api, the next released version of the module would be 100.2.0. That is, they’d increment the minor version number.
If Magento changed copyFieldsetToTarget so its behavior changed, even when called with identical parameters, this would be a non-backwards compatible change, and the next version of the module would be 101.0.0. That is, they’d increment the major version number.
As we said, Magento has a SemVer API and this is a good thing. However, where Magento’s SemVer API fails is in the comprehensive requirement. Only a minority of Magento’s methods are covered by @api. Changes to most Magento code will not create a component version change because that code is not covered by an @api annotation.
If a module developer stuck to the methods available in the SemVer API, they’d either have a useless module, or would have to reimplement non-@api functionality in their own code. The later defeats the purpose of adopting a platform like Magento in the first place. Worse, despite having real world feedback (in the form of released Magento 2 extensions), Magento 2.1 did not significantly increase the surface area of the SemVer API.
The practical net negative for third party developers is every Magento version change brings with it the possibility of breaking things. A responsible third party developer can never be ready on day one with an extension, even for a small patch change.
SemVer, Javascript, and DSLs
The final failure of SemVer in Magento 2 is its assumption of a PHP only universe. While better evangelism and a real, comprehensive API would be great improvements to Magento 2, SemVer only covers PHP methods, and there’s a huge component of Magento 2 that’s still programming, but falls outside of PHP.
While Magento 2 has adopted a number of modern javascript approaches, there’s no equivalent to SemVer for Magento’s many RequireJS modules. This leaves third party developers out in the cold when it comes to using core Magento RequireJS modules.
Similarly, the two XML based domain specific languages (DSL) Magento uses to generate layouts (Layout Handle XML files and UI Component XML files), have no concept of SemVer. Magento 2 developers have felt this acutely with Magento 2.1, which introduced backward incompatible changes to the backend UI Components that manage grids and forms.. Developers are faced with either leaving customers on the 2.0.x branch behind (a branch which Magento will support for 2 years), or maintaining two branches for their extensions. The multiple branch approach seems the only way forward, but its a labour intensive effort that puts the ROI of Magento 2 extension development in jeopardy.
Wrap Up
While the relationship between Magento 1 and its third party developer ecosystem was strained at times, the Magento 1 core team operated under an informal “don’t break things” policy that worked well (more or less) for the lifetime of the Magento 1 product. As a Magento 1 extension developer, I always worried about little breaking changes to an extension, but never worried about systems vanishing.
Magento 2, on the theoretical level, formalizes this with a semantic versioning policy. While the theory is sound, in practice the only people currently served well by the semantic versioning policy are the Magento core engineers, who now have something to point at when they break something. As a Magento 2 extension developer, version changes fill me with dread that entire sub-systems may be removed, leaving me with a short RC cycle to replace those sub-systems, or remove functionality for my customers.
This is usually the part where I try to find some small light of hope or improvement to point at, but until Magento Inc. can move beyond optics and addresses the needs of its third party developers, Magento 2 extension development will remain a tenuous, uncertain thing.