Slow Composer Followup

Last week’s Bypassing a slow Composer Repository post shook a few interesting things from the internet underbrush.

Compress all the Things

First, the FireGento folks got together and turned on gzip compression for http://packages.firegento.com/packages.json. This turned the painfully slow two to three minute download into something closer to 4 seconds

composer.phar --profile -vvv create-project --repository-url=http://packages.firegento.com magento-hackathon/magento-composer-installer
[4.2MB/0.02s] Downloading http://packages.firegento.com/packages.json
[70.6MB/3.58s] Writing /Users/alanstorm/.composer/cache/repo/http---packages.firegento.com/packages.json into cache    

It shouldn’t, but it always amazes me that gzip compression isn’t a default everywhere in this day and age — but fear of an old HTTP client being accidentally sent gzip data will probably keep this from ever happening. One more invisible but hugely important thing for us to keep track of during deployments.

A big thanks to the entire FireGento team for jumping on this, and for hosting the packagist repository itself. Public criticism, especially in the world of unclear expectations created by open source software, can sometime seem like an attack. I’m glad they know better than that.

Composer is not HTTP Only

The other interesting thing that came across my radar were a few tweets by Cyrill Schumacher. Cyrill is a developer who lives in Sydney, Australia. As an American who grew up in the 80s, it’s my understanding that packets are delivered to Australia users by Paul Hogan riding a Kangaroo, which makes the internet a little slow down there, which in turns makes developers like Cyrill ever vigilant for ways to reduce their day to day network latency issues.

Cyrill noticed two things. First, Composer is not limited to HTTP repositories. You can actually use a file:/// to point to a packages.json on your local system. If you’re not a fan of absolute file paths Composer also supports a zip:// protocol which allows you to point to a zip archive containing a packages.json file

Here’s an example of the file:// syntax

"repositories": [
    {
        "type": "composer",
        "url": "file:///path/to/packages.json"
    }
],

And the same for the zip:// syntax

"repositories": [
    {
        "type": "composer",
        "url": "zip://some-zip.zip#path/in/archive/to/packages.json"
    }
],

Neither of these options solve the problem of syncing the changes from the real FireGento package to your local copy, but local copies are almost always going to be faster than pulling something off the internet. The file:/// and zip:// options are worth considering if this sort of latency is a bottleneck in your Composer usage.

Sight-reading

Composer is a fantastic project, but like so much open source software today its documentation could use some work. It’s not that Composer is undocumented, it’s that the documentation often assumes familiarity with Composer’s implementation and important-to-newcomers information is often glossed over.

Take the file:// and zip:// protocols mentioned above. The only documentation I could find for these were the following two paragraphs

The packages.json file is loaded using a PHP stream. You can set extra options on that stream using the options parameter. You can set any valid PHP stream context option. See Context options and parameters for more information.

A composer repository is simply a packages.json file served via the network (HTTP, FTP, SSH), that contains a list of composer.json objects with additional dist and/or source information. The packages.json file is loaded using a PHP stream. You can set extra options on that stream using the options parameter.

To the the person who wrote this — saying “loaded using a PHP steam” was enough. Because they were familiar with both PHP streams and how Composer uses them, “loaded using a PHP steam” was the same as “You can use any PHP stream wrapper protocol to load a Composer repository”.

However, for a developer not familiar with Composer’s implementation, it’s not necessarily clear this is the case. Putting aside that steams are a less used feature of PHP, the above documentation doesn’t make it clear if Composer can use all the stream wrappers, or if the implementation is hard coded to a few protocols (HTTP, FTP, SSH, etc). Additionally, given all the examples in the composer documentation use http URLs, there’s a bias towards how a reader will interpret the brief mention of streams.

Wrap Up

Good documentation is more than a rote listing of what a project does, or what order the configuration flags go in. For software systems more complex that the latest mobile app — documentation is the first user interface your users encounter, the first evangelist for your code, and goes a long way towards setting the culture of your project.

And yes — ”the internet” and “user generated content” can go a long way toward filling this gap for you, my career are this article are living proof of this. However, if you have solid evangelical documentation up front, instead of spending their time writing your documentation, that same internet will take you project and do fantastic things with it.