Magento 2: Composer, Marketplace, and Satis

As Magento 2 approaches its first half-birthday, one thing is clear: Magento 2 is leaning heavily on PHP Composer for its developer workflow, and for the merchant facing Marketplace.

If you’re a developer working with Magento, you may be familiar with repo.magento.com. This is Magento 2’s composer repository, and up until Magento Imagine 2016, its main purpose was to provide modern PHP developers with a way to install Magento 2 using PHP composer.

At Imagine 2016 Magento unveiled their Magento Connect replacement, Magento Marketplace. Behind the scenes, Marketplace is running on PHP Composer, and repo.magento.com is the source repository for purchased extensions.

In this article, we’re going to show you how you can mirror your Magento specific composer packages on a local server. While not necessary, this is a useful precaution to take if you want to avoid any unscheduled maintenance bringing down your deployment and development pipelines. Along the way, we’ll also discuss Marketplace’s new composer based architecture, and end up touching on many lesser known composer features.

Understanding Composer

If you’re interested in an in-depth look at how composer works, my Laravel, Composer, and the State of Autoloading article is still a great starting point. If you don’t have time for that right now, here’s the basics.

Default Composer Behavior

As a developer, if you want to install a PHP package, you say

Hey, composer, get me the source files for the foo/bar package

i.e.

$ composer require foo/bar

When you do this, behind the scenes, composer says

Hey, Packagist, where can I find the foo/bar package?

In turn, Packagist answers

The package you want is at [THIS Git/SVN/Mercurial URL] and I found an archive at [THIS URL]

Packagist is a composer repository. One thing that catches a few composer newcomers by surprise is packagist, and other composer repositories, don’t actually host any packages. They just point to the location of a package on another server.

Other Repositories

Packagist support is baked into composer. However, it’s possible to point composer at other Composer repositories with a composer.json configuration that looks something like this

//File: composer.json    
{
    //...
    "repositories": [
        {
            "type": "composer",
            "url": "https://repo.magento.com/"
        }
    ],
    //...        
}

So, in reality, this

Hey, Packagist, where can I find the foo/bar package

Has a preamble of

“Hmm”, composer said to itself “Do I have any custom repositories configured? If so, I should ask them first for the location of the foo/bar package”

Magento’s Composer Repository

Magento Inc. hosts a composer repository at the URL https://repo.magento.com/. Magento developers were introduced to this repository via the composer meta-package installer. This installation method fetches and installs Magento’s source code via Composer. Notice the --repository-url flag in the command

$ composer create-project --repository-url=https://repo.magento.com/ magento/project-community-edition <installation directory name>

This flag ensures the magento/project-community-edition package is downloaded from repo.magento.com and not packagist.

The meta-package method downloads and installs Magento components (modules, themes, language packs, and libraries) into the vendor/magento/ folder. Thanks to Magento’s registration.php file and PSR-4 support, there’s no longer a strictly defined place where Magento modules, themes, language packs and libraries need to be located. These components can now be separated out into individual composer packages.

One thing that caught a lot of Magento developers off guard about this meta-package was the requirement for set of composer auth.json credentials, issued by Magento’s main website. It didn’t quite make sense that Magento would create this authenticated method for installing Magento CE when the source code was already publicly available via GitHub.

The key to understanding Magento’s need for credentials is Magento Marketplace.

Magento Marketplace

Magento Marketplace is Magento 2’s replacement for the old Magento Connect. Magento Connect only hosted free Magento modules — commercial module listings pointed off towards independent software vendor’s websites where you could purchase the extension or service directly from the independent vendor.

Magento Marketplace changes that. Marketplace has free extensions available, but it also provides a one-stop shop for purchasing your Magento extensions. Once purchased, you can download the extension package from the My Account section of Magento’s website.

In addition to this download, the extension will also be available via the Magento 2 Component Manager in Magento’s backend, at System -> Web Setup Wizard. The Component Manager is a GUI for installing composer packages from repo.magento.com. It turns out that repo.magento.com is a session-ed packagist repository.

When you log in to repo.magento.com using the aforementioned HTTP Auth credentials, Magento’s composer repository returns a custom list of packages for you to install. This will include

1. The base Magento 2 CE packages
2. Any Magento 2 EE versions your account has access to
3. Any package you’ve purchased

This is what enables you to fetch purchased Magento Marketplace packages via Component Manager. You can also simply add new packages to your composer.json file and then update your system via the command line (i.e. composer update)

Mirroring repo.magento.com

While Magento’s adoption of composer is welcomed, it does add an additional wrinkle to deploying Magento 2 projects. The repo.magento.com repository is a new, single point of failure that, when down, could block a composer based deployment from being updated, or prevent your development team from getting started on a new project. Additionally, unlike the other (still not great) single points of failure in a composer project (packagist.org, github.com, etc.), repo.magento.com isn’t yet a battle tested system.

Because of this, it makes sense to create a local mirror of repo.magento.com. In addition to protecting you from unplanned maintenance/downtime at repo.magento.com, hosting your Magento packages on your local network (or even local development machine) should speed up Magento deployment tremendously.

Mirroring with Satis

The composer project has a second, sibling project called satis. Satis was created to allow developers to create their own local mirrors of packagist.org content. It turns out a stock composer repository requires zero dynamic processing — a repository is just a collection of static json files, and (optionally) mirrored archive packages.

To use satis, you’ll need to clone the GitHub repository to your local development machine with one of the following commands

git clone https://github.com/composer/satis
git clone git@github.com:composer/satis.git

Once cloned, you can run the satis command with php bin/satis

$ php bin/satis 
Satis version 1.0.0-dev

Usage:
 command [options] [arguments]

Options:
 --help (-h)           Display this help message
 --quiet (-q)          Do not output any message
 --verbose (-v|vv|vvv) Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug
 --version (-V)        Display this application version
 --ansi                Force ANSI output
 --no-ansi             Disable ANSI output
 --no-interaction (-n) Do not ask any interactive question

Available commands:
 add     Add repository URL to satis JSON file
 build   Builds a composer repository out of a json file
 help    Displays help for a command
 init    Initialize Satis configuration file
 list    Lists commands
 purge   Purge packages

Satis needs two things to build a repository mirror.

1. A JSON configuration file
2. An output folder

Satis will output packages.json files and (optionally) package archives to the output folder you specify. You’ll be able to upload these files to any simple web server, and have a working packagist repository.

The JSON configuration file is where you tell satis which repositories you’d like to mirror, which packages in the repositories, as well as any other satis configuration needed.

The syntax for building a satis mirror is

$ php bin/satis build config.json build-folder

The file we’ve named config.json above is often named satis.json by convention.

Satis Configuration

Next up, we’re going to create a satis configuration file and review its options. While the composer manual has a section on satis, the file format isn’t as well documented as it could be. The following is one possible configuration, please get in touch or comment below if you see something egregiously wrong here — I’m learning with the rest of you here!

#File: satis.json
{
    "name": "Pulse Storm mirror of repo.magento.com",
    "homepage": "http://composer.pulsestorm.dev",
    "repositories": [
        { 
            "type": "composer", 
            "url": "https://repo.magento.com" 
        }
    ],
    "require-dependencies": true,
    "require-dev-dependencies": true, 
    "require-all": true,   
    "archive": {
        "directory": "dist",
        "format": "zip",
        "prefix-url": "http://composer.pulsestorm.dev",
        "skip-dev": false
    }    
}

The name property should be a simple description of your repository. It’s used in a static HTML file generated for the repository, so don’t use a name that would embarrass your mother or your supervisor.

The homepage property is the URL you’re planning on hosting your satis repository at. This can be a URL on the Internet, or a URL local to your network/dev machine. This URL will be used in a static HTML file generated for the repository, so make sure its accurate.

The repositories property is the first vital configuration field. This should be an array of the composer repositories you’d like to mirror with satis. In our case, this is the composer repository at repo.magento.com. Other types of repositories you might see are VCS or pear.

The require-dependencies and require-dev-dependencies flags make sure composer will require any dependencies a specific package may have. Not strictly necessary in our case (see require-all below) but it never hurts to be explicit.

The require-all property tells satis we want to grab every package on the repository. Since our goal is to create a complete local mirror, we want this here. If we were interested in creating a mirror of select packages, we’d use a require property and list out the packages.

The final top level archive property tells satis that, in addition to creating a packages.json file for us that points to packages, we’d like satis to grab and/or build a project archive for us as well.

In the archive object configuration, the directory property tells satis where the archived files should be copied to (i.e. build-folder/dist), the format property tells satis which archive format we should use, the prefix-url should be, again, the URL for your repository, and setting the skip-dev option to false ensures we get every file in a package.

The prefix-url key is important here, as it’s the URL satis will use in the generated packages.json file to point to our local mirrored archive.

If you’re building for repo.magento.com, you’ll want to keep the archive format set to zip. There are some bugs and inconsistent behavior with Magento 2’s repository and composer that were only just recently fixed for zip archives, and still exist for tar archives.

Building the Mirror

Once you’ve created your satis.json, you can create the mirror with the following

$ php bin/satis build satis.json public -vvv

If your repo.magento.com HTTP credentials aren’t stored in ~/.composer/auth.json, satis will prompt you for them. The -vvv flag is optional, but will ensure satis is verbose in its output. This can help point to problems in your configuration, or with your network connection.

When the command finishes running, you’ll want to upload the files in public to the web root of whatever URL you configured in satis.json (composer.pulsestorm.dev in our case).

When satis is done running, you can take a look at the repository files using the unix find command

$ find public -type f
public/dist/auctane-api-2.0.6.zip
public/dist/belvg-module-facebookfree-1.0.1.zip
public/dist/magento-composer-1.0.2.zip
public/dist/magento-data-migration-tool-2.0.0.zip
//...
public/dist/magento-updater-10.0.0.zip
public/include/all$e89e3a381f6a71df66912bf26c12b89db1200cd8.json
public/index.html
public/packages.json

You should see archives of all the community edition files, any EE files your account license grants you access to, and any extension files you’ve purchased. Upload the entire contents of the public folder to your web server, and you’ll have a local composer repository up and running.

With our mirror created, we’re ready to install Magento 2.

Installing Magento 2 with our Mirror

Normally, when installing Magento via composer, you use the following command

composer create-project --repository-url=https://repo.magento.com/ magento/project-community-edition <installation directory name>

We’re going to change this to

composer create-project --no-install --repository-url=http://composer.pulsestorm.dev/ magento/project-community-edition <installation directory name>

i.e. — the same command, but with our repository, and the no-install flag. To understand why we’re doing this, we need to talk about what the create-project command does.

Composer: Create Project

The create-project command is a shortcut command. When you use create-project, composer will

1. Fetch the latest version of the specified package (magento/project-community-edition) and extract it to the <installation directory name> folder
2. Run composer install from the <installation directory name> folder, installing all dependencies

If you’ve done any work in the modern PHP world, you’re probably familiar with this command from the various framework’s one-line installers.

This presents a problem for our mirror. If you unzip any of the magento/project-community-edition packages archived in our mirror.

$ ls public/dist/magento-project-community-edition-2.*
public/dist/magento-project-community-edition-2.0.0.zip
public/dist/magento-project-community-edition-2.0.1.zip
public/dist/magento-project-community-edition-2.0.2.zip
public/dist/magento-project-community-edition-2.0.3.zip
public/dist/magento-project-community-edition-2.0.4.zip

You’ll see that its composer.json file points to repo.magento.com. The --repository-url option only applies to the package that create-project grabs. Otherwise, composer will use whatever it finds in the project’s base composer.json file.

This is easy enough to work around — all we need to do is use the no-install flag — this way create-project will only download and extract the project, it won’t run composer install. This will give us a chance to edit the composer.json file before running install.

Installing Magento 2

OK! We’re ready. Step 1, lets clear our composer cache

$ composer clear-cache

This is not strictly necessary, but useful when you’re first setting up a mirror, and may have an invalid package reference stashed in cache somewhere.

Also, if your mirror’s not located on an https server, you may need to set composer’s global secure-http flag to false.

$ composer global config secure-http false

Recent versions of composer will refuse to run over non-encrypted HTTP.

Next, let’s run our create project command

$ composer create-project -vvv --no-install --repository-url=http://composer.pulsestorm.dev/ magento/project-community-edition
//...
Downloading http://composer.pulsestorm.dev/packages.json
Writing /Users/username/.composer/cache/repo/http---composer.pulsestorm.dev/packages.json into cache
Reading /Users/username/.composer/cache/repo/http---composer.pulsestorm.dev/include-all$e89e3a381f6a71df66912bf26c12b89db1200cd8.json from cache
Installing magento/project-community-edition (2.0.4)
  - Installing magento/project-community-edition (2.0.4)
Downloading http://composer.pulsestorm.dev/dist/magento-project-community-edition-2.0.4.zip
    Downloading: 100%         
Writing /Users/username/.composer/cache/files/magento/project-community-edition/648f32cf7a59f92769940d85435cf16d7385fa5f.zip into cache from /path/to/magento/project-community-edition//de123c8bffdb844a4093d235a37b66d3.zip
    Extracting archive
Executing command (CWD): unzip '/path/to/magento/project-community-edition//de123c8bffdb844a4093d235a37b66d3.zip' -d '/path/to/magento/vendor/composer/847ae1bb' && chmod -R u+w '/path/to/magento/vendor/composer/847ae1bb'

//...

Again, the -vvvs are optional, but viewing composer’s verbose output can help us ensure that no package was/is downloaded from repo.magento.com. When composer’s done running, change into the project-community-edition folder, and take a look at composer.json

$ cd project-community-edition
$ ls
README.md    composer.json    update
$ cat composer.json
{
    //...
    "repositories": [
        {
            "type": "composer",
            "url": "https://repo.magento.com/"
        }
    ],
    //...        
}

As mentioned, there’s repo.magento.com, getting in the way of our local mirror. Let’s edit that file to point at our repository.

#File: composer.json
//...
    "repositories": [
        {
            "type": "composer",
            "url": "http://composer.pulsestorm.dev/"
        }
    ],    
//...

With the above in place, run

$ composer install -vvv

and composer will grab all Magento’s packages from your local mirror — no access to repo.magento.com required.

Wrap Up

All of this, of course, is only a start to keeping and maintaining a local composer mirror. EE users will want to check their enterprise license agreement to make sure doing this falls within acceptable use of the EE source code, and regardless of which version of Magento you’re running you’ll want to make sure your mirror isn’t located anywhere online, as you may become an inadvertent distribution point for commercial extensions you don’t have the right to distribute.

You’ll also want to figure out a way to get satis running on a regular basis — otherwise you may miss important updates to the Magento repository. The very bold may want to expand their mirroring to packagist itself, but that’s a larger problem filled with all sorts of blind alleys and large hard drives.