Composer vs. History

Composer has been a boon for anyone working with PHP. The usefulness in being able to go from

Hey, I want to use code in someone’s library

and then run a single command to get that library

% composer require someones/library

can’t be overstated.

However — one area where composer’s uptake has been less smooth is with frameworks and platforms that predate composer’s existence. Prior to composer, PHP code was passed around in archive files (.zip, .tar.gz, etc.). As a user you’d unzip the archive and have a pile of files, including a folder to expose as your web-root. With the better frameworks there was a clear separation between files that belonged to the framework, and folders where you could add your own files.

Composer complicates things for these frameworks. While composer’s great for letting users get “the files that belong to the framework” into a vendor folder, it’s less great at providing an area where users can add their own code. It’s also not great at providing users with a public web-root or bootstrap (index.php) file.

Some projects with a long history and their own gravitational pull (looking at you WordPress) still haven’t fully embraced composer. Instead the communities behind these platforms do all sorts of ad-hoc things to get composer running. Modern projects that came up along side composer seem to have settled on the composer create-project command to solve this problem. In the middle there’s a — variety? — of different solutions that combine manual instructions and composer’s various attempts to solve this with plugins and installers.

In this article we’re going to look at how the Shopware e-commerce platform uses composer and what a working PHP developer will need to know if they want to go beyond the “hello dev setup” instructions and understand what their Shopware development environments are actually doing.

Getting Started

At the end of the day, any PHP system deployed to a server or development machine will have a web-root folder served up by a web server, and then one-or-many PHP files.

With that in mind, let’s look at the code we’re installing when we run through Shopware’s “production repo” installation. In order to deploy to production, Shopware asks that you

1. Clone the https://github.com/shopware/production repository
2. Run composer install on that repository

This shopware/production repo includes your public folder with a single PHP bootstrap file.

So, right off the bat, one thing to be aware of is that everything you clone out of of https://github.com/shopware/production is your responsibility to keep updated. This will mean merging in changes via git when Shopware updates https://github.com/shopware/production. It will also mean keeping the composer.json file you install up to date and clean. If you install packages via composer and those packages have dependency conflicts with updated packages from the shopware dependencies, it’s on you to fix that.

My general advice would be to, as much as you can, avoid changing or adding files to your clone of this repository.

Shopware’s Dependencies

If we take a look at the packages listed in this production template, we see the following.

"require": {
    "php": "^7.4 || ^8.0",
    "composer-runtime-api": "^2.0",
    "shopware/core": "~v6.4.0",
    "shopware/administration": "~v6.4.0",
    "shopware/storefront": "~v6.4.0"
    "shopware/elasticsearch": "~v6.4.0",
    "shopware/recovery": "~v6.4.0",
},

The ^7.4 || ^8.0 dependency is related to your version of PHP and not a specific package. The composer-runtime-api dependency is another special one: it forces you to use Composer 2 (vs. Composer 1) with this repository. Outside of these two requirements, the remaining five packages are all Shopware code.

shopware/core
shopware/administration
shopware/storefront
shopware/elasticsearch
shopware/recovery

If we look at at these packages on packagist.org.

• https://packagist.org/packages/shopware/core
• https://packagist.org/packages/shopware/administration
• https://packagist.org/packages/shopware/storefron
• https://packagist.org/packages/shopware/elasticsearch
• https://packagist.org/packages/shopware/recovery

we can link these up with the following GitHub repositories.

• http://github.com/shopware/core
• https://github.com/shopware/administration
• https://github.com/shopware/storefront
• https://github.com/shopware/elasticsearch
• https://github.com/shopware/recovery

(A tip of the hat to Shopware for keeping their package names and the repo names the same)

Interestingly, most of these GitHub repositories are marked as read-only.

While these are the repositories that composer will use to fetch your Shopware source, they’re not the repositories where Shopware works on these project.

Instead, shopware publishes their PHP code to these repositories. Actual development happens elsewhere. Specifically, it happens in the shopware/platform repository. This is a mono-repository that contains code for all the shopeware/* packages mentioned above.

If you’re curious how this code is published, these two tweets from the Shopware folks might be of interest.

some dark GitLab pipeline magic 😃
the code is here https://t.co/7D1QV6IJTU

don't ask which place exactly 🙈😁

— Michael T (@mitelg) May 7, 2021

We use https://t.co/ZQVraGrG2k . I can also recommend https://t.co/KxQ5TNKi55

— Shyim (@Shyim97) May 7, 2021

Development Template

While the shopware/production repository is how Shopware wants you to deploy your Shopware systems, it’s not intended for use by developers working on plugins or developers who want to work on the Shopware framework code itself. Instead, there’s a shopware/development template project.

If we take a look at this project’s requirements

"require": {
    "php": "^7.4 || ^8.0",
    "composer-runtime-api": "^2.0",
    "shopware/platform": "6.4.x@dev || dev-trunk"
},

we see that it’s required the shopware/platform package instead of the individual packages. However, it’s not be 100% clear how we’re meant to use this to work on the shopware/platform code. If this code is sitting in vendor, how can we edit it or how can we make PRs into the shopware/platform repo?

The key to understanding this is in the installation instructions, and composer path repositories.

Confusingly, the quick-start instructions for using the development template don’t live in the shopware/development repository. Instead they live in the shopware/platform repository. We’ll quote the relevant bits here

Only if you want to work with the Shopware platform code itself, e.g. in order to create a pull request for it, you should clone the platform code manually. Before doing so, empty the existing platform directory.

% rm platform/.gitkeep
% git clone git@github.com:shopware/platform
% git checkout @ platform/.gitkeep

So the quick start instructions tell you to clone the platform repository into your clone of the development repository — but how will PHP know to look here for the platform code instead of in your vendor folder? That’s where path repositories come in. If we look at the development template’s composer.json file again, we’ll see this section.

"repositories": [
    /* ... */
    {
        "type": "path",
        "url": "platform",
        "options": {
            "symlink": true
        }
    }
],

This path repository tells composer that the code in ./platform, if it exists, is a path repository. This means composer will look at ./platform/composer.json, see that it contains a shopware/platform repository, and prefer this version over the version hosted at packagist.org. (If you’re curious how path repositories work, we wrote up a primer specifically for this.)

There’s two other path repository configurations to be aware of in this development template. The first is here

"repositories": [
    {
        "type": "path",
        "url": "custom/plugins/*",
        "options": {
            "symlink": true
        }
    },
    /* ... */
],

and the second is here

"repositories": [
    // ...
    {
        "type": "path",
        "url": "custom/plugins/*/packages/*",
        "options": {
            "symlink": true
        }
    },
],

These path repositories also exist in the production template as well, and appear to turn every plugin into a single package composer repository. They also turn folders in the packages sub-folder into composer repositories.

It’s a little unclear how these folder are intended work or be used, and is a perfect example of the challenge pre-composer systems face when integrating composer and getting it to work along side their old “put the code that’s not ours into this folder” systems. There’s some nuance to how Shopware’s Extension Store downloads plugins to these folders and how those plugin’s dependencies interact with the main system that’s not clear to folks not already bought-in to the Shopware system.

IMPORTANT: If you’re going to take advantage of these path repositories to develop packages that also exist in packagist.org, your files will need to be in place before you run composer install or composer update. The generated composer.lock file will become the source of truth for where a package lives. If you add a new local folder that’s a path repository, you’ll need to delete and regenerate your composer.lock file.

Installing via Docker

The final thing to understand about this development template is that it’s not intended (or at least, not designed) to be used without docker. The quick start instructions never asks you to run composer install or composer update — all you need to do is run

./psh.phar docker:start

and then shell into the running container with

./psh.phar docker:ssh

and, inside the container, run the installer

./psh.phar docker:install

This uses Shopware’s Shell Runner to run the docker:start command. If we look at our .psh.yaml.dist file

#File: .psh.yaml.dist
environments:
  docker:
    paths:
      - "dev-ops/docker/actions"

We can see the docker: command namespace loads its scripts from the dev-ops/docker/actions folder, which means your docker:start command is here

% cat ./dev-ops/docker/actions/start.sh
#!/usr/bin/env bash

# check that the ~/.npm and ~/.composer folder are not owned by the root user
dev-ops/docker/scripts/check_permissions.sh

# Start docker-sync
if [ -n "__DOCKER_SYNC_ENABLED__" ]; then docker-sync start && echo "\n docker-sync is initially indexing files. It may take some minutes, until code changes take effect"; fi

docker-compose build --parallel && docker-compose up -d

If you’re familiar with docker and docker-compose this is a relatively straight forward script. Docker runs container images, which give you isolated instances of a piece of software that acts like a VM (but is implemented very differently). The docker-compose command is a way to build networks of multiple docker containers that users can start-up at once. The

% docker-compose build --parallel && docker-compose up -d

invocation will use the docker-compose.yml file to build your container images, and then run them all. It’s beyond the scope of this article to cover these in depth, but it looks like you’ll get

1. An app server that runs PHP and a webserver (image: shopware/development)
2. A MySQL server, built using a custom Dockerfile
3. A cypress host, presumably part of the continuous integration/testing framework(s) (image: cypress/included)
4. A mailhog host, presumably for testing system generated email functionality (image: mailhog/mailhog)
5. An adminer host for a MySQL UI (image: adminer)
6. An elasticsearch host, presumably for search features (image: elastic/elasticsearch)

The docker:ssh command is not actually using ssh

% cat ./dev-ops/docker/actions/ssh.sh
#!/usr/bin/env bash

TTY: docker exec -i --env COLUMNS=`tput cols` --env LINES=`tput lines` -u __USERKEY__ -t __APP_ID__ bash

This is a common docker technique to “shell in” to a running container image. Docker will look the container tagged with __APP_ID__ and run the bash command on that container. __APP_ID__ is psh.phar placeholder, defined here

.psh.yaml.dist
61:  APP_ID: docker-compose ps -q app_server

which will return the container ID of the app_server. So this behaves similarly to ssh (it gets you a shell on your running docker image) but doesn’t actually use ssh

Finally, from within the container, when you run psh.phar docker:install you’re running the following shell script.

% cat ./dev-ops/common/actions/install.sh

#!/usr/bin/env bash
#DESCRIPTION: Everything from "init" + demo data generation + administration build/deploy

# generate default SSL private/public key
php dev-ops/generate_ssl.php

INCLUDE: ./cache.sh
INCLUDE: ./init-composer.sh
INCLUDE: ./init-database.sh
INCLUDE: ./init-shopware.sh
INCLUDE: ./init-test-databases.sh
INCLUDE: ./demo-data.sh
INCLUDE: ../../administration/actions/init.sh
INCLUDE: ../../storefront/actions/init.sh
bin/console assets:install
bin/console theme:compile

The INCLUDE lines are special psh.phar directives that will include other bash scripts. As you can see, installing the Shopware development environment without Docker is no simple task. If we peek at the init-composer.sh script

% cat ./dev-ops/common/actions/init-composer.sh
#!/usr/bin/env bash

rm -rf vendor/shopware
rm -rf composer.lock
rm -rf dev-ops/analyze/vendor

composer update --no-interaction --optimize-autoloader --no-suggest --no-scripts
composer install --no-interaction --optimize-autoloader --no-suggest --working-dir=dev-ops/analyze

we see that not only are the main composer dependencies updated, but that there’s a second composer based application in dev-ops/analyze to install. We also see that this script automatically deletes composer.lock — likely to ensure that any new path repositories are automatically picked up.

If all of this seems like a lot — you’re not wrong, but Shopware’s not alone in this environmental complexity. The days of just setting up PHP, a web server, a database, and getting to work are long over.

Wrap Up

Way back in the day, when software was passed around in tar.gz archives, one of my breakthrough lessons was learning the “magic trinity” of

$ ./configure
$ make
$ make install

No matter how banana-cakes C programmers got with their programs, you could always rely on this invocation working. It was a cultural expectation that your software would work when someone input the above.

PHP feels like it’s regressing here. In the PHP4/PHP5 days there was an expectation you could take your archive of PHP files, unzip them into a web folder, and load index.php to install your application. This wasn’t perfect and I don’t want to go back to this, exactly, but it was a community norm around how you get your PHP software running.

These days there doesn’t seem to be a norm other than “hope you platform maintainers have good docs”. Composer’s done a lot to bring modern engineering workflows to PHP, but PHP’s continued reliance on external web servers (either mod_php or FastCGI proxies) mean most apps, especially legacy apps, can’t easily be started via the command line.

Put another way, most Node.js applications will let you do something like this

$ npm run start

But the following

$ composer.phar run start

remains an impossible dream for the PHP world.