N98-magerun: Development Environment

This article is part of a longer series covering the n98-magerun power tool

Today we’re going to run through setting up your own n98-magerun development environment. This will allow you to develop your own features and contribute bug fixes back to the main repository. This one’s a little longer than our normal Magento Quickies fare, so if you’re in a hurry you may want to save this for later.

There’s two (or maybe three) extra software packages you’ll need to install if you want to work with the n98-magerun source code. The first is PHP Composer, the second is the phing build system.

Composer

Composer is a software package for managing project dependencies. The n98-magerun program relies on many third party packages for its functionality. Rather than include these in the main source tree, n98-magerun includes a top level vendor/ folder. This is where third party dependencies are downloaded to.

This is a standard best practice in profesional software development. By separating your vendor packages from the main source tree, you make upgrading the packages easier. You also encourage/force a “no hacking the vendor package” discipline on your team.

There’s instructions for installing Composer on their website. The short version is there’s a composer.phar (similar to n98-magerun.phar) archive that needs to be in your shell path and needs to be executable. The composer team has a “download install script with curl and run it” one liner, or you may download the phar file directly and place it in your path.

You can test if composer is installed correctly by running the following

$ composer.phar --version
Composer version 0209bd31a0ac3aeb2a68fc81e2d03c71072bef33

Phing

Phing is a PHP build system based on apache ant. For The Kids™ in the audience, ant is a system that was created to give developers a “pure java” build system (as opposed to the powerful-but-tortuous options of the day like GNU make). Phing, in turn, is a PHP version of ant.

Phing implements an XML based build language that allows you to perform common build tasks. This XML format is an example of a Domain Specific Language (DSL). Folks using Magento’s layout system may get a sense of Déjà vu.

Installing phing is slightly trickier than installing PHP composer. The preferred way of installing phing is via PHP PEAR. PEAR (the third package we mentioned above) is PHP’s original dependencies and package management system. While powerful, it never became consistently adopted by the disperate PHP communities. A side effect of this is many PHP distributions don’t have PEAR properly installed, or have it only available to users with sudo permission.

If a fully operational PEAR isn’t available to you, phing is also available via a package download, as well as a stand alone phar file.

Regardless how you install it, at the end of the day you’ll have a phing (or phing.phar or phing-latest.phar) command executable from your shell.

$ phing -v
Phing 2.5.0

Downloading Dependencies with Composer

Now that our tools are in place, we’re ready to start. Step 1 for working with n98-magerun is to get the codebase from GitHub.

$ git clone https://github.com/netz98/n98-magerun.git

However, even after you’ve cloned the git source tree, you’re still missing a number of files n98-magerun needs to function. If you look in the vendor folder, you’ll see it’s empty

$ ls -la vendor/
total 16
drwxr-xr-x   3 alanstorm  staff   102 Apr 26 14:40 .
drwxr-xr-x  26 alanstorm  staff   884 Apr 25 23:24 ..

As mentioned earlier, the vendor folder is where third party code libraries are kept. To download these files, run the composer.phar install command from the root folder of your repository.

$ cd n98-magerun
$ composer.phar install
Loading composer repositories with package information
Installing dependencies from lock file
  - Installing symfony/process (v2.2.1)
    Loading from cache

  - Installing symfony/finder (v2.2.1)
    Loading from cache

  - Installing symfony/console (2.2.x-dev a10a2fb)
    Cloning a10a2fb156267b9b8f01d717a06979e6ebb69251

  - Installing seld/jsonlint (1.1.1)
    Loading from cache

  - Installing justinrainbow/json-schema (1.1.0)
    Loading from cache

  - Installing composer/composer (dev-master c655674)
    Cloning c65567447c84eb6a8c0f229c40b0865cd4dcecb5

  - Installing fzaninotto/faker (v1.1.0)
    Loading from cache

  - Installing n98/junit-xml (dev-master 6381433)
    Cloning 6381433a39498dc22bec60cebdb2576a9d2327b0

  - Installing symfony/translation (v2.2.1)
    Loading from cache

  - Installing symfony/validator (v2.2.1)
    Loading from cache

  - Installing symfony/yaml (v2.2.1)
    Loading from cache

symfony/translation suggests installing symfony/config (2.2.*)
symfony/validator suggests installing doctrine/common (~2.2)
symfony/validator suggests installing symfony/config (2.2.*)
symfony/validator suggests installing symfony/http-foundation (2.2.*)
symfony/validator suggests installing symfony/locale (2.2.*)
Generating autoload files

If you take a look at your vendor folder after running the install command, you’ll see the required vendor packages have been installed.

$ ls -1a vendor/
.
..    
autoload.php
bin
composer
fzaninotto
justinrainbow
n98
seld
symfony

Of course, this raises a few questions: How does composer.phar know which packages to download? And where are these packages coming from? While this knowledge isn’t strictly required to proceed, we’re going to cover it to put your (and our) minds at ease.

The composer.phar command looks for a file named composer.json. This file contains configuration information for a composer based project (which n98-magerun is). This configuration information will include a list of package dependencies

//File: composer.json
"require": {
    "php": ">=5.3.3",
    "symfony/console": "2.2.x-dev",
    "symfony/finder": "2.2.1",
    "symfony/yaml": "2.2.1",
    "symfony/process": "2.2.1",
    "symfony/validator": "2.2.1",
    "justinrainbow/json-schema": "1.1.*",
    "seld/jsonlint": "1.*",
    "n98/junit-xml": "dev-master",
    "fzaninotto/faker": "1.1.0",
    "composer/composer": "dev-master"
},

As you can see, the require key contains a list of packages and versions. These package names correspond with the downloaded packages from the install command. So this answers the “how does composer know” question. Before we move on the “where” question, it’s worth pointing out that composer is actually reading from the composer.lock file. This composer.lock file is generated with the information in composer.json, and is not meant to be human editable. If you want to get into the nitty gritty of this checkout the composer manual.

The “from where” question is a little trickier. There’s two grand arcs to the composer project. The first is the composer.phar application we downloaded, which contains the logic for dependency management. The second is a composer repository at packagist.org. Unless a different repository is listed in composer.json, packagist.org is where composer.phar will grab package information from.

Does that mean composer is downloading packages from packagist.org? No. Confused? Read on!

A composer repository only contains meta-data about a package. This meta-data includes a source code repository where the package source can be found. The packagist.org repository simply tells composer.phar where it should download the package source from.

Figuring out where composer.phar is grabbing packages from is a two step process. First, you can list out all the packages used in your project with the show command and and --self flag.

$ composer.phar show --self
name     : n98/magerun
descrip. : Tools for managing Magento projects and installations
keywords : magento, installer
versions : * dev-master
type     : library
license  : MIT
source   : []  
dist     : []  
names    : n98/magerun

support
issues : https://github.com/netz98/n98-magerun/issues

autoload
psr-0
N98 => src
Composer => src
Xanido => src

requires
php >=5.3.3
symfony/console 2.2.x-dev
symfony/finder 2.2.1
symfony/yaml 2.2.1
symfony/process 2.2.1
symfony/validator 2.2.1
justinrainbow/json-schema 1.1.*
seld/jsonlint 1.*
n98/junit-xml dev-master
fzaninotto/faker 1.1.0
composer/composer dev-master

requires (dev)
phpunit/phpunit 3.7.19
phpunit/phpunit-story 1.0.2
phpunit/phpunit-selenium 1.2.12
phpunit/dbunit 1.2.3
phpunit/php-invoker 1.1.2

Once you have a package name (such as symfony/yaml), you can see its repository by using the show command again, this time with the package name as an argument.

$ composer.phar show symfony/yaml
name     : symfony/yaml
descrip. : Symfony Yaml Component
keywords : 
versions : dev-master, 2.3.x-dev, 2.2.x-dev, * v2.2.1, v2.2.0, v2.2.0-RC3, v2.2.0-RC2, v2.2.0-RC1, v2.2.0-BETA2, v2.2.0-BETA1, 2.1.x-dev, v2.1.9, v2.1.8, v2.1.7, v2.1.6, v2.1.5, v2.1.4, v2.1.3, v2.1.2, v2.1.1, v2.1.0, v2.1.0-RC2, v2.1.0-RC1, v2.1.0-BETA4, v2.1.0-BETA3, v2.1.0-BETA2, v2.1.0-BETA1, 2.0.x-dev, v2.0.23, v2.0.22, v2.0.21, v2.0.20, v2.0.19, v2.0.18, v2.0.17, v2.0.16, v2.0.15, v2.0.14, v2.0.13, v2.0.12, v2.0.10, v2.0.9, 2.0.7, 2.0.6, 2.0.5, 2.0.4
type     : library
license  : MIT
source   : [git] https://github.com/symfony/Yaml.git v2.2.1
dist     : [zip] https://api.github.com/repos/symfony/Yaml/zipball/v2.2.1 v2.2.1
names    : symfony/yaml

autoload
psr-0
SymfonyComponentYaml => .

requires
php >=5.3.3

The two sections we’re interested in here are source and dist. The source lists the repository where the source code should be downloaded from, (git, svn, and hg repository types are supported). This is how composer.phar will grab the needed vendor packages.

The archive (as opposed to repository) at dist can be used to install a package instead, but only if the --prefer-dist flag is used with the install command.

If you’re paranoid about security, you’ll want to review these package locations — although at some point using a distributed packaging system like CPAN, ruby gems, or packagist requires a certain level of trust. That, however, is another subject better left to another time.

Composer Development Packages

There’s one last thing we should cover before moving on from composer. If you examined the composer.json file carefully, you may have noticed this section

//File: composer.json
"require-dev": {
    "phpunit/phpunit": "3.7.19",
    "phpunit/phpunit-story": "1.0.2",
    "phpunit/phpunit-selenium": "1.2.12",
    "phpunit/dbunit": "1.2.3",
    "phpunit/php-invoker": "1.1.2"
},

This appears to be a second list of packages. It turns out composer projects can have two sets of required packages. The first is the standard distribution you’ll need to use the package. The second, require-dev, is an additional set of packages you’ll need to work on (or develop) the project.

Since this whole article is about building our own version of n98-magerun.phar, we’ll want to install these development packages. Let’s run composer.phar with the --dev flag.

$ composer.phar install --dev
Loading composer repositories with package information
Installing dependencies (including require-dev) from lock file
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Installation request for phpunit/php-invoker 1.1.2 -> satisfiable by phpunit/php-invoker 1.1.2.
    - phpunit/php-invoker 1.1.2 requires ext-pcntl * -> the requested PHP extension pcntl is missing from your system.

Uh oh. At this point, you may (as we did above) run into an intractable problem. The phpunit/php-invoker package couldn’t be installed because it requires that PHP have the process control (ext-pcntl) extension installed. This extension is omitted from many standard PHP distributions.

We’ve now run into what I like to call the Software is Hard problem. In order to install these dependencies, we’d need to recompile our version of PHP. That’s something many people can’t, or won’t do. Unless we’re willing to make a massive change to our environment, we won’t be able to install the full suite of required phpunit tools.

On the flip side, we could remove phpunit/php-invoker as a requirement, and then successfully install the other development packages. However, this would mean we’re veering off from what the main team is doing. Our system may start to behave differently than theirs, causing subtle and hard to track down compatibility problems in the future.

There’s going to be times in your career where you run into problems where there is no ideal solution. The only option is to raise your concerns, and then press on with the less than ideal conditions. The trick to a long career is finding a way to not let these small defeats sap your motivation for other, more important tasks.

Fortunately, it’s still possible to work on n98-magerun code without the full phpunit suite, so we’ll carry on without it.

Update: From the sounds of this GitHub thread, the phpunit/php-invoker requirement will be dropped in the 1.64 release, so the previous whinging may not apply if you’re reading this in the future.

Phing

Now that we’ve discussed composer, our next step is using phing to build the project. Take a look at the build.xml file in the root of the project

<!-- File: build.xml -->
<project name="n98-magerun" default="dist">

    <fileset dir="." id="root_folder"><!-- ... --></fileset>
    <fileset dir="src"  id="src_folder"><!-- ... --></fileset>
    <fileset dir="res" id="res_folder"><!-- ... --></fileset>
    <fileset dir="vendor" id="vendor_folder"><!-- ... --></fileset>

    <target name="dist">
        <!-- ... -->
    </target>

    <target name="dist_unix">
        <!-- ... -->
    </target>

    <target name="install">
        <!-- ... -->
    </target>

</project>

This is our phing build file. It contains all the build instructions for our package. We’re interested in the <target/> nodes. Each <target/> node defines a single build command. For example, the install command can be used to install the n98-magerun.phar from the repository to your local computer. This is a relatively simple build target.

<target name="install">
    <exec command="sudo cp ${project.basedir}/n98-magerun.phar /usr/local/bin/n98-magerun.phar;" />
    <exec command="sudo chmod a+x /usr/local/bin/n98-magerun.phar;" />
</target>

As you can see, there’s two <exec/> nodes within this target. These are command nodes — think of each one like you would a single line in a normal programming language or shell script. The <exec/> command is one of the simplest build commands — it simply runs a unix shell script.

Let’s try running the install target.

$ phing install
Buildfile: /path/to/n98-magerun/build.xml

n98-magerun > install:

Password:
Sorry, try again.
Password:

BUILD FINISHED

Total time: 5.9880 seconds

As you can see, to run a phing command you simply pass it the target name. The phing command will look for a build.xml file in the current directory, and then run the target you’ve passed in.

As for the command output itself, it prompts you for a password (since it’s running the sudo command), and the finishes the build. Unfortunately, there’s no indication of what the command actually did. If you’d like a little more feedback, try the verbose flag.

$ phing -verbose install
Buildfile: /path/to/n98-magerun/build.xml
Override ignored for user property phing.file
Override ignored for user property phing.dir
parsing buildfile build.xml
Project base dir set to: /path/to/n98-magerun
Build sequence for target 'install' is: install 
Complete build sequence is: install dist dist_unix 

n98-magerun > install:

Property ${project.basedir} => /path/to/n98-magerun
     [exec] Executing command: sudo cp /path/to/n98-magerun/n98-magerun.phar /usr/local/bin/n98-magerun.phar; 2>&1
     [exec] Executing command: sudo chmod a+x /usr/local/bin/n98-magerun.phar; 2>&1

BUILD FINISHED

Total time: 0.1772 seconds

With the verbose flag, phing tells you everything it’s doing. From this we can figure out a file has been copied to /usr/local/bin/n98-magerun.phar. This places the phar in our *nix path, and is another way to install the n98-magerun.phar command.

Building the Project

Most phing build files have a default target which builds the project.

If you run phing with the -l option

$ phing -l
Buildfile: /path/to/n98-magerun/build.xml
Default target:
-------------------------------------------------------------------------------
 dist

Subtargets:
-------------------------------------------------------------------------------
 dist
 dist_unix
 install

you’ll get a list of each target in the application, as well as the default target (in this case, dist). You can also see the default target by looking at the root node of the build.xml file.

<!-- build.xml -->
<project name="n98-magerun" default="dist">

We’re going to run through a default build of the n98-magerun tool, including several problems we ran into and how to work around them. You may not run into these specific problems — and you may run into different problems not mentioned here. Setting up a build environment is always a trial and error process, and things will rarely work out of the gate. Part of the reason teams use a tool like phing is to provide a single entry point into their build system and force developers to download and install the same dependencies. If you run into trouble getting this up and running, the Magento Stack Exchange is a great place to ask questions to help get your system up and running.

To run the default build step, just run phing from the project root directory with no arguments

$ phing

The first thing you’ll see is phing going through the project dependencies and downloading the needed files to the vendor folder.

$ phing
Buildfile: /path/to/n98-magerun/build.xml

n98-magerun > dist:

Loading composer repositories with package information
Installing dependencies from lock file
  - Installing symfony/process (v2.2.1)
    Loading from cache

  - Installing symfony/finder (v2.2.1)
    Loading from cache

... snipped ...

If you’ve already downloaded these files with the composer.phar install command, these downloads will be skipped. The output for that looks like this

$ phing
Buildfile: /private/tmp/building/n98-magerun/build.xml

n98-magerun > dist:

Loading composer repositories with package information
Installing dependencies from lock file
Nothing to install or update
Generating autoload files

If you look at the build.xml file, you can see all this happens because the command in the dist target runs the composer install command.

<target name="dist">
    <exec command="composer.phar install" dir="${project.basedir}" passthru="true" />

So, it would be more accurate to say phing still attempts to update your dependencies, but if the latest versions are already downloaded composer will skip re-downloading them.

Writeable PHAR Archives

After phing runs the composer.phar command, it moves on to the next build steps, and (for some of you) will hit another error.

Execution of target “dist_unix” failed for the following reason: /path/to/n98-magerun/build.xml:45:65: Problem creating package: creating archive “/path/to/n98-magerun/n98-magerun.phar” disabled by the php.ini setting phar.readonly

Here we’ve discovered a constraint of PHP’s phar implementation. By default, PHP won’t let you edit an existing phar archive. While annoying, if you consider that a phar archive is meant to “emulate jars and dlls”, this makes sense. Allowing a program to modify its libraries would be a recipe for disaster.

The problem, of course, is that it’s PHP code that creates a phar archive. In order to solve this chicken/egg problem, PHP has a phar.readonly flag for php.ini. If you set this flag to 0 or Off, PHP will allow you to access the methods which modify a phar archive.

This flag is one you’ll need to set in your php.ini file, it can’t be set at runtime with ini_set. If you’re not sure where your php.ini file is located, try running the following

$ php --ini
Configuration File (php.ini) Path: /etc
Loaded Configuration File:         /private/etc/php.ini
Scan for additional .ini files in: (none)
Additional .ini files parsed:      (none)

Remember, it’s not uncommon for the command line version of PHP to have a different ini file than the web server version of PHP.

BZip Compression Problems

With the above in place, try running the build again. If you’re like me, you’re about to run into another problem

Execution of target “dist_unix” failed for the following reason: /path/to/n98-magerun/build.xml:45:65: Problem creating package: unable to create temporary file [wrapped: unable to create temporary file]

This one is a head scratcher, but I finally tracked down the problem. In short: While PHP’s phar format works across a variety of platforms and environments, the methods for creating phar archives are still very beta-ish. The specific problem here is, when compressing phar archives PHP likes to open a temporary file — and from the sound of things it opens a temporary file for each file in the archive, and doesn’t close them until the command is complete. All that library code means n98-magerun has over 1,000 files.

This quickly hits the limits that a non-server operating system has in place for the number of files each process is allowed to have open. To solve this problem, you’ll need to edit build.xml. Find the following XML node

<!-- File: build.xml -->
<pharpackage basedir="./" stub="_cli_stub.php" signature="sha512" compression="bzip2" destfile="./n98-magerun.phar">    

The pharpackage node is a phing build command that allows you to automatically create a phar application from a PHP codebase. We’re interested in the compression="bzip2" attribute. This tells phing to run the phar archive through the compressFiles method, and is done to reduce the file size of the archive that’s distributed. If you change this attribute to

compression="none"

then phing will skip the compression step, and we should be able to get a complete build.

$ phing
Buildfile: /path/to/n98-magerun/build.xml

n98-magerun > dist:

Loading composer repositories with package information
Installing dependencies from lock file
Nothing to install or update
Generating autoload files
[phingcall] Calling Buildfile '/path/to/n98-magerun/build.xml' with target 'dist_unix'

n98-magerun > dist_unix:

[pharpackage] Building package: /path/to/n98-magerun/n98-magerun.phar
    [chmod] Changed file mode on '/path/to/n98-magerun/n98-magerun.phar' to 775
Loading composer repositories with package information
Installing dependencies (including require-dev) from lock file

BUILD FINISHED

Total time: 3.5711 seconds

That looks like a clean build, let’s test out the n98-magerun.phar file that was just created.

$ ./n98-magerun.phar --version
n98-magerun version 1.63.0 by netz98 new media GmbH

Success! However, our n98-magerun.phar file

$ ls -lh n98-magerun.phar
-rwxrwxr-x  1 alanstorm  staff   3.4M Apr 25 15:50 n98-magerun.phar

is over 3 MB in size. Compare that to the 1 MB file distributed in the main github archive. So, while we can’t build exactly what the core n98-magerun team does, we can get a clean build in place with a testable phar, which is good enough for our needs.

While frustrating, you’ll find many points in your career where your development environment won’t match up exactly with that of the team you’re working with. While it’s true (especially among non-distributed teams) that a chaotic build environment can lead to development delays, these delays are subtle and complicated in nature, and not the sort of thing that shows up in ROI metrics. Unfair as it may be, as an individual developer working with a group, it will often fall on you to solve these problems for yourself in a way that doesn’t disrupt team flow.

Wrap Up

Regardless of the subtle incompatibilities, we’ve successfully built the n98-magerun.phar file from source. This means we can start hacking on bugs, as well as adding new features to the tool. Next time we’ll create our very first n98-magerun command.