OroCRM Frontend Asset Pipeline

I’m willing to concede this might be my inner frozen caveman developer talking, but right now “frontend web development” seems like it’s in a crises of chaos. Frontend developers, (the folks writing CSS, Javascript, and building our application UIs), have never had such a wealth of tools to choose from. Unfortunately, most of these tools tend to live in a universe by themselves, and getting these tools to play together and perform well in a production system is one of those thankless, yet vitally important tasks for any web application.

This chaos is on display in OroCRM. Today we’re going to cover Oro’s frontend asset pipeline. That is, we’ll cover the basics of how OroCRM pulls in its various frontend files so you, as a third party developer, can make intelligent choices about including javascript and CSS in your own bundles.

Symfony Asset Pipeline

Like most OroCRM features, the first thing we’ll want to review is Symfony 2’s frontend asset management systems. Symfony has two systems for managing frontend assets: The assets system, and the assetic system.

The assets system is the original system for managing frontend assets in Symfony. It gives you the ability to create asset links from templates, as well as include your asset files in your bundle.

In a twig template you’d write something like this

<script src="{{ asset('my/asset.js') }}" type="text/javascript"></script>

The asset twig function will automatically create the correct asset path. As a bundle developer, you distribute your frontend assets with your bundle. Specifically, they go the Resources/public folder.

For example, you would place the javascript file in the template above at

Resources/public/my/asset.js

Then, to copy the files to your publicly accessible web folder, you’d use the Symfony console application

$ php app/console assets:install

Running the above will copy all the Resource/public files to

$ /path/to/symfony/web/bundles/[bundle-name]

The asset function in the twig template handles converting the asset path to the correct http accessible path.

That’s the assets system. Its main goal is to enable linking to assets, and distributing frontend assets as part of a bundle. While useful, the needs of a modern frontend developer have evolved to the point where this simple system isn’t adequate. That’s why Symfony created the assetic system.

The Symfony documentation covers assetic well, so our description will be brief. Like the asset system, assetic gives developers a template syntax, and a way to get files from a bundle to the frontend web accessible system. The three main differences with assetic are the ability to do this dynamically at runtime, include filters for modifying the asset files (javascript minification/optimization/etc.), and automatically combine individual asset files into a single file for production deployments.

For templates, there are twig tags for each frontend file type. For example, to include a javascript file, you’d write something like this

{% javascripts '@AcmeFooBundle/Resources/public/js/file.js' %}
    <script type="text/javascript" src="{{ asset_url }}"></script>
{% endjavascripts %}

In addition to the {% javascripts %} tag, you get tags for {% stylesheets %} and an {% image %} tag.

In the example above, we’ve linked to an individual javascript file via a Symfony identifier, (@AcmeFooBundle/Resources/public/js/file.js). Unlike the assets system, the assetic system gives us the ability to link to frontend files from different bundles. Another advantage is the ability to link to multiple frontend asset files in one go. Consider the following

{% javascripts '@AcmeFooBundle/Resources/public/js/*' %}
    <script type="text/javascript" src="{{ asset_url }}"></script>
{% endjavascripts %}

The above code snippet will link to every javascript file in the AcmeFooBundle‘s Resources/public/js folder. Think of the {% javascripts %} tag as a foreach loop — one loop iteration for each file. This means adding a new file to the application is as simple as copying the file to a folder — no extra template code needed.

The key ah ha moment in understanding assetic comes from looking at its output. In the dev environment, Symfony will generate HTML something like this

<script type="text/javascript" src="/app_dev.php/js/file.js"></script>
<script type="text/javascript" src="/app_dev.php/js/other-file.js"></script>
//...

That is — requests for the frontend files are routed through the PHP app_dev.php front controller. This allows assetic to filter/change the output of these files.

So that’s the dev environment. However, when it comes time to deploy, you’ll see output that looks something like this

<script type="text/javascript" src="/web/js/7946a9a.js"></script>

In production, a single javascript file is served statically. This single file is a combination of all the javascript files specified by a single tag. Prior to deploying, a Symfony developer runs the app/console assetic:dump command.

$ php app/console assetic:dump

This command combines all the frontend files for each block, with filters applied, and spits out a single javascript file.

The end result is developers can tweak things to their heart’s content in the dev environment without regenerating a cache file, but the production deployment is a nice, static file, suitable for CDNs. A win/win as our business friends might say.

OroCRM’s Pipeline Customizations

The Symfony asset pipeline follows a common pattern in framework development. At first, Symfony tried to offer a single, straightforward way to manage javascript and CSS files (assets). When this didn’t prove adequate for every developer, they created an abstract system (assetic) that lets individual developers create their own asset pipeline.

While powerful, this means every project will end up using assetic a little bit differently. It should come as no surprise that this includes the OroCRM team.

OroCRM and Symfony

Because OroCRM is a Symfony application, they need to use asset and assetic. That’s because many bundles rely on these packages to include their frontend files. OroCRM’s installer includes the assets:install and assetic:dump commands.

There are plenty of Oro templates which use the simpler asset linking method. However, the framework makes limited use of the files generated by the assetic commands. From what I’ve seen, none of the {% javascripts %}, {% sytlesheets %}, etc. tags are used in Oro twig templates. The assetic generated javascript file(s) is/are ignored, and the assetic generated CSS file (web/css/oro.css) is linked via a twig template that sets up Oro’s use of the Less CSS pre-processor.

#File: vendor/oro/platform/src/Oro/Bundle/AsseticBundle/Resources/views/Assets/oro_css.html.twig
{% oro_css filter='cssrewrite, lessphp, cssmin' output='css/oro.css' %}
{% set isLess = ('less' in asset_url|split('.')) %}
{% if isLess %}
<script type="text/javascript">localStorage.clear();</script>
{% endif %}
<link {% if isLess %}rel="stylesheet/less"{% else %}rel="stylesheet"{% endif %} media="all" href="{{ asset_url }}" />
{% if isLess %}
<script type="text/javascript" src="{{ asset('bundles/oroui/lib/less-1.3.3.min.js') }}"></script>
{% endif %}
{% endoro_css %}

If you’re not familiar with it, Less is one of the many “I wish CSS had more features than it does so lets create a programming language on top of CSS” projects out there.

While maybe not what you hoped for, this is a valid use of Symfony’s assetic system. It does leave us with a question — how does OroCRM pull-in/combine its javascript files, and why aren’t they using assetic for this?

OroCRM Javascript and RequireJS

For the most part, when OroCRM pulls in third party libraries, they stick to the twig asset function. You can see one example of this here

#File: vendor/oro/platform/src/Oro/Bundle/UIBundle/Resources/views/Default/dialogPage.html.twig
//...
<script type="text/javascript" src="{{ asset('bundles/orowindows/js/jquery.dialog.extended.js') }}"></script>
//...

However, when it’s not a third party library — that is when OroCRM writes new custom javascript code — they do so with a framework called RequireJS.

RequireJS is a javascript Asynchronous module definition (AMD) implementation. It gives javascript a module system that’s similar to python’s, ruby’s, or NodeJS’s. A full explanation of RequireJS is beyond the scope of this article, but in short RequireJS allows you to define javascript module objects in individual files, and then include those modules in other scoped javascript code. Consider the following code

#File: web/bundles/orosync/js/sync/wamp.js
define(['jquery', 'underscore', 'backbone', 'autobahn'],
function ($, _, Backbone, ab) {
    //...
    return Wamp;
});

This defines a module named js/sync/wamp. The name comes from the javascript path of the file. This module uses (requires, imports, etc.) the modules jquery, underscore, backbone, and autobahn. The module objects are passed to the anonymous function as parameters ($, _, Backbone, ab). The returned Wamp object is the js/synx/wamp module.

The other top level function in RequireJS is require. You use the require function when you want to create an individual RequireJS program. You can see an example of this here

#File: vendor/oro/platform/src/Oro/Bundle/WorkflowBundle/Resources/views/Widget/widget/transitionForm.html.twig
require(['oro/widget-manager'],
function(widgetManager){
    widgetManager.getWidgetInstance({{ app.request.get('_wid')|json_encode|raw }}, function(widget) {
        widget.trigger(
            'formSave',
            {% if data is defined and data %}
                {{ data|json_encode|raw }}
            {% else %}
                null
            {% endif %}
        );
    });
});

In a pure RequireJS program, the require function is limited to the main entry point. OroCRM is not a pure RequireJS project, and uses require more like a document.ready/dom:loaded replacement.

Optimizing RequireJS

RequireJS presents a problem for a Symfony application. Specifically — RequireJS has its own optimization system, one that’s not compatible with the approach provided by assetic. This is why the OroCRM installer also includes the oro:requirejs:build command

$ php app/console oro:requirejs:build

This command builds the combined RequireJS files in web/js/oro.min.js. This is the file OroCRM links to when running in production mode. In development mode, OroCRM dynamically includes the RequireJS modules. The how/why of that is beyond the scope of this article but if you’re curious about OroCRM’s RequireJS setup, the scripts.html.twig template in the RequireJSBundle bundle is a good place to start.

#File: vendor/oro/platform/src/Oro/Bundle/RequireJSBundle/Resources/views/scripts.html.twig
{% if compressed and requirejs_build_exists() %}
    <script type="text/javascript">
        var require = (function(){
            var r=function(c){m(r.c,c)};r.c={};function m(a,b){
                for (var i in b)b[i].toString()==='[object Object]'?m(a[i]||(a[i]={}),b[i]):a[i]=b[i]}
            return r;
        }());
        {% placeholder requirejs_config_extend %}
        require = require.c;
    </script>
    <script type="text/javascript" src="{{ asset(get_requirejs_build_path()) }}"></script>
{% else %}
    <script type="text/javascript" src="{{ asset('bundles/ororequirejs/lib/require.js') }}"></script>
    <script type="text/javascript">
        {{ get_requirejs_config() }}
    </script>
    <script type="text/javascript">
        {% placeholder requirejs_config_extend %}
    </script>
{% endif %}

You’ll notice a RequireJS file is pulled in with the assets function.

<script type="text/javascript" src="{{ asset('bundles/ororequirejs/lib/require.js') }}"></script>

This means Oro’s RequireJS implementation is reliant on the Symfony assets system, and your having successfully run the assets:install command.

The next section takes a small detour for people interested in tracing how Symfony pulls in the RequireJS modules in development mode. It’s not required reading at this point, but we’re including it for the curious minded. If your brain’s already overloaded, skip ahead to the OroCRM Placeholders section.

OroCRM’s get_requirejs_config Twig Function

The dynamically generated modules come from the call to the get_requirejs_config twig function. This function is defined in an extension here

#File: vendor/oro/platform/src/Oro/Bundle/RequireJSBundle/Twig/OroRequireJSExtension.php
new \Twig_SimpleFunction('get_requirejs_config', function () use ($container) {
    return $container->get('oro_requirejs_config_provider')->getMainConfig();
}

In turn, this uses the getMainConfig method in the the oro_requirejs_config_provider Symfony service at

vendor/oro/platform/src/Oro/Bundle/RequireJSBundle/Provider/Config.php

to pull in the modules. This service ends up reading from a bundle’s Resources/config/requirejs.yml. These YML files contain configuration that tells OroCRM which javascript files contain RequireJS modules, as well as setting up RequireJS dependencies. You can see an example of one of these files here.

vendor/oro/platform/src/Oro/Bundle/SyncBundle/Resources/config/requirejs.yml

OroCRM Placeholders

Phew! That’s a lot of information to take in. For a frontend or full-stack developer, understanding an application’s frontend asset pipeline is hugely important. The way assets are included into a project heavily impacts the way an application works, and making sure your code lines up with expectations is vital to ensuring your code behaves itself.

However — for someone looking to make a simple CSS tweak, or include some third party javascript library, this can all seem like a bit much to process. Fortunately, OroCRM offers a much simpler way to add frontend files to your system, and that’s via the twig {% placeholder %} tag. If you take a look at the base OroBAP/OroCRM twig template file, you’ll see these tags

#File: vendor/oro/platform/src/Oro/Bundle/RequireJSBundle/Resources/views/scripts.html.twig

{% placeholder head_style %}
//...
{% placeholder scripts_before %}
//...
{% placeholder scripts_after %}

These placeholder scripts allow bundles to include any twig template wherever the {% placeholder %] tag is. This twig template can then reference any javascript or CSS you like. To use this feature, just include a placeholders.yml file in your bundle. If you’re curious about the syntax, checkout the OroBAP’s Twig Placeholder Tag post over on my Oro Quickies website.

Wrap Up

That’s Oro’s frontend asset pipeline. While seemingly chaotic, once you understand the constraints in place, and all the various moving parts, it should be simple enough to add your own frontend files to the system. While a lot has changed in software development over the past 20 years, the maxim that The Best System is the System in Use By Your Team remains truer than ever.

Every development shop will have a different approach to adding their frontend assets depending on their needs and expertise, and I hope this article helps you make your own decisions.

If you have any questions, or I’ve made a mistake above, please get in touch or let me know in the comments below. For super technical questions, open a thread over at Stack Overflow and mention it below.