Magento 2: UI Component Data Sources

Today we’ll discuss how Magento 2’s UI Component system gets configuration and data source data into a javascript and Knockout.js context. i.e. How is something configured via PHP, but used in the browser.

This article will focus on the Customer Listing grid. While we’ve experimented with creating XSD valid UI Components, it turns out our reliance on the root <container/> node makes Magento ignore the inner dataSource nodes when rendering the x-magento-init scripts. We’re going to focus on the Customer Listing grid, and hope that Magento 2.2 loosens its grip a little w/r/t to ui_component XML.

Also, if it’s not obvious from the above jargon, this article borrows heavily from concepts introduced in earlier series articles. While not absolutely required reading, if you get stuck you may want to start from the beginning. The specifics here are Magento 2.1.x, but the concepts should apply across versions.

The Magento_Ui/js/core/app Application

The Magento_Ui/js/core/app application is the program that handles instantiating Magento’s view models, and registering them in the uiRegistry. This means Magento_Ui/js/core/app is also the place where configuration data gets moved from the XML/x-magento-init script, and into our view models.

//File: vendor/magento//module-ui/view/base/web/js/core/app.js
/**
 * Copyright © 2016 Magento. All rights reserved.
 * See COPYING.txt for license details.
 */
define([
    './renderer/types',
    './renderer/layout',
    '../lib/knockout/bootstrap'
], function (types, layout) {
    'use strict';

    return function (data, merge) {
        types.set(data.types);
        layout(data.components, undefined, true, merge);
    };
});

This program is deceptively simple. There’s two function calls — the first, (a call to the set method on the Magento_Ui/js/core/renderer/types module object) is a simple registry/dictionary that’s used to serialize data from the types attribute of the rendered x-magento-init JSON.

//embeded in the HTML page
{
    "*": {
        "Magento_Ui/js/core/app": {
            "types": {/* handled by Magento_Ui/js/core/renderer/types */},
            "components": { /*...*/}
        }
    }
}

This is not a straight serialization of the data. You can peek at an individual entry in the registry with code like this

//try this in the javascript console on the customer grid listing page
typesReg = requirejs('Magento_Ui/js/core/renderer/types');
console.log(
    typesReg.get('text');
);

console.log(
    typesReg.get('customer_listing');
);

However, unlike the uiRegistry, there’s no way to view everything that’s stored in this types registry.

The majority of the work in Magento_Ui/js/core/app happens in the function returned by the Magento_Ui/js/core/renderer/layout module. Magento_Ui/js/core/renderer/layout is a — complicated — module. This module appears to be responsible for instantiating Knockout.js view model objects and registering them in the uiRegistry. It does this by using data from the components attributes of the rendered x-magento-init script.

//embeded in the HTML page    
{
    "*": {
        "Magento_Ui/js/core/app": {
            "types": { /*...*/},
            "components": { /* handled by Magento_Ui/js/core/renderer/layout */}
        }
    }
}

The javascript patterns used in Magento_Ui/js/core/renderer/layout are a little unorthodox (when compared to other modules).

#File: vendor/magento/module-ui/view/base/web/js/core/renderer/layout.js
define([
    'underscore',
    'jquery',
    'mageUtils',
    'uiRegistry',
    './types'
], function (_, $, utils, registry, types) {
    'use strict';

    var templates = registry.create(),
        layout = {},
        cachedConfig = {};

    function getNodeName(parent, node, name) {/*...*/}

    function getNodeType(parent, node) {/*...*/}

    function getDataScope(parent, node) {/*...*/}

    function loadDeps(node) {/*...*/}

    function loadSource(node) {/*...*/}

    function initComponent(node, Constr) {/*...*/}

    function run(nodes, parent, cached, merge) {/*...*/}

    _.extend(layout, /*... methods and properties ...*/);

    _.extend(layout, /*... methods and properties ...*/);

    _.extend(layout, /*... methods and properties ...*/);

    return run;
});      

The first thing you’ll notice about this module is, it returns a function (run). Most Magento 2 RequireJS modules return an object literal, an object created with .extend, or an object created with a javascript constructor function/new. The run function serves as a main entry point for the program, which (mostly) operates on the locally defined layout object.

#File: vendor/magento/module-ui/view/base/web/js/core/renderer/layout.js
function run(nodes, parent, cached, merge) {
    if (_.isBoolean(merge) && merge) {
        layout.merge(nodes);

        return false;
    }

    if (cached) {
        cachedConfig[_.keys(nodes)[0]] = JSON.parse(JSON.stringify(nodes));
    }

    _.each(nodes || [], layout.iterator.bind(layout, parent));
}

The layout object is the second weird thing about this file. You’ll see Magento uses the underscore JS library to extend a raw object literal with methods.

#File: vendor/magento/module-ui/view/base/web/js/core/renderer/layout.js
var /*...*/ layout = {}, /*...*/;
/*...*/
_.extend(layout, /*... methods and properties ...*/);

_.extend(layout, /*... methods and properties ...*/);

_.extend(layout, /*... methods and properties ...*/);    

There’s nothing odd about this in and of itself — but doing this in three separate statements seems a little off, and suggests a series of developers working on this module, each afraid to touch each other’s code.

Magento kremlinology aside, the run method kicks off processing with the following

#File: vendor/magento/module-ui/view/base/web/js/core/renderer/layout.js
_.each(nodes || [], layout.iterator.bind(layout, parent));

Magento uses underscore.js’s each method to run through each of the nodes from x-magento-init‘s components attribute, and call that node’s iterator method (using javascript’s bind method to call iterator). This module liberally uses bind, as well as apply, to call methods.

Running through the full execution chain of this module is beyond the scope of this article, and would probably require an entire new series. Instead we’re going to highlight a few important sections.

View Model Instantiation

The first part of layout.js that’s worth highlighting is the instantiation of our view models and their registration in the uiRegistry. All this happens here.

#File: vendor/magento/module-ui/view/base/web/js/core/renderer/layout.js
function initComponent(node, Constr) {
    var component = new Constr(_.omit(node, 'children'));
    registry.set(node.name, component);
}

Of course, you’re probably wondering what the node and Constr parameters are, and where they come from.

Regarding “what they are”, the node variable is an object that contains values from the components property of Magento’s x-magento-init array, and the Constr variable is a view model constructor object created via a view model constructor factory.

Regarding “where they come from”, that brings us to the other parts of layout.js worth highlighting. The data in the node object starts with the aforementioned x-magenti-init object, but it really comes to life in the build method.

#File: vendor/magento/module-ui/view/base/web/js/core/renderer/layout.js
build: function (parent, node, name) {
    var defaults    = parent && parent.childDefaults || {},
        children    = node.children,
        type        = getNodeType(parent, node),
        dataScope   = getDataScope(parent, node),
        nodeName;

    node.children = false;

    node = utils.extend({
    }, types.get(type), defaults, node);

    nodeName = getNodeName(parent, node, name);

    _.extend(node, node.config || {}, {
        index: node.name || name,
        name: nodeName,
        dataScope: dataScope,
        parentName: utils.getPart(nodeName, -2),
        parentScope: utils.getPart(dataScope, -2)
    });

    node.children = children;

    delete node.type;
    delete node.config;

    if (children) {
        node.initChildCount = _.size(children);
    }

    if (node.isTemplate) {
        node.isTemplate = false;

        templates.set(node.name, node);

        return false;
    }

    if (node.componentDisabled === true) {
        return false;
    }

    return node;
},

The component data from x-magento-init is a series of nested objects, and each parent/child object will make its way through this method. There’s a lot of default value setting in this method. This is also where Magento uses the values from the Magento_Ui/js/core/renderer/types registry to add values to node.

#File: vendor/magento/module-ui/view/base/web/js/core/renderer/layout.js
node = utils.extend({
}, types.get(type), defaults, node);

You’ll also notice layout.js looks at the node’s parent node to set default values. Magento’s core engineers manipulate this node object throughout layout.js, but build seems to be where the majority of the work happens.

The third area worth highlighting is the loadSource function. This is where Magento loads the view model constructor factory (i.e. the RequireJS model configured in component) to get a view model constructor object. The constr below is the Constr we pass in to initComponent.

#File: vendor/magento/module-ui/view/base/web/js/core/renderer/layout.js

function loadSource(node) {
    var loaded = $.Deferred(),
        source = node.component;

    require([source], function (constr) {
        loaded.resolve(node, constr);
    });

    return loaded.promise();
}

The loaded = $.Deferred(), loaded.resolve(node, constr), and loaded.promise() objects are part of the jQuery promises API. They don’t have anything to do with how Magento uses the view model constructor factory — they’re here as a consequence of the way Magento calls the loadSource function.

#File: vendor/magento/module-ui/view/base/web/js/core/renderer/layout.js

loadDeps(node)
    .then(loadSource)
    .done(initComponent);

Unfortunately, jQuery’s promises library and Magento’s use of the same is beyond the scope of what we can cover today.

Speaking broadly, the use of promises, the nested/recursive nature of parsing the components data, and some use of promise-ish like features in the registry (in methods like waitParent and waitTemplate) make reading through this module a bit more involved than your average code safari. That said, if you remember the ultimate job here is to get a view model constructor loaded, and a view model instantiated and registered, the above three methods should be enough to help you when it comes time to debug.

Practical Debugging Tips

Next, we’ll run through a few tips for those times when you need to trace down a piece of data from the backend to the view model/Knockout.js level.

When Magento encounters a bit of UI Component layout XML like this

<!-- File: vendor/magento/module-customer/view/adminhtml/ui_component/customer_listing.xml -->

<bookmark name="bookmarks">
    <argument name="data" xsi:type="array">
        <item name="config" xsi:type="array">
            <item name="storageConfig" xsi:type="array">
                <item name="namespace" xsi:type="string">customer_listing</item>
            </item>
        </item>
    </argument>
</bookmark>

Magento merges in the data from definition.xml

<!-- vendor/magento//module-ui/view/base/ui_component/etc/definition.xml -->
<bookmark class="Magento\Ui\Component\Bookmark">
    <argument name="data" xsi:type="array">
        <item name="config" xsi:type="array">
            <item name="component" xsi:type="string">Magento_Ui/js/grid/controls/bookmarks/bookmarks</item>
            <item name="displayArea" xsi:type="string">dataGridActions</item>
            <item name="storageConfig" xsi:type="array">
                <item name="saveUrl" xsi:type="url" path="mui/bookmark/save"/>
                <item name="deleteUrl" xsi:type="url" path="mui/bookmark/delete"/>
            </item>
        </item>
    </argument>
</bookmark>

and serializes the merged node out to the x-magento-init JSON/javascript. This data looks something like this

"bookmark": {
    "extends": "customer_listing",
    "current": {/* ... */},
    "activeIndex": "default",
    "views": {/* ... */},
    "component": "Magento_Ui\/js\/grid\/controls\/bookmarks\/bookmarks",
    "displayArea": "dataGridActions",
    "storageConfig": {/* ... */}
},

You’ll notice the storageConfig, component, and displayArea properties from the XML configuration are represented in the JSON. You’ll also notice some other properties (views, current, activeIndex) are not represented in the XML. Remember that the UI Component’s PHP class (in this case, Magento\Ui\Component\Bookmark) can render any JSON here. You may need to look at your rendered x-magento-init source rather than the XML files when you’re looking for a value.

When layout.js gets a hold of this node, it will load component (in this case, Magento_Ui/js/grid/controls/bookmarks/bookmarks) via RequireJS. We’ve been calling components view model constructor factories to better describe their role in the entire process. This component module returns our view model constructor. This view model constructor factory’s defaults looks something like this

//File: vendor/magento/module-ui/view/base/web/js/grid/controls/bookmarks/bookmarks.js

return Collection.extend({
    defaults: {
        template: 'ui/grid/controls/bookmarks/bookmarks',
        viewTmpl: 'ui/grid/controls/bookmarks/view',
        newViewLabel: $t('New View'),
        defaultIndex: 'default',
        activeIndex: 'default',
        viewsArray: [],
        storageConfig: {
            provider: '${ $.storageConfig.name }',
            name: '${ $.name }_storage',
            component: 'Magento_Ui/js/grid/controls/bookmarks/storage'
        },

You’ll remember from the UI Class data features article that the defaults array of a view model constructor controls default data values in an instantiated view model object. Lacking other context, the values set above will carry through to the instantiated view model object.

However, you’ll want to keep view model instantiation in mind as well. i.e. we don’t lack other context!

#File: vendor/magento/module-ui/view/base/web/js/core/renderer/layout.js
function initComponent(node, Constr) {
    var component = new Constr(_.omit(node, 'children'));
    registry.set(node.name, component);
}

When Magento instantiates the view model, Magento passes in an array of data to the constructor (node above — after using the omit method of the underscore.js library to remove the children property). This means values set via x-magento-init (which comes from the ui_component XML object’s config data property) will override the view model constructor’s defaults.

Data Source Data

If you’re not too dizzy, we have one last thing to talk about. The values we discussed above are a view model’s configuration values. There is, however, another source of data for backend UI Components. The data from the dataSource nodes.

<!-- File: vendor/magento/module-customer/view/adminhtml/ui_component/customer_listing.xml -->
<dataSource name="customer_listing_data_source">
    <argument name="dataProvider" xsi:type="configurableObject">
        <argument name="class" xsi:type="string">Magento\Customer\Ui\Component\DataProvider</argument>
        <argument name="name" xsi:type="string">customer_listing_data_source</argument>
        <argument name="primaryFieldName" xsi:type="string">entity_id</argument>
        <argument name="requestFieldName" xsi:type="string">id</argument>
        <argument name="data" xsi:type="array">
            <item name="config" xsi:type="array">
                <item name="component" xsi:type="string">Magento_Ui/js/grid/provider</item>
                <item name="update_url" xsi:type="url" path="mui/index/render"/>
            </item>
        </argument>
    </argument>
</dataSource>

You’ll remember from our Simplest UI Component that the dataSource node (when used inside a listing node) with the above configuration is responsible for the extra data node in the UI Component’s x-magento-init data

"customer_listing_data_source": {
    /* ... */
    "config": {
        "data": {
            "items": [/*... JSON representation of Grid Data ...*/],
            "totalRecords": 1
        },
        "component": "Magento_Ui\/js\/grid\/provider",
        "update_url": "http:\/\/magento-2-1-1.dev\/admin\/mui\/index\/render\/key\/a9a9aa9e29b8727c7eaa781ffc7e2619e0cf12b2cf36f7458d93ae6e2412e50b\/",
        "params": {
            "namespace": "customer_listing"
        }
    }
    /* ... */        
}

Despite this extra super power, a dataSource node is just another UI Component that will end up in the uiRegistry. Its view model constructor factory is Magento_Ui/js/grid/provider. The Magento_Ui/js/grid/provider view model factory returns a view model constructor with the following defaults

//File: vendor/magento//module-ui/view/base/web/js/grid/provider.js
return Element.extend({
    defaults: {
        firstLoad: true,
        storageConfig: {
            component: 'Magento_Ui/js/grid/data-storage',
            provider: '${ $.storageConfig.name }',
            name: '${ $.name }_storage',
            updateUrl: '${ $.update_url }'
        },
        listens: {
            params: 'onParamsChange',
            requestConfig: 'updateRequestConfig'
        }
    },
    //...

Accessing data source data directly view the uiRegistry is pretty straight forward. If you’re on a grid listing page, give the following a try in your javascript console.

requirejs(['uiRegistry'], function(reg){
    dataSource = reg.get('customer_listing.customer_listing_data_source')
    console.log(dataSource.data.items);    
});

Assuming you have customers registered in your system, you should see output something like the following

0:Object
    _rowIndex:0
    actions:Object
    billing_city:"Calder"
    billing_company:null
    billing_country_id:Array[1]
    billing_fax:null
    billing_firstname:"Veronica"
    billing_full:"6146 Honey Bluff Parkway Calder Michigan 49628-7978"
    billing_lastname:"Costello"
    billing_postcode:"49628-7978"
    billing_region:"Michigan"
    billing_street:"6146 Honey Bluff Parkway"
    billing_telephone:"(555) 229-3326"
    billing_vat_id:null
    confirmation:"Confirmation Not Required"
    created_at:"2016-09-01 10:01:38"
    created_in:"Default Store View"
    dob:"1973-12-15 00:00:00"
    email:"roni_cost@example.com"
    entity_id:"1"
    gender:Array[1]
    group_id:Array[1]
    id_field_name:"entity_id"
    lock_expires:"Unlocked"
    name:"Veronica Costello"
    //...    

If you need to access data source data via javascript, this is the most direct way to do it. However, this presents a problem for Magento’s “one view model, one Knockout.js template” front end model. If you look at a Knockout.js view

<!-- File: vendor/magento/module-ui/view/base/web/templates/grid/listing.html -->
<div class="admin__data-grid-wrap" data-role="grid-wrapper">
    <table class="data-grid" data-role="grid">
       <thead>
            <tr each="data: getVisible(), as: '$col'" render="getHeader()"/>
        </thead>
        <tbody>
            <tr class="data-row" repeat="foreach: rows, item: '$row'" css="'_odd-row': $index % 2">
                <td outerfasteach="data: getVisible(), as: '$col'"
                    css="getFieldClass($row())" click="getFieldHandler($row())" template="getBody()"/>
            </tr>
            <tr ifnot="hasData()" class="data-grid-tr-no-data">
                <td attr="colspan: countVisible()" translate="'We couldn\'t find any records.'"/>
            </tr>
        </tbody>
    </table>
</div>

You’ll see the view model gets at the data via a rows property, foreached over here

<!-- File: vendor/magento/module-ui/view/base/web/templates/grid/listing.html -->    
<tr class="data-row" repeat="foreach: rows, item: '$row'" css="'_odd-row': $index % 2">

To understand where this rows property comes from, you’ll need to look at the Knockout.js template’s view model constructor.

//File: vendor/magento/module-ui/view/base/web/js/grid/listing.js    
return Collection.extend({
    defaults: {
        //...
        imports: {
            rows: '${ $.provider }:data.items'
        },
        //...
    },     

The Magento_Ui/js/grid/listing view model uses the imports feature to load a rows property. We learned about imports in the UiClass Data Features article. The template literal (or template literal-ish thing, also covered in our ES6 Template Literals article) above expands to customer_listing.customer_listing_data_source, which means the values for rows will be imported by javascript that looks something like this

requirejs('uiRegistry') .
    get('customer_lising.customer_listing_data_source') .
    data .
    items

While indirect and hard to follow, this mechanism does allow a Knockout.js template to have easier access to this central store of data. Only time will tell if this pattern makes it easier, or harder, for Magento developers trying to create and maintain their expert system user interface elements.

Wrap Up

That brings us to the end of our article, and the near-end of our UI Components series. Next time we’ll reflect on where the UI Components system is at, how Magento’s core developers are coping with it, and some parting tips for surviving forays in to Magento’s front end systems.