Close [x]

Entities of the Magento Testing Framework

Edit this page on GitHub

Constraint

Constraint overview

The Magento Testing Framework (MTF) constraint performs assertions after a test flow. A test flow is a set of test steps without assertions. Each constraint name must be globally unique in Magento application and must be placed in the module to which it belongs. Constraints run automatically after test flow has finished.

Constraints and test flow

Constraint structure

Constraint directory

A module in functional tests (<magento2>/dev/tests/app/Magento/) stores constraints in the Constraint directory. The following image shows the Constraint directory of the Magento_Widget module.

Constraint class

The constraint PHP class must:

  • Have unique name created using the following template Assert{MagentoEntityName}{verification|action|place}. For example:

    • AssertUserSuccessDeleteMessage corresponds to Assert{entityName}{verification}
    • AssertOrderPlaced corresponds to Assert{entityName}{action}
    • AssertProductForm corresponds to Assert{entityName}{place}

  • Extend the Magento\Mtf\Constraint\AbstractConstraint class.

  • Contain the following methods:

    • processAssert() which contains assertions. A PHPUnit_Framework_Assert class (<magento2>/dev/tests/functional/vendor/phpunit/phpunit/src/Framework/Assert.php) can be used to simplify assertions.
    • toString() which returns a success message if the assertion is performed successfully

Constraint arguments

In the MTF, data set values are shared with a test class and constraints. A node name in data set can be complex like item1/item2/item3. The argument name in processAssert() must be the same as the item1 to transfer data from data set to constraint.

If a data set variable is used in the test, and is overwritten, it is transferred as altered to the constraint. Variables can be overwritten in the injectable test case class in test(), __inject() and __prepare() methods, and then passed to the constraint class by return. Furthermore, any returned value of these methods can be used as an argument in constraint.

An object that is not defined in the data set or isn鈥檛 returned from the test case is created using the Object Manager.

Let鈥檚 see the following images for the CreateSimpleProductEntityTest test and the AssertProductPricesOnCategoryPage constraint. Data set from the diagrams contains three variables with data: product, category and price.

Green arrows show that product value is transferred to the test and the constraint.

Orange arrows show that category variable is transferred to the test directly, overwritten by testCreate() method and only then transferred to constraint.

Blue arrow shows that price value is transferred to the constraint only.

Constraint in the test

A test case鈥檚 constraints are nodes in variations of a data set. The data set has references to the PHP classes with assertions.

Constraints are performed in order they listed in the data set. However, you can use prev (previous) and next attributes to set your custom order.

<constraint name="Magento\Catalog\Test\Constraint\AssertCategorySaveMessage" next="Magento\Catalog\Test\Constraint\AssertCategoryForm"/>
<constraint name="Magento\Catalog\Test\Constraint\AssertCategoryForm" prev="Magento\Catalog\Test\Constraint\AssertCategorySaveMessage" next="Magento\Catalog\Test\Constraint\AssertCategoryPage"/>
<constraint name="Magento\Catalog\Test\Constraint\AssertCategoryPage" prev="Magento\Catalog\Test\Constraint\AssertCategoryForm" />
聽聽聽聽

Constraint failure causes interruption of constraints execution within variation, and a test continues to perform from the next variation.

A test can contain constraints from different modules.

聽聽

Be careful when you use constraints from another module. A module that is referred by constraint can be disabled, that fails in the test execution. It is safe to use constraints of different modules in one test case if that modules have hard dependencies. 聽聽

The following example shows the <magento2>/dev/tests/functional/tests/app/Magento/Widget/Test/TestCase/DeleteWidgetEntityTest.xml data set with two constraints. 

<?xml version="1.0" encoding="utf-8"?>
<!--
/**
 * Copyright 漏 2015 Magento. All rights reserved.
 * See COPYING.txt for license details.
 */
 -->
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="../../../../../../vendor/magento/mtf/etc/variations.xsd">
    <testCase name="Magento\Widget\Test\TestCase\DeleteWidgetEntityTest" summary="Delete Widget" ticketId="MAGETWO-28459">
        <variation name="DeleteWidgetEntityTestVariation1">
            <data name="widget/dataset" xsi:type="string">default</data>
            <constraint name="Magento\Widget\Test\Constraint\AssertWidgetSuccessDeleteMessage" />
            <constraint name="Magento\Widget\Test\Constraint\AssertWidgetAbsentOnFrontendHome" />
        </variation>
    </testCase>
</config>

Immediately after the test steps complete, both constraints are performed in the order listed.

Tagging

Tagging enables you to indicate what constraints must be called.

You can tag constraints in <module>/Test/etc/di.xml using a severity argument. Severity tagging of constraints is used for the customization of test run.

You can use the following severity tags:

  • high
  • middle
  • low

To assign severity tags do the following:

  • Create di.xml file in Test/etc of the module.
  • Assign severity to constraints in the following format:
<type name="Magento\[Module_name]\Test\Constraint\Assert...">
    <arguments>
        <argument name="severity" xsi:type="string">high|middle|low</argument>
    </arguments>
</type>

For example, <magento2>/dev/tests/functional/tests/app/Magento/CatalogRule/Test/etc/di.xml:

<?xml version="1.0" encoding="utf-8"?>
<!--
/**
 * Copyright 漏 2015 Magento. All rights reserved.
 * See COPYING.txt for license details.
 */
 -->
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:ObjectManager/etc/config.xsd">
  <type name="Magento\CatalogRule\Test\Constraint\AssertCatalogPriceRuleAppliedProductPage">
    <arguments>
      <argument name="severity" xsi:type="string">high</argument>
    </arguments>
  </type>
  <type name="Magento\CatalogRule\Test\Constraint\AssertCatalogPriceRuleAppliedCatalogPage">
    <arguments>
      <argument name="severity" xsi:type="string">high</argument>
    </arguments>
  </type>
  <type name="Magento\CatalogRule\Test\Constraint\AssertCatalogPriceRuleAppliedShoppingCart">
    <arguments>
      <argument name="severity" xsi:type="string">high</argument>
    </arguments>
  </type>
</config>

How to create constraint

Use case: We want to assert widget availability in a widget grid.

Step 1. What module does it belong?

Widget grid and widget fixture are related to the Magento_Widget module. Thus, we need to create constraint in the Magento_Widget module, in <magento2>/dev/tests/functional/tests/app/Magento/Widget/Test/Constraint.

Step 2. What name should constraint have?

Using constraint naming principle, the constraint should be named as AssertWidgetInGrid.

Step 3. Create <magento2>/dev/tests/functional/tests/app/Magento/Widget/Test/Constraint/AssertWidgetInGrid.php with required structure

Step 4. Implement assertion in processAssert()

Assertion logic: Take title of the widget from the widget fixture, open the page with a grid, check if the grid has our title.

How to use constraint

To use constraint we鈥檝e created in previous section, add a corresponding node to the data set of your test

<constraint name="Magento\Widget\Test\Constraint\AssertWidgetInGrid" />

in the order that it must be performed.

<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="../../../../../../vendor/magento/mtf/etc/variations.xsd">
    <testCase name="Magento\Widget\Test\TestCase\CreateWidgetEntityTest" summary="Create Widget" ticketId="MAGETWO-27916">
        <variation name="CreateWidgetEntityTestVariation1">
            <data name="widget/data/code" xsi:type="string">CMS Static Block</data>
            <data name="widget/data/theme_id" xsi:type="string">Magento Luma</data>
            <data name="widget/data/title" xsi:type="string">Title_%isolation%</data>
            <data name="widget/data/store_ids/dataset" xsi:type="string">all_store_views</data>
            <data name="widget/data/widget_instance/dataset" xsi:type="string">on_category</data>
            <data name="widget/data/parameters/dataset" xsi:type="string">cmsStaticBlock</data>
            <constraint name="Magento\Widget\Test\Constraint\AssertWidgetSuccessSaveMessage" />
            <constraint name="Magento\Widget\Test\Constraint\AssertWidgetInGrid" />
            <constraint name="Magento\Widget\Test\Constraint\AssertWidgetOnFrontendInCatalog" />     
        </variation>
    </testCase>
</config>