Usage¶

The above is a trivial example, but ordering is respected across test files.

Note

The scope of the ordering is global per default, e.g. tests with lower ordinal numbers are always executed before tests with higher numbers in the same test session, regardless of the module and class they reside in. This can be changed by using the --order-scope option.

Ordering is done either absolutely, by using ordinal numbers that define the order, or relative to other tests, using the before and after attributes of the marker.

Ordering by numbers¶

The order can be defined by ordinal numbers, or by ordinal strings.

Order by index¶

As already shown above, the order can be defined using ordinal numbers. There is a long form that uses the keyword index, and a short form, that uses just the ordinal number–both are shown in the example below. The long form may be better readable if you want to combine it with a dependency marker (before or after, see below). Negative numbers are also allowed–they are used the same way as indices are used in Python lists, e.g. to count from the end:

import pytest


@pytest.mark.order(-2)
def test_three():
    assert True


@pytest.mark.order(index=-1)
def test_four():
    assert True


@pytest.mark.order(index=2)
def test_two():
    assert True


@pytest.mark.order(1)
def test_one():
    assert True

$ pytest test_foo.py -vv
============================= test session starts ==============================
platform darwin -- Python 3.7.1, pytest-5.4.3, py-1.8.1, pluggy-0.13.1 -- env/bin/python
plugins: order
collected 4 items

test_foo.py:17: test_one PASSED
test_foo.py:12: test_two PASSED
test_foo.py:3: test_three PASSED
test_foo.py:7: test_four PASSED

=========================== 4 passed in 0.02 seconds ===========================

There is no limit for the numbers that can be used in this way.

Order using ordinals¶

Instead of the numbers, you can use ordinal names such as “first”, “second”, “last”, and “second_to_last”. These are convenience notations, and have the same effect as the numbers 0, 1, -1 and -2, respectively, that have been shown above:

import pytest


@pytest.mark.order("second_to_last")
def test_three():
    assert True


@pytest.mark.order("last")
def test_four():
    assert True


@pytest.mark.order("second")
def test_two():
    assert True


@pytest.mark.order("first")
def test_one():
    assert True

$ pytest test_foo.py -vv
============================= test session starts ==============================
platform darwin -- Python 3.7.1, pytest-5.4.3, py-1.8.1, pluggy-0.13.1 -- env/bin/python
plugins: order
collected 4 items

test_foo.py:17: test_one PASSED
test_foo.py:12: test_two PASSED
test_foo.py:3: test_three PASSED
test_foo.py:7: test_four PASSED

=========================== 4 passed in 0.02 seconds ===========================

Convenience names are only defined for the first and the last 8 numbers. Here is the complete list with the corresponding numbers:

“first”: 0
“second”: 1
“third”: 2
“fourth”: 3
“fifth”: 4
“sixth”: 5
“seventh”: 6
“eighth”: 7
“last”: -1
“second_to_last”: -2
“third_to_last”: -3
“fourth_to_last”: -4
“fifth_to_last”: -5
“sixth_to_last”: -6
“seventh_to_last”: -7
“eighth_to_last”: -8

Markers on class level¶

If setting an order mark on class level, all tests in this class will be handled as having the same ordinal marker, e.g. the class as a whole will be reordered without changing the test order inside the test class:

import pytest


@pytest.mark.order(1)
class Test1:
    def test_1(self):
        assert True

    def test_2(self):
        assert True


@pytest.mark.order(0)
class Test2:
    def test_1(self):
        assert True

    def test_2(self):
        assert True

$ pytest -vv test_ordinal_class_mark.py
============================= test session starts ==============================
...
collected 4 items

test_ordinal_class_mark.py::Test2::test_1 PASSED
test_ordinal_class_mark.py::Test2::test_2 PASSED
test_ordinal_class_mark.py::Test1::test_1 PASSED
test_ordinal_class_mark.py::Test1::test_2 PASSED

Handling of unordered tests¶

By default, tests with no order mark are executed after all tests with positive ordinal numbers (or the respective names), and before tests with negative ordinal numbers. The order of these tests in relationship to each other is not changed. This behavior will slightly change if the option --sparse-ordering is used and the ordinals are not contiguous.

Order relative to other tests¶

The test order can be defined relative to other tests, which are referenced by their name. The marker attributes before and after can be used to define the order relative to these tests:

import pytest


@pytest.mark.order(after="test_second")
def test_third():
    assert True


def test_second():
    assert True


@pytest.mark.order(before="test_second")
def test_first():
    assert True

$ pytest test_foo.py -vv
============================= test session starts ==============================
platform darwin -- Python 3.7.1, pytest-5.4.3, py-1.8.1, pluggy-0.13.1 -- env/bin/python
plugins: order
collected 3 items

test_foo.py:11: test_first PASSED
test_foo.py:7: test_second PASSED
test_foo.py:4: test_third PASSED

=========================== 4 passed in 0.02 seconds ===========================

Referencing of tests in other classes or modules¶

If a test is referenced using the unqualified test name as shown in the example above, the test is assumed to be in the current module and the current class, if any. For tests in other classes in the same module the class name with a :: suffix has to be prepended to the test name:

import pytest


class TestA:
    @pytest.mark.order(after="TestB::test_c")
    def test_a(self):
        assert True

    def test_b(self):
        assert True


class TestB:
    def test_c(self):
        assert True

If the referenced test lives in another module, you have to use the nodeid of the test, or a part of the nodeid that is sufficient to make it uniquely identifiable (the nodeid is the test ID that pytest prints if you run it with the -v option). Let’s say we have the following module and test layout:

test_module_a.py
    TestA
        test_a
        test_b
test_module_b.py
    test_a
    test_b
test_module_c
    test_submodule.py
        test_1
        test_2

Suppose the tests in test_module_b shall depend on tests in the other modules, this could be expressed like:

test_module_b.py

import pytest


@pytest.mark.order(after="test_module_a.py::TestA::test_a")
def test_a():
    assert True


@pytest.mark.order(before="test_module_c/test_submodule.py::test_2")
def test_b():
    assert True

If an unknown test is referenced, a warning is issued and the execution order of the test in is not changed.

Markers on class level¶

As for ordinal markers, markers on class level are handled as if they are set to each individual test in the class. Additionally to referencing single tests, you can also reference test classes if using the before or after marker attributes:

import pytest


@pytest.mark.order(after="Test2")
class Test1:
    def test_1(self):
        assert True

    def test_2(self):
        assert True


class Test2:
    def test_1(self):
        assert True

    def test_2(self):
        assert True

In this case, the tests in the marked class will be ordered behind all tests in the referenced class:

$ pytest -vv test_relative_class_mark.py
============================= test session starts ==============================
...
collected 4 items

test_relative_class_marker.py::Test2::test_1 PASSED
test_relative_class_marker.py::Test2::test_2 PASSED
test_relative_class_marker.py::Test1::test_1 PASSED
test_relative_class_marker.py::Test1::test_2 PASSED

Combination of absolute and relative ordering¶

If you combine absolute and relative order markers, the ordering is first done for the absolute markers (e.g. the ordinals), and afterwards for the relative ones. This means that relative ordering always takes preference:

import pytest


@pytest.mark.order(index=0, after="test_second")
def test_first():
    assert True


@pytest.mark.order(1)
def test_second():
    assert True

In this case, test_second will be executed before test_first, regardless of the ordinal markers.

Several relationships for the same marker¶

If you need to order a certain test relative to more than one other test, you can add more than one test name to the before or after marker attributes by using a list or tuple of test names:

import pytest


@pytest.mark.order(after=["test_second", "other_module.py::test_other"])
def test_first():
    assert True


def test_second():
    assert True

This will ensure that test_first is executed both after test_second and after test_other which resides in the module other_module.py.

Relationships with parameterized tests¶

If you want to reference parametrized tests, you can just use the test name without the parameter part, for example:

import pytest


@pytest.mark.order(after=["test_second"])
def test_first():
    assert True


@pytest.mark.parametrize("param", [1, 2, 3])
def test_second(param):
    assert True

Note that using the fully qualified test name, which would include the parameter (in this case test_second[1], test_second[2] etc) is not supported.

Multiple test order markers¶

More than one order marker can be set for the test. In this scenario test will be executed several times in the defined order.

Combination of absolute and relative ordering¶

import pytest


@pytest.mark.order(1)
@pytest.mark.order(-1)
def test_one_and_seven():
    pass


@pytest.mark.order(2)
@pytest.mark.order(-2)
def test_two_and_six():
    pass


def test_four():
    pass


@pytest.mark.order(before="test_four")
@pytest.mark.order(after="test_four")
def test_three_and_five():
    pass

When a test has multiple order markers, each marker turns into a pytest ParameterSet, so it will be run multiple times.

============================= test session starts =============================
collecting ... collected 7 items
test_multiple_markers.py::test_one_and_seven[index=1]
test_multiple_markers.py::test_two_and_six[index=2]
test_multiple_markers.py::test_three_and_five[before=test_four]
test_multiple_markers.py::test_four
test_multiple_markers.py::test_three_and_five[after=test_four]
test_multiple_markers.py::test_two_and_six[index=-2]
test_multiple_markers.py::test_one_and_seven[index=-1]
============================== 7 passed in 0.02s ==============================

Parametrized tests¶

Although multiple test order markers create their own parametrization, it can be used with parametrized tests.

import pytest


@pytest.mark.order(1)
@pytest.mark.order(3)
@pytest.mark.parametrize("foo", ["aaa", "bbb"])
def test_one_and_three(foo):
    pass


@pytest.mark.order(4)
@pytest.mark.parametrize("bar", ["bbb", "ccc"])
@pytest.mark.order(2)
def test_two_and_four(bar):
    pass

collecting ... collected 8 items
test_multiple_markers.py::test_one_and_three[index=1-aaa] PASSED         [ 12%]
test_multiple_markers.py::test_one_and_three[index=1-bbb] PASSED         [ 25%]
test_multiple_markers.py::test_two_and_four[index=2-bbb] PASSED          [ 37%]
test_multiple_markers.py::test_two_and_four[index=2-ccc] PASSED          [ 50%]
test_multiple_markers.py::test_one_and_three[index=3-aaa] PASSED         [ 62%]
test_multiple_markers.py::test_one_and_three[index=3-bbb] PASSED         [ 75%]
test_multiple_markers.py::test_two_and_four[index=4-bbb] PASSED          [ 87%]
test_multiple_markers.py::test_two_and_four[index=4-ccc] PASSED          [100%]
============================== 8 passed in 0.02s ==============================

Usage¶

Ordering by numbers¶

Order by index¶

Order using ordinals¶

Markers on class level¶

Handling of unordered tests¶

Order relative to other tests¶

Referencing of tests in other classes or modules¶

Markers on class level¶

Combination of absolute and relative ordering¶

Several relationships for the same marker¶

Relationships with parameterized tests¶

Multiple test order markers¶

Combination of absolute and relative ordering¶

Parametrized tests¶

Table of Contents

Previous topic

Next topic

This Page