mirror of
https://github.com/saltstack/salt.git
synced 2025-04-17 10:10:20 +00:00
Update test related docs to use the newer nox sessions.
Signed-off-by: Pedro Algarvio <palgarvio@vmware.com>
This commit is contained in:
Pedro Algarvio
Megan Wilhite
parent
692c67817e
commit
8f4381f203
3 changed files with 76 additions and 107 deletions
|
|
@ -33,20 +33,20 @@ python3 you would need to specify the zeromq transport and python3.
|
|||
|
||||
.. code-block:: bash
|
||||
|
||||
nox -e 'pytest-zeromq-3(coverage=False)'
|
||||
nox -e 'test-zeromq-3(coverage=False)'
|
||||
|
||||
As a contrast, when using the deprecated ``runtests.py`` test runner, the
|
||||
command would be:
|
||||
And because zeromq is the default transport, the following nox session can also be used:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
nox -e 'runtests-zeromq-3(coverage=False)'
|
||||
nox -e 'test-zeromq-3(coverage=False)'
|
||||
|
||||
|
||||
To run all the tests but on the tcp transport, you would need to specify the tcp session.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
nox -e 'pytest-tcp-3(coverage=False)'
|
||||
nox -e 'test-tcp-3(coverage=False)'
|
||||
|
||||
As a contrast, when using the deprecated ``runtests.py`` test runner, the
|
||||
command would be:
|
||||
|
|
@ -94,7 +94,7 @@ Salt's test suite is located in the ``tests/`` directory in the root of
|
|||
Salt's codebase.
|
||||
|
||||
With the migration to PyTest, Salt has created a separate directory for tests
|
||||
that are written taking advantage of the full pottential of PyTest. These are
|
||||
that are written taking advantage of the full potential of PyTest. These are
|
||||
located under ``tests/pytests``.
|
||||
|
||||
As for the old test suite, it is divided into two main groups:
|
||||
|
|
@ -174,14 +174,14 @@ all of the tests included in Salt's test suite:
|
|||
|
||||
.. code-block:: bash
|
||||
|
||||
nox -e 'pytest-zeromq-3(coverage=False)'
|
||||
nox -e 'test-3(coverage=False)'
|
||||
|
||||
For more information about options you can pass the test runner, see the
|
||||
``--help`` option:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
nox -e 'pytest-zeromq-3(coverage=False)' -- --help
|
||||
nox -e 'test-3(coverage=False)' -- --help
|
||||
|
||||
.. _running-test-subsections:
|
||||
|
||||
|
|
@ -192,23 +192,17 @@ Instead of running the entire test suite all at once, which can take a long time
|
|||
there are several ways to run only specific groups of tests or individual tests:
|
||||
|
||||
* Run :ref:`unit tests only<running-unit-tests-no-daemons>`: ``nox -e
|
||||
'pytest-zeromq-3(coverage=False)' -- tests/unit/``, using the deprecated ``runtests.py`` the command would be ``nox -e
|
||||
'runtests-zeromq-3(coverage=False)' -- --unit-tests``
|
||||
'test-3(coverage=False)' -- tests/unit/``.
|
||||
* Run unit and integration tests for states: ``nox -e
|
||||
'pytest-zeromq-3(coverage=False)' -- tests/unit/states/ tests/integration/states/``, using the deprecated
|
||||
``runtests.py`` the command would be ``nox -e 'runtests-zeromq-3(coverage=False)' -- --state-tests``
|
||||
* Run integration tests for an individual module: ``nox -e 'pytest-zeromq-3(coverage=False)' --
|
||||
tests/integration/modules/test_virt.py``, using the deprecated ``runtests.py`` the command would be ``nox -e
|
||||
'runtests-zeromq-3(coverage=False)' -- -n integration.modules.test_virt``
|
||||
* Run unit tests for an individual module: ``nox -e 'pytest-zeromq-3(coverage=False)' --
|
||||
tests/unit/modules/test_virt.py``, using the deprecated ``runtests.py`` the command would be ``nox -e
|
||||
'runtests-zeromq-3(coverage=False)' -- -n unit.modules.test_virt``
|
||||
'test-3(coverage=False)' -- tests/unit/states/ tests/integration/states/``.
|
||||
* Run integration tests for an individual module: ``nox -e 'test-3(coverage=False)' --
|
||||
tests/pytests/integration/modules/test_virt.py``.
|
||||
* Run unit tests for an individual module: ``nox -e 'test-3(coverage=False)' --
|
||||
tests/unit/modules/test_virt.py``.
|
||||
* Run an individual test by using the class and test name (this example is for the
|
||||
``test_default_kvm_profile`` test in the ``tests/integration/module/test_virt.py``):
|
||||
``nox -e 'pytest-zeromq-3(coverage=False)' --
|
||||
tests/integration/modules/test_virt.py::VirtTest::test_default_kvm_profile``, using the deprecated ``runtests.py`` the
|
||||
command would be ``nox -e 'runtests-zeromq-3(coverage=False)' -- -n
|
||||
integration.modules.test_virt.VirtTest.test_default_kvm_profile``
|
||||
``test_default_kvm_profile`` test in the ``tests/pytests/integration/module/test_virt.py``):
|
||||
``nox -e 'test-3(coverage=False)' --
|
||||
tests/pytests/integration/modules/test_virt.py::VirtTest::test_default_kvm_profile``.
|
||||
|
||||
|
||||
For more specific examples of how to run various test subsections or individual
|
||||
|
|
@ -231,14 +225,7 @@ integration test daemons, simply add the unit directory as an argument:
|
|||
|
||||
.. code-block:: bash
|
||||
|
||||
nox -e 'pytest-zeromq-3(coverage=False)' -- tests/unit/
|
||||
|
||||
As a contrast, when using the deprecated ``runtests.py`` test runner, the
|
||||
command would be:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
nox -e 'runtests-zeromq-3(coverage=False)' -- --unit-tests
|
||||
nox -e 'test-3(coverage=False)' -- tests/unit/
|
||||
|
||||
All of the other options to run individual tests, entire classes of tests, or
|
||||
entire test modules still apply.
|
||||
|
|
@ -269,14 +256,7 @@ To run tests marked as destructive, set the ``--run-destructive`` flag:
|
|||
|
||||
.. code-block:: bash
|
||||
|
||||
nox -e 'pytest-zeromq-3(coverage=False)' -- --run-destructive
|
||||
|
||||
As a contrast, when using the deprecated ``runtests.py`` test runner, the
|
||||
command would be:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
nox -e 'runtests-zeromq-3(coverage=False)' -- --run-destructive
|
||||
nox -e 'test-3(coverage=False)' -- --run-destructive
|
||||
|
||||
|
||||
Running Cloud Provider Tests
|
||||
|
|
@ -319,18 +299,12 @@ must be provided:
|
|||
do not include the single quotes.
|
||||
|
||||
Once all of the valid credentials for the cloud provider have been supplied, the
|
||||
cloud provider tests can be run by setting the ``--cloud-provider-tests`` flag:
|
||||
cloud provider tests can be run like:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
nox -e 'pytest-cloud-3(coverage=False)'
|
||||
nox -e 'test-cloud-3(coverage=False)'
|
||||
|
||||
As a contrast, when using the deprecated ``runtests.py`` test runner, the
|
||||
command would be:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
nox -e 'runtests-cloud-3(coverage=False)'
|
||||
|
||||
Automated Test Runs
|
||||
===================
|
||||
|
|
@ -535,37 +509,27 @@ Test Helpers
|
|||
------------
|
||||
|
||||
Several Salt-specific helpers are available. A full list is available by inspecting
|
||||
functions exported in `tests.support.helpers`.
|
||||
functions exported under `tests/support/*.py`.
|
||||
|
||||
`@expensiveTest` -- Designates a test which typically requires a relatively costly
|
||||
external resource, like a cloud virtual machine. This decorator is not normally
|
||||
used by developers outside of the Salt core team.
|
||||
Test Markers
|
||||
------------
|
||||
|
||||
`@destructiveTest` -- Marks a test as potentially destructive. It will not be run
|
||||
by the test runner unless the ``-run-destructive`` test is expressly passed.
|
||||
`@pytest.mark.expensive_test` -- Designates a test which typically requires a
|
||||
relatively costly external resource, like a cloud virtual machine. This decorator
|
||||
is not normally used by developers outside of the Salt core team.
|
||||
|
||||
`@requires_network` -- Requires a network connection for the test to operate
|
||||
successfully. If a network connection is not detected, the test will not run.
|
||||
`@pytest.mark.destructive_test` -- Marks a test as potentially destructive. It
|
||||
will not be run unless the ``--run-destructive`` flag is expressly passed.
|
||||
|
||||
`@requires_salt_modules` -- Requires all the modules in a list of modules in
|
||||
order for the test to be executed. Otherwise, the test is skipped.
|
||||
`@pytest.mark.requires_network` -- Requires a network connection for the test to
|
||||
operate successfully. If a network connection is not detected, the test will not run.
|
||||
|
||||
`@requires_system_grains` -- Loads and passes the grains on the system as an
|
||||
keyword argument to the test function with the name `grains`.
|
||||
These are just a small preview of the supported marker. For a full listing, please
|
||||
run:
|
||||
|
||||
`@skip_if_binaries_missing(['list', 'of', 'binaries'])` -- If called from inside a test,
|
||||
the test will be skipped if the binaries are not all present on the system.
|
||||
.. code-block:: bash
|
||||
|
||||
`@skip_if_not_root` -- If the test is not executed as root, it will be skipped.
|
||||
|
||||
`@with_system_user` -- Creates and optionally destroys a system user within a test case.
|
||||
See implementation details in `tests.support.helpers` for details.
|
||||
|
||||
`@with_system_group` -- Creates and optionally destroys a system group within a test case.
|
||||
See implementation details in `tests.support.helpers` for details.
|
||||
|
||||
`@with_system_user_and_group` -- Creates and optionally destroys a system user and group
|
||||
within a test case. See implementation details in `tests.support.helpers` for details.
|
||||
nox -e 'test-3(coverage=False)' -- --markers
|
||||
|
||||
|
||||
.. _kitchen-salt jenkins setup: https://kitchen.saltproject.io/docs/file/docs/jenkins.md
|
||||
|
|
|
|||
|
|
@ -780,7 +780,7 @@ executed:
|
|||
|
||||
.. code-block:: bash
|
||||
|
||||
nox -e 'pytest-3(coverage=False)' -- -v tests/pytests/unit/modules/test_fib.py
|
||||
nox -e 'test-3(coverage=False)' -- -v tests/pytests/unit/modules/test_fib.py
|
||||
|
||||
This will report the status of the test: success, failure, or error. The
|
||||
``-v`` flag increases output verbosity.
|
||||
|
|
|
|||
|
|
@ -97,7 +97,7 @@ Once all of the :ref:`requirements <getting_set_up_for_tests>` are installed, th
|
|||
|
||||
.. code-block:: bash
|
||||
|
||||
nox -e 'pytest-zeromq-3(coverage=False)'
|
||||
nox -e 'test-3(coverage=False)'
|
||||
|
||||
The command above, if executed without any options, will run the entire suite of
|
||||
integration and unit tests. Some tests require certain flags to run, such as
|
||||
|
|
@ -111,10 +111,10 @@ You can pass any pytest options after the nox command like so:
|
|||
|
||||
.. code-block:: bash
|
||||
|
||||
nox -e 'pytest-zeromq-3(coverage=False)' -- tests/unit/modules/test_ps.py
|
||||
nox -e 'test-3(coverage=False)' -- tests/unit/modules/test_ps.py
|
||||
|
||||
The above command will run the ``test_ps.py`` test with the zeromq transport, python3,
|
||||
and pytest. The option after ``--`` can include any pytest options.
|
||||
and pytest. Pass any pytest options after `--`
|
||||
|
||||
Running Integration Tests
|
||||
-------------------------
|
||||
|
|
@ -160,7 +160,7 @@ integration module tests:
|
|||
|
||||
.. code-block:: bash
|
||||
|
||||
nox -e 'pytest-zeromq-3(coverage=False)' -- tests/integration/modules/
|
||||
nox -e 'test-3(coverage=False)' -- tests/integration/modules/
|
||||
|
||||
Running Unit Tests
|
||||
------------------
|
||||
|
|
@ -173,7 +173,7 @@ do and execute very quickly compared to the integration tests.
|
|||
|
||||
.. code-block:: bash
|
||||
|
||||
nox -e 'pytest-zeromq-3(coverage=False)' -- tests/unit/
|
||||
nox -e 'test-3(coverage=False)' -- tests/unit/
|
||||
|
||||
|
||||
.. _running-specific-tests:
|
||||
|
|
@ -190,7 +190,7 @@ integration test directory, you must provide the file path.
|
|||
|
||||
.. code-block:: bash
|
||||
|
||||
nox -e 'pytest-zeromq-3(coverage=False)' -- tests/integration/modules/test_pillar.py
|
||||
nox -e 'test-3(coverage=False)' -- tests/pytests/integration/modules/test_pillar.py
|
||||
|
||||
Some test files contain only one test class while other test files contain multiple
|
||||
test classes. To run a specific test class within the file, append the name of
|
||||
|
|
@ -198,14 +198,14 @@ the test class to the end of the file path:
|
|||
|
||||
.. code-block:: bash
|
||||
|
||||
nox -e 'pytest-zeromq-3(coverage=False)' -- tests/integration/modules/test_pillar.py::PillarModuleTest
|
||||
nox -e 'test-3(coverage=False)' -- tests/pytests/integration/modules/test_pillar.py::PillarModuleTest
|
||||
|
||||
To run a single test within a file, append both the name of the test class the
|
||||
individual test belongs to, as well as the name of the test itself:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
nox -e 'pytest-zeromq-3(coverage=False)' -- tests/integration/modules/test_pillar.py::PillarModuleTest::test_data
|
||||
nox -e 'test-3(coverage=False)' -- tests/pytests/integration/modules/test_pillar.py::PillarModuleTest::test_data
|
||||
|
||||
|
||||
The following command is an example of how to execute a single test found in
|
||||
|
|
@ -213,7 +213,7 @@ the ``tests/unit/modules/test_cp.py`` file:
|
|||
|
||||
.. code-block:: bash
|
||||
|
||||
nox -e 'pytest-zeromq-3(coverage=False)' -- tests/unit/modules/test_cp.py::CpTestCase::test_get_file_not_found
|
||||
nox -e 'test-3(coverage=False)' -- tests/pytests/unit/modules/test_cp.py::CpTestCase::test_get_file_not_found
|
||||
|
||||
Writing Tests for Salt
|
||||
======================
|
||||
|
|
@ -322,22 +322,15 @@ Tests can be written to alter the system they are running on. This capability
|
|||
is what fills in the gap needed to properly test aspects of system management
|
||||
like package installation.
|
||||
|
||||
To write a destructive test, import and use the ``destructiveTest`` decorator for
|
||||
the test method:
|
||||
To write a destructive test, decorate the test function with the
|
||||
``destructive_test``:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import integration
|
||||
from tests.support.helpers import destructiveTest
|
||||
|
||||
|
||||
class PkgTest(integration.ModuleCase):
|
||||
@destructiveTest
|
||||
def test_pkg_install(self):
|
||||
ret = self.run_function("pkg.install", name="finch")
|
||||
self.assertSaltTrueReturn(ret)
|
||||
ret = self.run_function("pkg.purge", name="finch")
|
||||
self.assertSaltTrueReturn(ret)
|
||||
@pytest.mark.destructive_test
|
||||
def test_pkg_install(salt_cli):
|
||||
ret = salt_cli.run("pkg.install", "finch")
|
||||
assert ret
|
||||
|
||||
|
||||
Writing Unit Tests
|
||||
|
|
@ -359,7 +352,7 @@ Salt's unit tests utilize Python's mock class as well as `MagicMock`_. The
|
|||
``@patch`` decorator is also heavily used when "blocking all the exits".
|
||||
|
||||
A simple example of a unit test currently in use in Salt is the
|
||||
``test_get_file_not_found`` test in the ``tests/unit/modules/test_cp.py`` file.
|
||||
``test_get_file_not_found`` test in the ``tests/pytests/unit/modules/test_cp.py`` file.
|
||||
This test uses the ``@patch`` decorator and ``MagicMock`` to mock the return
|
||||
of the call to Salt's ``cp.hash_file`` execution module function. This ensures
|
||||
that we're testing the ``cp.get_file`` function directly, instead of inadvertently
|
||||
|
|
@ -412,10 +405,10 @@ in the following docs:
|
|||
Add a python module dependency to the test run
|
||||
----------------------------------------------
|
||||
|
||||
The test dependencies for python modules are managed under the ``requirements/static``
|
||||
directory. You will need to add your module to the appropriate file under ``requirements/static``.
|
||||
The test dependencies for python modules are managed under the ``requirements/static/ci``
|
||||
directory. You will need to add your module to the appropriate file under ``requirements/static/ci``.
|
||||
When ``pre-commit`` is run it will create all of the needed requirement files
|
||||
under ``requirements/static/py3{5,6,7}``. Nox will then use these files to install
|
||||
under ``requirements/static/ci/py3{6,7,8,9}``. Nox will then use these files to install
|
||||
the requirements for the tests.
|
||||
|
||||
Add a system dependency to the test run
|
||||
|
|
@ -436,16 +429,28 @@ can be used
|
|||
|
||||
.. code-block:: python
|
||||
|
||||
# Import logging handler
|
||||
from tests.support.helpers import TstSuiteLoggingHandler
|
||||
def test_issue_58763_a(tmp_path, modules, state_tree, caplog):
|
||||
|
||||
venv_dir = tmp_path / "issue-2028-pip-installed"
|
||||
|
||||
sls_contents = """
|
||||
test.random_hash:
|
||||
module.run:
|
||||
- size: 10
|
||||
- hash_type: md5
|
||||
"""
|
||||
with pytest.helpers.temp_file("issue-58763.sls", sls_contents, state_tree):
|
||||
with caplog.at_level(logging.DEBUG):
|
||||
ret = modules.state.sls(
|
||||
mods="issue-58763",
|
||||
)
|
||||
assert len(ret.raw) == 1
|
||||
for k in ret.raw:
|
||||
assert ret.raw[k]["result"] is True
|
||||
assert (
|
||||
"Detected legacy module.run syntax: test.random_hash" in caplog.messages
|
||||
)
|
||||
|
||||
# .. inside test
|
||||
with TstSuiteLoggingHandler() as handler:
|
||||
for message in handler.messages:
|
||||
if message.startswith("ERROR: This is the error message we seek"):
|
||||
break
|
||||
else:
|
||||
raise AssertionError("Did not find error message")
|
||||
|
||||
|
||||
Automated Test Runs
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue