mirror of
https://github.com/saltstack/salt.git
synced 2025-04-16 09:40:20 +00:00
Merge freeze into master (#60396)
* Merge 3002.6 bugfix changes (#59822) * Pass `CI_RUN` as an environment variable to the test run. This allows us to know if we're running the test suite under a CI environment or not and adapt/adjust if needed * Migrate `unit.setup` to PyTest * Backportae36b15just for test_install.py * Only skip tests on CI runs * Always store git sha in _version.py during installation * Fix PEP440 compliance. The wheel metadata version 1.2 states that the package version MUST be PEP440 compliant. This means that instead of `3002.2-511-g033c53eccb`, the salt version string should look like `3002.2+511.g033c53eccb`, a post release of `3002.2` ahead by 511 commits with the git sha `033c53eccb` * Fix and migrate `tests/unit/test_version.py` to PyTest * Skip test if `easy_install` is not available * We also need to be PEP440 compliant when there's no git history * Allow extra_filerefs as sanitized kwargs for SSH client * Fix regression on cmd.run when passing tuples as cmd Co-authored-by: Alexander Graul <agraul@suse.com> * Add unit tests to ensure cmd.run accepts tuples * Add unit test to check for extra_filerefs on SSH opts * Add changelog file * Fix comment for test case * Fix unit test to avoid failing on Windows * Skip failing test on windows * Fix test to work on Windows * Add all ssh kwargs to sanitize_kwargs method * Run pre-commit * Fix pylint * Fix cmdmod loglevel and module_names tests * Fix pre-commit * Skip ssh tests if binary does not exist * Use setup_loader for cmdmod test * Prevent argument injection in restartcheck * Add changelog for restartcheck fix * docs_3002.6 * Add back tests removed in merge Co-authored-by: Pedro Algarvio <pedro@algarvio.me> Co-authored-by: Megan Wilhite <megan.wilhite@gmail.com> Co-authored-by: Bryce Larson <brycel@vmware.com> Co-authored-by: Pablo Suárez Hernández <psuarezhernandez@suse.com> Co-authored-by: Alexander Graul <agraul@suse.com> Co-authored-by: Frode Gundersen <fgundersen@saltstack.com> * Remove glance state module in favor of glance_image * update wording in changelog * bump deprecation warning to Silicon. * Updating warnutil version to Phosphorous. * Update salt/modules/keystone.py Co-authored-by: Megan Wilhite <megan.wilhite@gmail.com> * Check $HOMEBREW_PREFIX when linking against libcrypto When loading `libcrypto`, Salt checks for a Homebrew installation of `openssl` at Homebrew's default prefix of `/usr/local`. However, on Apple Silicon Macs, Homebrew's default installation prefix is `/opt/homebrew`. On all platforms, the prefix is configurable. If Salt doesn't find one of those `libcrypto`s, it will fall back on the un-versioned `/usr/lib/libcrypto.dylib`, which will cause the following crash: Application Specific Information: /usr/lib/libcrypto.dylib abort() called Invalid dylib load. Clients should not load the unversioned libcrypto dylib as it does not have a stable ABI. This commit checks $HOMEBREW_PREFIX instead of hard-coding `/usr/local`. * Add test case * Add changelog for 59808 * Add changelog entry * Make _find_libcrypto fail on Big Sur if it can't find a library Right now, if `_find_libcrypto` can't find any externally-managed versions of libcrypto, it will fall back on the pre-Catalina un-versioned system libcrypto. This does not exist on Big Sur and it would be better to raise an exception here rather than crashing later when trying to open it. * Update _find_libcrypto tests This commit simplifies the unit tests for _find_libcrypto by mocking out the host's filesystem and testing the common libcrypto installations (brew, ports, etc.) on Big Sur. It simplifies the tests for falling back on system versions of libcrypto on previous versions of macOS. * Fix description of test_find_libcrypto_with_system_before_catalina * Patch sys.platform for test_rsax931 tests * modules/match: add missing "minion_id" in Pillar example The documented Pillar example for `match.filter_by` lacks the `minion_id` parameter. Without it, the assignment won't work as expected. - fix documentation - add tests: - to prove the misbehavior of the documented example - to prove the proper behaviour when supplying `minion_id` - to ensure some misbehaviour observed with compound matchers doesn't occur * Fix for issue #59773 - When instantiating the loader grab values of grains and pillars if they are NamedLoaderContext instances. - The loader uses a copy of opts. - Impliment deepcopy on NamedLoaderContext instances. * Add changelog for #59773 * _get_initial_pillar function returns pillar * Fix linter issues * Clean up test * Bump deprecation release for neutron * Uncomment Sulfur release name * Removing the _ext_nodes deprecation warning and alias. * Adding changelog. * Renaming changelog file. * Update 59804.removed * Initial pass at fips_mode config option * Fix pre-commit * Fix tests and add changelog * update docs 3003 * update docs 3003 - newline * Fix warts in changelog * update releasenotes 3003 * add ubuntu-2004-amd64 m2crypto pycryptodome and tcp tests * add distro_arch * changing the cloud platforms file missed in1a9b7be0e2* Update __utils__ calls to import utils in azure * Add changelog for 59744 * Fix azure unit tests and move to pytest * Use contextvars from site-packages for thin If a contextvars package exists one of the site-packages locations use it for the generated thin tarball. This overrides python's builtin contextvars and allows salt-ssh to work with python <=3.6 even when the master's python is >3.6 (Fixes #59942) * Add regression test for #59942 * Add changelog for #59942 * Update filemap to include test_py_versions * Fix broken thin tests * Always install the `contextvars` backport, even on Py3.7+ Without this change, salt-ssh cannot target systems with Python <= 3.6 * Use salt-factories to handle the container. Don't override default roster * Fix thin tests on windows * No need to use warn log level here * Fix getsitepackages for old virtualenv versions * Add explicit pyobjc reqs * Add back the passthrough stuff * Remove a line so pre-commit will run * Bugfix release docs * Bugfix release docs * Removing pip-compile log files * Bump requirements to address a few security issues * Address traceback on macOS ``` Traceback (most recent call last): File "setup.py", line 1448, in <module> setup(distclass=SaltDistribution) File "/Users/jenkins/setup-tests/.venv/lib/python3.7/site-packages/setuptools/__init__.py", line 153, in setup return distutils.core.setup(**attrs) File "/opt/salt/lib/python3.7/distutils/core.py", line 108, in setup _setup_distribution = dist = klass(attrs) File "setup.py", line 1068, in __init__ self.update_metadata() File "setup.py", line 1074, in update_metadata attrvalue = getattr(self, attrname, None) File "setup.py", line 1182, in _property_install_requires install_requires += _parse_requirements_file(reqfile) File "setup.py", line 270, in _parse_requirements_file platform.python_version(), _parse_op(op), _parse_ver(ver) File "setup.py", line 247, in _check_ver return getattr(operator, "__{}__".format(op))(pyver, wanted) File "/opt/salt/lib/python3.7/distutils/version.py", line 46, in __eq__ c = self._cmp(other) File "/opt/salt/lib/python3.7/distutils/version.py", line 337, in _cmp if self.version < other.version: TypeError: '<' not supported between instances of 'str' and 'int' ``` * Replace `saltstack.com` with `saltproject.io` on URLs being tested * Add back support to load old entrypoints by iterating instead of type checking Fixes #59961 * Fix issue #59975 * Fix pillar serialization for jinja #60083 * Fix test * Add changelog for #60083 * Update changelog and release for 3003.1 * Remove the changelog source refs * Add connect to IPCMessageSubscriber's async_methods Fixes #60049 by making sure an IPCMessageSubscriber that is wrapped by SyncWrapper has a connect method that runs the coroutine rather than returns a fugure. * Add changelog for #60049 * Update 60049.fixed * Fix coroutine spelling error Co-authored-by: Wayne Werner <waynejwerner@gmail.com> * IPC on windows cannot use socket paths Fixes #60298 * Update Jinja2 and lxml due to security related bugfix releases Jinja2 ------ CVE-2020-28493 moderate severity Vulnerable versions: < 2.11.3 Patched version: 2.11.3 This affects the package jinja2 from 0.0.0 and before 2.11.3. The ReDOS vulnerability of the regex is mainly due to the sub-pattern [a-zA-Z0-9.-]+.[a-zA-Z0-9.-]+ This issue can be mitigated by Markdown to format user content instead of the urlize filter, or by implementing request timeouts and limiting process memory. lxml ---- CVE-2021-28957 moderate severity Vulnerable versions: < 4.6.3 Patched version: 4.6.3 An XSS vulnerability was discovered in the python lxml clean module versions before 4.6.3. When disabling the safe_attrs_only and forms arguments, the Cleaner class does not remove the formaction attribute allowing for JS to bypass the sanitizer. A remote attacker could exploit this flaw to run arbitrary JS code on users who interact with incorrectly sanitized HTML. This issue is patched in lxml 4.6.3. * fix github actions jobs on branch until bullseye comes out * Upgrade to `six==1.16.0` to avoid problems on CI runs ``` 13:59:02 nox > Session invoke-pre-commit was successful. 13:59:02 nox > Running session invoke-pre-commit 13:59:02 nox > pip install --progress-bar=off -r requirements/static/ci/py3.7/invoke.txt 13:59:02 Collecting blessings==1.7 13:59:02 Using cached blessings-1.7-py3-none-any.whl (18 kB) 13:59:02 Collecting invoke==1.4.1 13:59:02 Using cached invoke-1.4.1-py3-none-any.whl (210 kB) 13:59:02 Collecting pyyaml==5.3.1 13:59:02 Using cached PyYAML-5.3.1.tar.gz (269 kB) 13:59:02 Collecting six==1.15.0 13:59:02 Using cached six-1.15.0-py2.py3-none-any.whl (10 kB) 13:59:02 Building wheels for collected packages: pyyaml 13:59:02 Building wheel for pyyaml (setup.py) ... - \ | / - \ | done 13:59:02 Created wheel for pyyaml: filename=PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl size=546391 sha256=e42e1d66cc32087f4d33ceb81268c86b59f1a97029b19459f91b8d6ad1430167 13:59:02 Stored in directory: /var/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653 13:59:02 Successfully built pyyaml 13:59:02 Installing collected packages: six, pyyaml, invoke, blessings 13:59:02 Attempting uninstall: six 13:59:02 Found existing installation: six 1.16.0 13:59:02 Uninstalling six-1.16.0: 13:59:02 ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/var/jenkins/.cache/pre-commit/repomw8oee1s/py_env-python3/lib/python3.7/site-packages/__pycache__/six.cpython-37.pyc' 13:59:02 13:59:02 nox > Command pip install --progress-bar=off -r requirements/static/ci/py3.7/invoke.txt failed with exit code 1 13:59:02 nox > Session invoke-pre-commit failed. ``` * add changelog for https://github.com/saltstack/salt/issues/59982 * Regression test for #56273 * Fix race condition in batch. #56273 * Add changelog for #56273 * Update salt/client/__init__.py Co-authored-by: Pedro Algarvio <pedro@algarvio.me> * Update doc for salt/client * Update changelog/56273.fixed Thoreau said, "Simplify, Simplify" * Update docs * Update docs * Update CHANGELOG.md * Update 3003.1.rst * Fix changelog Co-authored-by: Daniel Wozniak <dwozniak@saltstack.com> Co-authored-by: Pedro Algarvio <pedro@algarvio.me> Co-authored-by: Bryce Larson <brycel@vmware.com> Co-authored-by: Pablo Suárez Hernández <psuarezhernandez@suse.com> Co-authored-by: Alexander Graul <agraul@suse.com> Co-authored-by: Frode Gundersen <fgundersen@saltstack.com> Co-authored-by: Gareth J. Greenaway <gareth@saltstack.com> Co-authored-by: Gareth J. Greenaway <gareth@wiked.org> Co-authored-by: Hoa-Long Tam <hoalong@apple.com> Co-authored-by: krionbsd <krion@freebsd.org> Co-authored-by: Elias Probst <e.probst@ssc-services.de> Co-authored-by: Daniel A. Wozniak <dwozniak@vmware.com> Co-authored-by: Frode Gundersen <frogunder@gmail.com> Co-authored-by: twangboy <slee@saltstack.com> Co-authored-by: twangboy <leesh@vmware.com> Co-authored-by: ScriptAutomate <derek@icanteven.io> Co-authored-by: Wayne Werner <waynejwerner@gmail.com>
This commit is contained in:
Megan Wilhite
GitHub
parent
f0a58b317f
commit
90ed88638a
20 changed files with 379 additions and 44 deletions
13
CHANGELOG.md
13
CHANGELOG.md
|
|
@ -7,7 +7,7 @@ Versions are `MAJOR.PATCH`.
|
|||
|
||||
# Changelog
|
||||
|
||||
Salt 3003.1 (2021-04-20)
|
||||
Salt 3003.1 (2021-06-08)
|
||||
========================
|
||||
|
||||
Removed
|
||||
|
|
@ -22,6 +22,17 @@ Fixed
|
|||
- Import salt.utils.azurearm instead of using __utils__ from loader in azure cloud. This fixes an issue where __utils__ would become unavailable when we are using the ThreadPool in azurearm. (#59744)
|
||||
- Use contextvars libary from site-packages if it is intalled. Fixes salt ssh for targets with python <=3.6 (#59942)
|
||||
|
||||
Fixed
|
||||
-----
|
||||
|
||||
- Fixed race condition in batch logic. Added `listen` option to `LocalClient` to prevent event subscriber from purging cached events during batch iteration. (#56273)
|
||||
- Fixed dependencies for Amazon Linux 2 on https://repo.saltproject.io since Amazon Linux 2 now provides some of the python libraries in their repos. (#59982)
|
||||
- IPCMessageSubscriber objects expose their connect method as a coroutine so they can be wrapped by SyncWrapper. (#60049)
|
||||
- Import salt.utils.azurearm instead of using __utils__ from loader in azure cloud. This fixes an issue where __utils__ would become unavailable when we are using the ThreadPool in azurearm. (#59744)
|
||||
- Use contextvars libary from site-packages if it is intalled. Fixes salt ssh for targets with python <=3.6 (#59942)
|
||||
- Add back support to load old entrypoints by iterating instead of type checking (#59961)
|
||||
- Pass the value of the `__grains__` NamedContext to salt.pillar.get_pillar, instead of the NamedContext object itself. (#59975)
|
||||
- Fix pillar serialization in jinja templates (#60083)
|
||||
|
||||
Salt 3003 (2021-03-05)
|
||||
======================
|
||||
|
|
|
|||
|
|
@ -7,13 +7,18 @@ Salt 3003.1 Release Notes
|
|||
Version 3003.1 is a bug fix release for :ref:`3003 <release-3003>`.
|
||||
|
||||
Removed
|
||||
=======
|
||||
-------
|
||||
|
||||
- Removed support for Ubuntu 16.04 (#59913)
|
||||
|
||||
|
||||
Fixed
|
||||
=====
|
||||
-----
|
||||
|
||||
- Fixed race condition in batch logic. Added `listen` option to `LocalClient` to prevent event subscriber from purging cached events during batch iteration. (#56273)
|
||||
- Fixed dependencies for Amazon Linux 2 on https://repo.saltproject.io since Amazon Linux 2 now provides some of the python libraries in their repos. (#59982)
|
||||
- IPCMessageSubscriber objects expose their connect method as a coroutine so they can be wrapped by SyncWrapper. (#60049)
|
||||
- Import salt.utils.azurearm instead of using __utils__ from loader in azure cloud. This fixes an issue where __utils__ would become unavailable when we are using the ThreadPool in azurearm. (#59744)
|
||||
- Use contextvars libary from site-packages if it is intalled. Fixes salt ssh for targets with python <=3.6 (#59942)
|
||||
- Use contextvars libary from site-packages if it is intalled. Fixes salt ssh for targets with python <=3.6 (#59942)
|
||||
- Add back support to load old entrypoints by iterating instead of type checking (#59961)
|
||||
- Pass the value of the ``__grains__`` NamedContext to salt.pillar.get_pillar, instead of the NamedContext object itself. (#59975)
|
||||
- Fix pillar serialization in jinja templates (#60083)
|
||||
|
|
|
|||
|
|
@ -26,7 +26,7 @@ ntlm-auth==1.3.0
|
|||
# smbprotocol
|
||||
profitbricks==4.1.3
|
||||
# via -r requirements/static/ci/cloud.in
|
||||
pyasn1==0.4.5
|
||||
pyasn1==0.4.8
|
||||
# via smbprotocol
|
||||
pycparser==2.19
|
||||
# via cffi
|
||||
|
|
@ -36,7 +36,7 @@ pywinrm==0.3.0
|
|||
# via -r requirements/static/ci/cloud.in
|
||||
requests-ntlm==1.1.0
|
||||
# via pywinrm
|
||||
requests==2.21.0
|
||||
requests==2.25.1
|
||||
# via
|
||||
# apache-libcloud
|
||||
# profitbricks
|
||||
|
|
@ -51,7 +51,7 @@ six==1.16.0
|
|||
# smbprotocol
|
||||
smbprotocol==0.1.1
|
||||
# via pypsexec
|
||||
urllib3==1.24.2
|
||||
urllib3==1.26.4
|
||||
# via requests
|
||||
xmltodict==0.12.0
|
||||
# via pywinrm
|
||||
|
|
|
|||
|
|
@ -26,7 +26,7 @@ ntlm-auth==1.3.0
|
|||
# smbprotocol
|
||||
profitbricks==4.1.3
|
||||
# via -r requirements/static/ci/cloud.in
|
||||
pyasn1==0.4.5
|
||||
pyasn1==0.4.8
|
||||
# via smbprotocol
|
||||
pycparser==2.19
|
||||
# via cffi
|
||||
|
|
@ -36,7 +36,7 @@ pywinrm==0.3.0
|
|||
# via -r requirements/static/ci/cloud.in
|
||||
requests-ntlm==1.1.0
|
||||
# via pywinrm
|
||||
requests==2.21.0
|
||||
requests==2.25.1
|
||||
# via
|
||||
# apache-libcloud
|
||||
# profitbricks
|
||||
|
|
@ -51,7 +51,7 @@ six==1.16.0
|
|||
# smbprotocol
|
||||
smbprotocol==0.1.1
|
||||
# via pypsexec
|
||||
urllib3==1.24.2
|
||||
urllib3==1.26.4
|
||||
# via requests
|
||||
xmltodict==0.12.0
|
||||
# via pywinrm
|
||||
|
|
|
|||
|
|
@ -26,7 +26,7 @@ ntlm-auth==1.3.0
|
|||
# smbprotocol
|
||||
profitbricks==4.1.3
|
||||
# via -r requirements/static/ci/cloud.in
|
||||
pyasn1==0.4.5
|
||||
pyasn1==0.4.8
|
||||
# via smbprotocol
|
||||
pycparser==2.19
|
||||
# via cffi
|
||||
|
|
@ -36,7 +36,7 @@ pywinrm==0.3.0
|
|||
# via -r requirements/static/ci/cloud.in
|
||||
requests-ntlm==1.1.0
|
||||
# via pywinrm
|
||||
requests==2.21.0
|
||||
requests==2.25.1
|
||||
# via
|
||||
# apache-libcloud
|
||||
# profitbricks
|
||||
|
|
@ -51,7 +51,7 @@ six==1.16.0
|
|||
# smbprotocol
|
||||
smbprotocol==0.1.1
|
||||
# via pypsexec
|
||||
urllib3==1.24.2
|
||||
urllib3==1.26.4
|
||||
# via requests
|
||||
xmltodict==0.12.0
|
||||
# via pywinrm
|
||||
|
|
|
|||
|
|
@ -26,7 +26,7 @@ ntlm-auth==1.3.0
|
|||
# smbprotocol
|
||||
profitbricks==4.1.3
|
||||
# via -r requirements/static/ci/cloud.in
|
||||
pyasn1==0.4.5
|
||||
pyasn1==0.4.8
|
||||
# via smbprotocol
|
||||
pycparser==2.19
|
||||
# via cffi
|
||||
|
|
@ -36,7 +36,7 @@ pywinrm==0.3.0
|
|||
# via -r requirements/static/ci/cloud.in
|
||||
requests-ntlm==1.1.0
|
||||
# via pywinrm
|
||||
requests==2.21.0
|
||||
requests==2.25.1
|
||||
# via
|
||||
# apache-libcloud
|
||||
# profitbricks
|
||||
|
|
@ -51,7 +51,7 @@ six==1.16.0
|
|||
# smbprotocol
|
||||
smbprotocol==0.1.1
|
||||
# via pypsexec
|
||||
urllib3==1.24.2
|
||||
urllib3==1.26.4
|
||||
# via requests
|
||||
xmltodict==0.12.0
|
||||
# via pywinrm
|
||||
|
|
|
|||
|
|
@ -26,7 +26,7 @@ ntlm-auth==1.3.0
|
|||
# smbprotocol
|
||||
profitbricks==4.1.3
|
||||
# via -r requirements/static/ci/cloud.in
|
||||
pyasn1==0.4.5
|
||||
pyasn1==0.4.8
|
||||
# via smbprotocol
|
||||
pycparser==2.19
|
||||
# via cffi
|
||||
|
|
@ -36,7 +36,7 @@ pywinrm==0.3.0
|
|||
# via -r requirements/static/ci/cloud.in
|
||||
requests-ntlm==1.1.0
|
||||
# via pywinrm
|
||||
requests==2.21.0
|
||||
requests==2.25.1
|
||||
# via
|
||||
# apache-libcloud
|
||||
# profitbricks
|
||||
|
|
@ -51,7 +51,7 @@ six==1.16.0
|
|||
# smbprotocol
|
||||
smbprotocol==0.1.1
|
||||
# via pypsexec
|
||||
urllib3==1.24.2
|
||||
urllib3==1.26.4
|
||||
# via requests
|
||||
xmltodict==0.12.0
|
||||
# via pywinrm
|
||||
|
|
|
|||
|
|
@ -31,7 +31,7 @@ msgpack==0.6.2
|
|||
pathtools==0.1.2 # via watchdog
|
||||
portend==2.6 # via cherrypy
|
||||
psutil==5.8.0
|
||||
pyasn1==0.4.5
|
||||
pyasn1==0.4.8
|
||||
pycparser==2.19
|
||||
pycryptodomex==3.9.7
|
||||
pycurl==7.43.0.5
|
||||
|
|
@ -45,13 +45,13 @@ pytz==2019.3 # via tempora
|
|||
pywin32==227
|
||||
pyyaml==5.4.1
|
||||
pyzmq==19.0.0
|
||||
requests==2.21.0
|
||||
requests==2.25.1
|
||||
setproctitle==1.1.10
|
||||
six==1.16.0 # via cheroot, cherrypy, cryptography, pyopenssl, python-dateutil, tempora
|
||||
smmap2==2.0.5
|
||||
tempora==1.14.1 # via portend
|
||||
timelib==0.2.5
|
||||
urllib3==1.24.3 # via requests
|
||||
urllib3==1.26.4 # via requests
|
||||
watchdog==0.9.0
|
||||
wheel==0.33.4
|
||||
wmi==1.4.9
|
||||
|
|
|
|||
|
|
@ -31,7 +31,7 @@ msgpack==0.6.2
|
|||
pathtools==0.1.2 # via watchdog
|
||||
portend==2.6 # via cherrypy
|
||||
psutil==5.8.0
|
||||
pyasn1==0.4.5
|
||||
pyasn1==0.4.8
|
||||
pycparser==2.19
|
||||
pycryptodomex==3.9.7
|
||||
pycurl==7.43.0.5
|
||||
|
|
@ -45,13 +45,13 @@ pytz==2019.3 # via tempora
|
|||
pywin32==227
|
||||
pyyaml==5.4.1
|
||||
pyzmq==19.0.0
|
||||
requests==2.21.0
|
||||
requests==2.25.1
|
||||
setproctitle==1.1.10
|
||||
six==1.16.0 # via cheroot, cherrypy, cryptography, pyopenssl, python-dateutil, tempora
|
||||
smmap2==2.0.5
|
||||
tempora==1.14.1 # via portend
|
||||
timelib==0.2.5
|
||||
urllib3==1.24.3 # via requests
|
||||
urllib3==1.26.4 # via requests
|
||||
watchdog==0.9.0
|
||||
wheel==0.33.4
|
||||
wmi==1.4.9
|
||||
|
|
|
|||
|
|
@ -19,18 +19,31 @@ log = logging.getLogger(__name__)
|
|||
class Batch:
|
||||
"""
|
||||
Manage the execution of batch runs
|
||||
|
||||
"""
|
||||
|
||||
def __init__(self, opts, eauth=None, quiet=False, parser=None):
|
||||
def __init__(self, opts, eauth=None, quiet=False, _parser=None):
|
||||
"""
|
||||
:param dict opts: A config options dictionary.
|
||||
|
||||
:param dict eauth: An eauth config to use.
|
||||
|
||||
The default is an empty dict.
|
||||
|
||||
:param bool quiet: Supress printing to stdout
|
||||
|
||||
The default is False.
|
||||
"""
|
||||
self.opts = opts
|
||||
self.eauth = eauth if eauth else {}
|
||||
self.pub_kwargs = eauth if eauth else {}
|
||||
self.quiet = quiet
|
||||
self.local = salt.client.get_local_client(opts["conf_file"])
|
||||
self.minions, self.ping_gen, self.down_minions = self.__gather_minions()
|
||||
self.options = parser
|
||||
self.options = _parser
|
||||
# Passing listen True to local client will prevent it from purging
|
||||
# cahced events while iterating over the batches.
|
||||
self.local = salt.client.get_local_client(opts["conf_file"], listen=True)
|
||||
|
||||
def __gather_minions(self):
|
||||
def gather_minions(self):
|
||||
"""
|
||||
Return a list of minions to use for the batch run
|
||||
"""
|
||||
|
|
@ -106,6 +119,7 @@ class Batch:
|
|||
"""
|
||||
Execute the batch run
|
||||
"""
|
||||
self.minions, self.ping_gen, self.down_minions = self.gather_minions()
|
||||
args = [
|
||||
[],
|
||||
self.opts["fun"],
|
||||
|
|
|
|||
|
|
@ -289,7 +289,7 @@ class SaltCMD(salt.utils.parsers.SaltCMDOptionParser):
|
|||
try:
|
||||
self.config["batch"] = self.options.batch
|
||||
batch = salt.cli.batch.Batch(
|
||||
self.config, eauth=eauth, parser=self.options
|
||||
self.config, eauth=eauth, _parser=self.options
|
||||
)
|
||||
except SaltClientError:
|
||||
# We will print errors to the console further down the stack
|
||||
|
|
|
|||
|
|
@ -71,6 +71,7 @@ def get_local_client(
|
|||
skip_perm_errors=False,
|
||||
io_loop=None,
|
||||
auto_reconnect=False,
|
||||
listen=False,
|
||||
):
|
||||
"""
|
||||
.. versionadded:: 2014.7.0
|
||||
|
|
@ -78,11 +79,38 @@ def get_local_client(
|
|||
Read in the config and return the correct LocalClient object based on
|
||||
the configured transport
|
||||
|
||||
:param str c_path: Path of config file to use for opts.
|
||||
|
||||
The default value is None.
|
||||
|
||||
:param bool mopts: When provided the local client will use this dictionary of
|
||||
options insead of loading a config file from the value
|
||||
of c_path.
|
||||
|
||||
The default value is None.
|
||||
|
||||
:param str skip_perm_errors: Ignore permissions errors while loading keys.
|
||||
|
||||
The default value is False.
|
||||
|
||||
:param IOLoop io_loop: io_loop used for events.
|
||||
Pass in an io_loop if you want asynchronous
|
||||
operation for obtaining events. Eg use of
|
||||
set_event_handler() API. Otherwise, operation
|
||||
will be synchronous.
|
||||
|
||||
:param bool keep_loop: Do not destroy the event loop when closing the event
|
||||
subsriber.
|
||||
|
||||
:param bool auto_reconnect: When True the event subscriber will reconnect
|
||||
automatically if a disconnect error is raised.
|
||||
|
||||
.. versionadded:: 3004
|
||||
:param bool listen: Listen for events indefinitly. When option is set the
|
||||
LocalClient object will listen for events until it's
|
||||
destroy method is called.
|
||||
|
||||
The default value is False.
|
||||
"""
|
||||
if mopts:
|
||||
opts = mopts
|
||||
|
|
@ -98,6 +126,7 @@ def get_local_client(
|
|||
skip_perm_errors=skip_perm_errors,
|
||||
io_loop=io_loop,
|
||||
auto_reconnect=auto_reconnect,
|
||||
listen=listen,
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -128,6 +157,7 @@ class LocalClient:
|
|||
|
||||
local = salt.client.LocalClient()
|
||||
local.cmd('*', 'test.fib', [10])
|
||||
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
|
|
@ -138,13 +168,41 @@ class LocalClient:
|
|||
io_loop=None,
|
||||
keep_loop=False,
|
||||
auto_reconnect=False,
|
||||
listen=False,
|
||||
):
|
||||
"""
|
||||
:param str c_path: Path of config file to use for opts.
|
||||
|
||||
The default value is None.
|
||||
|
||||
:param bool mopts: When provided the local client will use this dictionary of
|
||||
options insead of loading a config file from the value
|
||||
of c_path.
|
||||
|
||||
The default value is None.
|
||||
|
||||
:param str skip_perm_errors: Ignore permissions errors while loading keys.
|
||||
|
||||
The default value is False.
|
||||
|
||||
:param IOLoop io_loop: io_loop used for events.
|
||||
Pass in an io_loop if you want asynchronous
|
||||
operation for obtaining events. Eg use of
|
||||
set_event_handler() API. Otherwise,
|
||||
operation will be synchronous.
|
||||
set_event_handler() API. Otherwise, operation
|
||||
will be synchronous.
|
||||
|
||||
:param bool keep_loop: Do not destroy the event loop when closing the event
|
||||
subsriber.
|
||||
|
||||
:param bool auto_reconnect: When True the event subscriber will reconnect
|
||||
automatically if a disconnect error is raised.
|
||||
|
||||
.. versionadded:: 3004
|
||||
:param bool listen: Listen for events indefinitly. When option is set the
|
||||
LocalClient object will listen for events until it's
|
||||
destroy method is called.
|
||||
|
||||
The default value is False.
|
||||
"""
|
||||
if mopts:
|
||||
self.opts = mopts
|
||||
|
|
@ -162,12 +220,13 @@ class LocalClient:
|
|||
self.skip_perm_errors = skip_perm_errors
|
||||
self.key = self.__read_master_key()
|
||||
self.auto_reconnect = auto_reconnect
|
||||
self.listen = listen
|
||||
self.event = salt.utils.event.get_event(
|
||||
"master",
|
||||
self.opts["sock_dir"],
|
||||
self.opts["transport"],
|
||||
opts=self.opts,
|
||||
listen=False,
|
||||
listen=self.listen,
|
||||
io_loop=io_loop,
|
||||
keep_loop=keep_loop,
|
||||
)
|
||||
|
|
@ -1052,6 +1111,9 @@ class LocalClient:
|
|||
)
|
||||
yield raw
|
||||
|
||||
def returns_for_job(self, jid):
|
||||
return self.returners["{}.get_load".format(self.opts["master_job_cache"])](jid)
|
||||
|
||||
def get_iter_returns(
|
||||
self,
|
||||
jid,
|
||||
|
|
@ -1088,10 +1150,7 @@ class LocalClient:
|
|||
missing = set()
|
||||
# Check to see if the jid is real, if not return the empty dict
|
||||
try:
|
||||
if (
|
||||
self.returners["{}.get_load".format(self.opts["master_job_cache"])](jid)
|
||||
== {}
|
||||
):
|
||||
if not self.returns_for_job(jid):
|
||||
log.warning("jid does not exist")
|
||||
yield {}
|
||||
# stop the iteration, since the jid is invalid
|
||||
|
|
|
|||
|
|
@ -503,7 +503,7 @@ def ext(external, pillar=None):
|
|||
external = salt.utils.yaml.safe_load(external)
|
||||
pillar_obj = salt.pillar.get_pillar(
|
||||
__opts__,
|
||||
__grains__,
|
||||
__grains__.value(),
|
||||
__opts__["id"],
|
||||
__opts__["saltenv"],
|
||||
ext=external,
|
||||
|
|
|
|||
|
|
@ -26,6 +26,7 @@ from salt import __path__ as saltpath
|
|||
from salt.exceptions import CommandExecutionError, SaltInvocationError, SaltRenderError
|
||||
from salt.ext import six
|
||||
from salt.features import features
|
||||
from salt.loader_context import NamedLoaderContext
|
||||
from salt.utils.decorators.jinja import JinjaFilter, JinjaGlobal, JinjaTest
|
||||
from salt.utils.odict import OrderedDict
|
||||
from salt.utils.versions import LooseVersion
|
||||
|
|
@ -477,7 +478,10 @@ def render_jinja_tmpl(tmplstr, context, tmplpath=None):
|
|||
decoded_context = {}
|
||||
for key, value in context.items():
|
||||
if not isinstance(value, str):
|
||||
decoded_context[key] = value
|
||||
if isinstance(value, NamedLoaderContext):
|
||||
decoded_context[key] = value.value()
|
||||
else:
|
||||
decoded_context[key] = value
|
||||
continue
|
||||
|
||||
try:
|
||||
|
|
@ -490,7 +494,6 @@ def render_jinja_tmpl(tmplstr, context, tmplpath=None):
|
|||
SLS_ENCODING,
|
||||
)
|
||||
decoded_context[key] = salt.utils.data.decode(value)
|
||||
|
||||
try:
|
||||
template = jinja_env.from_string(tmplstr)
|
||||
template.globals.update(decoded_context)
|
||||
|
|
|
|||
185
tests/pytests/functional/cli/test_batch.py
Normal file
185
tests/pytests/functional/cli/test_batch.py
Normal file
|
|
@ -0,0 +1,185 @@
|
|||
"""
|
||||
tests.pytests.functional.cli.test_batch
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
"""
|
||||
import salt.cli.batch
|
||||
import salt.config
|
||||
import salt.utils.jid
|
||||
from tests.support.mock import Mock, patch
|
||||
|
||||
|
||||
class MockPub:
|
||||
"""
|
||||
Mock salt.client.LocalClient.pub method
|
||||
"""
|
||||
|
||||
calls = 0
|
||||
initial_ping = False
|
||||
batch1_jid = None
|
||||
batch1_tgt = None
|
||||
batch2_jid = None
|
||||
batch2_tgt = None
|
||||
batch3_jid = None
|
||||
batch3_tgt = None
|
||||
|
||||
def __call__(self, tgt, fun, *args, **kwargs):
|
||||
if tgt == "minion*" and fun == "test.ping":
|
||||
MockPub.calls += 1
|
||||
MockPub.initial_ping = salt.utils.jid.gen_jid({})
|
||||
pub_ret = {
|
||||
"jid": MockPub.initial_ping,
|
||||
"minions": ["minion0", "minion1", "minion2", "minion3"],
|
||||
}
|
||||
elif fun == "state.sls":
|
||||
if MockPub.calls == 1:
|
||||
MockPub.calls += 1
|
||||
MockPub.batch1_tgt = list(tgt)
|
||||
MockPub.batch1_jid = jid = salt.utils.jid.gen_jid({})
|
||||
pub_ret = {"jid": jid, "minions": tgt}
|
||||
elif MockPub.calls == 2:
|
||||
MockPub.calls += 1
|
||||
MockPub.batch2_tgt = tgt
|
||||
MockPub.batch2_jid = jid = salt.utils.jid.gen_jid({})
|
||||
pub_ret = {"jid": jid, "minions": tgt}
|
||||
elif MockPub.calls == 3:
|
||||
MockPub.calls += 1
|
||||
MockPub.batch3_tgt = tgt
|
||||
MockPub.batch3_jid = jid = salt.utils.jid.gen_jid({})
|
||||
pub_ret = {"jid": jid, "minions": tgt}
|
||||
elif fun == "saltutil.find_job":
|
||||
jid = salt.utils.jid.gen_jid({})
|
||||
pub_ret = {"jid": jid, "minions": tgt}
|
||||
return pub_ret
|
||||
|
||||
|
||||
class MockSubscriber:
|
||||
"""
|
||||
Mock salt.transport.ipc IPCMessageSubscriber in order to inject events into
|
||||
salt.utils.Event
|
||||
"""
|
||||
|
||||
calls = 0
|
||||
pubret = None
|
||||
|
||||
def __init__(self, *args, **kwargs):
|
||||
return
|
||||
|
||||
def read(self, timeout=None):
|
||||
"""
|
||||
Mock IPCMessageSubcriber read method.
|
||||
|
||||
- Return events for initial ping
|
||||
- Returns event for a minion in first batch to cause second batch to get sent.
|
||||
- Returns 5 null events on first iteration of second batch to go back to first batch.
|
||||
- On second iteration of first batch, send an event from second batch which will get cached.
|
||||
- Return events for the rest of the batches.
|
||||
"""
|
||||
if MockSubscriber.pubret.initial_ping:
|
||||
# Send ping responses for 4 minions
|
||||
jid = MockSubscriber.pubret.initial_ping
|
||||
if MockSubscriber.calls == 0:
|
||||
MockSubscriber.calls += 1
|
||||
return self._ret(jid, minion_id="minion0", fun="test.ping")
|
||||
elif MockSubscriber.calls == 1:
|
||||
MockSubscriber.calls += 1
|
||||
return self._ret(jid, minion_id="minion1", fun="test.ping")
|
||||
elif MockSubscriber.calls == 2:
|
||||
MockSubscriber.calls += 1
|
||||
return self._ret(jid, minion_id="minion2", fun="test.ping")
|
||||
elif MockSubscriber.calls == 3:
|
||||
MockSubscriber.calls += 1
|
||||
return self._ret(jid, minion_id="minion3", fun="test.ping")
|
||||
if MockSubscriber.pubret.batch1_jid:
|
||||
jid = MockSubscriber.pubret.batch1_jid
|
||||
tgt = MockSubscriber.pubret.batch1_tgt
|
||||
if MockSubscriber.calls == 4:
|
||||
# Send a return for first minion in first batch. This causes the
|
||||
# second batch to get sent.
|
||||
MockSubscriber.calls += 1
|
||||
return self._ret(jid, minion_id=tgt[0], fun="state.sls")
|
||||
if MockSubscriber.pubret.batch2_jid:
|
||||
if MockSubscriber.calls <= 10:
|
||||
# Skip the first iteration of the second batch; this will cause
|
||||
# batch logic to go back to iterating over the first batch.
|
||||
MockSubscriber.calls += 1
|
||||
return
|
||||
elif MockSubscriber.calls == 11:
|
||||
# Send the minion from the second batch, This event will get cached.
|
||||
jid = MockSubscriber.pubret.batch2_jid
|
||||
tgt = MockSubscriber.pubret.batch2_tgt
|
||||
MockSubscriber.calls += 1
|
||||
return self._ret(jid, minion_id=tgt[0], fun="state.sls")
|
||||
if MockSubscriber.calls == 12:
|
||||
jid = MockSubscriber.pubret.batch1_jid
|
||||
tgt = MockSubscriber.pubret.batch1_tgt
|
||||
MockSubscriber.calls += 1
|
||||
return self._ret(jid, minion_id=tgt[1], fun="state.sls")
|
||||
if MockSubscriber.pubret.batch3_jid:
|
||||
jid = MockSubscriber.pubret.batch3_jid
|
||||
tgt = MockSubscriber.pubret.batch3_tgt
|
||||
if MockSubscriber.calls == 13:
|
||||
MockSubscriber.calls += 1
|
||||
return self._ret(jid, minion_id=tgt[0], fun="state.sls")
|
||||
return
|
||||
|
||||
def _ret(self, jid, minion_id, fun, _return=True, _retcode=0):
|
||||
"""
|
||||
Create a mock return from a jid, minion, and fun
|
||||
"""
|
||||
serial = salt.payload.Serial({"serial": "msgpack"})
|
||||
dumped = serial.dumps(
|
||||
{
|
||||
"fun_args": [],
|
||||
"jid": jid,
|
||||
"return": _return,
|
||||
"retcode": 0,
|
||||
"success": True,
|
||||
"cmd": "_return",
|
||||
"fun": fun,
|
||||
"id": minion_id,
|
||||
"_stamp": "2021-05-24T01:23:25.373194",
|
||||
},
|
||||
use_bin_type=True,
|
||||
)
|
||||
tag = "salt/job/{}/ret".format(jid).encode()
|
||||
return b"".join([tag, b"\n\n", dumped])
|
||||
|
||||
def connect(self, timeout=None):
|
||||
pass
|
||||
|
||||
|
||||
def test_batch_issue_56273():
|
||||
"""
|
||||
Regression test for race condition in batch logic.
|
||||
https://github.com/saltstack/salt/issues/56273
|
||||
"""
|
||||
|
||||
mock_pub = MockPub()
|
||||
MockSubscriber.pubret = mock_pub
|
||||
|
||||
def returns_for_job(jid):
|
||||
return True
|
||||
|
||||
opts = {
|
||||
"conf_file": "",
|
||||
"tgt": "minion*",
|
||||
"fun": "state.sls",
|
||||
"arg": ["foo"],
|
||||
"timeout": 1,
|
||||
"gather_job_timeout": 1,
|
||||
"batch": 2,
|
||||
"extension_modules": "",
|
||||
"failhard": True,
|
||||
}
|
||||
with patch("salt.transport.ipc.IPCMessageSubscriber", MockSubscriber):
|
||||
batch = salt.cli.batch.Batch(opts, quiet=True)
|
||||
with patch.object(batch.local, "pub", Mock(side_effect=mock_pub)):
|
||||
with patch.object(
|
||||
batch.local, "returns_for_job", Mock(side_effect=returns_for_job)
|
||||
):
|
||||
ret = list(batch.run())
|
||||
assert len(ret) == 4
|
||||
for val in ret:
|
||||
values = list(val.values())
|
||||
assert len(values) == 1
|
||||
assert values[0] is True
|
||||
|
|
@ -579,3 +579,12 @@ def test_pillar_match_filter_by_minion_id(
|
|||
|
||||
# Refresh pillar once we're done
|
||||
salt_cli.run("saltutil.refresh_pillar", wait=True, minion_tgt=salt_minion.id)
|
||||
|
||||
|
||||
@pytest.mark.slow_test
|
||||
def test_pillar_ext_59975(salt_call_cli):
|
||||
"""
|
||||
pillar.ext returns result. Issue #59975
|
||||
"""
|
||||
ret = salt_call_cli.run("pillar.ext", '{"libvert": _}')
|
||||
assert "ext_pillar_opts" in ret.json
|
||||
|
|
|
|||
0
tests/pytests/integration/utils/__init__.py
Normal file
0
tests/pytests/integration/utils/__init__.py
Normal file
37
tests/pytests/integration/utils/test_templates.py
Normal file
37
tests/pytests/integration/utils/test_templates.py
Normal file
|
|
@ -0,0 +1,37 @@
|
|||
"""
|
||||
Tests for the templates utils
|
||||
"""
|
||||
import os
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
def test_issue_60083(
|
||||
salt_call_cli, tmp_path, base_env_state_tree_root_dir,
|
||||
):
|
||||
"""
|
||||
Validate that we can serialize pillars to json in states.
|
||||
Issue #60083
|
||||
"""
|
||||
target_path = tmp_path / "issue-60083-target.txt"
|
||||
assert not os.path.exists(str(target_path))
|
||||
sls_name = "issue-60083"
|
||||
sls_contents = """
|
||||
{{ pillar["target-path"] }}:
|
||||
file.managed:
|
||||
- contents: |
|
||||
{{ pillar|json }}
|
||||
"""
|
||||
sls_tempfile = pytest.helpers.temp_file(
|
||||
"{}.sls".format(sls_name), sls_contents, base_env_state_tree_root_dir
|
||||
)
|
||||
with sls_tempfile: # , issue_50221_ext_pillar_tempfile:
|
||||
ret = salt_call_cli.run(
|
||||
"state.apply", sls_name, pillar={"target-path": str(target_path)}
|
||||
)
|
||||
assert ret.stdout.find("Jinja error") == -1
|
||||
assert ret.json
|
||||
keys = list(ret.json.keys())
|
||||
assert len(keys) == 1
|
||||
key = keys[0]
|
||||
assert ret.json[key]["changes"]["diff"] == "New file"
|
||||
|
|
@ -81,7 +81,9 @@ class BatchTestCase(TestCase):
|
|||
"gather_job_timeout": 5,
|
||||
"ret": "my_return",
|
||||
}
|
||||
self.batch.minions = ["foo", "bar", "baz"]
|
||||
self.batch.gather_minions = MagicMock(
|
||||
return_value=[["foo", "bar", "baz"], [], []],
|
||||
)
|
||||
self.batch.local.cmd_iter_no_block = MagicMock(return_value=iter([]))
|
||||
ret = Batch.run(self.batch)
|
||||
# We need to fetch at least one object to trigger the relevant code path.
|
||||
|
|
@ -112,7 +114,9 @@ class BatchTestCase(TestCase):
|
|||
"gather_job_timeout": 5,
|
||||
"return": "my_return",
|
||||
}
|
||||
self.batch.minions = ["foo", "bar", "baz"]
|
||||
self.batch.gather_minions = MagicMock(
|
||||
return_value=[["foo", "bar", "baz"], [], []],
|
||||
)
|
||||
self.batch.local.cmd_iter_no_block = MagicMock(return_value=iter([]))
|
||||
ret = Batch.run(self.batch)
|
||||
# We need to fetch at least one object to trigger the relevant code path.
|
||||
|
|
|
|||
|
|
@ -192,6 +192,14 @@ class BadTestModuleNamesTestCase(TestCase):
|
|||
"unit.test_simple",
|
||||
"unit.test_virtualname",
|
||||
"unit.test_zypp_plugins",
|
||||
"unit.utils.scheduler.test_error",
|
||||
"unit.utils.scheduler.test_eval",
|
||||
"unit.utils.scheduler.test_helpers",
|
||||
"unit.utils.scheduler.test_maxrunning",
|
||||
"unit.utils.scheduler.test_postpone",
|
||||
"unit.utils.scheduler.test_run_job",
|
||||
"unit.utils.scheduler.test_schedule",
|
||||
"unit.utils.scheduler.test_skip",
|
||||
"unit.auth.test_auth",
|
||||
)
|
||||
errors = []
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue