saltstack/salt
1
0
Fork 0
mirror of https://github.com/saltstack/salt.git synced 2025-04-16 17:50:20 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 2

Merge pull request #62585 from MKLeb/merge-forward/3005

Browse source 
Merge forward 3005
This commit is contained in:
Megan Wilhite 2022-09-01 09:47:15 -06:00 • committed by GitHub
commit 9012911960
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
32 changed files with 1095 additions and 110 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

22
CHANGELOG.md
View file

@ -7,7 +7,7 @@ Versions are `MAJOR.PATCH`.
# Changelog
Salt 3005 (2022-06-30)
Salt 3005 (2022-08-22)
======================
Removed
@ -29,6 +29,11 @@ Removed
- tplfile: When using tplfile in a SLS file in the root directory of file roots it returns empty. Now it returns the filename. (#61697)
- Remove SaltMessageServer.shutdown in favor of close.
Remove LoadBalancerWorker.stop in favor of close. (#61698)
- Removed the PyObjC dependency.
This addresses problems with building a one dir build for macOS.
It became problematic because depending on the macOS version, it pulls different dependencies, and we would either have to build a macos onedir for each macOS supported release, or ship a crippled onedir(because it would be tied to the macOS version where the onedir was built).
Since it's currently not being used, it's removed. (#62432)
Deprecated
@ -43,6 +48,7 @@ Deprecated
- Deprecate all Azure cloud modules (#62183)
- Deprecated ``defaults`` and ``preserve_context`` for ``salt.utils.functools.namespaced_function``.
Additionally, the behavior when ``preserve_namespace=True`` was passed is now the default in order not to require duplicating imports on the modules that are namespacing functions. (#62272)
- Deprecated the cassandra module in favor of the cassandra_cql module/returner. (#62327)
Changed
@ -243,6 +249,19 @@ Fixed
- Fix variable reuse causing requisite_in problems (#62264)
- Adding -G option to pkgdd cmd_prefix list when current_zone_only is True. (#62206)
- Don't expect ``lsof`` to be installed when trying check which minions are connected. (#62303)
- Added a pyinstaller hook that traverses the python used on the tiamat package to add all possible modules as hidden imports. (#62362)
- Fix use of random shuffle and sample functions as Jinja filters (#62372)
- All of the requirements provided in the requirements files are now included. The job of evaluating platform markers is not Salt's it's pip's. (#62392)
- Update all platforms to use pycparser 2.21 or greater for Py 3.9 or higher, fixes fips fault with openssl v3.x (#62400)
- Due to changes in the Netmiko library for the exception paths, need to check the version of Netmiko python library and then import the exceptions from different locations depending on the result. (#62405)
- Fixed urlparse typo in rpmbuild_pkgbuild.py (#62442)
- Fixing changes dict in pkg state to be consistent when installing and test=True. (#60995)
- Use fire_event_async when expecting a coroutine (#62453)
- Fixes import error under windows. (#62459)
- account for revision number in formulas to account for difference between bottle and formula (#62466)
- Fixed stacktrace on Windows when running pkg.list_pkgs (#62479)
- Update sanitizing masking for Salt SSH to include additional password like strings. (#62483)
- Fixes an issue where the minion could not connect to a master after 2 failed attempts (#62489)
Added
@ -301,6 +320,7 @@ Added
* Fix issues when processing inventory and there are groups with no members.
* Allow new types of targets for Ansible roster (#60056)
- Add sample and shuffle functions from random (#62225)
- Add "<tiamat> python" subcommand to allow execution or arbitrary scripts via bundled Python runtime (#62381)
Salt 3004.2 (2022-05-12)

1
changelog/62362.fixed
View file

@ -1 +0,0 @@
Added a pyinstaller hook that traverses the python used on the tiamat package to add all possible modules as hidden imports.

1
changelog/62392.fixed
View file

@ -1 +0,0 @@
All of the requirements provided in the requirements files are now included. The job of evaluating platform markers is not Salt's it's pip's.

46
doc/topics/releases/3005.rst
View file

@ -151,20 +151,24 @@ For example:
Known issues
------------
To make use of Salt 3005 or later on a Salt master connected to SaltStack
Config, you must use SaltStack Config version 8.9.0 or later.
- To make use of Salt 3005 or later on a Salt master connected to SaltStack
Config, you must use SaltStack Config version 8.9.0 or later.
The root cause of the issue is a breaking change to
``AsyncClient._proc_function()``` in Salt, which is the function that the
raas-master uses to run ``salt-run`` commands. As this is a private API, there's
no expectation that the API should remain backward-compatible.
The root cause of the issue is a breaking change to
``AsyncClient._proc_function()``` in Salt, which is the function that the
raas-master uses to run ``salt-run`` commands. As this is a private API, there's
no expectation that the API should remain backward-compatible.
It is recommended to upgrade SaltStack Config before upgrading your Salt
masters. However, if a Salt master is upgraded to version 3005 before
upgrading SaltStack Config, the upgrade can still be completed.
It is recommended to upgrade SaltStack Config before upgrading your Salt
masters. However, if a Salt master is upgraded to version 3005 before
upgrading SaltStack Config, the upgrade can still be completed.
After upgrading SaltStack Config, including the SSC plugin on each Salt master,
restart the Salt masters.
After upgrading SaltStack Config, including the SSC plugin on each Salt master,
restart the Salt masters.
- Salt does not currently support napalm 4. Users will need to install napalm 3.x to
ensure they do not run into issue #62468
Removed
@ -186,6 +190,11 @@ Removed
- tplfile: When using tplfile in a SLS file in the root directory of file roots it returns empty. Now it returns the filename. (#61697)
- Remove SaltMessageServer.shutdown in favor of close.
Remove LoadBalancerWorker.stop in favor of close. (#61698)
- Removed the PyObjC dependency.
This addresses problems with building a one dir build for macOS.
It became problematic because depending on the macOS version, it pulls different dependencies, and we would either have to build a macos onedir for each macOS supported release, or ship a crippled onedir(because it would be tied to the macOS version where the onedir was built).
Since it's currently not being used, it's removed. (#62432)
Deprecated
@ -200,6 +209,12 @@ Deprecated
- Deprecate all Azure cloud modules (#62183)
- Deprecated ``defaults`` and ``preserve_context`` for ``salt.utils.functools.namespaced_function``.
Additionally, the behavior when ``preserve_namespace=True`` was passed is now the default in order not to require duplicating imports on the modules that are namespacing functions. (#62272)
- Added a pyinstaller hook that traverses the python used on the tiamat package to add all possible modules as hidden imports. (#62362)
- Fix use of random shuffle and sample functions as Jinja filters (#62372)
- All of the requirements provided in the requirements files are now included. The job of evaluating platform markers is not Salt's it's pip's. (#62392)
- Update all platforms to use pycparser 2.21 or greater for Py 3.9 or higher, fixes fips fault with openssl v3.x (#62400)
- Due to changes in the Netmiko library for the exception paths, need to check the version of Netmiko python library and then import the exceptions from different locations depending on the result. (#62405)
- Deprecated the cassandra module in favor of the cassandra_cql module/returner. (#62327)
Changed
@ -400,6 +415,14 @@ Fixed
- Fix variable reuse causing requisite_in problems (#62264)
- Adding -G option to pkgdd cmd_prefix list when current_zone_only is True. (#62206)
- Don't expect ``lsof`` to be installed when trying check which minions are connected. (#62303)
- Fixed urlparse typo in rpmbuild_pkgbuild.py (#62442)
- Fixing changes dict in pkg state to be consistent when installing and test=True. (#60995)
- Use fire_event_async when expecting a coroutine (#62453)
- Fixes import error under windows. (#62459)
- account for revision number in formulas to account for difference between bottle and formula (#62466)
- Fixed stacktrace on Windows when running pkg.list_pkgs (#62479)
- Update sanitizing masking for Salt SSH to include additional password like strings. (#62483)
- Fixes an issue where the minion could not connect to a master after 2 failed attempts (#62489)
Added
@ -458,3 +481,4 @@ Added
* Fix issues when processing inventory and there are groups with no members.
* Allow new types of targets for Ansible roster (#60056)
- Add sample and shuffle functions from random (#62225)
- Add "<tiamat> python" subcommand to allow execution or arbitrary scripts via bundled Python runtime (#62381)

2
pkg/osx/build.sh
View file

@ -99,7 +99,7 @@ echo "#########################################################################"
################################################################################
if [[ $(id -u) -ne 0 ]]; then
echo ">>>>>> Re-launching as sudo <<<<<<"
exec sudo /bin/bash -c "$(printf '%q ' "$BASH_SOURCE" "$@")"
exec sudo -E /bin/bash -c "$(printf '%q ' "$BASH_SOURCE" "$@")"
fi
################################################################################

2
pkg/osx/build_env.sh
View file

@ -34,7 +34,7 @@ echo "Build Environment Script"
################################################################################
if [[ $(id -u) -ne 0 ]]; then
echo ">>>>>> Re-launching as sudo <<<<<<"
exec sudo /bin/bash -c "$(printf '%q ' "$BASH_SOURCE" "$@")"
exec sudo -E /bin/bash -c "$(printf '%q ' "$BASH_SOURCE" "$@")"
fi
################################################################################

36
pkg/osx/notarize.sh
View file

@ -12,6 +12,13 @@
# This script will upload a copy of the package to apple and wait
# for the notarization to return. This can take several minutes.
#
# This script should be run with sudo. If not, it will attempt to
# elevate and run with sudo. In order for the environment variables
# to be available in sudo you need to pass the `-E` option. For
# example:
#
# sudo -E ./notarize.sh
#
# Requirements:
# - Full Xcode Installation
# I had issues installing Xcode after installing Command Line Tools. This
@ -27,7 +34,7 @@
# Example:
# The following will notarize the 'salt-v2015.8.3-signed.pkg' file
#
# ./notarize.sh salt-v2015.8.3-signed.pkg
# sudo ./notarize.sh salt-v2015.8.3-signed.pkg
#
# Environment Setup:
#
@ -43,6 +50,15 @@
echo "vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv"
echo "Notarize Salt Package"
# Make sure the script is launched with sudo
# We're doing this with notarize because the package we're notarizing was
# built with sudo... so, we need sudo here in order to staple the notarization
# to the package
if [[ $(id -u) -ne 0 ]]; then
echo ">>>>>> Re-launching as sudo <<<<<<"
exec sudo -E /bin/bash -c "$(printf '%q ' "${BASH_SOURCE[0]}" "$@")"
fi
################################################################################
# Check input parameters
################################################################################
@ -110,15 +126,27 @@ while sleep 30; do
# Uncomment for debugging
# cat "$NOTARIZE_INFO_LOG"
# once notarization is complete, run stapler and exit
# Continue checking until Status is no longer "in progress"
if ! grep -q "Status: in progress" "$NOTARIZE_INFO_LOG"; then
echo ""
echo "**** Stapling Notarization to the Package"
xcrun stapler staple "$PACKAGE" > /dev/null
break
fi
done
# Make sure the result is "success", then staple
if ! grep -q "Status: success" "$NOTARIZE_INFO_LOG"; then
echo "**** There was a problem notarizing the package"
echo "**** View the log for details:"
awk -F ': ' '/LogFileURL/ {print $2}' "$NOTARIZE_INFO_LOG"
exit 1
fi
echo "**** Stapling Notarization to the Package"
if ! xcrun stapler staple "$PACKAGE" > "$NOTARIZE_INFO_LOG"; then
cat "$NOTARIZE_INFO_LOG" 1>&2
exit 1
fi
echo "Notarize Salt Package Completed Successfully"
echo "^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"

2
pkg/osx/package.sh
View file

@ -54,7 +54,7 @@ echo "Building Salt Package"
################################################################################
if [[ $(id -u) -ne 0 ]]; then
echo ">>>>>> Re-launching as sudo <<<<<<"
exec sudo /bin/bash -c "$(printf '%q ' "$BASH_SOURCE" "$@")"
exec sudo -E /bin/bash -c "$(printf '%q ' "$BASH_SOURCE" "$@")"
fi
################################################################################

58
pkg/osx/sign_binaries.sh
View file

@ -49,7 +49,7 @@ echo "Signing Binaries"
################################################################################
if [[ $(id -u) -ne 0 ]]; then
echo ">>>>>> Re-launching as sudo <<<<<<"
exec sudo /bin/bash -c "$(printf '%q ' "$BASH_SOURCE" "$@")"
exec sudo -E /bin/bash -c "$(printf '%q ' "$BASH_SOURCE" "$@")"
fi
################################################################################
@ -69,48 +69,65 @@ echo "**** Setting Variables"
INSTALL_DIR=/opt/salt
PY_VERSION=3.9
PY_DOT_VERSION=3.9.12
CMD_OUTPUT=$(mktemp -t cmd.log)
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
################################################################################
# Add rpath to the Python binaries before signing
################################################################################
echo "**** Setting rpath in binaries"
install_name_tool $INSTALL_DIR/bin/python${PY_VERSION}m \
install_name_tool $INSTALL_DIR/bin/python${PY_VERSION} \
-add_rpath $INSTALL_DIR/.pyenv/versions/$PY_DOT_VERSION/lib \
-add_rpath $INSTALL_DIR/.pyenv/versions/$PY_DOT_VERSION/openssl/lib || echo "already present"
################################################################################
# Add rpath to the Python binaries before signing
################################################################################
echo "**** Setting rpath in binaries"
install_name_tool $INSTALL_DIR/bin/python3.7m \
-add_rpath $INSTALL_DIR/.pyenv/versions/3.7.12/lib \
-add_rpath $INSTALL_DIR/.pyenv/versions/3.7.12/openssl/lib || echo "already present"
################################################################################
# Sign python binaries in `bin` and `lib`
################################################################################
echo "**** Signing binaries that have entitlements (/opt/salt/.pyenv)"
find ${INSTALL_DIR}/.pyenv \
if ! find ${INSTALL_DIR}/.pyenv \
-type f \
-perm -u=x \
-follow \
! -name "*.so" \
! -name "*.dylib" \
! -name "*.py" \
! -name "*.sh" \
! -name "*.bat" \
! -name "*.pl" \
! -name "*.crt" \
! -name "*.key" \
-exec codesign --timestamp \
--options=runtime \
--verbose \
--entitlements ./entitlements.plist \
--sign "$DEV_APP_CERT" "{}" \;
--force \
--entitlements "$SCRIPT_DIR/entitlements.plist" \
--sign "$DEV_APP_CERT" "{}" \; > "$CMD_OUTPUT" 2>&1; then
echo "Failed to sign binaries"
echo "Failed to sign run with entitlements"
echo "output >>>>>>"
cat "$CMD_OUTPUT" 1>&2
echo "<<<<<< output"
exit 1
fi
echo "**** Signing dynamic libraries (*dylib) (/opt/salt/.pyenv)"
find ${INSTALL_DIR}/.pyenv \
if ! find ${INSTALL_DIR}/.pyenv \
-type f \
-name "*dylib" \
-follow \
-exec codesign --timestamp \
--options=runtime \
--verbose \
--sign "$DEV_APP_CERT" "{}" \;
--force \
--sign "$DEV_APP_CERT" "{}" \; > "$CMD_OUTPUT" 2>&1; then
echo "Failed to sign dynamic libraries"
echo "output >>>>>>"
cat "$CMD_OUTPUT" 1>&2
echo "<<<<<< output"
exit 1
fi
echo "**** Signing shared libraries (*.so) (/opt/salt/.pyenv)"
if ! echo "**** Signing shared libraries (*.so) (/opt/salt/.pyenv)"
find ${INSTALL_DIR}/.pyenv \
-type f \
-name "*.so" \
@ -118,7 +135,14 @@ find ${INSTALL_DIR}/.pyenv \
-exec codesign --timestamp \
--options=runtime \
--verbose \
--sign "$DEV_APP_CERT" "{}" \;
--force \
--sign "$DEV_APP_CERT" "{}" \; > "$CMD_OUTPUT" 2>&1; then
echo "Failed to sign shared libraries"
echo "output >>>>>>"
cat "$CMD_OUTPUT" 1>&2
echo "<<<<<< output"
exit 1
fi
echo "**** Signing Binaries Completed Successfully"
echo "^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"

2
run.py
View file

@ -59,6 +59,8 @@ def py_shell():
def python_runtime():
import traceback
# extract the absolute script path to alter sys.path and specific dunder variables
script = pathlib.Path(sys.argv[2]).expanduser().resolve()
sys.path.insert(0, str(script.parent))

2
salt/channel/client.py
View file

@ -295,7 +295,7 @@ class AsyncReqChannel:
ret = yield self._crypted_transfer(load, timeout=timeout, raw=raw)
break
except Exception as exc: # pylint: disable=broad-except
log.error("Failed to send msg %r", exc)
log.trace("Failed to send msg %r", exc)
if _try >= tries:
raise
else:

721
salt/cloud/deploy/bootstrap-salt.sh
View file

File diff suppressed because it is too large Load diff

53
salt/defaults/__init__.py
View file

@ -5,5 +5,58 @@ Do NOT, import any salt modules (salt.utils, salt.config, etc.) into this file,
as this may result in circular imports.
"""
class _Constant:
"""
This class implements a way to create constants in python.
NOTE:
- This is not really a constant, ie, the `is` check will not work, you'll
have to use `==`.
- This class SHALL NOT be considered public API and might change or even
go away at any given time.
"""
__slots__ = ("name", "value")
def __init__(self, name, value=None):
self.name = name
self.value = value
def __hash__(self):
return hash((self.name, self.value))
def __eq__(self, other):
if not isinstance(other, self.__class__):
return False
if self.name != other.name:
return False
return self.value == other.value
def __get_state__(self):
return {
"name": self.name,
"value": self.value,
}
def __set_state__(self, state):
return self.__class__(state["name"], state["value"])
def __repr__(self):
if self.value:
return "<Constant.{} value={}>".format(self.name, self.value)
return "<Constant.{}>".format(self.name)
# Default delimiter for multi-level traversal in targeting
DEFAULT_TARGET_DELIM = ":"
"""
Used in functions to define that a keyword default is not set.
It's used to differentiate from `None`, `True`, `False` which, in some
cases are proper defaults and are also proper values to pass.
"""
NOT_SET = _Constant("NOT_SET")

6
salt/minion.py
View file

@ -820,11 +820,11 @@ class MinionBase:
self.connected = True
raise salt.ext.tornado.gen.Return((opts["master"], pub_channel))
except SaltClientError:
if pub_channel:
pub_channel.close()
if attempts == tries:
# Exhausted all attempts. Return exception.
self.connected = False
if pub_channel:
pub_channel.close()
raise
def _discover_masters(self):
@ -1622,7 +1622,7 @@ class Minion(MinionBase):
with salt.utils.event.get_event(
"minion", opts=self.opts, listen=False
) as event:
ret = yield event.fire_event(
ret = yield event.fire_event_async(
load, "__master_req_channel_payload", timeout=timeout
)
raise salt.ext.tornado.gen.Return(ret)

13
salt/modules/cassandra_mod.py
View file

@ -1,4 +1,10 @@
"""
.. warning::
The `cassandra` module is deprecated in favor of the `cassandra_cql`
module.
Cassandra NoSQL Database Module
:depends: - pycassa Cassandra Python adapter
@ -14,6 +20,7 @@ Cassandra NoSQL Database Module
import logging
import salt.utils.path
from salt.utils.versions import warn_until_date
log = logging.getLogger(__name__)
@ -37,6 +44,12 @@ def __virtual__():
"The cassandra execution module cannot be loaded: pycassa not installed.",
)
warn_until_date(
"20240101",
"The cassandra returner is broken and deprecated, and will be removed"
" after {date}. Use the cassandra_cql returner instead",
)
if HAS_PYCASSA and salt.utils.path.which("nodetool"):
return "cassandra"
return (

13
salt/modules/kernelpkg_linux_yum.py
View file

@ -21,10 +21,11 @@ log = logging.getLogger(__name__)
__virtualname__ = "kernelpkg"
# Import functions from yumpkg
# pylint: disable=invalid-name, protected-access
_yum = salt.utils.functools.namespaced_function(salt.modules.yumpkg._yum, globals())
# pylint: enable=invalid-name, protected-access
if __IMPORT_ERROR is None:
# Import functions from yumpkg
# pylint: disable=invalid-name, protected-access
_yum = salt.utils.functools.namespaced_function(salt.modules.yumpkg._yum, globals())
# pylint: enable=invalid-name, protected-access
def __virtual__():
@ -33,7 +34,7 @@ def __virtual__():
"""
if __IMPORT_ERROR:
return (False, __IMPORT_ERROR)
return False, __IMPORT_ERROR
if __grains__.get("os_family", "") == "RedHat":
return __virtualname__
@ -45,7 +46,7 @@ def __virtual__():
):
return __virtualname__
return (False, "Module kernelpkg_linux_yum: no YUM based system detected")
return False, "Module kernelpkg_linux_yum: no YUM based system detected"
def active():

5
salt/modules/mac_brew_pkg.py
View file

@ -258,7 +258,10 @@ def latest_version(*names, **kwargs):
def get_version(pkg_info):
# Perhaps this will need an option to pick devel by default
return pkg_info["versions"]["stable"] or pkg_info["versions"]["devel"]
version = pkg_info["versions"]["stable"] or pkg_info["versions"]["devel"]
if pkg_info["versions"]["bottle"] and pkg_info["revision"] >= 1:
version = "{}_{}".format(version, pkg_info["revision"])
return version
versions_dict = {key: get_version(val) for key, val in _info(*names).items()}

6
salt/modules/pillar.py
View file

@ -14,7 +14,7 @@ import salt.utils.dictupdate
import salt.utils.functools
import salt.utils.odict
import salt.utils.yaml
from salt.defaults import DEFAULT_TARGET_DELIM
from salt.defaults import DEFAULT_TARGET_DELIM, NOT_SET
from salt.exceptions import CommandExecutionError
__proxyenabled__ = ["*"]
@ -24,7 +24,7 @@ log = logging.getLogger(__name__)
def get(
key,
default=None,
default=NOT_SET,
merge=False,
merge_nested_lists=None,
delimiter=DEFAULT_TARGET_DELIM,
@ -122,7 +122,7 @@ def get(
salt '*' pillar.get pkg:apache
salt '*' pillar.get abc::def|ghi delimiter='|'
"""
if default is None:
if default == NOT_SET:
default = KeyError
if not __opts__.get("pillar_raise_on_missing"):
if default is KeyError:

2
salt/modules/rpmbuild_pkgbuild.py
View file

@ -154,7 +154,7 @@ def _get_deps(deps, tree_base, saltenv="base"):
"'deps' must be a Python list or comma-separated string"
)
for deprpm in deps:
parsed = urllib.parse._urlparse(deprpm)
parsed = urllib.parse.urlparse(deprpm)
depbase = os.path.basename(deprpm)
dest = os.path.join(tree_base, depbase)
if parsed.scheme:

9
salt/modules/vault.py
View file

@ -178,12 +178,13 @@ Functions to interact with Hashicorp Vault.
import logging
import os
from salt.defaults import NOT_SET
from salt.exceptions import CommandExecutionError
log = logging.getLogger(__name__)
def read_secret(path, key=None, metadata=False, default=None):
def read_secret(path, key=None, metadata=False, default=NOT_SET):
"""
.. versionchanged:: 3001
The ``default`` argument has been added. When the path or path/key
@ -211,7 +212,7 @@ def read_secret(path, key=None, metadata=False, default=None):
first: {{ supersecret.first }}
second: {{ supersecret.second }}
"""
if default is None:
if default == NOT_SET:
default = CommandExecutionError
version2 = __utils__["vault.is_v2"](path)
if version2["v2"]:
@ -358,7 +359,7 @@ def destroy_secret(path, *args):
return False
def list_secrets(path, default=None):
def list_secrets(path, default=NOT_SET):
"""
.. versionchanged:: 3001
The ``default`` argument has been added. When the path or path/key
@ -374,7 +375,7 @@ def list_secrets(path, default=None):
salt '*' vault.list_secrets "secret/my/"
"""
if default is None:
if default == NOT_SET:
default = CommandExecutionError
log.debug("Listing vault secret keys for %s in %s", __grains__["id"], path)
version2 = __utils__["vault.is_v2"](path)

8
salt/modules/yaml.py
View file

@ -27,9 +27,13 @@ except ImportError:
def __virtual__():
if HAS_LINT:
version = salt.utils.yamllint.version()
version_cmp = salt.utils.versions.version_cmp(version, "1.20.0")
if salt.utils.yamllint.has_yamllint():
version = salt.utils.yamllint.version()
version_cmp = salt.utils.versions.version_cmp(version, "1.20.0")
else:
return (False, "yamllint not installed")
if version_cmp >= 0:
return __virtualname__
else:

10
salt/payload.py
View file

@ -14,6 +14,7 @@ import salt.transport.frame
import salt.utils.immutabletypes as immutabletypes
import salt.utils.msgpack
import salt.utils.stringutils
from salt.defaults import _Constant
from salt.exceptions import SaltDeserializationError, SaltReqTimeoutError
from salt.utils.data import CaseInsensitiveDict
@ -77,6 +78,9 @@ def loads(msg, encoding=None, raw=False):
if code == 78:
data = salt.utils.stringutils.to_unicode(data)
return datetime.datetime.strptime(data, "%Y%m%dT%H:%M:%S.%f")
if code == 79:
name, value = salt.utils.msgpack.loads(data, raw=False)
return _Constant(name, value)
return data
gc.disable() # performance optimization for msgpack
@ -144,6 +148,12 @@ def dumps(msg, use_bin_type=False):
78,
salt.utils.stringutils.to_bytes(obj.strftime("%Y%m%dT%H:%M:%S.%f")),
)
elif isinstance(obj, _Constant):
# Special case our constants.
return salt.utils.msgpack.ExtType(
79,
salt.utils.msgpack.dumps((obj.name, obj.value), use_bin_type=True),
)
# The same for immutable types
elif isinstance(obj, immutabletypes.ImmutableDict):
return dict(obj)

11
salt/returners/cassandra_return.py
View file

@ -1,4 +1,9 @@
"""
.. warning::
The `cassandra` returner is deprecated in favor of the `cassandra_cql`
returner.
Return data to a Cassandra ColumnFamily
Here's an example Keyspace / ColumnFamily setup that works with this
@ -22,6 +27,7 @@ Required python modules: pycassa
import logging
import salt.utils.jid
from salt.utils.versions import warn_until_date
try:
import pycassa # pylint: disable=import-error
@ -46,6 +52,11 @@ __virtualname__ = "cassandra"
def __virtual__():
if not HAS_PYCASSA:
return False, "Could not import cassandra returner; pycassa is not installed."
warn_until_date(
"20240101",
"The cassandra returner is broken and deprecated, and will be removed"
" after {date}. Use the cassandra_cql returner instead",
)
return __virtualname__

20
salt/utils/yamllint.py
View file

@ -1,14 +1,26 @@
import logging
import yamllint
from yamllint import linter
from yamllint.config import YamlLintConfig
import salt.utils.stringutils
try:
import yamllint
from yamllint import linter
from yamllint.config import YamlLintConfig
HAS_YAMLLINT = True
except ImportError:
HAS_YAMLLINT = False
log = logging.getLogger(__name__)
def has_yamllint():
"""
report if yamllint could be imported safly. allowing for clean import detection
"""
return HAS_YAMLLINT
def version():
"""
report version of yamllint installed for version comparison

34
tests/pytests/functional/modules/pillar/test_config.py Normal file
View file

@ -0,0 +1,34 @@
# This tests the pillar module with `pillar_raise_on_missing` set to True in the
# minion config. This effects all tests in this file
import pytest
pytestmark = [
pytest.mark.windows_whitelisted,
]
@pytest.fixture(scope="module")
def pillar(modules):
return modules.pillar
@pytest.fixture(scope="module")
def minion_config_overrides():
yield {"pillar_raise_on_missing": True}
def test_get_non_existing(pillar):
"""
Test pillar.get when the item does not exist. Should raise a KeyError when
`pillar_raise_on_missing` is True in the minion config
"""
with pytest.raises(KeyError):
pillar.get("non-existing-pillar-item")
def test_get_default_none(pillar):
"""
Tests pillar.get when default is set to `None`. Should return `None`
"""
result = pillar.get("non-existing-pillar-item", default=None)
assert result is None

42
tests/pytests/functional/modules/pillar/test_default.py Normal file
View file

@ -0,0 +1,42 @@
import pytest
pytestmark = [
pytest.mark.windows_whitelisted,
]
@pytest.fixture(scope="module")
def sys_mod(modules):
return modules.sys
@pytest.fixture(scope="module")
def pillar(modules):
return modules.pillar
def test_pillar_get_issue_61084(sys_mod):
"""
Test issue 61084. `sys.argspec` should return valid data and not throw a
TypeError due to pickling
This should probably be a pre-commit check or something
"""
result = sys_mod.argspec("pillar.get")
assert isinstance(result, dict)
assert isinstance(result.get("pillar.get"), dict)
def test_get_non_existing(pillar):
"""
Tests pillar.get when the item does not exist. Should return an empty string
"""
result = pillar.get("non-existing-pillar-item")
assert result == ""
def test_get_default_none(pillar):
"""
Tests pillar.get when default is set to `None`. Should return `None`
"""
result = pillar.get("non-existing-pillar-item", default=None)
assert result is None

21
tests/pytests/functional/modules/test_pillar.py
View file

@ -1,21 +0,0 @@
import pytest
pytestmark = [
pytest.mark.windows_whitelisted,
]
@pytest.fixture(scope="module")
def sys_mod(modules):
return modules.sys
def test_pillar_get_issue_61084(sys_mod):
"""
Test issue 61084. `sys.argspec` should return valid data and not throw a
TypeError due to pickling
This should probably be a pre-commit check or something
"""
result = sys_mod.argspec("pillar.get")
assert isinstance(result, dict)
assert isinstance(result.get("pillar.get"), dict)

3
tests/pytests/functional/transport/zeromq/test_pub_server_channel.py
View file

@ -31,9 +31,10 @@ log = logging.getLogger(__name__)
pytestmark = [
pytest.mark.skip_on_freebsd(reason="Temporarily skipped on FreeBSD."),
pytest.mark.skip_on_spawning_platform(
reason="These tests are currently broken on spawning platforms. Need to be rewritten.",
)
),
]

4
tests/pytests/functional/utils/yamllint/test_yamllint.py
View file

@ -64,3 +64,7 @@ def test_config():
def test_version():
assert versions.version_cmp(yamllint.version(), "1.26.3") >= 0
def test_has_yamllint():
assert yamllint.has_yamllint() is True

13
tests/pytests/unit/test_defaults.py Normal file
View file

@ -0,0 +1,13 @@
import pickle
from salt.defaults import _Constant
def test_pickle_constants():
"""
That that we can pickle and unpickle constants.
"""
constant = _Constant("Foo", 123)
sdata = pickle.dumps(constant)
odata = pickle.loads(sdata)
assert odata == constant

26
tests/pytests/unit/test_minion.py
View file

@ -86,6 +86,9 @@ def test_send_req_fires_completion_event(event):
load = {"load": "value"}
timeout = 60
# XXX This is buggy because "async" in event[0] will never evaluate
# to True and if it *did* evaluate to true the test would fail
# because you Mock isn't a co-routine.
if "async" in event[0]:
rtn = minion._send_req_async(load, timeout).result()
else:
@ -116,6 +119,29 @@ def test_send_req_fires_completion_event(event):
assert rtn
async def test_send_req_async_regression_62453():
event_enter = MagicMock()
event_enter.send.side_effect = (
lambda data, tag, cb=None, timeout=60: salt.ext.tornado.gen.maybe_future(True)
)
event = MagicMock()
event.__enter__.return_value = event_enter
opts = salt.config.DEFAULT_MINION_OPTS.copy()
opts["random_startup_delay"] = 0
opts["return_retry_tries"] = 30
opts["grains"] = {}
with patch("salt.loader.grains"):
minion = salt.minion.Minion(opts)
load = {"load": "value"}
timeout = 60
# We are just validating no exception is raised
rtn = await minion._send_req_async(load, timeout)
assert rtn is False
@patch("salt.channel.client.ReqChannel.factory")
def test_mine_send_tries(req_channel_factory):
channel_enter = MagicMock()

11
tests/pytests/unit/test_payload.py
View file

@ -9,6 +9,7 @@ import logging
import salt.exceptions
import salt.payload
from salt.defaults import _Constant
from salt.utils import immutabletypes
from salt.utils.odict import OrderedDict
@ -199,3 +200,13 @@ def test_raw_vs_encoding_utf8():
sdata = salt.payload.dumps(idata.copy())
odata = salt.payload.loads(sdata, encoding="utf-8")
assert isinstance(odata[dtvalue], str)
def test_constants():
"""
That that we handle encoding and decoding of constants.
"""
constant = _Constant("Foo", "bar")
sdata = salt.payload.dumps(constant)
odata = salt.payload.loads(sdata)
assert odata == constant