Merge remote-tracking branch 'saltstack/2018.3' into whitelist

This commit is contained in:
Daniel A. Wozniak 2018-09-21 15:39:45 -07:00
commit a44b37a510
No known key found for this signature in database
GPG key ID: 166B9D2C06C82D61
12 changed files with 297 additions and 113 deletions

View file

@ -4,6 +4,34 @@
Salt 2018.3.0 Release Notes - Codename Oxygen
=============================================
.. warning::
If you are using Jinja to dump lists or dictionaries in your SLS files,
this will now cause errors in Python 2 since Jinja does not produce
YAML-compatible output when strings in the data structures contain unicode
types. The dictionary must be passed through a Jinja filter to produce
YAML-compatible strings.
The below is an example of invalid SLS:
.. code-block:: yaml
/etc/foo.conf:
file.mangaged:
- source: salt://foo.conf
- template: jinja
- defaults: {{ mydict }}
To make it valid, use either one of Salt's own ``json`` or ``yaml``
filters:
.. code-block:: yaml
/etc/foo.conf:
file.mangaged:
- source: salt://foo.conf
- template: jinja
- defaults: {{ mydict | json }}
Unicode/Python 3 Compatibility Improvements
===========================================

View file

@ -14,6 +14,34 @@ Statistics
- Contributors: **55** (`Ch3LL`_, `DmitryKuzmenko`_, `Giandom`_, `Kimol`_, `L4rS6`_, `LukeCarrier`_, `OrlandoArcapix`_, `TamCore`_, `The-Loeki`_, `UtahDave`_, `aesposito91`_, `bbinet`_, `bdrung`_, `boltronics`_, `bosatsu`_, `clan`_, `corywright`_, `damon-atkins`_, `dincamihai`_, `dmurphy18`_, `dnABic`_, `douglasjreynolds`_, `dwoz`_, `edgan`_, `ejparker12`_, `esell`_, `ezh`_, `femnad`_, `folti`_, `garethgreenaway`_, `gtmanfred`_, `isbm`_, `jasperla`_, `johnj`_, `mateiw`_, `mcalmer`_, `mirceaulinic`_, `morganwillcock`_, `opdude`_, `pcn`_, `pruiz`_, `psagers`_, `psyer`_, `rallytime`_, `robinro`_, `s0undt3ch`_, `samodid`_, `shengis`_, `skjaro`_, `tankywoo`_, `terminalmage`_, `twangboy`_, `vutny`_, `yannj-fr`_, `zmedico`_)
.. warning::
If you are using Jinja to dump lists or dictionaries in your SLS files,
this will now cause errors in Python 2 since Jinja does not produce
YAML-compatible output when strings in the data structures contain unicode
types. The dictionary must be passed through a Jinja filter to produce
YAML-compatible strings.
The below is an example of invalid SLS:
.. code-block:: yaml
/etc/foo.conf:
file.mangaged:
- source: salt://foo.conf
- template: jinja
- defaults: {{ mydict }}
To make it valid, use either one of Salt's own ``json`` or ``yaml``
filters:
.. code-block:: yaml
/etc/foo.conf:
file.mangaged:
- source: salt://foo.conf
- template: jinja
- defaults: {{ mydict | json }}
Tornado 5.0 Support for Python 2 Only
=====================================

View file

@ -26,6 +26,33 @@ Statistics
- Contributors: **4** (`cro`_, `garethgreenaway`_, `gtmanfred`_, `rallytime`_)
.. warning::
If you are using Jinja to dump lists or dictionaries in your SLS files,
this will now cause errors in Python 2 since Jinja does not produce
YAML-compatible output when strings in the data structures contain unicode
types. The dictionary must be passed through a Jinja filter to produce
YAML-compatible strings.
The below is an example of invalid SLS:
.. code-block:: yaml
/etc/foo.conf:
file.mangaged:
- source: salt://foo.conf
- template: jinja
- defaults: {{ mydict }}
To make it valid, use either one of Salt's own ``json`` or ``yaml``
filters:
.. code-block:: yaml
/etc/foo.conf:
file.mangaged:
- source: salt://foo.conf
- template: jinja
- defaults: {{ mydict | json }}
Changelog for v2018.3.1..v2018.3.2
==================================

View file

@ -5,6 +5,35 @@ In Progress: Salt 2018.3.3 Release Notes
Version 2018.3.3 is an **unreleased** bugfix release for :ref:`2018.3.0 <release-2018-3-0>`.
This release is still in progress and has not been released yet.
.. warning::
If you are using Jinja to dump lists or dictionaries in your SLS files,
this will now cause errors in Python 2 since Jinja does not produce
YAML-compatible output when strings in the data structures contain unicode
types. The dictionary must be passed through a Jinja filter to produce
YAML-compatible strings.
The below is an example of invalid SLS:
.. code-block:: yaml
/etc/foo.conf:
file.mangaged:
- source: salt://foo.conf
- template: jinja
- defaults: {{ mydict }}
To make it valid, use either one of Salt's own ``json`` or ``yaml``
filters. Another option would be to use Jinja's :ref:`tojson
<release-2018-3-3-tojson-filter>` filter.
.. code-block:: yaml
/etc/foo.conf:
file.mangaged:
- source: salt://foo.conf
- template: jinja
- defaults: {{ mydict | tojson }}
Changes to win_timezone
=======================
@ -15,6 +44,8 @@ Improves timezone detection by using the pytz module.
Adds ``timezone.list`` to list supported timezones in either Windows or Unix
format.
.. _release-2018-3-3-tojson-filter:
New Jinja Filter
================
@ -22,15 +53,15 @@ The :jinja_ref:`tojson` filter (from Jinja 2.9 and later) has been ported to
Salt, and will be used when this filter is not available. This allows older LTS
releases such as CentOS 7 and Ubuntu 14.04 to use this filter.
You should use this filter any time you wish to dump a list or dictionary into
an SLS file, to ensure that the result is able to be loaded by the YAML
renderer. For example:
You can use this filter any time you wish to dump a list or dictionary into an
SLS file, to ensure that the result is able to be loaded by the YAML renderer.
For example:
.. code-block:: jinja
foo:
bar.baz:
- some_arg: {{ mydict|tojson }}
- some_arg: {{ mydict | tojson }}
MacOSX escape characters with runas
===================================

View file

@ -60,6 +60,7 @@ import logging
import pprint
import base64
import collections
import pkgutil
import salt.cache
import salt.config as config
import salt.utils.cloud
@ -67,7 +68,6 @@ import salt.utils.data
import salt.utils.files
import salt.utils.stringutils
import salt.utils.yaml
from salt.utils.versions import LooseVersion
from salt.ext import six
import salt.version
from salt.exceptions import (
@ -117,9 +117,12 @@ try:
from azure.mgmt.storage import StorageManagementClient
from azure.mgmt.web import WebSiteManagementClient
from msrestazure.azure_exceptions import CloudError
from azure.multiapi.storage.v2016_05_31 import CloudStorageAccount
from azure.cli import core
HAS_LIBS = LooseVersion(core.__version__) >= LooseVersion("2.0.12")
if pkgutil.find_loader('azure.multiapi'):
# use multiapi version if available
from azure.multiapi.storage.v2016_05_31 import CloudStorageAccount
else:
from azure.storage import CloudStorageAccount
HAS_LIBS = True
except ImportError:
pass
# pylint: enable=wrong-import-position,wrong-import-order
@ -152,8 +155,7 @@ def __virtual__():
False,
'The following dependencies are required to use the AzureARM driver: '
'Microsoft Azure SDK for Python >= 2.0rc5, '
'Microsoft Azure Storage SDK for Python >= 0.32, '
'Microsoft Azure CLI >= 2.0.12'
'Microsoft Azure Storage SDK for Python >= 0.32'
)
global cache # pylint: disable=global-statement,invalid-name

View file

@ -410,6 +410,13 @@ def master_event(type, master=None):
return event_map.get(type, None)
def service_name():
'''
Return the proper service name based on platform
'''
return 'salt_minion' if 'bsd' in sys.platform else 'salt-minion'
class MinionBase(object):
def __init__(self, opts):
self.opts = opts
@ -2619,8 +2626,17 @@ class Minion(MinionBase):
log.error('** Master Ping failed. Attempting to restart minion**')
delay = self.opts.get('random_reauth_delay', 5)
log.info('delaying random_reauth_delay %ss', delay)
# regular sys.exit raises an exception -- which isn't sufficient in a thread
os._exit(salt.defaults.exitcodes.SALT_KEEPALIVE)
try:
self.functions['service.restart'](service_name())
except KeyError:
# Probably no init system (running in docker?)
log.warning(
'ping_interval reached without response '
'from the master, but service.restart '
'could not be run to restart the minion '
'daemon. ping_interval requires that the '
'minion is running under an init system.'
)
self._fire_master('ping', 'minion_ping', sync=False, timeout_handler=ping_timeout_handler)
except Exception:

View file

@ -6,6 +6,9 @@ not all of the options may have been provided yet. For a complete reference,
see the `Parallels Desktop Reference Guide
<http://download.parallels.com/desktop/v9/ga/docs/en_US/Parallels%20Command%20Line%20Reference%20Guide.pdf>`_.
This module requires the prlctl binary to be installed to run most functions.
To run parallels.prlsrvctl, the prlsrvctl binary is required.
What has not been implemented yet can be accessed through ``parallels.prlctl``
and ``parallels.prlsrvctl`` (note the preceding double dash ``--`` as
necessary):
@ -29,7 +32,7 @@ import shlex
import salt.utils.locales
import salt.utils.path
import salt.utils.yaml
from salt.exceptions import SaltInvocationError
from salt.exceptions import SaltInvocationError, CommandExecutionError
# Import 3rd party libs
from salt.ext import six
@ -43,17 +46,6 @@ log = logging.getLogger(__name__)
GUID_REGEX = re.compile(r'{?([0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12})}?', re.I)
def __virtual__():
'''
Load this module if prlctl is available
'''
if not salt.utils.path.which('prlctl'):
return (False, 'prlctl utility not available')
if not salt.utils.path.which('prlsrvctl'):
return (False, 'prlsrvctl utility not available')
return __virtualname__
def _normalize_args(args):
'''
Return args as a list of strings
@ -111,6 +103,9 @@ def prlsrvctl(sub_cmd, args=None, runas=None):
salt '*' parallels.prlsrvctl usb list runas=macdev
salt -- '*' parallels.prlsrvctl set '--mem-limit auto' runas=macdev
'''
if not salt.utils.path.which('prlsrvctl'):
raise CommandExecutionError('prlsrvctl utility not available')
# Construct command
cmd = ['prlsrvctl', sub_cmd]
if args:
@ -141,6 +136,9 @@ def prlctl(sub_cmd, args=None, runas=None):
salt '*' parallels.prlctl exec 'macvm uname' runas=macdev
salt -- '*' parallels.prlctl capture 'macvm --file macvm.display.png' runas=macdev
'''
if not salt.utils.path.which('prlctl'):
raise CommandExecutionError('prlctl utility not available')
# Construct command
cmd = ['prlctl', sub_cmd]
if args:

View file

@ -1174,17 +1174,15 @@ def get_pending_component_servicing():
salt '*' system.get_pending_component_servicing
'''
vname = '(Default)'
key = r'SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\RebootPending'
reg_ret = __salt__['reg.read_value']('HKLM', key, vname)
# So long as the registry key exists, a reboot is pending.
if reg_ret['success']:
log.debug('Found key: %s', key)
if __utils__['reg.key_exists']('HKLM', key):
log.debug('Key exists: %s', key)
return True
else:
log.debug('Unable to access key: %s', key)
log.debug('Key does not exist: %s', key)
return False
@ -1205,29 +1203,24 @@ def get_pending_domain_join():
salt '*' system.get_pending_domain_join
'''
vname = '(Default)'
base_key = r'SYSTEM\CurrentControlSet\Services\Netlogon'
avoid_key = r'{0}\AvoidSpnSet'.format(base_key)
join_key = r'{0}\JoinDomain'.format(base_key)
# If either the avoid_key or join_key is present,
# then there is a reboot pending.
avoid_reg_ret = __salt__['reg.read_value']('HKLM', avoid_key, vname)
if avoid_reg_ret['success']:
log.debug('Found key: %s', avoid_key)
if __utils__['reg.key_exists']('HKLM', avoid_key):
log.debug('Key exists: %s', avoid_key)
return True
else:
log.debug('Unable to access key: %s', avoid_key)
log.debug('Key does not exist: %s', avoid_key)
join_reg_ret = __salt__['reg.read_value']('HKLM', join_key, vname)
if join_reg_ret['success']:
log.debug('Found key: %s', join_key)
if __utils__['reg.key_exists']('HKLM', join_key):
log.debug('Key exists: %s', join_key)
return True
else:
log.debug('Unable to access key: %s', join_key)
log.debug('Key does not exist: %s', join_key)
return False
@ -1321,17 +1314,15 @@ def get_pending_update():
salt '*' system.get_pending_update
'''
vname = '(Default)'
key = r'SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update\RebootRequired'
reg_ret = __salt__['reg.read_value']('HKLM', key, vname)
# So long as the registry key exists, a reboot is pending.
if reg_ret['success']:
log.debug('Found key: %s', key)
if __utils__['reg.key_exists']('HKLM', key):
log.debug('Key exists: %s', key)
return True
else:
log.debug('Unable to access key: %s', key)
log.debug('Key does not exist: %s', key)
return False
@ -1421,7 +1412,9 @@ def get_pending_reboot():
'''
# Order the checks for reboot pending in most to least likely.
checks = (get_pending_update, get_pending_file_rename, get_pending_servermanager,
checks = (get_pending_update,
get_pending_file_rename,
get_pending_servermanager,
get_pending_component_servicing,
get_reboot_required_witnessed,
get_pending_computer_name,

View file

@ -2266,7 +2266,7 @@ def list_provides(**kwargs):
'''
ret = __context__.get('pkg.list_provides')
if not ret:
cmd = ['rpm', '-qa', '--queryformat', '[%{PROVIDES}_|-%{NAME}\n]']
cmd = ['rpm', '-qa', '--queryformat', '%{PROVIDES}_|-%{NAME}\n']
ret = dict()
for line in __salt__['cmd.run'](cmd, output_loglevel='trace', python_shell=False).splitlines():
provide, realname = line.split('_|-')
@ -2331,7 +2331,7 @@ def resolve_capabilities(pkgs, refresh, **kwargs):
try:
result = search(name, provides=True, match='exact')
if len(result) == 1:
name = result.keys()[0]
name = next(iter(result.keys()))
elif len(result) > 1:
log.warn("Found ambiguous match for capability '%s'.", pkg)
except CommandExecutionError as exc:

View file

@ -39,6 +39,7 @@ try:
import win32gui
import win32api
import win32con
import pywintypes
HAS_WINDOWS_MODULES = True
except ImportError:
HAS_WINDOWS_MODULES = False
@ -161,7 +162,7 @@ class Registry(object): # pylint: disable=R0903
def key_exists(hive, key, use_32bit_registry=False):
'''
Check that the key is found in the registry. This refers to keys and not
value/data pairs.
value/data pairs. To check value/data pairs, use ``value_exists``
Args:
@ -193,10 +194,74 @@ def key_exists(hive, key, use_32bit_registry=False):
try:
handle = win32api.RegOpenKeyEx(hkey, local_key, 0, access_mask)
win32api.RegCloseKey(handle)
return True
except Exception: # pylint: disable=E0602
return False
except pywintypes.error as exc:
if exc.winerror == 2:
return False
raise
finally:
win32api.RegCloseKey(handle)
def value_exists(hive, key, vname, use_32bit_registry=False):
'''
Check that the value/data pair is found in the registry.
.. version-added:: 2018.3.4
Args:
hive (str): The hive to connect to
key (str): The key to check in
vname (str): The name of the value/data pair you're checking
use_32bit_registry (bool): Look in the 32bit portion of the registry
Returns:
bool: True if exists, otherwise False
Usage:
.. code-block:: python
import salt.utils.win_reg
winreg.key_exists(hive='HKLM', key='SOFTWARE\\Microsoft')
'''
local_hive = _to_unicode(hive)
local_key = _to_unicode(key)
local_vname = _to_unicode(vname)
registry = Registry()
try:
hkey = registry.hkeys[local_hive]
except KeyError:
raise CommandExecutionError('Invalid Hive: {0}'.format(local_hive))
access_mask = registry.registry_32[use_32bit_registry]
try:
handle = win32api.RegOpenKeyEx(hkey, local_key, 0, access_mask)
except pywintypes.error as exc:
if exc.winerror == 2:
# The key containing the value/data pair does not exist
return False
raise
try:
# RegQueryValueEx returns and accepts unicode data
_, _ = win32api.RegQueryValueEx(handle, local_vname)
# value/data pair exists
return True
except pywintypes.error as exc:
if exc.winerror == 2 and vname is None:
# value/data pair exists but is empty
return True
else:
# value/data pair not found
return False
finally:
win32api.RegCloseKey(handle)
def broadcast_change():

View file

@ -398,6 +398,7 @@ class WinSystemModuleTest(ModuleCase):
now = datetime.datetime.now()
self.assertEqual(now.strftime("%I:%M"), ret.rsplit(':', 1)[0])
@flaky
@destructiveTest
def test_set_system_time(self):
'''

View file

@ -6,7 +6,7 @@ import textwrap
# Import Salt Libs
import salt.modules.parallels as parallels
from salt.exceptions import SaltInvocationError
from salt.exceptions import SaltInvocationError, CommandExecutionError
# Import Salt Testing Libs
from tests.support.mixins import LoaderModuleMockMixin
@ -25,27 +25,6 @@ class ParallelsTestCase(TestCase, LoaderModuleMockMixin):
def setup_loader_modules(self):
return {parallels: {}}
def test___virtual__(self):
'''
Test parallels.__virtual__
'''
mock_true = MagicMock(return_value=True)
mock_false = MagicMock(return_value=False)
# Validate false return
with patch('salt.utils.path.which', mock_false):
ret = parallels.__virtual__()
self.assertTrue(isinstance(ret, tuple))
self.assertEqual(len(ret), 2)
self.assertFalse(ret[0])
self.assertTrue(isinstance(ret[1], six.string_types))
# Validate true return
with patch('salt.utils.path.which', mock_true):
ret = parallels.__virtual__()
self.assertTrue(ret)
self.assertEqual(ret, 'parallels')
def test__normalize_args(self):
'''
Test parallels._normalize_args
@ -94,26 +73,34 @@ class ParallelsTestCase(TestCase, LoaderModuleMockMixin):
'''
runas = 'macdev'
# Validate 'prlsrvctl info'
info_cmd = ['prlsrvctl', 'info']
info_fcn = MagicMock()
with patch.dict(parallels.__salt__, {'cmd.run': info_fcn}):
parallels.prlsrvctl('info', runas=runas)
info_fcn.assert_called_once_with(info_cmd, runas=runas)
# Test missing prlsrvctl binary
with patch('salt.utils.path.which', MagicMock(return_value=False)):
with self.assertRaises(CommandExecutionError):
parallels.prlsrvctl('info', runas=runas)
# Validate 'prlsrvctl usb list'
usb_cmd = ['prlsrvctl', 'usb', 'list']
usb_fcn = MagicMock()
with patch.dict(parallels.__salt__, {'cmd.run': usb_fcn}):
parallels.prlsrvctl('usb', 'list', runas=runas)
usb_fcn.assert_called_once_with(usb_cmd, runas=runas)
# Simulate the existence of prlsrvctl
with patch('salt.utils.path.which', MagicMock(return_value="/usr/bin/prlsrvctl")):
# Validate 'prlsrvctl set "--mem-limit auto"'
set_cmd = ['prlsrvctl', 'set', '--mem-limit', 'auto']
set_fcn = MagicMock()
with patch.dict(parallels.__salt__, {'cmd.run': set_fcn}):
parallels.prlsrvctl('set', '--mem-limit auto', runas=runas)
set_fcn.assert_called_once_with(set_cmd, runas=runas)
# Validate 'prlsrvctl info'
info_cmd = ['prlsrvctl', 'info']
info_fcn = MagicMock()
with patch.dict(parallels.__salt__, {'cmd.run': info_fcn}):
parallels.prlsrvctl('info', runas=runas)
info_fcn.assert_called_once_with(info_cmd, runas=runas)
# Validate 'prlsrvctl usb list'
usb_cmd = ['prlsrvctl', 'usb', 'list']
usb_fcn = MagicMock()
with patch.dict(parallels.__salt__, {'cmd.run': usb_fcn}):
parallels.prlsrvctl('usb', 'list', runas=runas)
usb_fcn.assert_called_once_with(usb_cmd, runas=runas)
# Validate 'prlsrvctl set "--mem-limit auto"'
set_cmd = ['prlsrvctl', 'set', '--mem-limit', 'auto']
set_fcn = MagicMock()
with patch.dict(parallels.__salt__, {'cmd.run': set_fcn}):
parallels.prlsrvctl('set', '--mem-limit auto', runas=runas)
set_fcn.assert_called_once_with(set_cmd, runas=runas)
def test_prlctl(self):
'''
@ -121,26 +108,34 @@ class ParallelsTestCase(TestCase, LoaderModuleMockMixin):
'''
runas = 'macdev'
# Validate 'prlctl user list'
user_cmd = ['prlctl', 'user', 'list']
user_fcn = MagicMock()
with patch.dict(parallels.__salt__, {'cmd.run': user_fcn}):
parallels.prlctl('user', 'list', runas=runas)
user_fcn.assert_called_once_with(user_cmd, runas=runas)
# Test missing prlctl binary
with patch('salt.utils.path.which', MagicMock(return_value=False)):
with self.assertRaises(CommandExecutionError):
parallels.prlctl('info', runas=runas)
# Validate 'prlctl exec "macvm uname"'
exec_cmd = ['prlctl', 'exec', 'macvm', 'uname']
exec_fcn = MagicMock()
with patch.dict(parallels.__salt__, {'cmd.run': exec_fcn}):
parallels.prlctl('exec', 'macvm uname', runas=runas)
exec_fcn.assert_called_once_with(exec_cmd, runas=runas)
# Simulate the existence of prlctl
with patch('salt.utils.path.which', MagicMock(return_value="/usr/bin/prlctl")):
# Validate 'prlctl capture "macvm --file macvm.display.png"'
cap_cmd = ['prlctl', 'capture', 'macvm', '--file', 'macvm.display.png']
cap_fcn = MagicMock()
with patch.dict(parallels.__salt__, {'cmd.run': cap_fcn}):
parallels.prlctl('capture', 'macvm --file macvm.display.png', runas=runas)
cap_fcn.assert_called_once_with(cap_cmd, runas=runas)
# Validate 'prlctl user list'
user_cmd = ['prlctl', 'user', 'list']
user_fcn = MagicMock()
with patch.dict(parallels.__salt__, {'cmd.run': user_fcn}):
parallels.prlctl('user', 'list', runas=runas)
user_fcn.assert_called_once_with(user_cmd, runas=runas)
# Validate 'prlctl exec "macvm uname"'
exec_cmd = ['prlctl', 'exec', 'macvm', 'uname']
exec_fcn = MagicMock()
with patch.dict(parallels.__salt__, {'cmd.run': exec_fcn}):
parallels.prlctl('exec', 'macvm uname', runas=runas)
exec_fcn.assert_called_once_with(exec_cmd, runas=runas)
# Validate 'prlctl capture "macvm --file macvm.display.png"'
cap_cmd = ['prlctl', 'capture', 'macvm', '--file', 'macvm.display.png']
cap_fcn = MagicMock()
with patch.dict(parallels.__salt__, {'cmd.run': cap_fcn}):
parallels.prlctl('capture', 'macvm --file macvm.display.png', runas=runas)
cap_fcn.assert_called_once_with(cap_cmd, runas=runas)
def test_list_vms(self):
'''