New pillar/master_tops saltclass module

This commit is contained in:
Olivier Mauras 2017-07-12 17:29:22 +02:00
parent f1a93401d1
commit 139e065ce9
17 changed files with 806 additions and 0 deletions

View file

@ -97,6 +97,194 @@ file. For example:
These commands will run in sequence **before** the bootstrap script is executed.
New pillar/master_tops module called saltclass
----------------------------------------------
This module clones the behaviour of reclass (http://reclass.pantsfullofunix.net/), without the need of an external app, and add several features to improve flexibility.
Saltclass lets you define your nodes from simple ``yaml`` files (``.yml``) through hierarchical class inheritance with the possibility to override pillars down the tree.
**Features**
- Define your nodes through hierarchical class inheritance
- Reuse your reclass datas with minimal modifications
- applications => states
- parameters => pillars
- Use Jinja templating in your yaml definitions
- Access to the following Salt objects in Jinja
- ``__opts__``
- ``__salt__``
- ``__grains__``
- ``__pillars__``
- ``minion_id``
- Chose how to merge or override your lists using ^ character (see examples)
- Expand variables ${} with possibility to escape them if needed \${} (see examples)
- Ignores missing node/class and will simply return empty without breaking the pillar module completely - will be logged
An example subset of datas is available here: http://git.mauras.ch/salt/saltclass/src/master/examples
========================== ===========
Terms usable in yaml files Description
========================== ===========
classes A list of classes that will be processed in order
states A list of states that will be returned by master_tops function
pillars A yaml dictionnary that will be returned by the ext_pillar function
environment Node saltenv that will be used by master_tops
========================== ===========
A class consists of:
- zero or more parent classes
- zero or more states
- any number of pillars
A child class can override pillars from a parent class.
A node definition is a class in itself with an added ``environment`` parameter for ``saltenv`` definition.
**class names**
Class names mimic salt way of defining states and pillar files.
This means that ``default.users`` class name will correspond to one of these:
- ``<saltclass_path>/classes/default/users.yml``
- ``<saltclass_path>/classes/default/users/init.yml``
**Saltclass tree**
A saltclass tree would look like this:
.. code-block:: text
<saltclass_path>
├── classes
│ ├── app
│ │ ├── borgbackup.yml
│ │ └── ssh
│ │ └── server.yml
│ ├── default
│ │ ├── init.yml
│ │ ├── motd.yml
│ │ └── users.yml
│ ├── roles
│ │ ├── app.yml
│ │ └── nginx
│ │ ├── init.yml
│ │ └── server.yml
│ └── subsidiaries
│ ├── gnv.yml
│ ├── qls.yml
│ └── zrh.yml
└── nodes
├── geneva
│ └── gnv.node1.yml
├── lausanne
│ ├── qls.node1.yml
│ └── qls.node2.yml
├── node127.yml
└── zurich
├── zrh.node1.yml
├── zrh.node2.yml
└── zrh.node3.yml
**Examples**
``<saltclass_path>/nodes/lausanne/qls.node1.yml``
.. code-block:: yaml
environment: base
classes:
{% for class in ['default'] %}
- {{ class }}
{% endfor %}
- subsidiaries.{{ __grains__['id'].split('.')[0] }}
``<saltclass_path>/classes/default/init.yml``
.. code-block:: yaml
classes:
- default.users
- default.motd
states:
- openssh
pillars:
default:
network:
dns:
srv1: 192.168.0.1
srv2: 192.168.0.2
domain: example.com
ntp:
srv1: 192.168.10.10
srv2: 192.168.10.20
``<saltclass_path>/classes/subsidiaries/gnv.yml``
.. code-block:: yaml
pillars:
default:
network:
sub: Geneva
dns:
srv1: 10.20.0.1
srv2: 10.20.0.2
srv3: 192.168.1.1
domain: gnv.example.com
users:
adm1:
uid: 1210
gid: 1210
gecos: 'Super user admin1'
homedir: /srv/app/adm1
adm3:
uid: 1203
gid: 1203
gecos: 'Super user adm
Variable expansions:
Escaped variables are rendered as is - ``${test}``
Missing variables are rendered as is - ``${net:dns:srv2}``
.. code-block:: yaml
pillars:
app:
config:
dns:
srv1: ${default:network:dns:srv1}
srv2: ${net:dns:srv2}
uri: https://application.domain/call?\${test}
prod_parameters:
- p1
- p2
- p3
pkg:
- app-core
- app-backend
List override:
Not using ``^`` as the first entry will simply merge the lists
.. code-block:: yaml
pillars:
app:
pkg:
- ^
- app-frontend
**Known limitation**
Currently you can't have both a variable and an escaped variable in the same string as the escaped one will not be correctly rendered - '\${xx}' will stay as is instead of being rendered as '${xx}'
Newer PyWinRM Versions
----------------------

62
salt/pillar/saltclass.py Normal file
View file

@ -0,0 +1,62 @@
# -*- coding: utf-8 -*-
'''
SaltClass Pillar Module
.. code-block:: yaml
ext_pillar:
- saltclass:
- path: /srv/saltclass
'''
# import python libs
from __future__ import absolute_import
import salt.utils.saltclass as sc
import logging
log = logging.getLogger(__name__)
def __virtual__():
'''
This module has no external dependencies
'''
return True
def ext_pillar(minion_id, pillar, *args, **kwargs):
'''
Node definitions path will be retrieved from args - or set to default -
then added to 'salt_data' dict that is passed to the 'get_pillars' function.
'salt_data' dict is a convenient way to pass all the required datas to the function
It contains:
- __opts__
- __salt__
- __grains__
- __pillar__
- minion_id
- path
If successfull the function will return a pillar dict for minion_id
'''
# If path has not been set, make a default
for i in args:
if 'path' not in i:
path = '/srv/saltclass'
args[i]['path'] = path
log.warning('path variable unset, using default: {0}'.format(path))
else:
path = i['path']
# Create a dict that will contain our salt dicts to pass it to reclass
salt_data = {
'__opts__': __opts__,
'__salt__': __salt__,
'__grains__': __grains__,
'__pillar__': pillar,
'minion_id': minion_id,
'path': path
}
return sc.get_pillars(minion_id, salt_data)

69
salt/tops/saltclass.py Normal file
View file

@ -0,0 +1,69 @@
# -*- coding: utf-8 -*-
'''
SaltClass master_tops Module
.. code-block:: yaml
master_tops:
saltclass:
path: /srv/saltclass
'''
# import python libs
from __future__ import absolute_import
import logging
import salt.utils.saltclass as sc
log = logging.getLogger(__name__)
def __virtual__():
'''
Only run if properly configured
'''
if __opts__['master_tops'].get('saltclass'):
return True
return False
def top(**kwargs):
'''
Node definitions path will be retrieved from __opts__ - or set to default -
then added to 'salt_data' dict that is passed to the 'get_tops' function.
'salt_data' dict is a convenient way to pass all the required datas to the function
It contains:
- __opts__
- empty __salt__
- __grains__
- empty __pillar__
- minion_id
- path
If successfull the function will return a top dict for minion_id
'''
# If path has not been set, make a default
_opts = __opts__['master_tops']['saltclass']
if 'path' not in _opts:
path = '/srv/saltclass'
log.warning('path variable unset, using default: {0}'.format(path))
else:
path = _opts['path']
# Create a dict that will contain our salt objects
# to send to get_tops function
if 'id' not in kwargs['opts']:
log.warning('Minion id not found - Returning empty dict')
return {}
else:
minion_id = kwargs['opts']['id']
salt_data = {
'__opts__': kwargs['opts'],
'__salt__': {},
'__grains__': kwargs['grains'],
'__pillar__': {},
'minion_id': minion_id,
'path': path
}
return sc.get_tops(minion_id, salt_data)

296
salt/utils/saltclass.py Normal file
View file

@ -0,0 +1,296 @@
# -*- coding: utf-8 -*-
from __future__ import absolute_import
import os
import re
import logging
from salt.ext.six import iteritems
import yaml
from jinja2 import FileSystemLoader, Environment
log = logging.getLogger(__name__)
# Renders jinja from a template file
def render_jinja(_file, salt_data):
j_env = Environment(loader=FileSystemLoader(os.path.dirname(_file)))
j_env.globals.update({
'__opts__': salt_data['__opts__'],
'__salt__': salt_data['__salt__'],
'__grains__': salt_data['__grains__'],
'__pillar__': salt_data['__pillar__'],
'minion_id': salt_data['minion_id'],
})
j_render = j_env.get_template(os.path.basename(_file)).render()
return j_render
# Renders yaml from rendered jinja
def render_yaml(_file, salt_data):
return yaml.safe_load(render_jinja(_file, salt_data))
# Returns a dict from a class yaml definition
def get_class(_class, salt_data):
l_files = []
saltclass_path = salt_data['path']
straight = '{0}/classes/{1}.yml'.format(saltclass_path, _class)
sub_straight = '{0}/classes/{1}.yml'.format(saltclass_path,
_class.replace('.', '/'))
sub_init = '{0}/classes/{1}/init.yml'.format(saltclass_path,
_class.replace('.', '/'))
for root, dirs, files in os.walk('{0}/classes'.format(saltclass_path)):
for l_file in files:
l_files.append('{0}/{1}'.format(root, l_file))
if straight in l_files:
return render_yaml(straight, salt_data)
if sub_straight in l_files:
return render_yaml(sub_straight, salt_data)
if sub_init in l_files:
return render_yaml(sub_init, salt_data)
log.warning('{0}: Class definition not found'.format(_class))
return {}
# Return environment
def get_env_from_dict(exp_dict_list):
environment = ''
for s_class in exp_dict_list:
if 'environment' in s_class:
environment = s_class['environment']
return environment
# Merge dict b into a
def dict_merge(a, b, path=None):
if path is None:
path = []
for key in b:
if key in a:
if isinstance(a[key], list) and isinstance(b[key], list):
if b[key][0] == '^':
b[key].pop(0)
a[key] = b[key]
else:
a[key].extend(b[key])
elif isinstance(a[key], dict) and isinstance(b[key], dict):
dict_merge(a[key], b[key], path + [str(key)])
elif a[key] == b[key]:
pass
else:
a[key] = b[key]
else:
a[key] = b[key]
return a
# Recursive search and replace in a dict
def dict_search_and_replace(d, old, new, expanded):
for (k, v) in iteritems(d):
if isinstance(v, dict):
dict_search_and_replace(d[k], old, new, expanded)
if v == old:
d[k] = new
return d
# Retrieve original value from ${xx:yy:zz} to be expanded
def find_value_to_expand(x, v):
a = x
for i in v[2:-1].split(':'):
if i in a:
a = a.get(i)
else:
a = v
return a
return a
# Return a dict that contains expanded variables if found
def expand_variables(a, b, expanded, path=None):
if path is None:
b = a.copy()
path = []
for (k, v) in iteritems(a):
if isinstance(v, dict):
expand_variables(v, b, expanded, path + [str(k)])
else:
if isinstance(v, str):
vre = re.search(r'(^|.)\$\{.*?\}', v)
if vre:
re_v = vre.group(0)
if re_v.startswith('\\'):
v_new = v.replace(re_v, re_v.lstrip('\\'))
b = dict_search_and_replace(b, v, v_new, expanded)
expanded.append(k)
elif not re_v.startswith('$'):
v_expanded = find_value_to_expand(b, re_v[1:])
v_new = v.replace(re_v[1:], v_expanded)
b = dict_search_and_replace(b, v, v_new, expanded)
expanded.append(k)
else:
v_expanded = find_value_to_expand(b, re_v)
b = dict_search_and_replace(b, v, v_expanded, expanded)
expanded.append(k)
return b
def expand_classes_in_order(minion_dict,
salt_data,
seen_classes,
expanded_classes,
classes_to_expand):
# Get classes to expand from minion dictionnary
if not classes_to_expand and 'classes' in minion_dict:
classes_to_expand = minion_dict['classes']
# Now loop on list to recursively expand them
for klass in classes_to_expand:
if klass not in seen_classes:
seen_classes.append(klass)
expanded_classes[klass] = get_class(klass, salt_data)
# Fix corner case where class is loaded but doesn't contain anything
if expanded_classes[klass] is None:
expanded_classes[klass] = {}
# Now replace class element in classes_to_expand by expansion
if 'classes' in expanded_classes[klass]:
l_id = classes_to_expand.index(klass)
classes_to_expand[l_id:l_id] = expanded_classes[klass]['classes']
expand_classes_in_order(minion_dict,
salt_data,
seen_classes,
expanded_classes,
classes_to_expand)
else:
expand_classes_in_order(minion_dict,
salt_data,
seen_classes,
expanded_classes,
classes_to_expand)
# We may have duplicates here and we want to remove them
tmp = []
for t_element in classes_to_expand:
if t_element not in tmp:
tmp.append(t_element)
classes_to_expand = tmp
# Now that we've retrieved every class in order,
# let's return an ordered list of dicts
ord_expanded_classes = []
ord_expanded_states = []
for ord_klass in classes_to_expand:
ord_expanded_classes.append(expanded_classes[ord_klass])
# And be smart and sort out states list
# Address the corner case where states is empty in a class definition
if 'states' in expanded_classes[ord_klass] and expanded_classes[ord_klass]['states'] is None:
expanded_classes[ord_klass]['states'] = {}
if 'states' in expanded_classes[ord_klass]:
ord_expanded_states.extend(expanded_classes[ord_klass]['states'])
# Add our minion dict as final element but check if we have states to process
if 'states' in minion_dict and minion_dict['states'] is None:
minion_dict['states'] = []
if 'states' in minion_dict:
ord_expanded_states.extend(minion_dict['states'])
ord_expanded_classes.append(minion_dict)
return ord_expanded_classes, classes_to_expand, ord_expanded_states
def expanded_dict_from_minion(minion_id, salt_data):
_file = ''
saltclass_path = salt_data['path']
# Start
for root, dirs, files in os.walk('{0}/nodes'.format(saltclass_path)):
for minion_file in files:
if minion_file == '{0}.yml'.format(minion_id):
_file = os.path.join(root, minion_file)
# Load the minion_id definition if existing, else an exmpty dict
node_dict = {}
if _file:
node_dict[minion_id] = render_yaml(_file, salt_data)
else:
log.warning('{0}: Node definition not found'.format(minion_id))
node_dict[minion_id] = {}
# Get 2 ordered lists:
# expanded_classes: A list of all the dicts
# classes_list: List of all the classes
expanded_classes, classes_list, states_list = expand_classes_in_order(
node_dict[minion_id],
salt_data, [], {}, [])
# Here merge the pillars together
pillars_dict = {}
for exp_dict in expanded_classes:
if 'pillars' in exp_dict:
dict_merge(pillars_dict, exp_dict)
return expanded_classes, pillars_dict, classes_list, states_list
def get_pillars(minion_id, salt_data):
# Get 2 dicts and 2 lists
# expanded_classes: Full list of expanded dicts
# pillars_dict: dict containing merged pillars in order
# classes_list: All classes processed in order
# states_list: All states listed in order
(expanded_classes,
pillars_dict,
classes_list,
states_list) = expanded_dict_from_minion(minion_id, salt_data)
# Retrieve environment
environment = get_env_from_dict(expanded_classes)
# Expand ${} variables in merged dict
# pillars key shouldn't exist if we haven't found any minion_id ref
if 'pillars' in pillars_dict:
pillars_dict_expanded = expand_variables(pillars_dict['pillars'], {}, [])
else:
pillars_dict_expanded = expand_variables({}, {}, [])
# Build the final pillars dict
pillars_dict = {}
pillars_dict['__saltclass__'] = {}
pillars_dict['__saltclass__']['states'] = states_list
pillars_dict['__saltclass__']['classes'] = classes_list
pillars_dict['__saltclass__']['environment'] = environment
pillars_dict['__saltclass__']['nodename'] = minion_id
pillars_dict.update(pillars_dict_expanded)
return pillars_dict
def get_tops(minion_id, salt_data):
# Get 2 dicts and 2 lists
# expanded_classes: Full list of expanded dicts
# pillars_dict: dict containing merged pillars in order
# classes_list: All classes processed in order
# states_list: All states listed in order
(expanded_classes,
pillars_dict,
classes_list,
states_list) = expanded_dict_from_minion(minion_id, salt_data)
# Retrieve environment
environment = get_env_from_dict(expanded_classes)
# Build final top dict
tops_dict = {}
tops_dict[environment] = states_list
return tops_dict

View file

@ -0,0 +1,6 @@
classes:
- app.ssh.server
pillars:
sshd:
root_access: yes

View file

@ -0,0 +1,4 @@
pillars:
sshd:
root_access: no
ssh_port: 22

View file

@ -0,0 +1,17 @@
classes:
- default.users
- default.motd
states:
- openssh
pillars:
default:
network:
dns:
srv1: 192.168.0.1
srv2: 192.168.0.2
domain: example.com
ntp:
srv1: 192.168.10.10
srv2: 192.168.10.20

View file

@ -0,0 +1,3 @@
pillars:
motd:
text: "Welcome to {{ __grains__['id'] }} system located in ${default:network:sub}"

View file

@ -0,0 +1,16 @@
states:
- user_mgt
pillars:
default:
users:
adm1:
uid: 1201
gid: 1201
gecos: 'Super user admin1'
homedir: /home/adm1
adm2:
uid: 1202
gid: 1202
gecos: 'Super user admin2'
homedir: /home/adm2

View file

@ -0,0 +1,21 @@
states:
- app
pillars:
app:
config:
dns:
srv1: ${default:network:dns:srv1}
srv2: ${default:network:dns:srv2}
uri: https://application.domain/call?\${test}
prod_parameters:
- p1
- p2
- p3
pkg:
- app-core
- app-backend
# Safe minion_id matching
{% if minion_id == 'zrh.node3' %}
safe_pillar: '_only_ zrh.node3 will see this pillar and this cannot be overriden like grains'
{% endif %}

View file

@ -0,0 +1,7 @@
states:
- nginx_deployment
pillars:
nginx:
pkg:
- nginx

View file

@ -0,0 +1,7 @@
classes:
- roles.nginx
pillars:
nginx:
pkg:
- nginx-module

View file

@ -0,0 +1,20 @@
pillars:
default:
network:
sub: Geneva
dns:
srv1: 10.20.0.1
srv2: 10.20.0.2
srv3: 192.168.1.1
domain: gnv.example.com
users:
adm1:
uid: 1210
gid: 1210
gecos: 'Super user admin1'
homedir: /srv/app/adm1
adm3:
uid: 1203
gid: 1203
gecos: 'Super user admin3'
homedir: /home/adm3

View file

@ -0,0 +1,17 @@
classes:
- app.ssh.server
- roles.nginx.server
pillars:
default:
network:
sub: Lausanne
dns:
srv1: 10.10.0.1
domain: qls.example.com
users:
nginx_adm:
uid: 250
gid: 200
gecos: 'Nginx admin user'
homedir: /srv/www

View file

@ -0,0 +1,24 @@
classes:
- roles.app
# This should validate that we process a class only once
- app.borgbackup
# As this one should not be processed
# and would override in turn overrides from app.borgbackup
- app.ssh.server
pillars:
default:
network:
sub: Zurich
dns:
srv1: 10.30.0.1
srv2: 10.30.0.2
domain: zrh.example.com
ntp:
srv1: 10.0.0.127
users:
adm1:
uid: 250
gid: 250
gecos: 'Super user admin1'
homedir: /srv/app/1

View file

@ -0,0 +1,6 @@
environment: base
classes:
{% for class in ['default'] %}
- {{ class }}
{% endfor %}

View file

@ -0,0 +1,43 @@
# -*- coding: utf-8 -*-
# Import python libs
from __future__ import absolute_import
import os
# Import Salt Testing libs
from tests.support.mixins import LoaderModuleMockMixin
from tests.support.unit import TestCase, skipIf
from tests.support.mock import NO_MOCK, NO_MOCK_REASON
# Import Salt Libs
import salt.pillar.saltclass as saltclass
base_path = os.path.dirname(os.path.realpath(__file__))
fake_minion_id = 'fake_id'
fake_pillar = {}
fake_args = ({'path': '{0}/../../integration/files/saltclass/examples'.format(base_path)})
fake_opts = {}
fake_salt = {}
fake_grains = {}
@skipIf(NO_MOCK, NO_MOCK_REASON)
class SaltclassPillarTestCase(TestCase, LoaderModuleMockMixin):
'''
Tests for salt.pillar.saltclass
'''
def setup_loader_modules(self):
return {saltclass: {'__opts__': fake_opts,
'__salt__': fake_salt,
'__grains__': fake_grains
}}
def _runner(self, expected_ret):
full_ret = saltclass.ext_pillar(fake_minion_id, fake_pillar, fake_args)
parsed_ret = full_ret['__saltclass__']['classes']
self.assertListEqual(parsed_ret, expected_ret)
def test_succeeds(self):
ret = ['default.users', 'default.motd', 'default']
self._runner(ret)