saltstack-formulas/template-formula
1
0
Fork 0
mirror of https://github.com/saltstack-formulas/template-formula.git synced 2025-04-17 10:10:28 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

feat(map): update to v5 map.jinja

Browse source 
The v5 `map.jinja` is a generic and configurable system to load
configuration values, exposed as the `mapdata` variable, from
different places:

- YAML files and templates from the fileserver for non-secret data
- pillars or SDB are preferred for secret data
- grains or `config.get`

The `map.jinja` optional sources are configured with compound targeting like
syntax `[<TYPE>[:<OPTION>[:<DELIMITER>]]@]<KEY>` with the following
default ordered sources:

- `Y:G@osarch`: YAML file and Jinja template named after `osarch` grain
- `Y:G@os_family`: YAML file and Jinja template named after `os_family` grain
- `Y:G@os` YAML file and Jinja template named after `os` grain
- `Y:G@osfinger` YAML file and Jinja template named after `osfinger` grain
- `C@{{ tplroot ~ ':lookup' }}`: dict retrieved with `salt["config.get"]`
- `C@{{ tplroot }}`: dict retrieved with `salt["config.get"]`
- `Y:G@id`: YAML file and Jinja template named after `osarch` grain

This is done by two new libraries:

- `libmatchers.jinja` provides the `parse_matchers` macro to parse
  strings looking like compound matchers, for example `Y:G@osarch`

- `libmapstack.jinja` provides the `mapstack` macro to load
  configuration from different sources described by matchers

Post-processing of `mapdata` variable can be done in a `parameters/post-map.jinja`.

The v5 `map.jinja` is documented in `docs/map.jinja.rst`.

BREAKING CHANGE: `map.jinja` now exports a generic `mapdata` variable

BREAKING CHANGE: The per grain parameter values are now under `TEMPLATE/parameters/`
This commit is contained in:
Daniel Dehennin 2021-02-16 11:10:09 +01:00
parent 41d222e30c
commit 42e19322c9
39 changed files with 1488 additions and 232 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
TEMPLATE/_mapdata/init.sls
View file

@ -3,10 +3,10 @@
--- ---
{#- Get the `tplroot` from `tpldir` #} {#- Get the `tplroot` from `tpldir` #}
{%- set tplroot = tpldir.split("/")[0] %} {%- set tplroot = tpldir.split("/")[0] %}
{%- from tplroot ~ "/map.jinja" import TEMPLATE with context %} {%- from tplroot ~ "/map.jinja" import mapdata with context %}
{%- set _mapdata = { {%- set _mapdata = {
"values": TEMPLATE, "values": mapdata,
} %} } %}
{%- do salt["log.debug"]("### MAP.JINJA DUMP ###\n" ~ _mapdata | yaml(False)) %} {%- do salt["log.debug"]("### MAP.JINJA DUMP ###\n" ~ _mapdata | yaml(False)) %}

2
TEMPLATE/config/clean.sls
View file

@ -4,7 +4,7 @@
{#- Get the `tplroot` from `tpldir` #} {#- Get the `tplroot` from `tpldir` #}
{%- set tplroot = tpldir.split('/')[0] %} {%- set tplroot = tpldir.split('/')[0] %}
{%- set sls_service_clean = tplroot ~ '.service.clean' %} {%- set sls_service_clean = tplroot ~ '.service.clean' %}
{%- from tplroot ~ "/map.jinja" import TEMPLATE with context %} {%- from tplroot ~ "/map.jinja" import mapdata as TEMPLATE with context %}
include: include:
- {{ sls_service_clean }} - {{ sls_service_clean }}

2
TEMPLATE/config/file.sls
View file

@ -4,7 +4,7 @@
{#- Get the `tplroot` from `tpldir` #} {#- Get the `tplroot` from `tpldir` #}
{%- set tplroot = tpldir.split('/')[0] %} {%- set tplroot = tpldir.split('/')[0] %}
{%- set sls_package_install = tplroot ~ '.package.install' %} {%- set sls_package_install = tplroot ~ '.package.install' %}
{%- from tplroot ~ "/map.jinja" import TEMPLATE with context %} {%- from tplroot ~ "/map.jinja" import mapdata as TEMPLATE with context %}
{%- from tplroot ~ "/libtofs.jinja" import files_switch with context %} {%- from tplroot ~ "/libtofs.jinja" import files_switch with context %}
include: include:

310
TEMPLATE/libmapstack.jinja Normal file
View file

@ -0,0 +1,310 @@
{#- -*- coding: utf-8 -*- #}
{#- vim: ft=jinja #}
{#- Get the `tplroot` from `tpldir` #}
{%- set tplroot = tpldir.split("/")[0] %}
{%- from tplroot ~ "/libmatchers.jinja" import parse_matchers, query_map %}
{%- set _default_config_dirs = [
"parameters/",
tplroot ~ "/parameters"
] %}
{%- macro mapstack(
matchers,
defaults=None,
dirs=_default_config_dirs,
log_prefix="libmapstack: "
) %}
{#-
Load configuration in the order of `matchers` and merge
successively the values with `defaults`.
The `matchers` are processed using `libmatchers.jinja` to select
the configuration sources from where the values are loaded.
Parameters:
- `matchers`: list of matchers in the form
`[<TYPE>[:<OPTION>[:<DELIMITER>]]@]<QUERY>`
- `defaults`: dictionary of default values to start the merging,
they are considered built-ins. It must conform to the same
layout as the YAML files: a mandatory `values` key and two
optional `strategy` and `merge_lists` keys.
- `dirs`: list of directory where to look-up the configuration
file matching the matchers, by default a global `salt://parameters/`
and a per formula `salt://<tplroot>/parameters`
- `log_prefix`: prefix used in the log outputs, by default it is
`libmapstack: `
Example: On a Debian system with `roles=["nginx/server", "telegraf"]`
{%- set settings = mapstack(
matchers=[
"Y:G@os_family",
"I@" ~ tplroot,
"Y:C@roles",
],
dirs=["defaults", tplroot ~ "/parameters"],
)
| load_yaml %}
This will merge the values:
- starting with the default empty dictionary `{}` (no
`defaults` parameter)
- from the YAML files
- `salt://defaults/os_family/Debian.yaml`
- `salt://{{ tplroot }}/parameters/os_family/Debian.yaml`
- from the pillar `salt["pillar.get"](tplroot)`
- from the `nginx/server` YAML files:
- `salt://defaults/roles/nginx/server.yaml`
- `salt://{{ tplroot }}/parameters/roles/nginx/server.yaml`
- from the `telegraf` YAML files:
- `salt://defaults/roles/telegraf.yaml`
- `salt://{{ tplroot }}/parameters/roles/telegraf.yaml`
Each YAML file and the `defaults` parameters must conform to the
following layout:
- a mandatory `values` key to store the configuration values
- two optional keys to configure the use of `salt.slsutil.merge`
- an optional `strategy` key to configure the merging
strategy, for example `strategy: 'recurse'`, the default is
`smart`
- an optional `merge_lists` key to configure if lists should
be merged or overridden for the `recurse` and `overwrite`
strategies, for example `merge_lists: 'true'`
#}
{%- set stack = defaults | default({"values": {} }, boolean=True) %}
{#- Build configuration file names based on matchers #}
{%- set config_get_strategy = salt["config.get"](tplroot ~ ":strategy", None) %}
{%- set matchers = parse_matchers(
matchers,
config_get_strategy=config_get_strategy,
log_prefix=log_prefix
)
| load_yaml %}
{%- do salt["log.debug"](
log_prefix
~ "built-in configuration:\n"
~ {"values": defaults | traverse("values")}
| yaml(False)
) %}
{%- for param_dir in dirs %}
{%- for matcher in matchers %}
{#- `slsutil.merge` options from #}
{#- 1. the `value` #}
{#- 2. the `defaults` #}
{#- 3. the built-in #}
{%- set strategy = matcher.value
| traverse(
"strategy",
defaults
| traverse(
"strategy",
"smart"
)
) %}
{%- set merge_lists = matcher.value
| traverse(
"merge_lists",
defaults
| traverse(
"merge_lists",
False
)
)
| to_bool %}
{%- if matcher.type in query_map.keys() %}
{#- No value is an empty list, must be a dict for `stack.update` #}
{%- set normalized_value = matcher.value | default({}, boolean=True) %}
{#- Merge in `mapdata.<query>` instead of directly in `mapdata` #}
{%- set is_sub_key = matcher.option | default(False) == "SUB" %}
{%- if is_sub_key %}
{#- Merge values with `mapdata.<key>`, `<key>` and `<key>:lookup` are merged together #}
{%- set value = { matcher.query | regex_replace(":lookup$", ""): normalized_value } %}
{%- else %}
{%- set value = normalized_value %}
{%- endif %}
{%- do salt["log.debug"](
log_prefix
~ "merge "
~ "sub key " * is_sub_key
~ "'"
~ matcher.query
~ "' retrieved with '"
~ matcher.query_method
~ "', merge: strategy='"
~ strategy
~ "', lists='"
~ merge_lists
~ "':\n"
~ value
| yaml(False)
) %}
{%- do stack.update(
{
"values": salt["slsutil.merge"](
stack["values"],
value,
strategy=strategy,
merge_lists=merge_lists,
)
}
) %}
{%- else %}
{#- Load YAML file matching the grain/pillar/... #}
{#- Fallback to use the source name as a direct filename #}
{%- if matcher.value | length == 0 %}
{#- Mangle `matcher.value` to use it as literal path #}
{%- set query_parts = matcher.query.split("/") %}
{%- set yaml_dirname = query_parts[0:-1] | join("/") %}
{%- set yaml_names = query_parts[-1] %}
{%- else %}
{%- set yaml_dirname = matcher.query %}
{%- set yaml_names = matcher.value %}
{%- endif %}
{#- Some configuration return list #}
{%- if yaml_names is string %}
{%- set yaml_names = [yaml_names] %}
{%- endif %}
{#- Try to load a `.yaml.jinja` file for each `.yaml` file #}
{%- set all_yaml_names = [] %}
{%- for name in yaml_names %}
{%- set extension = name.rpartition(".")[2] %}
{%- if extension not in ["yaml", "jinja"] %}
{%- do all_yaml_names.extend([name ~ ".yaml", name ~ ".yaml.jinja"]) %}
{%- elif extension == "yaml" %}
{%- do all_yaml_names.extend([name, name ~ ".jinja"]) %}
{%- else %}
{%- do all_yaml_names.append(name) %}
{%- endif %}
{%- endfor %}
{#- `yaml_dirname` can be an empty string with literal path like `myconf.yaml` #}
{%- set yaml_dir = [
param_dir,
yaml_dirname
]
| select
| join("/") %}
{%- for yaml_name in all_yaml_names %}
{%- set yaml_filename = [
yaml_dir.rstrip("/"),
yaml_name
]
| select
| join("/") %}
{%- do salt["log.debug"](
log_prefix
~ "load configuration values from "
~ yaml_filename
) %}
{%- load_yaml as yaml_values %}
{%- include yaml_filename ignore missing %}
{%- endload %}
{%- if yaml_values %}
{%- do salt["log.debug"](
log_prefix
~ "loaded configuration values from "
~ yaml_filename
~ ":\n"
~ yaml_values
| yaml(False)
) %}
{#- `slsutil.merge` options from #}
{#- 1. the `value` #}
{#- 2. the `defaults` #}
{#- 3. the built-in #}
{%- set strategy = yaml_values
| traverse(
"strategy",
defaults
| traverse(
"strategy",
"smart"
)
) %}
{%- set merge_lists = yaml_values
| traverse(
"merge_lists",
defaults
| traverse(
"merge_lists",
False
)
)
| to_bool %}
{%- do stack.update(
{
"values": salt["slsutil.merge"](
stack["values"],
yaml_values
| traverse("values", {}),
strategy=strategy,
merge_lists=merge_lists,
)
}
) %}
{%- do salt["log.debug"](
log_prefix
~ "merged configuration values from "
~ yaml_filename
~ ", merge: strategy='"
~ strategy
~ "', merge_lists='"
~ merge_lists
~ "':\n"
~ {"values": stack["values"]}
| yaml(False)
) %}
{%- endif %}
{%- endfor %}
{%- endif %}
{%- endfor %}
{%- endfor %}
{%- do salt["log.debug"](
log_prefix
~ "final configuration values:\n"
~ {"values": stack["values"]}
| yaml(False)
) %}
{#- Output stack as YAML, caller should use with something like #}
{#- `{%- set config = mapstack(matchers=["foo"]) | load_yaml %}` #}
{{ stack | yaml }}
{%- endmacro %}

222
TEMPLATE/libmatchers.jinja Normal file
View file

@ -0,0 +1,222 @@
{#- -*- coding: utf-8 -*- #}
{#- vim: ft=jinja #}
{#- Get the `tplroot` from `tpldir` #}
{%- set tplroot = tpldir.split("/")[0] %}
{%- from tplroot ~ "/libsaltcli.jinja" import cli %}
{%- set query_map = {
"C": "config.get",
"G": "grains.get",
"I": "pillar.get",
} %}
{#- When no part before `@` is provided: #}
{#- - define a filename path, noted `F` #}
{#- - use `salt["config.get"]`, noted `C` #}
{#- - use colon `:` delimiter for querying #}
{%- set _defaults = {
"type": "F",
"query_type": "C",
"query_delimiter": ":"
} %}
{%- macro parse_matchers(
matchers,
config_get_strategy=None,
log_prefix="libmatchers: "
) %}
{#- matcher format is `[<TYPE>[:<OPTION>[:DELIMITER]]@]<KEY>` #}
{#- each matcher has a type: #}
{#- - `F` to build a file name (the default when no type is set) #}
{#- - `C` to lookup values with `config.get` #}
{#- - `G` to lookup values with `grains.get` #}
{#- - `I` to lookup values with `pillar.get` #}
{#- The `FILE` type option can define query type to build the file name: #}
{#- - `C` for query with `config.get` (the default when to query type is set) #}
{#- - `G` for query with `grains.get` #}
{#- - `I` for query with `pillar.get` #}
{#- With `DELIMITER`, you can choose a different delimiter when doing queries #}
{%- set parsed_matchers = [] %}
{%- for matcher in matchers %}
{%- do salt["log.debug"](
log_prefix
~ "process matcher: '"
~ matcher
~ "'"
) %}
{%- set parsed = {} %}
{%- set matcher_parts = matcher.split('@') %}
{%- if matcher_parts | length == 1 %}
{#- By default we load YAML files for config looked up by `config.get` #}
{%- do parsed.update(
{
"type": _defaults["type"],
"option": None,
"query_method": query_map[_defaults["query_type"]],
"query_delimiter": _defaults["query_delimiter"],
"query": matcher
}
) %}
{%- do salt["log.debug"](
log_prefix
~ "use built-in defaults for matcher:\n"
~ parsed
| yaml(False)
| indent(4, True)
) %}
{%- else %}
{%- do salt["log.debug"](
log_prefix
~ "parse matcher: '"
~ matcher
~ "'"
) %}
{%- set metadatas = matcher_parts[0].split(":") %}
{%- do parsed.update(
{
"query": matcher_parts[1]
}
) %}
{%- if metadatas | length == 1 %}
{%- do parsed.update(
{
"type": metadatas[0],
"option": "C",
"query_delimiter": ":"
}
) %}
{%- do salt["log.debug"](
log_prefix
~ "parse as 1 metadata matcher:\n"
~ parsed
| yaml(False)
| indent(4, True)
) %}
{%- elif metadatas | length == 2 %}
{%- do parsed.update(
{
"type": metadatas[0],
"option": metadatas[1],
"query_delimiter": ":"
}
) %}
{%- do salt["log.debug"](
log_prefix
~ "parse as 2 metadata matcher:\n"
~ parsed
| yaml(False)
| indent(4, True)
) %}
{%- elif metadatas | length == 3 %}
{%- do parsed.update(
{
"type": metadatas[0],
"option": metadatas[1],
"query_delimiter": metadatas[2] | default(":", boolean=True)
}
) %}
{%- do salt["log.debug"](
log_prefix
~ "parse as 3 metadata matcher:\n"
~ parsed
| yaml(False)
| indent(4, True)
) %}
{%- elif metadatas | length == 4 %}
{#- The delimiter is `:` #}
{%- do parsed.update(
{
"type": metadatas[0],
"option": metadatas[1],
"query_delimiter": ":"
}
) %}
{%- do salt["log.debug"](
log_prefix
~ "parse as 4 metadata matcher:\n"
~ parsed
| yaml(False)
| indent(4, True)
) %}
{%- endif %}
{%- endif %}
{#- The `<OPTION>` has different meaning based on type #}
{%- if query_map.get(parsed.type, False) %}
{%- do parsed.update(
{
"query_method": query_map[parsed.type]
}
) %}
{%- else %}
{%- do parsed.update(
{
"query_method": query_map[
parsed.option
| default("C", boolean=True)
]
}
) %}
{%- endif %}
{#- Add `merge:` option to `salt["config.get"]` if configured #}
{%- if cli in ["minion", "local"] and parsed.query_method == "config.get" and config_get_strategy %}
{%- set query_opts = {
"merge": config_get_strategy,
"delimiter": parsed.query_delimiter,
} %}
{%- set query_opts_msg = (
", delimiter='"
~ parsed.query_delimiter
~ "', merge: strategy='"
~ config_get_strategy
~ "'"
) %}
{%- else %}
{%- if cli not in ["minion", "local"] %}
{%- do salt["log.error"](
log_prefix
~ "the 'delimiter' and 'merge' options of 'config.get' are skipped when the salt command type is '"
~ cli
~ "'"
) %}
{%- endif %}
{%- set query_opts = {} %}
{%- set query_opts_msg = "" %}
{%- endif %}
{%- do salt["log.debug"](
log_prefix
~ "lookup '"
~ parsed.query
~ "' with '"
~ parsed.query_method
~ "'"
~ query_opts_msg
) %}
{%- set values = salt[parsed.query_method](
parsed.query,
default=[],
**query_opts
) %}
{%- do parsed.update(
{
"value": values
}
) %}
{%- do parsed_matchers.append(parsed) %}
{%- endfor %}
{%- do salt["log.debug"](
log_prefix
~ "parsed matchers:\n"
~ parsed_matchers
| yaml(False)
| indent(4, True)
) %}
{{ parsed_matchers | yaml }}
{%- endmacro %}

102
TEMPLATE/map.jinja
View file

@ -2,57 +2,65 @@
# vim: ft=jinja # vim: ft=jinja
{#- Get the `tplroot` from `tpldir` #} {#- Get the `tplroot` from `tpldir` #}
{%- set tplroot = tpldir.split('/')[0] %} {%- set tplroot = tpldir.split("/")[0] %}
{#- Start imports as #} {%- from tplroot ~ "/libmapstack.jinja" import mapstack %}
{%- import_yaml tplroot ~ "/defaults.yaml" as default_settings %}
{%- import_yaml tplroot ~ "/osarchmap.yaml" as osarchmap %}
{%- import_yaml tplroot ~ "/osfamilymap.yaml" as osfamilymap %}
{%- import_yaml tplroot ~ "/osmap.yaml" as osmap %}
{%- import_yaml tplroot ~ "/osfingermap.yaml" as osfingermap %}
{#- Retrieve the config dict only once #} {#- Where to lookup parameters source files #}
{%- set _config = salt['config.get'](tplroot, default={}) %} {%- set formula_param_dir = tplroot ~ "/parameters" %}
{%- set defaults = salt['grains.filter_by']( {#- List of sources to lookup for parameters #}
default_settings, {#- Fallback to previously used grains plus minion `id` #}
default=tplroot, {%- set map_sources = [
merge=salt['grains.filter_by']( "Y:G@osarch",
osarchmap, "Y:G@os_family",
grain='osarch', "Y:G@os",
merge=salt['grains.filter_by']( "Y:G@osfinger",
osfamilymap, "C@" ~ tplroot ~ ":lookup",
grain='os_family', "C@" ~ tplroot,
merge=salt['grains.filter_by']( "Y:G@id",
osmap, ] %}
grain='os',
merge=salt['grains.filter_by']( {%- set _map_settings = mapstack(
osfingermap, matchers=["map_jinja.yaml"],
grain='osfinger', defaults={
merge=salt['grains.filter_by']( "values": {"sources": map_sources}
_config, },
default='lookup' log_prefix="map.jinja configuration: ",
)
)
)
)
)
) )
%} | load_yaml %}
{%- set config = salt['grains.filter_by']( {%- set map_sources = _map_settings | traverse("values:sources") %}
{'defaults': defaults}, {%- do salt["log.debug"](
default='defaults', "map.jinja: load parameters from sources:\n"
merge=_config ~ map_sources
| yaml(False)
) %}
{#- Load formula parameters values #}
{%- set _formula_matchers = ["defaults.yaml"] + map_sources %}
{%- set _formula_settings = mapstack(
matchers=_formula_matchers,
dirs=[formula_param_dir],
defaults={
"values": {},
"merge_strategy": salt["config.get"](tplroot ~ ":strategy", None),
"merge_lists": salt["config.get"](tplroot ~ ":merge_lists", False),
},
log_prefix="map.jinja: ",
) )
%} | load_yaml %}
{#- <REMOVEME #} {#- Make sure to track `map.jinja` configuration with `_mapdata` #}
{#- Change **TEMPLATE** to match with your formula's name and then remove this line #} {%- do _formula_settings["values"].update(
{#- REMOVEME> #} {
{%- set TEMPLATE = config %} "map_jinja": _map_settings["values"]
}
) %}
{#- Post-processing for specific non-YAML customisations #} {%- do salt["log.debug"]("map.jinja: save parameters in variable 'mapdata'") %}
{%- if grains.os == 'MacOS' %} {%- set mapdata = _formula_settings["values"] %}
{%- set macos_group = salt['cmd.run']("stat -f '%Sg' /dev/console") %}
{%- do TEMPLATE.update({'rootgroup': macos_group}) %} {#- Per formula post-processing of `mapdata` if it exists #}
{%- endif %} {%- do salt["log.debug"]("map.jinja: post-processing of 'mapdata'") %}
{%- include tplroot ~ "/post-map.jinja" ignore missing %}

35
TEMPLATE/osarchmap.yaml
View file

@ -1,35 +0,0 @@
# -*- coding: utf-8 -*-
# vim: ft=yaml
#
# Setup variables using grains['osarch'] based logic.
# You just need to add the key:values for an `osarch` that differ
# from `defaults.yaml`.
# Only add an `osarch` which is/will be supported by the formula.
#
# If you do not need to provide defaults via the `osarch` grain,
# you will need to provide at least an empty dict in this file, e.g.
# osarch: {}
---
amd64:
arch: amd64
x86_64:
arch: amd64
386:
arch: 386
arm64:
arch: arm64
armv6l:
arch: armv6l
armv7l:
arch: armv7l
ppc64le:
arch: ppc64le
s390x:
arch: s390x

49
TEMPLATE/osfamilymap.yaml
View file

@ -1,49 +0,0 @@
# -*- coding: utf-8 -*-
# vim: ft=yaml
#
# Setup variables using grains['os_family'] based logic.
# You just need to add the key:values for an `os_family` that differ
# from `defaults.yaml` + `osarch.yaml`.
# Only add an `os_family` which is/will be supported by the formula.
#
# If you do not need to provide defaults via the `os_family` grain,
# you will need to provide at least an empty dict in this file, e.g.
# osfamilymap: {}
---
Debian:
pkg:
name: TEMPLATE-debian
config: /etc/TEMPLATE.d/custom.conf
RedHat:
pkg:
name: TEMPLATE-redhat
config: /etc/TEMPLATE.conf
Suse:
pkg:
name: TEMPLATE-suse
Gentoo:
pkg:
name: TEMPLATE-gentoo
Arch:
pkg:
name: TEMPLATE-arch
service:
name: service-arch
Alpine: {}
FreeBSD:
rootgroup: wheel
OpenBSD:
rootgroup: wheel
Solaris: {}
Windows: {}
MacOS: {}

49
TEMPLATE/osfingermap.yaml
View file

@ -1,49 +0,0 @@
# -*- coding: utf-8 -*-
# vim: ft=yaml
#
# Setup variables using grains['osfinger'] based logic.
# You just need to add the key:values for an `osfinger` that differ
# from `defaults.yaml` + `osarch.yaml` + `os_family.yaml` + `osmap.yaml`.
# Only add an `osfinger` which is/will be supported by the formula.
#
# If you do not need to provide defaults via the `os_finger` grain,
# you will need to provide at least an empty dict in this file, e.g.
# osfingermap: {}
---
# os: Debian
Debian-10: {}
Debian-9: {}
Debian-8: {}
# os: Ubuntu
Ubuntu-18.04:
config: /etc/TEMPLATE.d/custom-ubuntu-18.04.conf
Ubuntu-16.04: {}
# os: Fedora
Fedora-31: {}
Fedora-30: {}
# os: CentOS
CentOS Linux-8: {}
CentOS Linux-7: {}
CentOS-6:
pkg:
name: TEMPLATE-centos-6
config: /etc/TEMPLATE.d/custom-centos-6.conf
# os: Amazon
Amazon Linux-2: {}
Amazon Linux AMI-2018: {}
# os: SUSE
Leap-15: {}
# os: FreeBSD
FreeBSD-12: {}
# os: Windows
Windows-8.1: {}
# os: Gentoo
Gentoo-2: {}

40
TEMPLATE/osmap.yaml
View file

@ -1,40 +0,0 @@
# -*- coding: utf-8 -*-
# vim: ft=yaml
#
# Setup variables using grains['os'] based logic.
# You just need to add the key:values for an `os` that differ
# from `defaults.yaml` + `osarch.yaml` + `os_family.yaml`.
# Only add an `os` which is/will be supported by the formula.
#
# If you do not need to provide defaults via the `os` grain,
# you will need to provide at least an empty dict in this file, e.g.
# osmap: {}
---
# os_family: Debian
Ubuntu:
pkg:
name: TEMPLATE-ubuntu
config: /etc/TEMPLATE.d/custom-ubuntu.conf
Raspbian: {}
# os_family: RedHat
Fedora:
pkg:
name: TEMPLATE-fedora
service:
name: service-fedora
CentOS: {}
Amazon: {}
# os_family: Suse
SUSE: {}
openSUSE: {}
# os_family: Gentoo
Funtoo: {}
# os_family: Arch
Manjaro: {}
# os_family: Solaris
SmartOS: {}

2
TEMPLATE/package/clean.sls
View file

@ -4,7 +4,7 @@
{#- Get the `tplroot` from `tpldir` #} {#- Get the `tplroot` from `tpldir` #}
{%- set tplroot = tpldir.split('/')[0] %} {%- set tplroot = tpldir.split('/')[0] %}
{%- set sls_config_clean = tplroot ~ '.config.clean' %} {%- set sls_config_clean = tplroot ~ '.config.clean' %}
{%- from tplroot ~ "/map.jinja" import TEMPLATE with context %} {%- from tplroot ~ "/map.jinja" import mapdata as TEMPLATE with context %}
include: include:
- {{ sls_config_clean }} - {{ sls_config_clean }}

2
TEMPLATE/package/install.sls
View file

@ -3,7 +3,7 @@
{#- Get the `tplroot` from `tpldir` #} {#- Get the `tplroot` from `tpldir` #}
{%- set tplroot = tpldir.split('/')[0] %} {%- set tplroot = tpldir.split('/')[0] %}
{%- from tplroot ~ "/map.jinja" import TEMPLATE with context %} {%- from tplroot ~ "/map.jinja" import mapdata as TEMPLATE with context %}
TEMPLATE-package-install-pkg-installed: TEMPLATE-package-install-pkg-installed:
pkg.installed: pkg.installed:

5
TEMPLATE/defaults.yaml → TEMPLATE/parameters/defaults.yaml
View file

@ -1,7 +1,9 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
# vim: ft=yaml # vim: ft=yaml
#
# Set default values.
--- ---
TEMPLATE: values:
pkg: pkg:
name: TEMPLATE name: TEMPLATE
rootgroup: root rootgroup: root
@ -13,3 +15,4 @@ TEMPLATE:
# Just here for testing # Just here for testing
added_in_defaults: defaults_value added_in_defaults: defaults_value
winner: defaults winner: defaults
...

19
TEMPLATE/parameters/os/Fedora.yaml Normal file
View file

@ -0,0 +1,19 @@
# -*- coding: utf-8 -*-
# vim: ft=yaml
#
# Set values specific to:
# salt['config.get']('os') == Fedora.
#
# You just need to add the key:values for this `os` that differ
# from `defaults.yaml` + `<osarch>.yaml` + `<os_family>.yaml`.
#
# If you do not need to provide defaults via the `os` config,
# you can remove this file or provide at least an empty dict, e.g.
# values: {}
---
values:
pkg:
name: TEMPLATE-fedora
service:
name: service-fedora
...

20
TEMPLATE/parameters/os/MacOS.yaml.jinja Normal file
View file

@ -0,0 +1,20 @@
{#- -*- coding: utf-8 -*- #}
{#- vim: ft=jinja #}
{#-
Set values specific to:
salt['config.get']('os') == MacOS.
You just need to add the key:values for this `os` that differ
from `defaults.yaml` + `<osarch>.yaml` + `<os_family>.yaml`.
This jinja2 file must return a valid `map.jinja` YAML.
If you do not need to provide calculated values via the `os`
config, you can remove this file or provide at least an empty
dict, e.g.
values: {}
#}
---
values:
rootgroup: {{ salt['cmd.run']("stat -f '%Sg' /dev/console") }}
...

18
TEMPLATE/parameters/os/Ubuntu.yaml Normal file
View file

@ -0,0 +1,18 @@
# -*- coding: utf-8 -*-
# vim: ft=yaml
#
# Set values specific to:
# salt['config.get']('os') == Ubuntu.
#
# You just need to add the key:values for this `os` that differ
# from `defaults.yaml` + `<osarch>.yaml` + `<os_family>.yaml`.
#
# If you do not need to provide defaults via the `os` config,
# you can remove this file or provide at least an empty dict, e.g.
# values: {}
---
values:
pkg:
name: TEMPLATE-ubuntu
config: /etc/TEMPLATE.d/custom-ubuntu.conf
...

19
TEMPLATE/parameters/os_family/Arch.yaml Normal file
View file

@ -0,0 +1,19 @@
# -*- coding: utf-8 -*-
# vim: ft=yaml
#
# Set values specific to:
# salt['config.get']('os_family') == Arch.
#
# You just need to add the key:values for this `os_family` that differ
# from `defaults.yaml` + `<osarch>.yaml`.
#
# If you do not need to provide defaults via the `os_family` config,
# you can remove this file or provide at least an empty dict, e.g.
# values: {}
---
values:
pkg:
name: TEMPLATE-arch
service:
name: service-arch
...

18
TEMPLATE/parameters/os_family/Debian.yaml Normal file
View file

@ -0,0 +1,18 @@
# -*- coding: utf-8 -*-
# vim: ft=yaml
#
# Set values specific to:
# salt['config.get']('os_family') == Debian.
#
# You just need to add the key:values for this `os_family` that differ
# from `defaults.yaml` + `<osarch>.yaml`.
#
# If you do not need to provide defaults via the `os_family` config,
# you can remove this file or provide at least an empty dict, e.g.
# values: {}
---
values:
pkg:
name: TEMPLATE-debian
config: /etc/TEMPLATE.d/custom.conf
...

16
TEMPLATE/parameters/os_family/FreeBSD.yaml Normal file
View file

@ -0,0 +1,16 @@
# -*- coding: utf-8 -*-
# vim: ft=yaml
#
# Set values specific to:
# salt['config.get']('os_family') == FreeBSD.
#
# You just need to add the key:values for this `os_family` that differ
# from `defaults.yaml` + `<osarch>.yaml`.
#
# If you do not need to provide defaults via the `os_family` config,
# you can remove this file or provide at least an empty dict, e.g.
# values: {}
---
values:
rootgroup: wheel
...

17
TEMPLATE/parameters/os_family/Gentoo.yaml Normal file
View file

@ -0,0 +1,17 @@
# -*- coding: utf-8 -*-
# vim: ft=yaml
#
# Set values specific to:
# salt['config.get']('os_family') == Gentoo.
#
# You just need to add the key:values for this `os_family` that differ
# from `defaults.yaml` + `<osarch>.yaml`.
#
# If you do not need to provide defaults via the `os_family` config,
# you can remove this file or provide at least an empty dict, e.g.
# values: {}
---
values:
pkg:
name: TEMPLATE-gentoo
...

16
TEMPLATE/parameters/os_family/OpenBSD.yaml Normal file
View file

@ -0,0 +1,16 @@
# -*- coding: utf-8 -*-
# vim: ft=yaml
#
# Set values specific to:
# salt['config.get']('os_family') == OpenBSD.
#
# You just need to add the key:values for this `os_family` that differ
# from `defaults.yaml` + `<osarch>.yaml`.
#
# If you do not need to provide defaults via the `os_family` config,
# you can remove this file or provide at least an empty dict, e.g.
# values: {}
---
values:
rootgroup: wheel
...

18
TEMPLATE/parameters/os_family/RedHat.yaml Normal file
View file

@ -0,0 +1,18 @@
# -*- coding: utf-8 -*-
# vim: ft=yaml
#
# Set values specific to:
# salt['config.get']('os_family') == RedHat.
#
# You just need to add the key:values for this `os_family` that differ
# from `defaults.yaml` + `<osarch>.yaml`.
#
# If you do not need to provide defaults via the `os_family` config,
# you can remove this file or provide at least an empty dict, e.g.
# values: {}
---
values:
pkg:
name: TEMPLATE-redhat
config: /etc/TEMPLATE.conf
...

17
TEMPLATE/parameters/os_family/Suse.yaml Normal file
View file

@ -0,0 +1,17 @@
# -*- coding: utf-8 -*-
# vim: ft=yaml
#
# Set values specific to:
# salt['config.get']('os_family') == Suse.
#
# You just need to add the key:values for this `os_family` that differ
# from `defaults.yaml` + `<osarch>.yaml`.
#
# If you do not need to provide defaults via the `os_family` config,
# you can remove this file or provide at least an empty dict, e.g.
# values: {}
---
values:
pkg:
name: TEMPLATE-suse
...

16
TEMPLATE/parameters/osarch/386.yaml Normal file
View file

@ -0,0 +1,16 @@
# -*- coding: utf-8 -*-
# vim: ft=yaml
#
# Set values specific to:
# salt['config.get']('osarch') == 386.
#
# You just need to add the key:values for this `osarch` that differ
# from `defaults.yaml`.
#
# If you do not need to provide defaults via the `osarch` config,
# you can remove this file or provide at least an empty dict, e.g.
# values: {}
---
values:
arch: 386
...

16
TEMPLATE/parameters/osarch/amd64.yaml Normal file
View file

@ -0,0 +1,16 @@
# -*- coding: utf-8 -*-
# vim: ft=yaml
#
# Set values specific to:
# salt['config.get']('osarch') == amd64.
#
# You just need to add the key:values for this `osarch` that differ
# from `defaults.yaml`.
#
# If you do not need to provide defaults via the `osarch` config,
# you can remove this file or provide at least an empty dict, e.g.
# values: {}
---
values:
arch: amd64
...

16
TEMPLATE/parameters/osarch/arm64.yaml Normal file
View file

@ -0,0 +1,16 @@
# -*- coding: utf-8 -*-
# vim: ft=yaml
#
# Set values specific to:
# salt['config.get']('osarch') == arm64.
#
# You just need to add the key:values for this `osarch` that differ
# from `defaults.yaml`.
#
# If you do not need to provide defaults via the `osarch` config,
# you can remove this file or provide at least an empty dict, e.g.
# values: {}
---
values:
arch: arm64
...

16
TEMPLATE/parameters/osarch/armv6l.yaml Normal file
View file

@ -0,0 +1,16 @@
# -*- coding: utf-8 -*-
# vim: ft=yaml
#
# Set values specific to:
# salt['config.get']('osarch') == armv6l.
#
# You just need to add the key:values for this `osarch` that differ
# from `defaults.yaml`.
#
# If you do not need to provide defaults via the `osarch` config,
# you can remove this file or provide at least an empty dict, e.g.
# values: {}
---
values:
arch: armv6l
...

16
TEMPLATE/parameters/osarch/armv7l.yaml Normal file
View file

@ -0,0 +1,16 @@
# -*- coding: utf-8 -*-
# vim: ft=yaml
#
# Set values specific to:
# salt['config.get']('osarch') == armv7l.
#
# You just need to add the key:values for this `osarch` that differ
# from `defaults.yaml`.
#
# If you do not need to provide defaults via the `osarch` config,
# you can remove this file or provide at least an empty dict, e.g.
# values: {}
---
values:
arch: armv7l
...

16
TEMPLATE/parameters/osarch/ppc64le.yaml Normal file
View file

@ -0,0 +1,16 @@
# -*- coding: utf-8 -*-
# vim: ft=yaml
#
# Set values specific to:
# salt['config.get']('osarch') == ppc64le.
#
# You just need to add the key:values for this `osarch` that differ
# from `defaults.yaml`.
#
# If you do not need to provide defaults via the `osarch` config,
# you can remove this file or provide at least an empty dict, e.g.
# values: {}
---
values:
arch: ppc64le
...

16
TEMPLATE/parameters/osarch/s390x.yaml Normal file
View file

@ -0,0 +1,16 @@
# -*- coding: utf-8 -*-
# vim: ft=yaml
#
# Set values specific to:
# salt['config.get']('osarch') == s390x.
#
# You just need to add the key:values for this `osarch` that differ
# from `defaults.yaml`.
#
# If you do not need to provide defaults via the `osarch` config,
# you can remove this file or provide at least an empty dict, e.g.
# values: {}
---
values:
arch: s390x
...

16
TEMPLATE/parameters/osarch/x86_64.yaml Normal file
View file

@ -0,0 +1,16 @@
# -*- coding: utf-8 -*-
# vim: ft=yaml
#
# Set values specific to:
# salt['config.get']('osarch') == x86_64.
#
# You just need to add the key:values for this `osarch` that differ
# from `defaults.yaml`.
#
# If you do not need to provide defaults via the `osarch` config,
# you can remove this file or provide at least an empty dict, e.g.
# values: {}
---
values:
arch: amd64
...

18
TEMPLATE/parameters/osfinger/CentOS-6.yaml Normal file
View file

@ -0,0 +1,18 @@
# -*- coding: utf-8 -*-
# vim: ft=yaml
#
# Set values specific to:
# salt['config.get']('osfinger') == CentOS-6.
#
# You just need to add the key:values for this `osfinger` that differ
# from `defaults.yaml`.
#
# If you do not need to provide defaults via the `osfinger` config,
# you can remove this file or provide at least an empty dict, e.g.
# values: {}
---
values:
pkg:
name: TEMPLATE-centos-6
config: /etc/TEMPLATE.d/custom-centos-6.conf
...

16
TEMPLATE/parameters/osfinger/Ubuntu-18.04.yaml Normal file
View file

@ -0,0 +1,16 @@
# -*- coding: utf-8 -*-
# vim: ft=yaml
#
# Set values specific to:
# salt['config.get']('osfinger') == Ubuntu-18.04.
#
# You just need to add the key:values for this `osfinger` that differ
# from `defaults.yaml`.
#
# If you do not need to provide defaults via the `osfinger` config,
# you can remove this file or provide at least an empty dict, e.g.
# values: {}
---
values:
config: /etc/TEMPLATE.d/custom-ubuntu-18.04.conf
...

2
TEMPLATE/service/clean.sls
View file

@ -3,7 +3,7 @@
{#- Get the `tplroot` from `tpldir` #} {#- Get the `tplroot` from `tpldir` #}
{%- set tplroot = tpldir.split('/')[0] %} {%- set tplroot = tpldir.split('/')[0] %}
{%- from tplroot ~ "/map.jinja" import TEMPLATE with context %} {%- from tplroot ~ "/map.jinja" import mapdata as TEMPLATE with context %}
TEMPLATE-service-clean-service-dead: TEMPLATE-service-clean-service-dead:
service.dead: service.dead:

2
TEMPLATE/service/running.sls
View file

@ -4,7 +4,7 @@
{#- Get the `tplroot` from `tpldir` #} {#- Get the `tplroot` from `tpldir` #}
{%- set tplroot = tpldir.split('/')[0] %} {%- set tplroot = tpldir.split('/')[0] %}
{%- set sls_config_file = tplroot ~ '.config.file' %} {%- set sls_config_file = tplroot ~ '.config.file' %}
{%- from tplroot ~ "/map.jinja" import TEMPLATE with context %} {%- from tplroot ~ "/map.jinja" import mapdata as TEMPLATE with context %}
include: include:
- {{ sls_config_file }} - {{ sls_config_file }}

2
TEMPLATE/subcomponent/config/clean.sls
View file

@ -4,7 +4,7 @@
{#- Get the `tplroot` from `tpldir` #} {#- Get the `tplroot` from `tpldir` #}
{%- set tplroot = tpldir.split('/')[0] %} {%- set tplroot = tpldir.split('/')[0] %}
{%- set sls_service_clean = tplroot ~ '.service.clean' %} {%- set sls_service_clean = tplroot ~ '.service.clean' %}
{%- from tplroot ~ "/map.jinja" import TEMPLATE with context %} {%- from tplroot ~ "/map.jinja" import mapdata as TEMPLATE with context %}
include: include:
- {{ sls_service_clean }} - {{ sls_service_clean }}

2
TEMPLATE/subcomponent/config/file.sls
View file

@ -4,7 +4,7 @@
{#- Get the `tplroot` from `tpldir` #} {#- Get the `tplroot` from `tpldir` #}
{%- set tplroot = tpldir.split('/')[0] %} {%- set tplroot = tpldir.split('/')[0] %}
{%- set sls_config_file = tplroot ~ '.config.file' %} {%- set sls_config_file = tplroot ~ '.config.file' %}
{%- from tplroot ~ "/map.jinja" import TEMPLATE with context %} {%- from tplroot ~ "/map.jinja" import mapdata as TEMPLATE with context %}
{%- from tplroot ~ "/libtofs.jinja" import files_switch with context %} {%- from tplroot ~ "/libtofs.jinja" import files_switch with context %}
include: include:

6
docs/README.rst
View file

@ -38,7 +38,11 @@ which contains the currently released version. This formula is versioned accordi
See `Formula Versioning Section <https://docs.saltstack.com/en/latest/topics/development/conventions/formulas.html#versioning>`_ for more details. See `Formula Versioning Section <https://docs.saltstack.com/en/latest/topics/development/conventions/formulas.html#versioning>`_ for more details.
If you need (non-default) configuration, please pay attention to the ``pillar.example`` file and/or `Special notes`_ section. If you need (non-default) configuration, please refer to:
- `how to configure the formula with map.jinja <map.jinja.rst>`_
- the ``pillar.example`` file
- the `Special notes`_ section
Contributing to this repo Contributing to this repo
------------------------- -------------------------

542
docs/map.jinja.rst Normal file
View file

@ -0,0 +1,542 @@
.. _map.jinja:
``map.jinja``: gather formula configuration values
==================================================
The `documentation`_ explains the use of a ``map.jinja`` to gather parameters values for a formula.
As `pillars`_ are rendered on the Salt master for every minion, this increases the load on the master as the pillar values and the number of minions grows.
As a good practice, you should:
- store non-secret data in YAML files distributed by the `fileserver`_
- store secret data in:
- `pillars`_ (and look for the use of something like `pillar.vault`_)
- `SDB`_ (and look for the use of something like `sdb.vault`_)
Current best practice is to let ``map.jinja`` handle parameters from all sources, to minimise the use of pillars, grains or configuration from ``sls`` files and templates directly.
.. contents:: **Table of Contents**
For formula users
-----------------
Quick start: configure per role and per DNS domain name values
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
We will see a quick setup to configure the ``TEMPLATE`` formula for different DNS domain names and several roles.
For this example, I'll define 2 kinds of `fileserver`_ sources:
1. formulas git repositories with hard-coded version reference to avoid breaking my setup randomly at upstream update. they are the last sources where files are looked up
2. parameters of the formulas in the file backend `roots`_
Configure the fileserver backends
`````````````````````````````````
I configure the `fileserver`_ backends to serve:
1. files from `roots`_ first
2. `gitfs`_ repositories last
Create the file ``/etc/salt/master.d/fileserver.conf`` and restart the ``master``:
.. code-block:: yaml
---
##
## file server
##
fileserver_backend:
# parameters values and override
- roots
# formulas
- gitfs
# The files in this directory will take precedence over git repositories
file_roots:
base:
- /srv/salt
# List of formulas I'm using
gitfs_remotes:
- https://github.com/saltstack-formulas/template-formula.git:
- base: v4.1.1
- https://github.com/saltstack-formulas/openssh-formula.git:
- base: v2.0.1
...
Create per DNS configuration for ``TEMPLATE`` formula
`````````````````````````````````````````````````````
Now, we can provides the per DNS domain name configuration files for the ``TEMPLATE`` formulas under ``/srv/salt/TEMPLATE/parameters/``.
We create the directory for ``dns:domain`` grain and we add a symlink for the ``domain`` grain which is extracted from the minion ``id``:
.. code-block:: console
mkdir -p /srv/salt/TEMPLATE/parameters/dns:domain/
ln -s dns:domain /srv/salt/TEMPLATE/parameters/domain
We create a configuration for the DNS domain ``example.net`` in ``/srv/salt/TEMPLATE/parameters/dns:domain/example.net.yaml``:
.. code-block:: yaml
---
values:
config: /etc/template-formula-example-net.conf
...
We create another configuration for the DNS domain ``example.com`` in the Jinja YAML template ``/srv/salt/TEMPLATE/parameters/dns:domain/example.com.yaml.jinja``:
.. code-block:: yaml
---
values:
config: /etc/template-formula-{{ grains['os_family'] }}.conf
...
Create per role configuration for ``TEMPLATE`` formula
``````````````````````````````````````````````````````
Now, we can provides the per role configuration files for the ``TEMPLATE`` formulas under ``/srv/salt/TEMPLATE/parameters/``.
We create the directory for roles:
.. code-block:: console
mkdir -p /srv/salt/TEMPLATE/parameters/roles
We will define 2 roles:
- ``TEMPLATE/server``
- ``TEMPLATE/client``
We create a configuration for the role ``TEMPLATE/server`` in ``/srv/salt/TEMPLATE/parameters/roles/TEMPLATE/server.yaml``:
.. code-block:: yaml
---
values:
config: /etc/template-formula-server.conf
...
We create another configuration for the role ``TEMPLATE/client`` in ``/srv/salt/TEMPLATE/parameters/roles/TEMPLATE/client.yaml``:
.. code-block:: yaml
---
values:
config: /etc/template-formula-client.conf
...
Enable roles and the ``dns:domain`` and ``domain`` grains for ``map.jinja``
```````````````````````````````````````````````````````````````````````````
We need to redefine the sources for ``map.jinja`` to load values from our new configuration files, we provide a global configuration for all our minions.
We create the global parameters file ``/srv/salt/parameters/map_jinja.yaml``:
.. code-block:: yaml
---
values:
sources:
# default values
- "Y:G@osarch"
- "Y:G@os_family"
- "Y:G@os"
- "Y:G@osfinger"
- "C@{{ tplroot ~ ':lookup' }}"
- "C@{{ tplroot }}"
# Roles activate/deactivate things
# then thing are configured depending on environment
# So roles comes before `dns:domain`, `domain` and `id`
- "Y:C@roles"
# DNS domain configured (DHCP or resolv.conf)
- "Y:G@dns:domain"
# Based on minion ID
- "Y:G@domain"
# default values
- "Y:G@id"
...
The syntax is explained later at `Sources of configuration values`_.
Bind roles to minions
`````````````````````
We associate roles `grains`_ to minion using `grains.append`_.
For the servers:
.. code-block:: console
salt 'server-*' grains.append roles TEMPLATE/server
For the clients:
.. code-block:: console
salt 'client-*' grains.append roles TEMPLATE/client
.. note::
Since we used ``Y:C@roles``, ``map.jinja`` will do a ``salt['config.get']('roles')`` to retrieve the roles so you could use any other method to bind roles to minions (`pillars`_ or `SDB`_) but `grains`_ seems to be the preferred method.
Note for Microsoft Windows systems
``````````````````````````````````
If you have a minion running under windows, you can't use colon ``:`` as a delimiter for grain path query (see `bug 58726`_) in which case you should use an alternate delimiter:
Modify ``/srv/salt/parameters/map_jinja.yaml`` to change the query for ``dns:domain`` to define the `alternate delimiter`_:
.. code-block:: yaml
---
values:
sources:
# default values
- "Y:G@osarch"
- "Y:G@os_family"
- "Y:G@os"
- "Y:G@osfinger"
- "C@{{ tplroot ~ ':lookup' }}"
- "C@{{ tplroot }}"
# Roles activate/deactivate things
# then thing are configured depending on environment
# So roles comes before `dns:domain`, `domain` and `id`
- "Y:C@roles"
# DNS domain configured (DHCP or resolv.conf)
- "Y:G:!@dns!domain"
# Based on minion ID
- "Y:G@domain"
# default values
- "Y:G@id"
...
And then, rename the directory:
.. code-block:: console
mv /srv/salt/TEMPLATE/parameters/dns:domain/ '/srv/salt/TEMPLATE/parameters/dns!domain/'
Format of configuration YAML files
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When you write a new YAML file, note that it must conform to the following layout:
- a mandatory ``values`` key to store the configuration values
- two optional keys to configure the use of `salt.slsutil.merge`_
- an optional ``strategy`` key to configure the merging strategy, for example ``strategy: 'recurse'``, the default is ``smart``
- an optional ``merge_lists`` key to configure if lists should be merged or overridden for the ``recurse`` and ``overwrite`` strategy, for example ``merge_lists: 'true'``
Here is a valid example:
.. code-block:: yaml
---
strategy: 'recurse'
merge_lists: 'false'
values:
pkg:
name: 'some-package'
config: '/path/to/a/configuration/file'
...
Using Jinja2 YAML template
``````````````````````````
You can provide a Jinja2 YAML template file with a name suffixed with ``.yaml.jinja``, it must produce a YAML file conform to the `Format of configuration YAML files`_, for example:
.. code-block:: jinja2
---
strategy: 'overwrite'
merge_lists: 'true'
values:
{%- if grains["os"] == "Debian" %}
output_dir: /tmp/{{ grains["id"] }}
{%- endif %}
...
Sources of configuration values
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The ``map.jinja`` file aggregates configuration values from several sources:
- YAML files stored in the `fileserver`_
- `pillars`_
- `grains`_
- configuration gathered with `salt['config.get']`_
For the values loaded from YAML files, ``map.jinja`` will automatically try to load a Jinja2 template with the same name as the YAML file with the addition of the ``.jinja`` extension, for example ``foo/bar/quux.yaml.jinja``.
After loading values from all sources, it will try to include the ``salt://parameters/post-map.jinja`` Jinja file if it exists which can post-process the ``mapdata`` variable.
Configuring ``map.jinja`` sources
`````````````````````````````````
The ``map.jinja`` file uses several sources where to lookup parameter values. The list of sources can be configured in two places:
1. globally
1. with a plain YAML file ``salt://parameters/map_jinja.yaml``
2. with a Jinja2 YAML template file ``salt://parameters/map_jinja.yaml.jinja``
2. per formula
1. with a plain YAML file ``salt://{{ tplroot }}/parameters/map_jinja.yaml``
2. with a Jinja2 YAML template file ``salt://{{ tplroot }}/parameters/map_jinja.yaml.jinja``
.. note::
The ``map.jinja`` configuration files must conform to the `format of configuration YAML files`_.
Each source definition has the form ``[<TYPE>[:<OPTION>[:<DELIMITER>]]@]<KEY>`` where ``<TYPE>`` can be one of:
- ``Y`` to load values from YAML files from the `fileserver`_, this is the default when no type is defined
- ``C`` to lookup values with `salt['config.get']`_
- ``G`` to lookup values with `salt['grains.get']`_
- ``I`` to lookup values with `salt['pillar.get']`_
The YAML type option can define the query method to lookup the key value to build the file name:
- ``C`` to query with `salt['config.get']`_, this is the default when no query method is defined
- ``G`` to query with `salt['grains.get']`_
- ``I`` to query with `salt['pillar.get']`_
The ``C``, ``G`` or ``I`` types can define the ``SUB`` option to store values in the sub key ``mapdata.<KEY>`` instead of directly in ``mapdata``.
All types can define the ``<DELIMITER>`` option to use an `alternate delimiter`_ of the ``<KEY>``, for example: on windows system you can't use colon ``:`` for YAML file path name and you should use something else like exclamation mark ``!``.
Finally, the ``<KEY>`` describes what to lookup to either build the YAML filename or gather values using one of the query methods.
.. note::
For the YAML type:
- if the ``<KEY>`` can't be looked up, then it's used a literal string path to a YAML file, for example: ``any/path/can/be/used/here.yaml`` will result in the loading of ``salt://{{ tplroot }}/parameters/any/path/can/be/used/here.yaml`` if it exists
- ``map.jinja`` will automatically try to load a Jinja2 template, after the corresponding YAML file, with the same name as the YAML file extended with the ``.jinja`` extension, for example ``any/path/can/be/used/here.yaml.jinja``
The built-in ``map.jinja`` sources are:
.. code-block:: yaml
- "Y:G@osarch"
- "Y:G@os_family"
- "Y:G@os"
- "Y:G@osfinger"
- "C@{{ tplroot ~ ':lookup' }}"
- "C@{{ tplroot }}"
- "Y:G@id"
This is strictly equivalent to the following ``map_jinja.yaml.jinja``:
.. code-block:: sls
values:
sources:
- "parameters/osarch/{{ salt['grains.get']('osarch') }}.yaml"
- "parameters/osarch/{{ salt['grains.get']('osarch') }}.yaml.jinja"
- "parameters/os_family/{{ salt['grains.get']('os_family') }}.yaml"
- "parameters/os_family/{{ salt['grains.get']('os_family') }}.yaml.jinja"
- "parameters/os/{{ salt['grains.get']('os') }}.yaml"
- "parameters/os/{{ salt['grains.get']('os') }}.yaml.jinja"
- "parameters/osfinger/{{ salt['grains.get']('osfinger') }}.yaml"
- "parameters/osfinger/{{ salt['grains.get']('osfinger') }}.yaml.jinja"
- "C@{{ tplroot ~ ':lookup' }}"
- "C@{{ tplroot }}"
- "parameters/id/{{ salt['grains.get']('id') }}.yaml"
- "parameters/id/{{ salt['grains.get']('id') }}.yaml.jinja"
Loading values from the configuration sources
`````````````````````````````````````````````
For each configuration source defined, ``map.jinja`` will:
#. load values depending on the source type:
- for YAML file sources
- if the ``<KEY>`` can be looked up:
- load values from the YAML file named ``salt://{{ tplroot }}/paramaters/<KEY>/{{ salt['<QUERY_METHOD>']('<KEY>') }}.yaml`` if it exists
- load values from the Jinja2 YAML template file named ``salt://{{ tplroot }}/paramaters/<KEY>/{{ salt['<QUERY_METHOD>']('<KEY>') }}.yaml.jinja`` if it exists
- otherwise:
- load the YAML file named ``salt://{{ tplroot }}/parameters/<KEY>.yaml`` if it exists
- load the Jinja2 YAML template file named ``salt://{{ tplroot }}/parameters/<KEY>.yaml.jinja`` if it exists
- for ``C``, ``G`` or ``I`` source type, lookup the value of ``salt['<QUERY_METHOD>']('<KEY>')``
#. merge the loaded values with the previous ones using `salt.slsutil.merge`_
There will be no error if a YAML or Jinja2 file does not exists, they are all optional.
Configuration values from ``salt['config.get']``
````````````````````````````````````````````````
For sources with of type ``C`` declared in ``map_jinja:sources``, you can configure the ``merge`` option of `salt['config.get']`_ by defining per formula ``strategy`` configuration key (retrieved with ``salt['config.get'](tplroot ~ ':strategy')`` with one of the following values:
- ``recurse`` merge recursively dictionaries. Non dictionary values replace already defined values
- ``overwrite`` new value completely replace old ones
By default, no merging is done, the first value found is returned.
Global view of the order of preferences
```````````````````````````````````````
To summarise, here is a complete example of the load order of formula configuration values for an ``AMD64`` ``Ubuntu 18.04`` minion named ``minion1.example.net`` for the ``libvirt`` formula:
#. ``parameters/defaults.yaml``
#. ``parameters/defaults.yaml.jinja``
#. ``parameters/osarch/amd64.yaml``
#. ``parameters/osarch/amd64.yaml.jinja``
#. ``parameters/os_family/Debian.yaml``
#. ``parameters/os_family/Debian.yaml.jinja``
#. ``parameters/os/Ubuntu.yaml``
#. ``parameters/os/Ubuntu.yaml.jinja``
#. ``parameters/osfinger/Ubuntu-18.04.yaml``
#. ``parameters/osfinger/Ubuntu-18.04.yaml.jinja``
#. ``salt['config.get']('libvirt:lookup')``
#. ``salt['config.get']('libvirt')``
#. ``parameters/id/minion1.example.net.yaml``
#. ``parameters/id/minion1.example.net.yaml.jinja``
Remember that the order is important, for example, the value of ``key1:subkey1`` loaded from ``parameters/os_family/Debian.yaml`` is overridden by a value loaded from ``parameters/id/minion1.example.net.yaml``.
For formula authors and contributors
------------------------------------
Dependencies
^^^^^^^^^^^^
``map.jinja`` requires:
- salt minion 2018.3.3 minimum to use the `traverse`_ jinja filter
- to be located at the root of the formula named directory (e.g. ``libvirt-formula/libvirt/map.jinja``)
- the ``libsaltcli.jinja`` library, stored in the same directory, to disable the ``merge`` option of `salt['config.get']`_ over `salt-ssh`_
- the ``libmapstack.jinja`` library to load the configuration values
- the ``libmatchers.jinja`` library used by ``libmapstack.jinja`` to parse compound like matchers
Use formula configuration values in ``sls``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The ``map.jinja`` exports a unique ``mapdata`` variable which could be renamed during import.
Here is the best way to use it in an ``sls`` file:
.. code-block:: sls
{#- Get the `tplroot` from `tpldir` #}
{%- set tplroot = tpldir.split("/")[0] %}
{%- from tplroot ~ "/map.jinja" import mapdata as TEMPLATE with context %}
test-does-nothing-but-display-TEMPLATE-as-json:
test.nop:
- name: {{ TEMPLATE | json }}
Use formula configuration values in templates
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When you need to process salt templates, you should avoid calling `salt['config.get']`_ (or `salt['pillar.get']`_ and `salt['grains.get']`_) directly from the template. All the needed values should be available within the ``mapdata`` variable exported by ``map.jinja``.
Here is an example based on `template-formula/TEMPLATE/config/file.sls`_:
.. code-block:: sls
# -*- coding: utf-8 -*-
# vim: ft=sls
{#- Get the `tplroot` from `tpldir` #}
{%- set tplroot = tpldir.split('/')[0] %}
{%- set sls_package_install = tplroot ~ '.package.install' %}
{%- from tplroot ~ "/map.jinja" import mapdata as TEMPLATE with context %}
{%- from tplroot ~ "/libtofs.jinja" import files_switch with context %}
include:
- {{ sls_package_install }}
TEMPLATE-config-file-file-managed:
file.managed:
- name: {{ TEMPLATE.config }}
- source: {{ files_switch(['example.tmpl'],
lookup='TEMPLATE-config-file-file-managed'
)
}}
- mode: 644
- user: root
- group: {{ TEMPLATE.rootgroup }}
- makedirs: True
- template: jinja
- require:
- sls: {{ sls_package_install }}
- context:
TEMPLATE: {{ TEMPLATE | json }}
This ``sls`` file expose a ``TEMPLATE`` context variable to the jinja template which could be used like this:
.. code-block:: jinja
########################################################################
# File managed by Salt at <{{ source }}>.
# Your changes will be overwritten.
########################################################################
This is another example file from SaltStack template-formula.
# This is here for testing purposes
{{ TEMPLATE | json }}
winner of the merge: {{ TEMPLATE['winner'] }}
.. _documentation: https://docs.saltproject.io/en/latest/topics/development/conventions/formulas.html#writing-formulas
.. _fileserver: https://docs.saltproject.io/en/latest/ref/file_server
.. _salt['config.get']: https://docs.saltproject.io/en/latest/ref/modules/all/salt.modules.config.html#salt.modules.config.get
.. _salt['grains.get']: https://docs.saltproject.io/en/latest/ref/modules/all/salt.modules.grains.html#salt.modules.grains.get
.. _salt['pillar.get']: https://docs.saltproject.io/en/latest/ref/modules/all/salt.modules.pillar.html#salt.modules.pillar.get
.. _alternate delimiter: https://docs.saltproject.io/en/latest/topics/targeting/compound.html#alternate-delimiters
.. _pillar.vault: https://docs.saltproject.io/en/latest/ref/pillar/all/salt.pillar.vault.html
.. _pillars: https://docs.saltproject.io/en/latest/topics/pillar/
.. _grains: https://docs.saltproject.io/en/latest/topics/grains/
.. _grains.append: https://docs.saltproject.io/en/latest/ref/modules/all/salt.modules.grains.html#salt.modules.grains.append
.. _SDB: https://docs.saltproject.io/en/latest/topics/sdb/index.html
.. _sdb.vault: https://docs.saltproject.io/en/latest/ref/sdb/all/salt.sdb.vault.html
.. _Jinja: https://docs.saltproject.io/en/latest/topics/jinja
.. _roots: https://docs.saltproject.io/en/latest/ref/file_server/all/salt.fileserver.roots.html
.. _gitfs: https://docs.saltproject.io/en/latest/topics/tutorials/gitfs.html
.. _salt.slsutil.merge: https://docs.saltproject.io/en/latest/ref/modules/all/salt.modules.slsutil.html#salt.modules.slsutil.merge
.. _traverse: https://docs.saltproject.io/en/latest/topics/jinja/index.html#traverse
.. _salt-ssh: https://docs.saltproject.io/en/latest/topics/ssh/
.. _template-formula/TEMPLATE/config/file.sls: https://github.com/saltstack-formulas/template-formula/blob/master/TEMPLATE/config/file.sls
.. _bug 58726: https://github.com/saltstack/salt/issues/58726