salt/doc/topics/tutorials/http.rst

HTTP Modules

This tutorial demonstrates using the various HTTP modules available in Salt. These modules wrap the Python tornado, urllib2, and requests libraries, extending them in a manner that is more consistent with Salt workflows.

The salt.utils.http Library

This library forms the core of the HTTP modules. Since it is designed to be used from the minion as an execution module, in addition to the master as a runner, it was abstracted into this multi-use library. This library can also be imported by 3rd-party programs wishing to take advantage of its extended functionality.

Core functionality of the execution, state, and runner modules is derived from this library, so common usages between them are described here. Documentation specific to each module is described below.

This library can be imported with:

import salt.utils.http

Configuring Libraries

This library can make use of either tornado, which is required by Salt, urllib2, which ships with Python, or requests, which can be installed separately. By default, tornado will be used. In order to switch to urllib2, set the following variable:

backend: urllib2

In order to switch to requests, set the following variable:

backend: requests

This can be set in the master or minion configuration file, or passed as an option directly to any http.query() functions.

salt.utils.http.query()

This function forms a basic query, but with some add-ons not present in the tornado, urllib2, and requests libraries. Not all functionality currently available in these libraries has been added, but can be in future iterations.

HTTPS Request Methods

A basic query can be performed by calling this function with no more than a single URL:

salt.utils.http.query("http://example.com")

By default the query will be performed with a GET method. The method can be overridden with the method argument:

salt.utils.http.query("http://example.com/delete/url", "DELETE")

When using the POST method (and others, such as PUT), extra data is usually sent as well. This data can be sent directly (would be URL encoded when necessary), or in whatever format is required by the remote server (XML, JSON, plain text, etc).

salt.utils.http.query(
    "http://example.com/post/url", method="POST", data=json.dumps(mydict)
)

Data Formatting and Templating

Bear in mind that the data must be sent pre-formatted; this function will not format it for you. However, a templated file stored on the local system may be passed through, along with variables to populate it with. To pass through only the file (untemplated):

salt.utils.http.query(
    "http://example.com/post/url", method="POST", data_file="/srv/salt/somefile.xml"
)

To pass through a file that contains jinja + yaml templating (the default):

salt.utils.http.query(
    "http://example.com/post/url",
    method="POST",
    data_file="/srv/salt/somefile.jinja",
    data_render=True,
    template_dict={"key1": "value1", "key2": "value2"},
)

To pass through a file that contains mako templating:

salt.utils.http.query(
    "http://example.com/post/url",
    method="POST",
    data_file="/srv/salt/somefile.mako",
    data_render=True,
    data_renderer="mako",
    template_dict={"key1": "value1", "key2": "value2"},
)

Because this function uses Salt's own rendering system, any Salt renderer can be used. Because Salt's renderer requires __opts__ to be set, an opts dictionary should be passed in. If it is not, then the default __opts__ values for the node type (master or minion) will be used. Because this library is intended primarily for use by minions, the default node type is minion. However, this can be changed to master if necessary.

salt.utils.http.query(
    "http://example.com/post/url",
    method="POST",
    data_file="/srv/salt/somefile.jinja",
    data_render=True,
    template_dict={"key1": "value1", "key2": "value2"},
    opts=__opts__,
)

salt.utils.http.query(
    "http://example.com/post/url",
    method="POST",
    data_file="/srv/salt/somefile.jinja",
    data_render=True,
    template_dict={"key1": "value1", "key2": "value2"},
    node="master",
)

Headers

Headers may also be passed through, either as a header_list, a header_dict, or as a header_file. As with the data_file, the header_file may also be templated. Take note that because HTTP headers are normally syntactically-correct YAML, they will automatically be imported as an a Python dict.

salt.utils.http.query(
    "http://example.com/delete/url",
    method="POST",
    header_file="/srv/salt/headers.jinja",
    header_render=True,
    header_renderer="jinja",
    template_dict={"key1": "value1", "key2": "value2"},
)

Because much of the data that would be templated between headers and data may be the same, the template_dict is the same for both. Correcting possible variable name collisions is up to the user.

Authentication

The query() function supports basic HTTP authentication. A username and password may be passed in as username and password, respectively.

salt.utils.http.query(
    "http://example.com", username="larry", password="5700g3543v4r",
)

Cookies and Sessions

Cookies are also supported, using Python's built-in cookielib. However, they are turned off by default. To turn cookies on, set cookies to True.

salt.utils.http.query("http://example.com", cookies=True)

By default cookies are stored in Salt's cache directory, normally /var/cache/salt, as a file called cookies.txt. However, this location may be changed with the cookie_jar argument:

salt.utils.http.query(
    "http://example.com", cookies=True, cookie_jar="/path/to/cookie_jar.txt"
)

By default, the format of the cookie jar is LWP (aka, lib-www-perl). This default was chosen because it is a human-readable text file. If desired, the format of the cookie jar can be set to Mozilla:

salt.utils.http.query(
    "http://example.com",
    cookies=True,
    cookie_jar="/path/to/cookie_jar.txt",
    cookie_format="mozilla",
)

Because Salt commands are normally one-off commands that are piped together, this library cannot normally behave as a normal browser, with session cookies that persist across multiple HTTP requests. However, the session can be persisted in a separate cookie jar. The default filename for this file, inside Salt's cache directory, is cookies.session.p. This can also be changed.

salt.utils.http.query(
    "http://example.com", persist_session=True, session_cookie_jar="/path/to/jar.p"
)

The format of this file is msgpack, which is consistent with much of the rest of Salt's internal structure. Historically, the extension for this file is .p. There are no current plans to make this configurable.

Proxy

If the tornado backend is used (tornado is the default), proxy information configured in proxy_host, proxy_port, proxy_username, proxy_password and no_proxy from the __opts__ dictionary will be used. Normally these are set in the minion configuration file.

proxy_host: proxy.my-domain
proxy_port: 31337
proxy_username: charon
proxy_password: obolus
no_proxy: ['127.0.0.1', 'localhost']

salt.utils.http.query("http://example.com", opts=__opts__, backend="tornado")

Return Data

Note

Return data encoding

If decode is set to True, query() will attempt to decode the return data. decode_type defaults to auto. Set it to a specific encoding, xml, for example, to override autodetection.

Because Salt's http library was designed to be used with REST interfaces, query() will attempt to decode the data received from the remote server when decode is set to True. First it will check the Content-type header to try and find references to XML. If it does not find any, it will look for references to JSON. If it does not find any, it will fall back to plain text, which will not be decoded.

JSON data is translated into a dict using Python's built-in json library. XML is translated using salt.utils.xml_util, which will use Python's built-in XML libraries to attempt to convert the XML into a dict. In order to force either JSON or XML decoding, the decode_type may be set:

salt.utils.http.query("http://example.com", decode_type="xml")

Once translated, the return dict from query() will include a dict called dict.

If the data is not to be translated using one of these methods, decoding may be turned off.

salt.utils.http.query("http://example.com", decode=False)

If decoding is turned on, and references to JSON or XML cannot be found, then this module will default to plain text, and return the undecoded data as text (even if text is set to False; see below).

The query() function can return the HTTP status code, headers, and/or text as required. However, each must individually be turned on.

salt.utils.http.query("http://example.com", status=True, headers=True, text=True)

The return from these will be found in the return dict as status, headers and text, respectively.

Writing Return Data to Files

It is possible to write either the return data or headers to files, as soon as the response is received from the server, but specifying file locations via the text_out or headers_out arguments. text and headers do not need to be returned to the user in order to do this.

salt.utils.http.query(
    "http://example.com",
    text=False,
    headers=False,
    text_out="/path/to/url_download.txt",
    headers_out="/path/to/headers_download.txt",
)

SSL Verification

By default, this function will verify SSL certificates. However, for testing or debugging purposes, SSL verification can be turned off.

salt.utils.http.query(
    "https://example.com", verify_ssl=False,
)

CA Bundles

The requests library has its own method of detecting which CA (certificate authority) bundle file to use. Usually this is implemented by the packager for the specific operating system distribution that you are using. However, urllib2 requires a little more work under the hood. By default, Salt will try to auto-detect the location of this file. However, if it is not in an expected location, or a different path needs to be specified, it may be done so using the ca_bundle variable.

salt.utils.http.query(
    "https://example.com", ca_bundle="/path/to/ca_bundle.pem",
)

Updating CA Bundles

The update_ca_bundle() function can be used to update the bundle file at a specified location. If the target location is not specified, then it will attempt to auto-detect the location of the bundle file. If the URL to download the bundle from does not exist, a bundle will be downloaded from the cURL website.

CAUTION: The target and the source should always be specified! Failure to specify the target may result in the file being written to the wrong location on the local system. Failure to specify the source may cause the upstream URL to receive excess unnecessary traffic, and may cause a file to be download which is hazardous or does not meet the needs of the user.

salt.utils.http.update_ca_bundle(
    target="/path/to/ca-bundle.crt",
    source="https://example.com/path/to/ca-bundle.crt",
    opts=__opts__,
)

The opts parameter should also always be specified. If it is, then the target and the source may be specified in the relevant configuration file (master or minion) as ca_bundle and ca_bundle_url, respectively.

ca_bundle: /path/to/ca-bundle.crt
ca_bundle_url: https://example.com/path/to/ca-bundle.crt

If Salt is unable to auto-detect the location of the CA bundle, it will raise an error.

The update_ca_bundle() function can also be passed a string or a list of strings which represent files on the local system, which should be appended (in the specified order) to the end of the CA bundle file. This is useful in environments where private certs need to be made available, and are not otherwise reasonable to add to the bundle file.

salt.utils.http.update_ca_bundle(
    opts=__opts__,
    merge_files=[
        "/etc/ssl/private_cert_1.pem",
        "/etc/ssl/private_cert_2.pem",
        "/etc/ssl/private_cert_3.pem",
    ],
)

Test Mode

This function may be run in test mode. This mode will perform all work up until the actual HTTP request. By default, instead of performing the request, an empty dict will be returned. Using this function with TRACE logging turned on will reveal the contents of the headers and POST data to be sent.

Rather than returning an empty dict, an alternate test_url may be passed in. If this is detected, then test mode will replace the url with the test_url, set test to True in the return data, and perform the rest of the requested operations as usual. This allows a custom, non-destructive URL to be used for testing when necessary.

Execution Module

The http execution module is a very thin wrapper around the salt.utils.http library. The opts can be passed through as well, but if they are not specified, the minion defaults will be used as necessary.

Because passing complete data structures from the command line can be tricky at best and dangerous (in terms of execution injection attacks) at worse, the data_file, and header_file are likely to see more use here.

All methods for the library are available in the execution module, as kwargs.

salt myminion http.query http://example.com/restapi method=POST \
    username='larry' password='5700g3543v4r' headers=True text=True \
    status=True decode_type=xml data_render=True \
    header_file=/tmp/headers.txt data_file=/tmp/data.txt \
    header_render=True cookies=True persist_session=True

Runner Module

Like the execution module, the http runner module is a very thin wrapper around the salt.utils.http library. The only significant difference is that because runners execute on the master instead of a minion, a target is not required, and default opts will be derived from the master config, rather than the minion config.

All methods for the library are available in the runner module, as kwargs.

salt-run http.query http://example.com/restapi method=POST \
    username='larry' password='5700g3543v4r' headers=True text=True \
    status=True decode_type=xml data_render=True \
    header_file=/tmp/headers.txt data_file=/tmp/data.txt \
    header_render=True cookies=True persist_session=True

State Module

The state module is a wrapper around the runner module, which applies stateful logic to a query. All kwargs as listed above are specified as usual in state files, but two more kwargs are available to apply stateful logic. A required parameter is match, which specifies a pattern to look for in the return text. By default, this will perform a string comparison of looking for the value of match in the return text. In Python terms this looks like:

def myfunc():
    if match in html_text:
        return True

If more complex pattern matching is required, a regular expression can be used by specifying a match_type. By default this is set to string, but it can be manually set to pcre instead. Please note that despite the name, this will use Python's re.search() rather than re.match().

Therefore, the following states are valid:

http://example.com/restapi:
  http.query:
    - match: 'SUCCESS'
    - username: 'larry'
    - password: '5700g3543v4r'
    - data_render: True
    - header_file: /tmp/headers.txt
    - data_file: /tmp/data.txt
    - header_render: True
    - cookies: True
    - persist_session: True

http://example.com/restapi:
  http.query:
    - match_type: pcre
    - match: '(?i)succe[ss|ed]'
    - username: 'larry'
    - password: '5700g3543v4r'
    - data_render: True
    - header_file: /tmp/headers.txt
    - data_file: /tmp/data.txt
    - header_render: True
    - cookies: True
    - persist_session: True

In addition to, or instead of a match pattern, the status code for a URL can be checked. This is done using the status argument:

http://example.com/:
  http.query:
    - status: 200

If both are specified, both will be checked, but if only one is True and the other is False, then False will be returned. In this case, the comments in the return data will contain information for troubleshooting.

Because this is a monitoring state, it will return extra data to code that expects it. This data will always include text and status. Optionally, headers and dict may also be requested by setting the headers and decode arguments to True, respectively.