saltstack/salt
1
0
Fork 0
mirror of https://github.com/saltstack/salt.git synced 2025-04-17 10:10:20 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 2

Merge pull request #34951 from vutny/fix-schedule-dateutil

Browse source 
Fix #34873
This commit is contained in:
Mike Place 2016-07-26 11:07:47 -06:00 • committed by GitHub
commit f23e8c525e
2 changed files with 334 additions and 192 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

207
doc/topics/jobs/index.rst
View file

@ -94,6 +94,7 @@ that have already, or partially returned.
Scheduling Jobs
===============
Salt's scheduling system allows incremental executions on minions or the
master. The schedule system exposes the execution of any execution function on
minions or any runner on the master.
@ -113,13 +114,8 @@ have to restart the application in order for the schedule to be implemented.
A scheduled run has no output on the minion unless the config is set to info level
or higher. Refer to :conf_minion:`minion-logging-settings`.
Specify ``maxrunning`` to ensure that there are no more than N copies of
a particular routine running. Use this for jobs that may be long-running
and could step on each other or otherwise double execute. The default for
``maxrunning`` is 1.
States are executed on the minion, as all states are. You can pass positional
arguments and provide a yaml dict of named arguments.
arguments and provide a YAML dict of named arguments.
.. code-block:: yaml
@ -132,8 +128,8 @@ arguments and provide a yaml dict of named arguments.
kwargs:
test: True
This will schedule the command: state.sls httpd test=True every 3600 seconds
(every hour)
This will schedule the command: ``state.sls httpd test=True`` every 3600 seconds
(every hour).
.. code-block:: yaml
@ -147,8 +143,8 @@ This will schedule the command: state.sls httpd test=True every 3600 seconds
test: True
splay: 15
This will schedule the command: state.sls httpd test=True every 3600 seconds
(every hour) splaying the time between 0 and 15 seconds
This will schedule the command: ``state.sls httpd test=True`` every 3600 seconds
(every hour) splaying the time between 0 and 15 seconds.
.. code-block:: yaml
@ -164,14 +160,17 @@ This will schedule the command: state.sls httpd test=True every 3600 seconds
start: 10
end: 15
This will schedule the command: state.sls httpd test=True every 3600 seconds
(every hour) splaying the time between 10 and 15 seconds
This will schedule the command: ``state.sls httpd test=True`` every 3600 seconds
(every hour) splaying the time between 10 and 15 seconds.
Schedule by Date and Time
-------------------------
.. versionadded:: 2014.7.0
Frequency of jobs can also be specified using date strings supported by
the python dateutil library. This requires python-dateutil to be installed on
the minion.
the Python ``dateutil`` library. This requires the Python ``dateutil`` library
to be installed.
.. code-block:: yaml
@ -184,7 +183,7 @@ the minion.
test: True
when: 5:00pm
This will schedule the command: state.sls httpd test=True at 5:00pm minion
This will schedule the command: ``state.sls httpd test=True`` at 5:00 PM minion
localtime.
.. code-block:: yaml
@ -197,14 +196,14 @@ localtime.
kwargs:
test: True
when:
- Monday 5:00pm
- Tuesday 3:00pm
- Wednesday 5:00pm
- Thursday 3:00pm
- Friday 5:00pm
- Monday 5:00pm
- Tuesday 3:00pm
- Wednesday 5:00pm
- Thursday 3:00pm
- Friday 5:00pm
This will schedule the command: state.sls httpd test=True at 5pm on Monday,
Wednesday, and Friday, and 3pm on Tuesday and Thursday.
This will schedule the command: ``state.sls httpd test=True`` at 5:00 PM on
Monday, Wednesday and Friday, and 3:00 PM on Tuesday and Thursday.
.. code-block:: yaml
@ -217,13 +216,32 @@ Wednesday, and Friday, and 3pm on Tuesday and Thursday.
kwargs:
test: True
range:
start: 8:00am
end: 5:00pm
start: 8:00am
end: 5:00pm
This will schedule the command: state.sls httpd test=True every 3600 seconds
(every hour) between the hours of 8am and 5pm. The range parameter must be a
dictionary with the date strings using the dateutil format. This requires
python-dateutil to be installed on the minion.
This will schedule the command: ``state.sls httpd test=True`` every 3600 seconds
(every hour) between the hours of 8:00 AM and 5:00 PM. The range parameter must
be a dictionary with the date strings using the ``dateutil`` format.
.. code-block:: yaml
schedule:
job1:
function: state.sls
seconds: 3600
args:
- httpd
kwargs:
test: True
range:
invert: True
start: 8:00am
end: 5:00pm
Using the invert option for range, this will schedule the command
``state.sls httpd test=True`` every 3600 seconds (every hour) until the current
time is between the hours of 8:00 AM and 5:00 PM. The range parameter must be
a dictionary with the date strings using the ``dateutil`` format.
.. code-block:: yaml
@ -235,29 +253,96 @@ python-dateutil to be installed on the minion.
refresh: true
once: '2016-01-07T14:30:00'
This will schedule the command pkg.install to be executed once at the specified
time. The schedule entry ``job1`` will not be removed after the job completes,
therefore use ``schedule.delete`` to manually remove it afterwards.
This will schedule the function ``pkg.install`` to be executed once at the
specified time. The schedule entry ``job1`` will not be removed after the job
completes, therefore use ``schedule.delete`` to manually remove it afterwards.
The default date format is ISO 8601 but can be overridden by also specifying the
``once_fmt`` option.
``once_fmt`` option, like this:
.. code-block:: yaml
schedule:
job1:
function: test.ping
once: 2015-04-22T20:21:00
once_fmt: '%Y-%m-%dT%H:%M:%S'
Maximum Parallel Jobs Running
-----------------------------
.. versionadded:: 2014.7.0
The scheduler also supports ensuring that there are no more than N copies of
a particular routine running. Use this for jobs that may be long-running
a particular routine running. Use this for jobs that may be long-running
and could step on each other or pile up in case of infrastructure outage.
The default for maxrunning is 1.
The default for ``maxrunning`` is 1.
.. code-block:: yaml
schedule:
long_running_job:
function: big_file_transfer
jid_include: True
function: big_file_transfer
jid_include: True
maxrunning: 1
run_on_start
Cron-like Schedule
------------------
.. versionadded:: 2014.7.0
.. code-block:: yaml
schedule:
job1:
function: state.sls
cron: '*/15 * * * *'
args:
- httpd
kwargs:
test: True
The scheduler also supports scheduling jobs using a cron like format.
This requires the Python ``croniter`` library.
Job Data Return
---------------
.. versionadded:: 2015.5.0
By default, data about jobs runs from the Salt scheduler is returned to the
master. Setting the ``return_job`` parameter to False will prevent the data
from being sent back to the Salt master.
.. code-block:: yaml
schedule:
job1:
function: scheduled_job_function
return_job: False
Job Metadata
------------
.. versionadded:: 2015.5.0
It can be useful to include specific data to differentiate a job from other
jobs. Using the metadata parameter special values can be associated with
a scheduled job. These values are not used in the execution of the job,
but can be used to search for specific jobs later if combined with the
``return_job`` parameter. The metadata parameter must be specified as a
dictionary, othewise it will be ignored.
.. code-block:: yaml
schedule:
job1:
function: scheduled_job_function
metadata:
foo: bar
Run on Start
------------
.. versionadded:: 2015.5.0
@ -265,7 +350,7 @@ run_on_start
By default, any job scheduled based on the startup time of the minion will run
the scheduled job when the minion starts up. Sometimes this is not the desired
situation. Using the ``run_on_start`` parameter set to ``False`` will cause the
scheduler to skip this first run and wait until the next scheduled run.
scheduler to skip this first run and wait until the next scheduled run:
.. code-block:: yaml
@ -279,6 +364,48 @@ scheduler to skip this first run and wait until the next scheduled run.
kwargs:
test: True
Until and After
---------------
.. versionadded:: 2015.8.0
.. code-block:: yaml
schedule:
job1:
function: state.sls
seconds: 15
until: '12/31/2015 11:59pm'
args:
- httpd
kwargs:
test: True
Using the until argument, the Salt scheduler allows you to specify
an end time for a scheduled job. If this argument is specified, jobs
will not run once the specified time has passed. Time should be specified
in a format supported by the ``dateutil`` library.
This requires the Python ``dateutil`` library to be installed.
.. versionadded:: 2015.8.0
.. code-block:: yaml
schedule:
job1:
function: state.sls
seconds: 15
after: '12/31/2015 11:59pm'
args:
- httpd
kwargs:
test: True
Using the after argument, the Salt scheduler allows you to specify
an start time for a scheduled job. If this argument is specified, jobs
will not run until the specified time has passed. Time should be specified
in a format supported by the ``dateutil`` library.
This requires the Python ``dateutil`` library to be installed.
Scheduling States
-----------------
@ -297,6 +424,7 @@ Scheduling States
Scheduling Highstates
---------------------
To set up a highstate to run on a minion every 60 minutes set this in the
minion config or pillar:
@ -311,6 +439,7 @@ Time intervals can be specified as seconds, minutes, hours, or days.
Scheduling Runners
------------------
Runner executions can also be specified on the master within the master
configuration file:
@ -329,6 +458,7 @@ The above configuration is analogous to running
Scheduler With Returner
-----------------------
The scheduler is also useful for tasks like gathering monitoring data about
a minion, this schedule option will gather status data and send it to a MySQL
returner database:
@ -348,4 +478,3 @@ returner database:
Since specifying the returner repeatedly can be tiresome, the
``schedule_returner`` option is available to specify one or a list of global
returners to be used by the minions when scheduling.

319
salt/utils/schedule.py
View file

@ -15,8 +15,8 @@ file or for the minion via config or pillar)
kwargs:
test: True
This will schedule the command: state.sls httpd test=True every 3600 seconds
(every hour)
This will schedule the command: ``state.sls httpd test=True`` every 3600 seconds
(every hour).
.. code-block:: yaml
@ -30,8 +30,8 @@ This will schedule the command: state.sls httpd test=True every 3600 seconds
test: True
splay: 15
This will schedule the command: state.sls httpd test=True every 3600 seconds
(every hour) splaying the time between 0 and 15 seconds
This will schedule the command: ``state.sls httpd test=True`` every 3600 seconds
(every hour) splaying the time between 0 and 15 seconds.
.. code-block:: yaml
@ -47,13 +47,14 @@ This will schedule the command: state.sls httpd test=True every 3600 seconds
start: 10
end: 15
This will schedule the command: state.sls httpd test=True every 3600 seconds
(every hour) splaying the time between 10 and 15 seconds
This will schedule the command: ``state.sls httpd test=True`` every 3600 seconds
(every hour) splaying the time between 10 and 15 seconds.
.. versionadded:: 2014.7.0
Frequency of jobs can also be specified using date strings supported by
the python dateutil library.
the Python ``dateutil`` library. This requires the Python ``dateutil`` library
to be installed.
.. code-block:: yaml
@ -66,7 +67,7 @@ the python dateutil library.
test: True
when: 5:00pm
This will schedule the command: state.sls httpd test=True at 5:00pm minion
This will schedule the command: ``state.sls httpd test=True`` at 5:00 PM minion
localtime.
.. code-block:: yaml
@ -79,15 +80,69 @@ localtime.
kwargs:
test: True
when:
- Monday 5:00pm
- Tuesday 3:00pm
- Wednesday 5:00pm
- Thursday 3:00pm
- Friday 5:00pm
- Monday 5:00pm
- Tuesday 3:00pm
- Wednesday 5:00pm
- Thursday 3:00pm
- Friday 5:00pm
This will schedule a job to run once on the specified date. The default date
format is ISO 8601 but can be overridden by also specifying the ``once_fmt``
option.
This will schedule the command: ``state.sls httpd test=True`` at 5:00 PM on
Monday, Wednesday and Friday, and 3:00 PM on Tuesday and Thursday.
.. code-block:: yaml
schedule:
job1:
function: state.sls
seconds: 3600
args:
- httpd
kwargs:
test: True
range:
start: 8:00am
end: 5:00pm
This will schedule the command: ``state.sls httpd test=True`` every 3600 seconds
(every hour) between the hours of 8:00 AM and 5:00 PM. The range parameter must
be a dictionary with the date strings using the ``dateutil`` format.
.. code-block:: yaml
schedule:
job1:
function: state.sls
seconds: 3600
args:
- httpd
kwargs:
test: True
range:
invert: True
start: 8:00am
end: 5:00pm
Using the invert option for range, this will schedule the command
``state.sls httpd test=True`` every 3600 seconds (every hour) until the current
time is between the hours of 8:00 AM and 5:00 PM. The range parameter must be
a dictionary with the date strings using the ``dateutil`` format.
.. code-block:: yaml
schedule:
job1:
function: pkg.install
kwargs:
pkgs: [{'bar': '>1.2.3'}]
refresh: true
once: '2016-01-07T14:30:00'
This will schedule the function ``pkg.install`` to be executed once at the
specified time. The schedule entry ``job1`` will not be removed after the job
completes, therefore use ``schedule.delete`` to manually remove it afterwards.
The default date format is ISO 8601 but can be overridden by also specifying the
``once_fmt`` option, like this:
.. code-block:: yaml
@ -97,67 +152,21 @@ option.
once: 2015-04-22T20:21:00
once_fmt: '%Y-%m-%dT%H:%M:%S'
This will schedule the command: state.sls httpd test=True at 5pm on Monday,
Wednesday and Friday, and 3pm on Tuesday and Thursday.
.. code-block:: yaml
schedule:
job1:
function: state.sls
seconds: 3600
args:
- httpd
kwargs:
test: True
range:
start: 8:00am
end: 5:00pm
This will schedule the command: state.sls httpd test=True every 3600 seconds
(every hour) between the hours of 8am and 5pm. The range parameter must be a
dictionary with the date strings using the dateutil format.
.. versionadded:: 2014.7.0
.. code-block:: yaml
The scheduler also supports ensuring that there are no more than N copies of
a particular routine running. Use this for jobs that may be long-running
and could step on each other or pile up in case of infrastructure outage.
schedule:
job1:
function: state.sls
seconds: 3600
args:
- httpd
kwargs:
test: True
range:
invert: True
start: 8:00am
end: 5:00pm
Using the invert option for range, this will schedule the command: state.sls
httpd test=True every 3600 seconds (every hour) until the current time is
between the hours of 8am and 5pm. The range parameter must be a dictionary
with the date strings using the dateutil format.
By default any job scheduled based on the startup time of the minion will run
the scheduled job when the minion starts up. Sometimes this is not the desired
situation. Using the 'run_on_start' parameter set to False will cause the
scheduler to skip this first run and wait until the next scheduled run.
.. versionadded:: 2015.5.0
The default for ``maxrunning`` is 1.
.. code-block:: yaml
schedule:
job1:
function: state.sls
seconds: 3600
run_on_start: False
args:
- httpd
kwargs:
test: True
long_running_job:
function: big_file_transfer
jid_include: True
maxrunning: 1
.. versionadded:: 2014.7.0
@ -173,9 +182,60 @@ scheduler to skip this first run and wait until the next scheduled run.
test: True
The scheduler also supports scheduling jobs using a cron like format.
This requires the python-croniter library.
This requires the Python ``croniter`` library.
... versionadded:: 2015.8.0
.. versionadded:: 2015.5.0
By default, data about jobs runs from the Salt scheduler is returned to the
master. Setting the ``return_job`` parameter to False will prevent the data
from being sent back to the Salt master.
.. code-block:: yaml
schedule:
job1:
function: scheduled_job_function
return_job: False
.. versionadded:: 2015.5.0
It can be useful to include specific data to differentiate a job from other
jobs. Using the metadata parameter special values can be associated with
a scheduled job. These values are not used in the execution of the job,
but can be used to search for specific jobs later if combined with the
``return_job`` parameter. The metadata parameter must be specified as a
dictionary, othewise it will be ignored.
.. code-block:: yaml
schedule:
job1:
function: scheduled_job_function
metadata:
foo: bar
.. versionadded:: 2015.5.0
By default any job scheduled based on the startup time of the minion will run
the scheduled job when the minion starts up. Sometimes this is not the desired
situation. Using the ``run_on_start`` parameter set to ``False`` will cause the
scheduler to skip this first run and wait until the next scheduled run:
.. code-block:: yaml
schedule:
job1:
function: state.sls
seconds: 3600
run_on_start: False
args:
- httpd
kwargs:
test: True
.. versionadded:: 2015.8.0
.. code-block:: yaml
schedule:
job1:
@ -188,12 +248,14 @@ This requires the python-croniter library.
test: True
Using the until argument, the Salt scheduler allows you to specify
an end time for a scheduled job. If this argument is specified, jobs
will not run once the specified time has passed. Time should be specified
in a format support by the dateutil library.
This requires the python-dateutil library.
an end time for a scheduled job. If this argument is specified, jobs
will not run once the specified time has passed. Time should be specified
in a format supported by the ``dateutil`` library.
This requires the Python ``dateutil`` library to be installed.
... versionadded:: 2015.8.0
.. versionadded:: 2015.8.0
.. code-block:: yaml
schedule:
job1:
@ -207,49 +269,9 @@ This requires the python-dateutil library.
Using the after argument, the Salt scheduler allows you to specify
an start time for a scheduled job. If this argument is specified, jobs
will not run until the specified time has passed. Time should be specified
in a format support by the dateutil library.
This requires the python-dateutil library.
The scheduler also supports ensuring that there are no more than N copies of
a particular routine running. Use this for jobs that may be long-running
and could step on each other or pile up in case of infrastructure outage.
The default for maxrunning is 1.
.. code-block:: yaml
schedule:
long_running_job:
function: big_file_transfer
jid_include: True
maxrunning: 1
By default, data about jobs runs from the Salt scheduler is returned to the
master. Setting the ``return_job`` parameter to False will prevent the data
from being sent back to the Salt master.
.. versionadded:: 2015.5.0
schedule:
job1:
function: scheduled_job_function
return_job: False
It can be useful to include specific data to differentiate a job from other
jobs. Using the metadata parameter special values can be associated with
a scheduled job. These values are not used in the execution of the job,
but can be used to search for specific jobs later if combined with the
return_job parameter. The metadata parameter must be specified as a
dictionary, othewise it will be ignored.
.. versionadded:: 2015.5.0
schedule:
job1:
function: scheduled_job_function
metadata:
foo: bar
will not run until the specified time has passed. Time should be specified
in a format supported by the ``dateutil`` library.
This requires the Python ``dateutil`` library to be installed.
'''
@ -265,6 +287,7 @@ import threading
import logging
import errno
import random
import yaml
# Import Salt libs
import salt.config
@ -283,7 +306,6 @@ from salt.utils.odict import OrderedDict
from salt.utils.process import os_is_running, default_signals, SignalHandlingMultiprocessingProcess
# Import 3rd-party libs
import yaml
import salt.ext.six as six
# pylint: disable=import-error
@ -833,7 +855,7 @@ class Schedule(object):
if 'until' in data:
if not _WHEN_SUPPORTED:
log.error('Missing python-dateutil.'
log.error('Missing python-dateutil. '
'Ignoring until.')
else:
until__ = dateutil_parser.parse(data['until'])
@ -846,7 +868,7 @@ class Schedule(object):
if 'after' in data:
if not _WHEN_SUPPORTED:
log.error('Missing python-dateutil.'
log.error('Missing python-dateutil. '
'Ignoring after.')
else:
after__ = dateutil_parser.parse(data['after'])
@ -868,7 +890,7 @@ class Schedule(object):
if any(i <= schedule_keys for i in invalid_sched_combos):
log.error('Unable to use "{0}" options together. Ignoring.'
.format('", "'.join(scheduling_elements)))
.format('", "'.join(scheduling_elements)))
continue
invalid_time_combos = []
@ -879,8 +901,8 @@ class Schedule(object):
if any(set(x) <= schedule_keys for x in invalid_time_combos):
log.error('Unable to use "{0}" with "{1}" options. Ignoring'
.format('", "'.join(time_elements),
'", "'.join(scheduling_elements)))
.format('", "'.join(time_elements),
'", "'.join(scheduling_elements)))
continue
if True in [True for item in time_elements if item in data]:
@ -907,8 +929,8 @@ class Schedule(object):
elif 'when' in data:
if not _WHEN_SUPPORTED:
log.error('Missing python-dateutil.'
'Ignoring job {0}'.format(job))
log.error('Missing python-dateutil. '
'Ignoring job {0}.'.format(job))
continue
if isinstance(data['when'], list):
@ -918,7 +940,7 @@ class Schedule(object):
i in self.opts['pillar']['whens']):
if not isinstance(self.opts['pillar']['whens'],
dict):
log.error('Pillar item "whens" must be dict.'
log.error('Pillar item "whens" must be dict. '
'Ignoring')
continue
__when = self.opts['pillar']['whens'][i]
@ -944,7 +966,7 @@ class Schedule(object):
try:
when__ = dateutil_parser.parse(i)
except ValueError:
log.error('Invalid date string {0}.'
log.error('Invalid date string {0}. '
'Ignoring job {1}.'.format(i, job))
continue
when = int(time.mktime(when__.timetuple()))
@ -963,8 +985,8 @@ class Schedule(object):
data['_when'] = when
seconds = when - now
# scheduled time is in the past
if seconds < 0:
# scheduled time is in the past and the run was not triggered before
if seconds < 0 and not data.get('_when_run', False):
continue
if '_when_run' not in data:
@ -1016,8 +1038,8 @@ class Schedule(object):
now = int(time.time())
seconds = when - now
# scheduled time is in the past
if seconds < 0:
# scheduled time is in the past and the run was not triggered before
if seconds < 0 and not data.get('_when_run', False):
continue
if '_when_run' not in data:
@ -1073,26 +1095,17 @@ class Schedule(object):
else:
data['_seconds'] = 0
if job in self.intervals:
if 'when' in data:
if seconds == 0:
if data['_when_run']:
data['_when_run'] = False
run = True
elif 'cron' in data:
if seconds == 1:
run = True
else:
if now - self.intervals[job] >= seconds:
run = True
if 'when' in data:
# scheduled time is now or in the past, and the run was triggered before
if seconds <= 0 and data['_when_run']:
data['_when_run'] = False
run = True
elif 'cron' in data:
if seconds == 1:
run = True
else:
if 'when' in data:
if seconds == 0:
if data['_when_run']:
data['_when_run'] = False
run = True
elif 'cron' in data:
if seconds == 1:
if job in self.intervals:
if now - self.intervals[job] >= seconds:
run = True
else:
# If run_on_start is True, the job will run when the Salt