tox

tox is a tool to automatically manage virtual environments, usually for tests and builds. It is used to make sure that those run in well-defined environments and is smart about caching them to reduce churn. True to its roots as a test-running tool, tox is configured in test environments. 

tox itself is a PyPI package usually installed in a virtual environment. Because tox creates ad hoc temporary virtual environments for testing, the virtual environment tox is installed in can be common to many projects. A common pattern is to create a virtual environment dedicated to tox.

$ python -m venv ~/.venvs/tox

$ ~/.venvx/tox/bin/python -m pip install tox

$ alias tox=~/.venvs/tox/bin/tox

It uses a unique ini-based configuration format. This can make writing configurations difficult since remembering the subtleties of the file format can be hard. However, while hard to tap, there is a lot of power that can certainly configure tests and build clear and concise runs.

One thing that tox lacks is a notion of dependencies between build steps. This means that those are usually managed from the outside by running specific test runs after others and sharing artifacts somewhat ad hoc.

A tox environment more or less corresponds to a section in the configuration file. By default, tox uses the tox.ini file.

[testenv:some-name]

.

.

.

Note that if the name of the environment contains pyNM (for example, py36), then tox defaults to using CPython, the standard Python implementation, version N.M (3.6, in this case) as the Python interpreter for that test environment.

tox also supports name-based environment guessing for more esoteric implementations of Python. For example, PyPy, an implementation of Python in Python, is supported with the name pypyNM.

If the name does not include one of the supported short names, or if there is a need to override the default, a basepython field in the section can be used to indicate a specific Python version. By default, tox looks for Python available in the path. However, if the plug-in tox-pyenv is installed in the virtual environment that tox itself is installed in, tox will query pyenv if it cannot find the right Python on the path.

Read More

Execution and utility modules

For historical reasons, execution modules go in the file roots _modules subdirectory. Similar to execution modules, they are also synchronized when state.highstate is applied and when explicitly synchronized via saltutil. sync_all.

As an example, let’s write an execution module to delete several files to simplify the state module.

def multiremove(files):

for fname in files:

__salt__['file.remove'](fname)

Note that Salt is usable in execution modules as well. However, while it can cross-call other execution modules (in this example, file) it cannot cross-call into state modules.

You put this code in _modules/multifile, and you can change the state module to have

__salt__['multifile.mutiremove'](mean_files)

instead of

for fname in mean_files:

__salt__['file.remove'](fname)

Execution modules are often simpler than state modules, as in this example. In this toy example, the execution module barely does anything except coordinate calls to other execution modules.

This is not completely atypical, however. Salt has so much logic for managing machines that all an execution module often has to do is coordinate calls to other execution modules. 

Utility

When writing several execution or state modules, sometimes there is common code that can be factored out.

This code can sit in utility modules under the root file _utils directory. It is available as the __utils__ dictionary.

As an example, you can factor out the calculation of the return value in the state module.

def return_value(name, old_files):

if len(old_files) == 0:

comment = "No changes made"

result = True

elif __opts__['test']:

comment = f"{name} will be changed"

result = None

else:

comment = f"{name} has been changed"

result = True

changes = dict(old=old_files, new=[])

return dict(

name=name,

comment=comment,

result=result,

changes=changes,

)

You get a simpler state module if you use the execution module and the utility modules.

def enforce_no_mean_files(name):

mean_files = __salt__['files.find'](name,

path="*mean*")

if len(mean_files) == 0 or __opts__['test']:

return __utils__['removal.return_value']

(name, mean_files)

__salt__['multifile.mutiremove'](mean_files)

return __utils__['removal.return_value'](name,mean_files)

In this case, you could have put the function as a regular function in the module. Putting it in a utility module was used to show how to call functions in utility modules.

Sometimes it is useful to have third-party dependencies, especially when writing new state and execution modules. This is straightforward to do when installing a minion. You just make sure to install the minion in a virtual environment with those third-party dependencies.

When using Salt with SSH, this is significantly less trivial. In that case, it is sometimes best to bootstrap from SSH to a real minion. One way to achieve that is to have a persistent state in the SSH minion directory and have the installation of the minion set a grain of completely_disable in the SSH minion. This would ensure that the SSH configuration does not crosstalk with the regular minion configuration.

Read More

Salt Extensions

Since Salt is written in Python, it is fully extensible in Python. The easiest way to extend Salt for new things is to put files in the file_roots directory on the Salt master. Unfortunately, there is no package manager for Salt extensions yet. Those files automatically get synchronized to the minions, either when running state.apply or explicitly running saltutil.sync_state. The latter is useful if you want to test, for example, a dry run of the state without causing any changes but with the modified modules.

States

State modules go under the root directory for the environment. If you want to share State modules between environments, it is possible to make a custom root and share that root between the right environments.

The following is an example of a module that ensures there are no files that have the name mean in them under a specific directory. It is probably not very useful, although making sure that unneeded files are not there could be important. For example, you might want to enforce no .git directories.

def enforce_no_mean_files(name):

mean_files = __salt__['files.find'](name,

path="*mean*")

# ...continues below...

The name of the function maps to the name of the state in the SLS state file. If you put this code in mean.py, the appropriate way to address this state would be mean.enforce_no_mean_files.

The right way to find files or do anything in a Salt state extension is to call Salt executors. In most non-toy examples, this means writing a matching pair: a Salt executor extension and a Salt state extension.

Since you want to progress one thing at a time, you use a prewritten Salt executor: the file module, which has the find function.

def enforce_no_mean_files(name):

# ...continued... 

if mean_files = []:

return dict(

name=name,

result=True,

comment='No mean files detected',

changes=[],

)

# ...continues below...

One of the things the state module is responsible for, and often the most important thing, is doing nothing if the state is already achieved. This is what being a convergence loop is all about—optimizing to achieve convergence.

def enforce_no_mean_files(name):

# ...continued...

changes = dict(

old=mean_files,

new=[],

)

# ...continues below...

You now know what the changes are going to be. Calculating it here means you can guarantee consistency between the responses in the test vs. non-test mode.

def enforce_no_mean_files(name):

# ...continued...

changes = dict(

if __opts__['test']:

return dict(

name=name,

result=None,

comment=f"The state of {name} will be

changed",

changes=changes,

)

# ...continues below...

The next important responsibility is to support the test mode. It is considered a best practice to always test before applying a state. You want to clearly articulate the changes that this module does if activated.

def enforce_no_mean_files(name):

# ...continued...

changes = dict(

for fname in mean_files:

__salt__['file.remove'](fname)

# ...continues below...

In general, you should only be calling one function from the execution module that matches the state module. Since you are using file as the execution module in this example, you call the remove function in a loop.

def enforce_no_mean_files(name):

# ...continued...

changes = dict(

return dict(

name=name,

changes=changes,

result=True,

comment=f"The state of {name} was

changed",

)

# ...continues below...

Finally, you return a dictionary with the same changes as those documented in the test mode but with a comment indicating that these have already run.

This is the typical structure of a state module: one (or more) functions that accept a name (and possibly more arguments) and then return a result. The structure of checking if changes are needed and whether you are in test mode, and then performing the changes is also typical.

Read More

py renderer

Let’s indicate that a file should be parsed with the py renderer with #!py at the top.

In that case, the file is interpreted as a Python file. Salt looks for a run function, runs it, and treats the return value as the state.

When running, __grains__ and __pillar__ contain the grain and pillar data.

As an example, you can implement the same logic with a py renderer.

#!py

def run():

if __grains__['os'] == 'CentOS':

package_name = 'python-devel'

elif __grains__['os'] == 'Debian':

package_name = 'python-dev'

else:

raise ValueError("Unrecognized operating

system",

__grains__['os'])

return { package_name: dict(pkg='installed') }

Since the py renderer is not a combination of two unrelated parsers, mistakes are sometimes easier to diagnose.

You get the following if you reintroduce the bug from the first version.

#!py

def run():

if __grains__['os'] == 'CentOS':

package_name = 'python-devel'

elif __grains__['os'] == 'Debian':

package_name = 'python-dev'

return { package_name: dict(pkg='installed') }

In this case, the result is a NameError pinpointing the erroneous line and the missing name.

The trade-off is that reading it in YAML form is more straightforward if the configuration is big and mostly static.

Read More