Salt Formats

So far, the example SLS files were YAML files. However, Salt interprets YAML files as Jinja templates of YAML files. This is useful for customizing fields based on grains or pillars.

For example, the name of the package containing the things you need to build Python packages differs between CentOS and Debian. The following SLS snippet shows how to target different packages to different machines in a heterogeneous environment.

{% if grains['os'] == 'CentOs' %}

python-devel:

{% elif grains['os'] == 'Debian' %}

python-dev:

{% endif %}

pkg:

- installed

It is important to notice that the Jinja processing step is completely ignorant of the YAML formatting. It treats the file as plain text, does the formatting, and then Salt uses the YAML parser on the result. This means that Jinja can make an invalid file only in some cases. Indeed, you embedded such a bug in the preceding example. If the OS is neither CentOS nor Debian, the result would be an incorrectly indented YAML file, which fails to parse in strange ways.

To fix it, you want to raise an explicit exception.

{% if grains['os'] == 'CentOs' %}

python-devel:

{% elif grains['os'] == 'Debian' %}

python-dev:

{% else %}

{{ raise('Unrecognized operating system',

grains['os']) }}

{% endif %}

pkg:

- installed

This raises an exception at the right point if a machine is added to the roster that is not one of the supported distributions. Otherwise, the YAML would be incorrectly formatted. In that case, the symptom would be that Salt would complain a parse error the YAML file, making it harder to troubleshoot the issue.

Such care is important whenever doing something non-trivial with Jinja because the two layers, the Jinja interpolation, and the YAML parsing, are not aware of each other: Jinja does not know it is supposed to produce YAML, and the YAML parser does not know what the Jinja source looked like.

Jinja supports filtering to process values. Some filters are built into Jinja, but Salt extends them with a custom list. Among the interesting filters is YAML_ENCODE. Sometimes you need to have a value in the .sls file, which is YAML itself; for example, the content of a YAML configuration file that you need to be copied over.

Embedding YAML in YAML is often unpleasant; special care must be given to proper escaping. With YAML_ENCODE, it can encode values written in the native YAML.

For a similar reason, JSON_ENCODE_DICT and JSON_ENCODE_LIST are useful for systems that take JSON as input. The list of custom filters is long, and this is one of the frequent things that changes from release to release. The canonical documentation is on the Salt documentation site, docs.saltstack.com, under Jinja → Filters.

Until now, we referred to SLS files as files that are processed by Jinja and then YAML; however, this is inaccurate. It is the default processing, but it can override the processing with a special instruction. Salt only cares that the final result is a YAML-like (or, equivalently in our case, JSON-like) data structure: a dictionary containing recursively dictionaries, lists, strings, and numbers.

Converting the text into such a data structure is called rendering in Salt parlance. This is opposed to common usage, where rendering means transforming to text and parsing means transforming from text, so it is important to note when reading Salt documentation.

A thing that can do rendering is a renderer. It is possible to write a custom renderer, but the most interesting is the py renderer among the built-in renderers.

Read More

Salt Concepts

Salt introduces quite a bit of terminology and quite a few concepts.A minion is the Salt agent. Even in the agentless SSH-based communication, there is still a minion. The first thing that Salt does is send over code for a minion and then start it.

A Salt master sends commands to minions. A Salt state is a file with the .sls extension, which contains state declarations.

name_of_state:

    state.identifier:

        - parameters

        - to

        - state

For example:

process_tools:

    pkg.installed:

    - pkgs:

    - procps

This ensures the procps package (which includes the ps command among others) is installed. Most Salt states are written to be idempotent to have no effect if they are already in effect. For example, if the package is already installed, Salt does nothing.

Salt modules are different from Python modules. Internally, they do correspond to modules, but only some modules.

Unlike states, modules run things. This means that there is no guarantee, or even attempt at, idempotence. Often, a Salt state wraps a module with some logic to decide whether it needs to run the module; for example, before installing a package, pkg.installed checks if the package is already installed.

A pillar is a way of attaching parameters to specific minions, which different states can then reuse. If a pillar filters out some minions, then these minions are guaranteed to never be exposed to the values in the pillar. This means that pillars are ideal for storing secrets since they are not sent to the wrong minions.

For better protection of secrets, it is possible to use gpg to encrypt secrets in pillars. Since gpg is based on asymmetric encryption, it is possible to advertise the public key; for example, in the same source control repository that holds the states and pillars.

This means anyone can add secrets to the configuration, but the private key is needed, on the master, to apply those configurations.

Since GPG is flexible, it is possible to target the encryption to several keys. As a best practice, it is best to load the keys into a gpg-agent.

When the master needs the secrets, it uses gpg, which communicates with the gpg-agent. This means the private keys are never exposed to the Salt master directly.

In general, Salt processes directives in states in order. However, a state can always specify require. When specifying dependencies, it is best to have the dependent state have a custom, readable name. This makes dependencies more readable.

Read More

Desired state in Salt

The technical term for the desired state in Salt is highstate, which is shortened from high-level state. It describes the goal of the state. The name is a frequent cause of confusion because it seems to be the opposite of a low state, which is described almost nowhere.

The low states, or the low-level states, are the steps Salt takes to get to the goal. Since the compilation of the goal to the low-level states is done internally, nothing in the user-facing documentation talks about a low state, thus leading to confusion.

The following applies the desired state.

$ salt '*' state.highstate

Because a lot of confusion is caused by the name highstate, an alias was created.

$ salt '*' state.apply

Again, both do the same thing: figure out the desired state for all machines and then issue commands to reach it.

The state is described in SLS files. These files are usually in the YAML format and describe the desired state.

The usual way to configure is one file top.sls, which describes which other files apply to which machines. The top.sls name is used by default as the top-level file.

A simple homogenous environment might be as follows.

# top.sls

base:

    '*':

        - core

        - monitoring

        - kubelet

This example would have all machines apply the configuration from core.sls (presumably, making sure the basic packages are installed, the right users are configured, etc.), from monitoring.sls (presumably, making sure that tools that monitor the machine are installed and running), and kubelet.sls, defining how to install and configure the kubelet.

Indeed, much of the time, Salt configures machines for workload orchestration tools such as Kubernetes or Docker Swarm.

Read More

Salt local testing

For testing locally, use salt-call --local instead of salt '*'. This needs privileged access, so it should probably be done in a VM or a container.

The following command sends a ping command to all the machines on the roster (or, later, all connected minions).

$ salt '*' test.ping

They are all supposed to return True. This command fails if machines are unreachable, SSH credentials are wrong, or other common configuration problems.

Because this command does not affect the remote machines, it is a good idea to run it first before starting to perform any changes. This ensures that the system is correctly configured.

Several other test functions are used for more sophisticated checks of the system.

The test.false command intentionally fails, which is useful to see what failures look like. For example, when running Salt via a higher-level abstraction, such as a continuous deployment system, this can be useful to see visible failures (for example, send appropriate notifications).

The test.collatz and test.fib functions perform heavy computations and return the time it took and a result. It is used to test performance; for example, this might be useful if machines dynamically tune CPU speed according to available power or external temperature. You want to test whether this is the cause of performance problems.

On the salt command line, many things are parsed into Python objects. The interaction of the shell parsing rules and the Salt parsing rules can sometimes be hard to predict. The test.kwarg command can be useful when checking how things are parsed. It returns the value the dictionary passed in as keyword arguments; for example, the following returns the dictionary of the keywords.

Since the combination of the shell parsing rules and the Salt parsing rules can be, at times, hard to predict, this is a useful command to be able to debug those combinations and figure out what things are over- or underquoted.

Instead of '*' you can target a specific machine by logical name. This is often useful when seeing a problem with a specific machine. It allows a quick feedback mechanism when trying various fixes (for example, changing firewall settings or SSH private keys).

While testing that the connection works well is important, the reason to use Salt is to control machines remotely. While the main usage of Salt is to synchronize to a known state, Salt can also be used to run ad hoc commands.

$ salt '*' cmd.run 'mkdir /src'

This causes all connected machines to create a /src directory. More sophisticated commands are possible, and it is possible to only target specific machines.

Read More

Using Salt

There are a few ways to use Salt.

• Locally: Run a local command that takes the desired steps.
• SSH: The server will ssh into clients and run commands that take the desired steps.
• Native protocol: Clients connect to the server and take whatever steps the server instructs them.

Using the ssh mode removes the need to install a dedicated client on the remote hosts since, in most configurations, an SSH server is already installed. However, Salts native protocol for managing remote hosts has several advantages.

It allows the clients to connect to the server, thus simplifying discovery. All you need for discovery is just for clients to the server. It also scales better. Finally, it allows you to control which Python modules are installed in the remote client, which is sometimes essential for Salt extensions.

If some Salt configuration requires an extension that needs a custom module, you can take a hybrid approach. Use the SSH-based configuration to bring a host to the point where it knows where the server is and how to connect to it, and then specify how to bring that host to the desired configuration.

This means there are two parts to the server: one that uses SSH to bring up the system to a basic configuration which, among other things, has a Salt client, with the second part waiting for the client to connect to send the rest of the configuration.

This has the advantage of solving the secret bootstrapping problem. You verify the client hosts SSH key using a different mechanism, and when connecting to it via Salt, inject the Salt secret to allow the host to connect to it.

When you choose the hybrid approach, there needs to be a way to find all machines. When using some cloud infrastructure, it is possible to do this using API queries; however, you need to make it accessible to Salt if you get this information.

This is done using a roster. The roster is a YAML file. The top level is the logical machine name, which is important since this is how the machine is addressed using Salt.

In ideal circumstances, all parameters are identical for the machines. The user is the SSH user. The sudo boolean states whether sudo is needed, which is almost always true. The only exception is if it is an administrative user (usually root). Since it is a best practice to avoid SSH as root, this is set to True in most environments.

The priv field is a path to the private key. Alternatively, it can be agent-forwarding to use SSH agent. This is often a good idea since it presents an extra barrier to key leakage.

The roster can go anywhere, but Salt looks for it in /etc/salt/roster by default. Putting this file in a different location is subtle. salt-ssh finds its configuration, by default, from /etc/salt/master. Since the usual reason to put the roster elsewhere is to avoid touching the /etc/salt directory, you usually need to configure an explicit master configuration file using the -c option.

Alternatively, a Saltfile can be used. salt-ssh looks to a Saltfile in the current directory for options.

salt-ssh:

config_dir: some/directory

If you put in the value. In config_dir, it looks in the current directory for a master file. You can set the roster_file field in the master file to a local path (for example, roster) to make sure the entire configuration is local and locally accessible. This can help if a version control system is managing things.

After defining the roster, it is useful to check that the Salt system is functioning. It is also possible to run the commands to test Salt locally.

Read More

SaltStack

Salt belongs to a class of configuration management systems intended to make administrating a large number of machines easier. It does so by applying the same rules to different machines, making sure that any differences in their configuration are intentional.

It is written in Python and, more importantly, extensible in Python. For example, wherever a YAML file is used, Salt allows a Python file that defines a dictionary.

Salt adopts an open source model for its core code. The code can be cloned from the Salt source code repository. There are two PyPI packages available for Salt.

• The salt package includes the client/server code. It depends on pyzmq, which, in turn, relies on the libzmq C library.
• The salt-ssh package only includes the local and SSH-based client. Because of that, it does not depend on the libzmq library. When only local/SSH support is needed, it is better to install salt-ssh.

Other than this distinction, the two packages are identical.

The Salt (or sometimes SaltStack) system is a system configuration management framework. It is designed to bring operating systems into a specific configuration. It is based on the convergence loop concept. When running Salt, it does three things.

• Calculates the desired configuration
• Calculates how the system differs from the desired configuration
• Issues commands to bring the system to the desired configuration

Some extensions to Salt go beyond the operating system concept to configure some SaaS products into the desired configuration; for example, there is support for Amazon Web Services, PagerDuty, or some DNS services (those supported by libcloud).

Since, in a typical environment, not all operating systems need to be configured the same way, Salt allows detecting properties of systems and specifying which configurations apply to which systems. Salt uses them at runtime to decide the complete desired state and enforce it.

Read More

Python data types continued.....

Python provides a set data type for storing sets of data. A set is not a sequence because it does not have a specific order.

Python also provides a single mapping data type, dict, which is used for creating dictionaries. A dictionary in Python is not a dictionary in the everyday sense, although there are some similarities between the two: A key in the dictionary maps to a particular value, enabling you to look up that value.

Understanding the Set Data Type

In Python, the set data type enables you to store multiple values in a single variable. The set data type has the following characteristics:

• It contains elements. The elements, also called members, are the discrete objects that make up the set.

• Each element is unique. A set cannot have duplicate elements. By contrast, a list or a tuple can have duplicate elements.

• It is unordered. The elements in a set have no specific order. This means you cannot refer to an element in a set by its index or position.

• It is immutable. Once you have created a set, you cannot change its existing items, but you can add further items to the set if you need to.

Understanding the Mapping Data Type

Python’s mapping category contains a single data type, dict, which is used for dictionaries. A dictionary consists of key/value pairs, with the key in each pair giving you access to set, retrieve, or modify the associated collection of information in the value.

A dictionary is unordered; you access the data by supplying the appropriate key rather than an index  value.

A dictionary is mutable, so you can change its contents after creating it.

Python’s Classes

In Python, a class is a kind of template you use for creating a new object of a particular type. You can create a class object to organize the functions and other code in a particular project.

That sounds nebulous, but if you work with office productivity software, you are likely used to a similar paradigm. For example, if you need to create many memos of the same type in Microsoft Word, you may create a custom memo template containing the layout and formatting for the memo, and perhaps some VBA code for automation. That memo template is analogous to a Python class.

Understanding the Instance Data Type

In Python, an instance is an individual object created from a particular class. For example, say you create a class that contains the functions needed to run a particular data‐aggregation and assessment task. When you want to work on that data, you create an instance of the class — or, to use the formal term, you instantiate the class.

Continuing the previous example, when you need to produce a memo, you create a new document based on your memo template rather than using the memo template itself. The document is analogous to an instance of the template class.

Understanding the Exception Data Type

In Python, an exception is an object representing an error that occurred during code. Will will see later how to work with Python’s built‐in exceptions to handle errors when they occur and how to create custom exceptions.

Read More