Thursday, November 10, 2022

pandas Series

A pandas Series is a 1D labeled array. By default, elements in a Series are labeled with integers according to their position, like in a Python list.

However, you can specify custom labels instead. These labels need not be unique, but they must be of a hashable type, such as integers, floats, strings, or tuples.

The elements of a Series can be of any type (integers, strings, floats, Python objects, and so on), but a Series works best if all its elements are of the same type. Ultimately, a Series may become one column in a larger DataFrame, and it’s unlikely you’ll want to store different kinds of data in the same column. 

Creating a Series

There are several ways to create a Series. In most cases, you feed it some kind of 1D dataset. Here’s how you create a Series from a Python list:

import pandas as pd

data = ['Jeff Russell','Jane Boorman','Tom Heints']

emps_names = pd.Series(data)

print(emps_names)

You start by importing the pandas library and aliasing it as pd. Then you create a list of items to be used as the data for the Series. Finally, you create the Series, passing the list in to the Series constructor method.

This gives you a single list with numeric indices set by default, starting from 0:

0 Jeff Russell

1 Jane Boorman

2 Tom Heints

dtype: object

The dtype attribute indicates the type of the underlying data for the given Series. By default, pandas uses the data type object to store strings.

You can create a Series with user-defined indices as follows:

data = ['Jeff Russell','Jane Boorman','Tom Heints']

emps_names = pd.Series(data,index=[9001,9002,9003])

print(emps_names)

This time the data in the emps_names Series object appears as follows:

9001 Jeff Russell

9002 Jane Boorman

9003 Tom Heints

dtype: object

You start by importing the pandas library and aliasing it as pd. Then you create a list of items to be used as the data for the Series. Finally, you create the Series, passing the list in to the Series constructor method .

This gives you a single list with numeric indices set by default, starting from 0:

0 Jeff Russell

1 Jane Boorman

2 Tom Heints

dtype: object

The dtype attribute indicates the type of the underlying data for the given Series. By default, pandas uses the data type object to store strings.

You can create a Series with user-defined indices as follows:

data = ['Jeff Russell','Jane Boorman','Tom Heints']

emps_names = pd.Series(data,index=[9001,9002,9003])

print(emps_names)

This time the data in the emps_names Series object appears as follows:

9001 Jeff Russell

9002 Jane Boorman

9003 Tom Heints

dtype: object

Share:

Monday, November 7, 2022

Using NumPy Statistical Functions

NumPy’s statistical functions allow you to analyze the contents of an array. For example, you can find the maximum value of an entire array or the maximum value of an array along a given axis.

Let’s say you want to find the maximum value in the salary_bonus array you created in the previous post. You can do this with the NumPy array’s max() function:

print(salary_bonus.max())

The function returns the maximum amount paid in the past three months to any employee in the dataset:

3400

NumPy can also find the maximum value of an array along a given axis. If you want to determine the maximum amount paid to each employee in the past three months, you can use NumPy’s amax() function, as shown here:

print(np.amax(salary_bonus, axis = 1))

By specifying axis = 1, you instruct amax() to search horizontally across the columns for a maximum in the salary_bonus array, thus applying the function across each row. This calculates the maximum monthly amount paid to each employee in the past three months:

[3400 3200 3000]

Similarly, you can calculate the maximum amount paid each month to any employee by changing the axis parameter to 0: 

print(np.amax(salary_bonus, axis = 0))

The results are as follows:

[3200 3400 3400] 

Share:

Thursday, November 3, 2022

Performing Element-Wise Operations on NumPy arrays

It’s easy to perform element-wise operations on multiple NumPy arrays of the same dimensions. For example, you can add the base_salary and bonus arrays together to determine the total amount paid each month to each employee:

salary_bonus = base_salary + bonus

print(type(salary_bonus))

print(salary_bonus)

As you can see, the addition operation is a one-liner. The resulting dataset is a NumPy array too, in which each element is the sum of the corresponding elements in the base_salary and bonus arrays:    

<class 'NumPy.ndarray'>

[[3200 3400 3400]

[3200 3100 3200]

[2500 3000 2900]]


Share:

Wednesday, October 26, 2022

tox

tox is a tool to automatically manage virtual environments, usually for tests and builds. It is used to make sure that those run in well-defined environments and is smart about caching them to reduce churn. True to its roots as a test-running tool, tox is configured in test environments. 

tox itself is a PyPI package usually installed in a virtual environment. Because tox creates ad hoc temporary virtual environments for testing, the virtual environment tox is installed in can be common to many projects. A common pattern is to create a virtual environment dedicated to tox.

$ python -m venv ~/.venvs/tox

$ ~/.venvx/tox/bin/python -m pip install tox

$ alias tox=~/.venvs/tox/bin/tox

It uses a unique ini-based configuration format. This can make writing configurations difficult since remembering the subtleties of the file format can be hard. However, while hard to tap, there is a lot of power that can certainly configure tests and build clear and concise runs.

One thing that tox lacks is a notion of dependencies between build steps. This means that those are usually managed from the outside by running specific test runs after others and sharing artifacts somewhat ad hoc.

A tox environment more or less corresponds to a section in the configuration file. By default, tox uses the tox.ini file.

[testenv:some-name]

.

.

.

Note that if the name of the environment contains pyNM (for example, py36), then tox defaults to using CPython, the standard Python implementation, version N.M (3.6, in this case) as the Python interpreter for that test environment.

tox also supports name-based environment guessing for more esoteric implementations of Python. For example, PyPy, an implementation of Python in Python, is supported with the name pypyNM.

If the name does not include one of the supported short names, or if there is a need to override the default, a basepython field in the section can be used to indicate a specific Python version. By default, tox looks for Python available in the path. However, if the plug-in tox-pyenv is installed in the virtual environment that tox itself is installed in, tox will query pyenv if it cannot find the right Python on the path.

Share:

Friday, October 21, 2022

Execution and utility modules

For historical reasons, execution modules go in the file roots _modules subdirectory. Similar to execution modules, they are also synchronized when state.highstate is applied and when explicitly synchronized via saltutil. sync_all.

As an example, let’s write an execution module to delete several files to simplify the state module.

def multiremove(files):

for fname in files:

__salt__['file.remove'](fname)

Note that Salt is usable in execution modules as well. However, while it can cross-call other execution modules (in this example, file) it cannot cross-call into state modules.

You put this code in _modules/multifile, and you can change the state module to have

__salt__['multifile.mutiremove'](mean_files)

instead of

for fname in mean_files:

__salt__['file.remove'](fname)

Execution modules are often simpler than state modules, as in this example. In this toy example, the execution module barely does anything except coordinate calls to other execution modules.

This is not completely atypical, however. Salt has so much logic for managing machines that all an execution module often has to do is coordinate calls to other execution modules. 

Utility

When writing several execution or state modules, sometimes there is common code that can be factored out.

This code can sit in utility modules under the root file _utils directory. It is available as the __utils__ dictionary.

As an example, you can factor out the calculation of the return value in the state module.

def return_value(name, old_files):

if len(old_files) == 0:

comment = "No changes made"

result = True

elif __opts__['test']:

comment = f"{name} will be changed"

result = None

else:

comment = f"{name} has been changed"

result = True

changes = dict(old=old_files, new=[])

return dict(

name=name,

comment=comment,

result=result,

changes=changes,

)

You get a simpler state module if you use the execution module and the utility modules.

def enforce_no_mean_files(name):

mean_files = __salt__['files.find'](name,

path="*mean*")

if len(mean_files) == 0 or __opts__['test']:

return __utils__['removal.return_value']

(name, mean_files)

__salt__['multifile.mutiremove'](mean_files)

return __utils__['removal.return_value'](name,mean_files)

In this case, you could have put the function as a regular function in the module. Putting it in a utility module was used to show how to call functions in utility modules.

Sometimes it is useful to have third-party dependencies, especially when writing new state and execution modules. This is straightforward to do when installing a minion. You just make sure to install the minion in a virtual environment with those third-party dependencies.

When using Salt with SSH, this is significantly less trivial. In that case, it is sometimes best to bootstrap from SSH to a real minion. One way to achieve that is to have a persistent state in the SSH minion directory and have the installation of the minion set a grain of completely_disable in the SSH minion. This would ensure that the SSH configuration does not crosstalk with the regular minion configuration.

Share:

Monday, October 17, 2022

Salt Extensions

Since Salt is written in Python, it is fully extensible in Python. The easiest way to extend Salt for new things is to put files in the file_roots directory on the Salt master. Unfortunately, there is no package manager for Salt extensions yet. Those files automatically get synchronized to the minions, either when running state.apply or explicitly running saltutil.sync_state. The latter is useful if you want to test, for example, a dry run of the state without causing any changes but with the modified modules.

States

State modules go under the root directory for the environment. If you want to share State modules between environments, it is possible to make a custom root and share that root between the right environments.

The following is an example of a module that ensures there are no files that have the name mean in them under a specific directory. It is probably not very useful, although making sure that unneeded files are not there could be important. For example, you might want to enforce no .git directories.

def enforce_no_mean_files(name):

mean_files = __salt__['files.find'](name,

path="*mean*")

# ...continues below...

The name of the function maps to the name of the state in the SLS state file. If you put this code in mean.py, the appropriate way to address this state would be mean.enforce_no_mean_files.

The right way to find files or do anything in a Salt state extension is to call Salt executors. In most non-toy examples, this means writing a matching pair: a Salt executor extension and a Salt state extension.

Since you want to progress one thing at a time, you use a prewritten Salt executor: the file module, which has the find function.

def enforce_no_mean_files(name):

# ...continued... 

if mean_files = []:

return dict(

name=name,

result=True,

comment='No mean files detected',

changes=[],

)

# ...continues below...

One of the things the state module is responsible for, and often the most important thing, is doing nothing if the state is already achieved. This is what being a convergence loop is all about—optimizing to achieve convergence.

def enforce_no_mean_files(name):

# ...continued...

changes = dict(

old=mean_files,

new=[],

)

# ...continues below...

You now know what the changes are going to be. Calculating it here means you can guarantee consistency between the responses in the test vs. non-test mode.

def enforce_no_mean_files(name):

# ...continued...

changes = dict(

if __opts__['test']:

return dict(

name=name,

result=None,

comment=f"The state of {name} will be

changed",

changes=changes,

)

# ...continues below...

The next important responsibility is to support the test mode. It is considered a best practice to always test before applying a state. You want to clearly articulate the changes that this module does if activated.

def enforce_no_mean_files(name):

# ...continued...

changes = dict(

for fname in mean_files:

__salt__['file.remove'](fname)

# ...continues below...

In general, you should only be calling one function from the execution module that matches the state module. Since you are using file as the execution module in this example, you call the remove function in a loop.

def enforce_no_mean_files(name):

# ...continued...

changes = dict(

return dict(

name=name,

changes=changes,

result=True,

comment=f"The state of {name} was

changed",

)

# ...continues below...

Finally, you return a dictionary with the same changes as those documented in the test mode but with a comment indicating that these have already run.

This is the typical structure of a state module: one (or more) functions that accept a name (and possibly more arguments) and then return a result. The structure of checking if changes are needed and whether you are in test mode, and then performing the changes is also typical.

Share:

Wednesday, October 12, 2022

py renderer

Let’s indicate that a file should be parsed with the py renderer with #!py at the top.

In that case, the file is interpreted as a Python file. Salt looks for a run function, runs it, and treats the return value as the state.

When running, __grains__ and __pillar__ contain the grain and pillar data.

As an example, you can implement the same logic with a py renderer.

#!py

def run():

if __grains__['os'] == 'CentOS':

package_name = 'python-devel'

elif __grains__['os'] == 'Debian':

package_name = 'python-dev'

else:

raise ValueError("Unrecognized operating

system",

__grains__['os'])

return { package_name: dict(pkg='installed') }

Since the py renderer is not a combination of two unrelated parsers, mistakes are sometimes easier to diagnose.

You get the following if you reintroduce the bug from the first version.

#!py

def run():

if __grains__['os'] == 'CentOS':

package_name = 'python-devel'

elif __grains__['os'] == 'Debian':

package_name = 'python-dev'

return { package_name: dict(pkg='installed') }

In this case, the result is a NameError pinpointing the erroneous line and the missing name.

The trade-off is that reading it in YAML form is more straightforward if the configuration is big and mostly static.

Share:

Friday, September 30, 2022

Salt Formats

So far, the example SLS files were YAML files. However, Salt interprets YAML files as Jinja templates of YAML files. This is useful for customizing fields based on grains or pillars.

For example, the name of the package containing the things you need to build Python packages differs between CentOS and Debian. The following SLS snippet shows how to target different packages to different machines in a heterogeneous environment.

{% if grains['os'] == 'CentOs' %}

python-devel:

{% elif grains['os'] == 'Debian' %}

python-dev:

{% endif %}

pkg:

- installed

It is important to notice that the Jinja processing step is completely ignorant of the YAML formatting. It treats the file as plain text, does the formatting, and then Salt uses the YAML parser on the result. This means that Jinja can make an invalid file only in some cases. Indeed, you embedded such a bug in the preceding example. If the OS is neither CentOS nor Debian, the result would be an incorrectly indented YAML file, which fails to parse in strange ways.

To fix it, you want to raise an explicit exception.

{% if grains['os'] == 'CentOs' %}

python-devel:

{% elif grains['os'] == 'Debian' %}

python-dev:

{% else %}

{{ raise('Unrecognized operating system',

grains['os']) }}

{% endif %}

pkg:

- installed

This raises an exception at the right point if a machine is added to the roster that is not one of the supported distributions. Otherwise, the YAML would be incorrectly formatted. In that case, the symptom would be that Salt would complain a parse error the YAML file, making it harder to troubleshoot the issue.

Such care is important whenever doing something non-trivial with Jinja because the two layers, the Jinja interpolation, and the YAML parsing, are not aware of each other: Jinja does not know it is supposed to produce YAML, and the YAML parser does not know what the Jinja source looked like.

Jinja supports filtering to process values. Some filters are built into Jinja, but Salt extends them with a custom list. Among the interesting filters is YAML_ENCODE. Sometimes you need to have a value in the .sls file, which is YAML itself; for example, the content of a YAML configuration file that you need to be copied over.

Embedding YAML in YAML is often unpleasant; special care must be given to proper escaping. With YAML_ENCODE, it can encode values written in the native YAML.

For a similar reason, JSON_ENCODE_DICT and JSON_ENCODE_LIST are useful for systems that take JSON as input. The list of custom filters is long, and this is one of the frequent things that changes from release to release. The canonical documentation is on the Salt documentation site, docs.saltstack.com, under Jinja → Filters.

Until now, we referred to SLS files as files that are processed by Jinja and then YAML; however, this is inaccurate. It is the default processing, but it can override the processing with a special instruction. Salt only cares that the final result is a YAML-like (or, equivalently in our case, JSON-like) data structure: a dictionary containing recursively dictionaries, lists, strings, and numbers.

Converting the text into such a data structure is called rendering in Salt parlance. This is opposed to common usage, where rendering means transforming to text and parsing means transforming from text, so it is important to note when reading Salt documentation.

A thing that can do rendering is a renderer. It is possible to write a custom renderer, but the most interesting is the py renderer among the built-in renderers.

Share:

Friday, September 23, 2022

Salt Concepts

Salt introduces quite a bit of terminology and quite a few concepts.A minion is the Salt agent. Even in the agentless SSH-based communication, there is still a minion. The first thing that Salt does is send over code for a minion and then start it.

A Salt master sends commands to minions. A Salt state is a file with the .sls extension, which contains state declarations.

name_of_state:

    state.identifier:

        - parameters

        - to

        - state

For example:

process_tools:

    pkg.installed:

    - pkgs:

    - procps

This ensures the procps package (which includes the ps command among others) is installed. Most Salt states are written to be idempotent to have no effect if they are already in effect. For example, if the package is already installed, Salt does nothing.

Salt modules are different from Python modules. Internally, they do correspond to modules, but only some modules.

Unlike states, modules run things. This means that there is no guarantee, or even attempt at, idempotence. Often, a Salt state wraps a module with some logic to decide whether it needs to run the module; for example, before installing a package, pkg.installed checks if the package is already installed.

A pillar is a way of attaching parameters to specific minions, which different states can then reuse. If a pillar filters out some minions, then these minions are guaranteed to never be exposed to the values in the pillar. This means that pillars are ideal for storing secrets since they are not sent to the wrong minions.

For better protection of secrets, it is possible to use gpg to encrypt secrets in pillars. Since gpg is based on asymmetric encryption, it is possible to advertise the public key; for example, in the same source control repository that holds the states and pillars.

This means anyone can add secrets to the configuration, but the private key is needed, on the master, to apply those configurations.

Since GPG is flexible, it is possible to target the encryption to several keys. As a best practice, it is best to load the keys into a gpg-agent.

When the master needs the secrets, it uses gpg, which communicates with the gpg-agent. This means the private keys are never exposed to the Salt master directly.

In general, Salt processes directives in states in order. However, a state can always specify require. When specifying dependencies, it is best to have the dependent state have a custom, readable name. This makes dependencies more readable.

Share:

Monday, September 19, 2022

Desired state in Salt

The technical term for the desired state in Salt is highstate, which is shortened from high-level state. It describes the goal of the state. The name is a frequent cause of confusion because it seems to be the opposite of a low state, which is described almost nowhere.

The low states, or the low-level states, are the steps Salt takes to get to the goal. Since the compilation of the goal to the low-level states is done internally, nothing in the user-facing documentation talks about a low state, thus leading to confusion.

The following applies the desired state.

$ salt '*' state.highstate

Because a lot of confusion is caused by the name highstate, an alias was created.

$ salt '*' state.apply

Again, both do the same thing: figure out the desired state for all machines and then issue commands to reach it.

The state is described in SLS files. These files are usually in the YAML format and describe the desired state.

The usual way to configure is one file top.sls, which describes which other files apply to which machines. The top.sls name is used by default as the top-level file.

A simple homogenous environment might be as follows.

# top.sls

base:

    '*':

        - core

        - monitoring

        - kubelet

This example would have all machines apply the configuration from core.sls (presumably, making sure the basic packages are installed, the right users are configured, etc.), from monitoring.sls (presumably, making sure that tools that monitor the machine are installed and running), and kubelet.sls, defining how to install and configure the kubelet.

Indeed, much of the time, Salt configures machines for workload orchestration tools such as Kubernetes or Docker Swarm.

Share:

Thursday, September 15, 2022

Salt local testing

For testing locally, use salt-call --local instead of salt '*'. This needs privileged access, so it should probably be done in a VM or a container.

The following command sends a ping command to all the machines on the roster (or, later, all connected minions).

$ salt '*' test.ping

They are all supposed to return True. This command fails if machines are unreachable, SSH credentials are wrong, or other common configuration problems.

Because this command does not affect the remote machines, it is a good idea to run it first before starting to perform any changes. This ensures that the system is correctly configured.

Several other test functions are used for more sophisticated checks of the system.

The test.false command intentionally fails, which is useful to see what failures look like. For example, when running Salt via a higher-level abstraction, such as a continuous deployment system, this can be useful to see visible failures (for example, send appropriate notifications).

The test.collatz and test.fib functions perform heavy computations and return the time it took and a result. It is used to test performance; for example, this might be useful if machines dynamically tune CPU speed according to available power or external temperature. You want to test whether this is the cause of performance problems.

On the salt command line, many things are parsed into Python objects. The interaction of the shell parsing rules and the Salt parsing rules can sometimes be hard to predict. The test.kwarg command can be useful when checking how things are parsed. It returns the value the dictionary passed in as keyword arguments; for example, the following returns the dictionary of the keywords.


Since the combination of the shell parsing rules and the Salt parsing rules can be, at times, hard to predict, this is a useful command to be able to debug those combinations and figure out what things are over- or underquoted.

Instead of '*' you can target a specific machine by logical name. This is often useful when seeing a problem with a specific machine. It allows a quick feedback mechanism when trying various fixes (for example, changing firewall settings or SSH private keys).

While testing that the connection works well is important, the reason to use Salt is to control machines remotely. While the main usage of Salt is to synchronize to a known state, Salt can also be used to run ad hoc commands.

$ salt '*' cmd.run 'mkdir /src'

This causes all connected machines to create a /src directory. More sophisticated commands are possible, and it is possible to only target specific machines.

Share: