Automation with Salt (SaltStack)

Unknown — Thu, 18 Apr 2024 00:00:00 +0000
Salt - also known as SaltStack - is an orchestration tool used for automated configuration management and simultaneous task execution on remote machines. While salt-ssh</code> is included in recent versions for agentless operation, Salt typically makes use of agents (minions) installed on configured managed systems, which are controlled from a central command server (master) This guide walks through the steps that are required to establish communication between a master and multiple minions in an agent-based architecture, and provides instructions on how to perform tasks remotely using grains, modules and states. Further information can be found at docs.saltproject.io</a>.
In order to better demonstrate the capabilities of Salt, typically, a setup with more than one minion machine is required. For the purposes of this guide, I will be working with a two machines running CentOS Stream 9, and one machine running Ubuntu 22.04. One of the CentOS machines will serve as the master, from which commands will be passed to the minions, while the other CentOS machine and the Ubuntu machine will be our minions. I will be running all three systems as virtual machines (VMs) in VirtualBox on bridged network mode. This setup can be achieved using physical machines and different Linux distributions as well (some instructions such as package manager commands may be different), so feel free to do so. What is important is that all three machines have different IP and MAC addresses, and are on the same network. Cloned VMs will not work.

Initial Requirements</h3>
To begin, we need to download disk images for CentOS Stream</a> and Ubuntu</a>.
We can then create the three virtual machines we need by clicking "New" and following the setup instructions in VirtualBox, and install the operating systems by going through with the graphical installation inside the VMs.
These processes are fairly straightforward and the default settings are most likely fine, but feel free to customize as you see fit.</p>
</div>
Next, we need to configure bridged mode for all the network interfaces.
The default setting uses NAT (network address translation), which puts every VM on its own separate network.
Bridged mode will allow the VM to work with our network interface using the device driver on the host machine, assign it an IP address on the same network using DHCP, and filter packets accordingly.
Therefore, all of our VMs will be on the same network, of which our host machine is also a part.
After going to "Settings", we can configure bridged mode in the "Network" tab shown below.
For all three VMs, we need to select the "Bridged adapter" setting and choose our regular network interface (which is a wireless interface in my case).</p>
</div>
Installing Salt</h3>
To proceed, Salt needs to be installed.
The project website has instructions for installation using the bootstrap script.
Installation needs to be done on all three VMs.</strong></p>
Download the bootstrap script:</p>
root@...:</span></span> ~</span></span>$ curl -</span>o</span> install-salt.sh -</span>L</span> https://bootstrap.saltproject.io</span>
</span></code></pre>
Make the script executable:</p>
root@...:</span></span> ~</span></span>$ chmod +x install-salt.sh</span>
</span></code></pre>

For the CentOS minion, simply run the script:</p>
root@centos:</span></span> ~</span></span>$ ./install-salt.sh</span>
</span></code></pre>
For the Ubuntu minion, run with the -P</code> option to install needed packages through pip</code>:</p>
root@ubuntu</span></span> ~</span></span>$ ./install-salt.sh -</span>P</span></span>
</span></code></pre>
For the master, run with the -M</code> option to install master services as well:</p>
root@master</span></span> ~</span></span>$ ./install-salt.sh -</span>M</span></span>
</span></code></pre>
</div>
Setting up the Salt Communication</h3>
In order to set up the connection between the master and the minions, we need to use the IP of our CentOS master machine, which can be obtained by running ifconfig</code>.</p>
root@master:</span></span> ~</span></span>$ ifconfig</span>
</span></code></pre>
I will assume that the IP address is 192.168.100.100</code>. We need to bind this address as the interface in the master configuration file.</p>
root@master:</span></span> ~</span></span>$ vi /etc/salt/master</span>
</span></code></pre>
Find the interface</code> line, uncomment if necessary, and add the master IP address.</p>
interface</span></span>:</span> 192.</span>168.100.100</span>
</span></code></pre>
Restart the master service to apply the change.</p>
root@master:</span></span> ~</span></span>$ systemctl restart salt-master</span>
</span></code></pre>
Next, for both minions, we need to go into the minion configuration file, find the master</code> line, uncomment if necessary, add the same master IP address, and then restart the minion service for the changes to take effect.</p>
root@...:</span></span> ~</span></span>$ vi /etc/salt/minion</span>
</span></code></pre>
master</span></span>:</span> 192.</span>168.100.100</span>
</span></code></pre>
root@...:</span></span> ~</span></span>$ systemctl restart salt-minion</span>
</span></code></pre>
The minions now know the master, but the master does not know the minions. This is because Salt communications are initiated by minions, and this eliminates the need to open ports or alter SELinux configurations on the minions, minimizing the attack surface. Minions will send their keys to the master address in the configuration file (which they should already have after the changes above). We can list the incoming keys with salt-key</code>.</p>
root@master:</span></span> ~</span></span>$ salt-key -</span>L</span></span>
</span></code></pre>
To accept all keys, run the command below.</p>
root@master:</span></span> ~</span></span>$ salt-key -</span>A</span></span>
</span></code></pre>
The connection should now be established. We can confirm this by pinging the minions from the master using Salt.</p>
root@master:</span></span> ~</span></span>$ salt '</span>*'</span></span> test.ping</span>
</span></code></pre>
Salt Grains</h3>
Salt uses the grains</em> interface for obtaining information about the underlying system running a minion service.
This is crucial, since we might want to execute different commands depending on what kind of system we are working with.
Grains can provide information about the operating system, kernel, network, cpu, memory and many other properties of the system.
In fact, we can simply return the operating system families of our minions by running the following command on the master.</p>
root@master:</span></span> ~</span></span>$ salt '</span>*'</span></span> grains.get '</span>os_family'</span></span></span>
</span></code></pre>
Note:</b>
Note that we are asking for the OS family, not the OS.
Ubuntu will return "Debian" since it is a Debian-based system.
CentOS Stream, being upstream of Red Hat Enterprise Linux, will return "RedHat".
</small></div>
This is useful, as we can now selectively run commands on systems, using Salt's -G</code> option for grains, while managing everything from the master.
For example, we can run the yum</code> command - which is the package manager for Red Hat systems - on relevant minions only, without causing errors and unnecessary load on other machines.
In our case, salt should run yum on CentOS only, and make no changes on Ubuntu.
The command below will install the epel-release</code> package for CentOS, which configures the "Extra Packages for Enterprise Linux" repository.</p>
root@master:</span></span> ~</span></span>$ salt -</span>G</span> '</span>os_family:RedHat'</span></span> cmd.run '</span>yum -y install epel-release'</span></span></span>
</span></code></pre>
For a full list all grains and their data, we can run the following command.</p>
root@master:</span></span> ~</span></span>$ salt '</span>*'</span></span> grains.items</span>
</span></code></pre>
Execution Modules</h3>
Execution modules in Salt are functions that are called by the salt</code> command to perform certain tasks.
In fact, we have already used a module: grains</code> is technically an execution module which is used to interact with grains data.
We can make use of other modules following a similar syntax.
Here is one way to obtain minion IP addresses using the network</code> module.</p>
root@master:</span></span> ~</span></span>$ salt '</span>*'</span></span> network.ip_addrs</span>
</span></code></pre>
This next example makes use of the file</code> module and its file_exists</code> method to check if the .bashrc</code> file for the root</code> account exists on our systems.</p>
root@master:</span></span> ~</span></span>$ salt '</span>*'</span></span> file.file_exists /root/.bashrc</span>
</span></code></pre>
If true, we could modify the file to add a setting of our own for the next time we log in to the system directly, using the append</code> method of the same module.</p>
root@master:</span></span> ~</span></span>$ salt '</span>*'</span></span> file.append /root/.bashrc "</span>alias grep='grep --color=auto'
</span></span></span></code></pre>
Next, leveraging grain data, we can check if the epel-release</code> package was installed successfully.</p>
root@master:</span></span> ~</span></span>$ salt -</span>G</span> '</span>os_family:RedHat'</span></span> pkg.list_pkgs</span> |</span> grep</span></span> -</span>A</span> 1 "</span>epel-release"</span></span></span>
</span></code></pre>
In fact, the pkg</code> module above is a virtual</em> module.
Virtual modules can be used to provide a convenient standard interface for tasks handled by different programs on different systems.
For example, we can install the package nginx</code> on both minions with a single command.
In this case, pkg</code> will internally use apt</code> for Ubuntu, and yum</code>  for CentOS.</p>
root@master:</span></span> ~</span></span>$ salt '</span>*'</span></span> pkg.install nginx</span>
</span></code></pre>
Salt has built-in modules which can be used to perform a variety of tasks on target machines.
A full list of these modules and their usage instructions can be found on the official documentation pages.</p>
State Files</h3>
Execution modules are typically used for executing immediate tasks or gathering information, but in order to define desired states and perform a large number of changes to systems, we need to make use of state files</em>.
State files in Salt have the .sls</code> extension (Structured Layered State), and are written in the YAML format, with the addition of jinja2 templating.
By default, they are stored in the /srv/salt</code> directory.</p>
States also make use of modules, but state modules and execution modules are not the same.
The official documentation has separate pages for state modules, which can be referred to when working with Salt states.</p>
We can create a simple state file to install git</code> on both minions using the pkg</code> state module.</p>
root@master:</span></span> ~</span></span>$ vim /srv/salt/mystate.sls</span>
</span></code></pre>
git</span></span>:</span>
</span>  pkg.installed</span>
</span></code></pre>
To apply a state for all minions, we need to run the following command, giving it the name of our state file without the extension:</p>
root@master:</span></span> ~</span></span>$ salt '</span>*'</span></span> state.apply mystate</span>
</span></code></pre>
We can extend the file to install both git</code> and nginx</code>, and make it look more neat.</p>
packages</span></span>:</span>
</span>  pkg.installed</span></span>:</span>
</span>    -</span> pkgs</span></span>:</span>
</span>      -</span> git</span>
</span>      -</span> nginx</span>
</span></code></pre>
This time, when we apply the state file, nothing will be installed.
This is because both packages are already installed on the minions.
Most modules in Salt work in a smart way which ensures that the system is in the desired state.
In this case, Salt will install the desired packages if they are not present on the system, and simply verify without making changes if they already exist.
This reduces overhead, as Salt will not attempt to make changes if the target system is already in the desired state.
It also prevents certain errors which might result from overwriting files or changing existing configurations.</p>
root@master:</span></span> ~</span></span>$ salt '</span>*'</span></span> state.apply mystate</span>
</span></code></pre>
We can also work with the file</code> module in state files, which comes in handy when deploying and managing configuration changes.
To demonstrate, this time we will source a file directly from the master.
We start by making a new directory and creating an exemplary file on our master machine.</p>
root@master:</span></span> ~</span></span>$ mkdir /srv/salt/files</span>
</span>root@master:</span></span> ~</span></span>$ echo "</span>Hello Kraken."</span></span> ></span> /srv/salt/files/hello</span>
</span></code></pre>
To deliver this file to the /etc</code> directory on both of our minions, we can apply the following state.</p>
example_file</span></span>:</span>
</span>  file.managed</span></span>:</span>
</span>    -</span> name</span></span>:</span> /etc/hello</span>
</span>    -</span> source</span></span>:</span> salt://files/hello</span>
</span></code></pre>
Note:</b>
The salt://</code> prefix tells Salt to look for the file in the /srv/salt</code> directory on the master system by default.
This is not the only option; in fact, we can source remote files using http://</code> or ftp://</code> using the exact same syntax.
</small></div>
root@master:</span></span> ~</span></span>$ salt '</span>*'</span></span> state.apply mystate</span>
</span></code></pre>
Here is another state which we can append to our mystate.sls</code> file.
This one will ensure that the group morgan</code> with the group ID of 1944 exists, and the user</code> morgan with the user ID of 1944, home directory of /home/morgan</code>, default shell of /bin/bash</code> and password of captain</code> is also present on the system.</p>
morgan</span></span>:</span>
</span>  group.present</span></span>:</span>
</span>    -</span> gid</span></span>:</span> 1944</span>
</span>  user.present</span></span>:</span>
</span>    -</span> gid</span></span>:</span> 1944</span>
</span>    -</span> uid</span></span>:</span> 1944</span>
</span>    -</span> home</span></span>:</span> /home/morgan</span>
</span>    -</span> shell</span></span>:</span> /bin/bash</span>
</span>    -</span> password</span></span>:</span> captain</span>
</span>    -</span> hash_password</span></span>:</span> True</span>
</span></code></pre>
root@master:</span></span> ~</span></span>$ salt '</span>*'</span></span> state.apply mystate</span>
</span></code></pre>
Pillar Data</h3>
In the example above, declaring the password in the state file is not ideal.
This is a good time to introduce pillars</em> in Salt.
Pillars are data structures which allow confidential data to be delivered only to relevant minions in a secure way.
Pillar data stored on the master needs to be made available to the desired minions in the /srv/pillar/top.sls</code> file.</p>
Pillar files also have the .sls</code> extension, and are stored in /srv/pillar</code> by default.
To proceed, first we need to provide some data - in this case a password which we will use later - in a pillar file.</p>
root@master:</span></span> ~</span></span>$ vim /srv/pillar/secret.sls</span>
</span></code></pre>
We write the following line into the file.</p>
password</span></span>:</span> black</span>
</span></code></pre>
Next, we need to make the pillar data available to minions.
For demonstration purposes, we will make the "secret" pillar available for the CentOS minion only, and create a new user there while doing nothing for Ubuntu.
Again, we can achieve this by using grains.
Here is the top.sls</code> file for this configuration.</p>
base</span></span>:</span>
</span>  '</span>os_family:RedHat'</span></span>:</span>
</span>    -</span> match</span></span>:</span> grain</span>
</span>    -</span> secret</span>
</span></code></pre>
Note:</b>
We are working with the default environment in Salt, which is called "base".
</small></div>
In order to work with this in a state file, we can to make use of conditional statements provided by jinja2.
We go back to our mystate.sls</code> file.</p>
root@master:</span></span> ~</span></span>$ vim /srv/salt/mystate.sls</span>
</span></code></pre>
We add the following lines to the file and apply the state.</p>
{</span>% if grains</span>[</span>'</span>os_family'</span></span>]</span></span> == 'RedHat' %</span>}</span></span>
</span>kraken</span></span>:</span>
</span>  group.present</span></span>:</span>
</span>    -</span> gid</span></span>:</span> 2010</span>
</span>  user.present</span></span>:</span>
</span>    -</span> gid</span></span>:</span> 2010</span>
</span>    -</span> uid</span></span>:</span> 2010</span>
</span>    -</span> home</span></span>:</span> /home/kraken</span>
</span>    -</span> shell</span></span>:</span> /bin/sh</span>
</span>    -</span> password</span></span>:</span> {</span>{</span> pillar</span>[</span>'</span>password'</span></span>]</span></span> }</span></span>}</span></span>
</span>    -</span> hash_password</span></span>:</span> True</span>
</span>{</span>% endif %</span>}</span></span>
</span></code></pre>
root@master:</span></span> ~</span></span>$ salt '</span>*'</span></span> state.apply mystate</span>
</span></code></pre>
This will create the kraken</code> group with the group ID of 2010, and the kraken</code> user with the user ID of 2010, home directory of /home/kraken</code>, default shell of /bin/sh</code> and password of black</code>, which will be obtained from the pillar file.</p>
Salt Requisites</h3>
You might remember that the hello</code> file that we created and delivered to the minions was actually addressed to the kraken</code> user.
However, we delivered the file to the /etc</code> directory on both minions, including the one which does not have the user kraken</code>.
A wiser choice would be to put the file to the user's home directory, on the relevant machine.
One way to achieve this would be to use the requisites system to make sure that the user is present before attempting to place a file in her home directory.
This would not only prevent errors on all irrelevant minions, but it would also make sure that the user and the directory are present before the file operation by creating dependencies between states, thereby eliminating the risk of failure due to an inconsistent order of execution.
With these considerations in mind, we can rewrite the state in the following way.</p>
example_file</span></span>:</span>
</span>  file.managed</span></span>:</span>
</span>    -</span> name</span></span>:</span> /home/kraken/hello</span>
</span>    -</span> source</span></span>:</span> salt://files/hello</span>
</span>    -</span> require</span></span>:</span>
</span>      -</span> user</span></span>:</span> kraken</span>
</span></code></pre>
There are multiple requisite directives in Salt, and require</code> is only one of them.
In the final section, we will use require</code> as well as watch</code> - a requisite type which allows us to trigger certain actions when certain changes occur.</p>
Setting up a Working Web Server</h3>
As a final example, we will configure and start a Web server on the Ubuntu minion, using some of the modules we have seen so far, and a couple of new concepts.
First, we need a working nginx.conf</code> file.</p>
root@master:</span></span> ~</span></span>$ vim /srv/salt/files/nginx.conf</span>
</span></code></pre>
Below is a very minimal configuration.</p>
user www-data;
</span>worker_processes auto;
</span>
</span>events {
</span>}
</span>
</span>http {
</span>  server {
</span>    listen 80 default_server;
</span>    root /var/www/hello;
</span>    index index.html;
</span>    location / {
</span>    }
</span>  }
</span>}
</span></code></pre>
Next, we need a basic index.html</code> file.</p>
root@master:</span></span> ~</span></span>$ vim /srv/salt/files/index.html</span>
</span></code></pre>
<!</span>DOCTYPE</span> html</span>></span></span>
</span><</span>html</span>></span></span>
</span>    <</span>head</span>></span></span>
</span>        <</span>title</span>></span></span>Hello</</span>title</span>></span></span>
</span>    </</span>head</span>></span></span>
</span>    <</span>body</span>></span></span>
</span>        <</span>p</span>></span></span>Configured by Salt.</</span>p</span>></span></span>
</span>    </</span>body</span>></span></span>
</span></</span>html</span>></span></span>
</span></code></pre>
Finally, we put everything together in a Salt state file.
The state file below does the following:</p>

Ensure that our custom Web root directory (/var/www/hello</code>) exists.</li>
Source the index.html</code> file from the master.</li>
Install nginx</code>.</li>
Source the nginx.conf</code> file from the master (require nginx</code> installation).</li>
Enable the nginx</code> service (require nginx</code> installation).</li>
Reload the service every time the configuration changes.</li>
</ul>
{</span>% if grains</span>[</span>'</span>os'</span></span>]</span></span> == 'Ubuntu' %</span>}</span></span>
</span>web_root</span></span>:</span>
</span>  file.directory</span></span>:</span>
</span>    -</span> name</span></span>:</span> /var/www/hello</span>
</span>    -</span> makedir</span></span>:</span> True</span>
</span>
</span>web_index</span></span>:</span>
</span>  file.managed</span></span>:</span>
</span>    -</span> name</span></span>:</span> /var/www/hello/index.html</span>
</span>    -</span> source</span></span>:</span> salt://files/index.html</span>
</span>    -</span> require</span></span>:</span>
</span>      -</span> file</span></span>:</span> web_root</span>
</span>
</span>web_server</span></span>:</span>
</span>  pkg.installed</span></span>:</span>
</span>    -</span> name</span></span>:</span> nginx</span>
</span>  file.managed</span></span>:</span>
</span>    -</span> name</span></span>:</span> /etc/nginx/nginx.conf</span>
</span>    -</span> source</span></span>:</span> salt://files/nginx.conf</span>
</span>    -</span> require</span></span>:</span>
</span>      -</span> pkg</span></span>:</span> nginx</span>
</span>  service.running</span></span>:</span>
</span>    -</span> name</span></span>:</span> nginx</span>
</span>    -</span> enable</span></span>:</span> True</span>
</span>    -</span> reload</span></span>:</span> True</span>
</span>    -</span> watch</span></span>:</span>
</span>      -</span> file</span></span>:</span> /etc/nginx/nginx.conf</span>
</span>    -</span> require</span></span>:</span>
</span>      -</span> pkg</span></span>:</span> nginx</span>
</span>{</span>% endif %</span>}</span></span>
</span></code></pre>
We should now have a working Web server on our Ubuntu machine, accessible on port 80!</p>
</div>
Automation with Salt (SaltStack)

Linux/OpenBSD Cheatsheet

Installing and Running umurmur (for Mumble)

A Brief Apologia for Wine

Wireguard VPN for OpenBSD/Linux

A Christian Perspective on Extraterrestrial Intelligence

Hosting a Website (with OpenBSD httpd)

Deploying & Securing a VPS

New Software Project: refer-styles

New Software Project: Blackbubble

Three Arguments for the Elimination of DRM

Offline & Encrypted Personal E-Mail

wpa_supplicant: Wireless Connections

Thoughts on Feature Phones and KaiOS
