Chapter 1. Introduction

1.1. Introduction to Configuration Management with Puppet

1.1.1. What is Puppet?

The perfect description of what Puppet is exactly comes from the upstream product website:

	The Puppet framework provides a means to describe IT infrastructure as policy, execute that policy to build services then audit and enforce ongoing changes to the policy. Puppet helps accomplish the goal of a hands-off, automated infrastructure. The benefits of automated infrastructure go beyond policy-enforced consistency and auditing. The impact of hardware failure and other disaster scenarios can be mitigated, as services can be quickly restored by Puppet. In conjunction with virtualizaton, the ability to reliably create new systems running consistent services can be leveraged to create autoscaling applications as well as test systems identical to production environments.
	--http://reductivelabs.com/products/puppet/

1.1.2. What is Configuration Management?

Within virtually every organization, there's probably a number of systems running Linux, Solaris, Mac OS X and/or HP-UX. These systems need to be configured appropriately to be able to function properly. Some will need special drivers, and all of them will need correct DNS settings, certain packages installed and certain other packages removed, users created, and SSH host keys exchanged. The more systems, the more these diverge in the configuration they need, diverge in the way this configuration needs to be applied, and the more these configurations will show discrepancies arising over time.

More specifically, an organization may have a couple of webservers, fileservers, a DNS and a DHCP server, a number of desktop PCs, and a number of laptops. The laptops may need slightly different system configuration (no LDAP authentication, and with a VPN client installed, for example), and the desktop PCs may need different applications installed then the servers, and so forth. Yet, between, say, a hundred desktop PCs, you would want the configuration to be as similar as possible. You may want to diverge between a software developer's desktop PC and a desktop PC in Human Resources, but in essence these are desktop profiles diverging on the application level, applied upon a stable system configuration which remains the same, or similar at least.

By the time the organization grows, replaces the hardware, upgrades to another version of the operating system, or applies changes, the challenge to making everything work yet maintain a similar configuration between all nodes becomes bigger. While every attempt made to control the situation can be called a form of configuration management, the solution without a configuration management framework is often comprised of:

a number of scripts (with or without revision control), to move around files, install packages, perform daily check-ups,
NFS mounts with programs pre-installed, so that nodes can mount these NFS shares and the software needs to be provided once, in one location, for all to share,
file server shares with pre-compiled drivers, or driver sources being compiled on the nodes by scripts running on the nodes,
terminal servers or desktop servers like with FreeNX, so that configuration concentrates on a smaller number of boxes

This means that workarounds for actual (user) problems maybe require an additional if-then-else in one or the other script, and updates to programs installed require manual compilation and installation. The success rate of these solutions never reaches 100%, and as it turns out the longer such a implemented solution runs, the more exotic problems become and the more machines will fail to remain up-to-date regardless of any attempt made to fix the issue; simply because it becomes to diverse and unmaintainable.

1.1.2.1. Configuration Management

Generally speaking, with configuration management, it's about managing the configuration of one or more organizational resources in order to have it be in a state in which it can perform the operations required by, and possibly critical to, the organization's operations. In addition to that, configuration management often concerns administrative tasks as to what systems provide a service and what SLA or OLA is applicable to that service, as well as the purchase date, location of the system, responsible party, etcetera.

In this workshop though, we are not going to explore configuration management of a coffee machine. Instead we look at the computers in a network running any platform but the one from a prominent proprietary North America-based vendor. We are talking automation and further enhancement of Computer Systems Administration.

When managing the operating system and software running on mainframes, servers, desktop PCs and laptops, you may find yourself looking for answers to questions such as:

How do I manage what packages are installed on a given system?
- How do I manage the configuration of those packages (this software)?
- How do I make sure these packages are updated?
How do I make sure the services that every machine needs to run are actually running?
How do I manage monitoring the services or a machine's state?
A job needs to run periodically (maybe via crontab), but how do I make sure it is run, and how can I change or remove the job later?
Given different operating systems and operating system versions, how do I make sure I apply the correct routine for adding a user, starting a service, install/update/remove a package?

1.1.2.2. Configuration Management Requirements

This section is about what you would want Configuration Management to do for you, as a system administrator for the systems within your organization. These could very well include:

Maintain consistency across systems
Consistency across systems is key in understanding where a problem might come from and assessing where problems may be first introduced. If each and every system is unique, you may end up searching for unique aspects of the system's configuration in order to determine the cause of a problem, while if systems are mostly consistent and the exceptions to the rule are easily determined, you may have found the problem even before your users experience the consequences.
Consistency !== Equality
Of course keeping systems consistent in their configuration doesn't say all your systems should be entirely equal, because that would not be feasible for many organizations and defeat the purpose of configuration management. Needless to say though, having all systems be entirely unique defeats part of the purpose of configuration management as well.
Categorize systems
Grouping systems into categories like (for example) desktop, server and/or laptop, helps in applying changes to one category, such as installing GNOME or keeping systems up-to-date according to a schedule that may (servers) or may not (desktops, laptops) need a service or maintenance window.
Different profiles
More generally speaking, different profiles for each of these categories may be defined as well. A developer's desktop most likely has different requirements then a publicly accessible information booth at the reception desk.
Version Control
Version control lets you keep track of changes applied to the overall configuration management framework, which is important because if you are managing different aspects of a (large) number of systems, and something goes wrong, the changes applied to the configuration Puppet uses will most likely be the first clue as to what caused the new problem and lets you recover relatively fast. Additionally, version control adds a layer that also gives you the chance to perform access control, to have notifications of changes applied sent to interested people, and to branch off.
Overview of systems' tasks and services
Being able to quickly tell what a system does exactly, and how it differs from another system not only aids in performing risk assessments (impact of a given change), but may also help in determining the impact of a change beforehand, as well as determine the impact of an unexpected system or service interruption. Providing an example to the latter; if you update httpd across systems (whether tested or untested), but the new software version doesn't work as expected, a configuration management framework should be able to quickly give you an overview of impacted systems and services.
Updating systems
Some systems can be updated irregularly, such as desktop PCs, but need to be kept up-to-date nonetheless. Other systems have service and/or maintenance windows, such as servers, and thus need a very regular and strict update schema, compliant with the update policies in place.

1.1.3. Problems without Configuration Management

There's a number of challenges an organization might encounter when not implementing some form of configuration management, such as:

Different operating systems
If you have a diverse organization in terms of the operating systems your systems run, applying the same or similar configuration to a set of different operating systems is challenging in terms of adding a user or setting a password on one operating system is not the same as adding a user or setting a password on another operating system. The same applies to installing, updating or removing a package, and so forth. Additionally the more different operating systems you have, the harder managing any given system resource becomes. Some commands for day-to-day administrative tasks may be equal, or similar, but most of them are and/or behave different.
Different distributions
Although an organization may not have different distributions running right now, sooner or later, an organization will migrate from one distribution to another; That is practically inevitable. If an organization does have different distributions running, practical problems such as the location of certain files become evident, as well as different interfaces to resource-management (like adding a user with useradd or adduser).
Different versions of distributions
Different versions of distributions, or, more accurately, the different versions of the applications shipped with each distribution version, as well as the configuration settings for updated programs that come with the distributions, can form a challenge when or if the organization does not have a proper configuration management framework in place. Note that even though an organization may not have different versions of a distribution right now, at some point the organization will need to upgrade to the next available release.
Different tasks to perform
Each different system in an organization is performing one or more tasks that may be unique to the system or may be shared between a group of systems, but with many different tasks being performed throughout the organization's infrastructure, keeping track of what system performs which task, keeping these systems up to date and configuring them to have the required packages installed for each of the tasks they perform, tackling the problem becomes harder.
Different ways to perform a task
Within an organization that has multiple servers performing the same task, keeping a similar state or perform a task in a similar manner is challenging in the sense that without configuration management, you are most likely to find more and more ways to purging old files from /tmp/ and /var/tmp/, for example. The same differentiation may apply to how webservers' VirtualHosts are configured, or how a NFS share is mounted (mount options in particular).
Different nodes
This one goes to hardware-specific needs and configuration. When the systems in an organization are not all of the same brand, make and model, or each system has different harddisk layouts, or needs different videocard drivers, you are basically keeping lists and making choices based on those lists.
Different services
Different services of course are configured differently, as far as configuration file locations and syntax are concerned. However, figuring out the best way to apply certain configuration to a system for each service is less efficient without configuration management. You might adjust a script or two and/or adjust the source repository from which you pull updates to each system, but the changes may turn out to only apply to that system that needed the exception to the rule instead of focussing on a more general solution to the problem once, and apply that solution multiple times, over and over again. Overview is key here, even if your standard environment exists of numerous different profiles.
Interfaces to a system resource
This is probably the hardest one if you are not using any configuration management framework. Given different operating systems, distributions and/or distribution versions, in which case any combination of the three only makes the problem harder to solve, you are most likely to encounter so many different ways to manage a given system resource, that a simple script or routine cannot cover all of them -and remain comprehensible and maintainable. One example is adding a user to the system, and making the user a group member of several groups. You may find routines ranging from using useradd or adduser depending on the distribution used, to writing out ldifs from a template and using ldapadd or ldapmodify depending on whether the user already exists or not, and whether the user is a member of the group already.
Fast pace changes
Changes that need to be applied sooner rather then later are often only applied by the time a crontab job polls for new configuration, or when a system reboots. This does not work in cases where changes need to be applied quickly, such as when a package installed on some or all systems exposes vulnerabilities that make the system remotely exploitable.

1.1.4. Not So Technical Aspects

In addition to the problems you may encounter with or without configuration management, there's a number of problems or challenges that are not so technical, but you may want to see resolved by a configuration management utility;

Applying changes
Applying changes to multiple machines at once may become a problem depending on the size of the organization or the amount of remote and direct control that you have over your systems. There was a time when changing the DNS servers for a set of systems required one to log on to the console of each system and edit /etc/resolv.conf manually. You can see the problem become bigger if the organization does not have 20 systems, but 1200.
Keeping track of changes
Another challenge is keeping track of the changes applied to each system. Even with configuration management, errors can be made and systems may behave unexpectedly, in which case you will want to know what changed on these systems, and how to recover to an operational state. Keeping track of changes without a configuration management framework however is a little harder, but with configuration management, you have reports (changes applied to a system in a nice overview), and most advisebly you have the configuration for Puppet stored in a Source Control Management system, or SCM system, like CVS, SVN, Mercurial, or GIT.
Staging changes
Staging changes is a huge must-have in case changes are radical or might destroy a normal system's operation (even if temporary). For such changes, you would want to test the changes first, and with Puppet, you get this in the form of environments. Additionally, in case any critical component needs to change, proper Change Management then requires you to Build & Test the solution prior to implementation. This is often not a very bad idea to relieve stress in case the implemented solution does not work, especially if the change is time-constrained such as with service windows.
Exceptions to the rule
As important as keeping systems consistent is being able to name and point out the exceptions to the rules that you set. Of course, every organization has exceptions and until the point where high-availability, high-performance, load-balancing or virtualization clusters are deployed (or storage pools for that matter), no two systems are alike. Trying to keep a consistent configuration amongst all your systems doesn't change that. Note though, being able to reproduce the reasoning behind the exceptions to the rule is important in recovering from (unexpected) system or service interruptions.
Restoring a system
Restoring a system to it's normal operations often requires you to have a backup of the most recent configuration files and transactional data for the machine. Although configuration management with puppet does not facilitate the backup of transactional data, it does facilitate the backup of configuration files, taking away the rather boring task of having to manually select which items need to be backed up on a regular basis, and restored when recovering the system.