PuppetLabs Primer

Scalable Systems
Management with
Puppet
Nick Jones and Stephen McNally
HPC Operations Group
August 2, 2010
National Institute for
Computational Sciences
NICS is a collaboration between UT and ORNL
Awarded the NSF Track 2B ($65M)
Phased deployment of Cray XT systems
Staffed with 25 FTEs, funding for 15 more
Total JICS funding ~$92M

#4 Top500
June 2010
4 of 77
Topics
Challenges that System Administrators Face
Why Puppet?
Puppet Installation and Configuration
Managing Your Infrastructure with Puppet
Advanced Puppet Topics
Hands On Exercises

5 of 77
Wireless

GlobalMeetingWireless

Passcode: TG10
Challenges that
System
Administrators
Face
7 of 77
What do Systems Administrators do?
Watch YouTube
Eat Fritos and Mountain Dew
Have big warm fuzzy secret heart
Rearrange Netflix queue

8 of 77
Case Studies : Your Prototypical
SysAdmin
Has to manage lots of servers
Responsible for installation, maintenance,
updatesthe whole shebang
Lets call him (or her) Geppetto
9 of 77
Case Study 1 : Sudo
Organization gets new interns
Need to give interns sudo privileges on
appropriate machines

10 of 77
Case Study 1 : Sudo
The old way:
Edit the sudoers file on one server
Use scp, pdsh, rsync or some combination of the
above to deploy to necessary systems

Pitfalls:
What if you have different distros, and they store
the sudoers file in different locations?
How do you detect errors?

11 of 77
Case Study 1 : Sudo
The Puppet way:
Edit the sudoers file on one server
Every other server automatically pulls down the
updated Puppet file and installs it into the proper
location for its distro

Advantages:
Easy
Centralized
Logged

12 of 77
Case Study 2 : iptables
Need to collaborate with outside group
Must allow them SSH access to internal
network

13 of 77
The old way:
Update the iptables rules to allow access from the
right ip ranges
Copy the rules to each machine
Restart iptables on each machine, and check that
the rule worked

14 of 77
The Puppet way:
Update the iptables rule to allow access from the
right ip ranges
Tell Puppet to ensure that the iptables service is
running
Each client machine automatically pulls the
updated iptables rule
Each client intelligently knows that iptables must
be restarted after a rule update, and it does this
automatically using dependencies

15 of 77
Case Study 3 : Doomsday
Centralized web server goes down due to
hardware failure
Must get the website back up quickly

16 of 77
The old way:
Multiple sysadmins get stressed out
Reinstall the server and reconfigure by hand
Restore Apache configs by hand from tape or other
backup (you did backup right?)

17 of 77
The Puppet way:
Take any other spare server
Change the MAC address in DHCP to the new
server
Puppet automatically enforces and deploys the
new configuration on the new server
Installs and updates all necessary configurations,
including ensuring that the proper services are
running

Why Puppet?
19 of 77
What is Puppet?
Puppet is a configuration management utility
Puppet is not the only configuration
management utility

20 of 77
Puppet Competitors
cFengine
Bcfg2
Chef
..and the list goes on

21 of 77
Puppet vs cFengine
cFengine focuses on managing textfiles
Puppet focuses on managing constructs like
users, services, and packages

22 of 77
Puppet vs cFengine
Puppet is model driven
The lowest layer is responsible for resource
modeling
Eg. User on Solaris vs User on Linux

The language handles high level relationships
Think users instead of /etc/passwd

23 of 77
Puppet Language
Puppet uses Ruby
Puppets Language is declarative
You specify the configuration Puppet
handles the implementation
Must use detailed specification
Need things like dependencies to get the full power
of Puppet

24 of 77
Layers
Image: Official Puppet Documentation
25 of 77
How does Puppet work?
Image: Official Puppet Documentation
26 of 77
Idempotency
Puppet configurations are idempotent they
can safely be run multiple times
By default, Puppet runs every 30 minutes

Unlike say --- kickstart --- Puppet can detect
the current state of the system
Wont make changes unless necessary

27 of 77
Cross Platform Abstraction
Puppet doesnt care about specifics unique to
your system
All handled automatically (facter!)

Manage users, files, packages, etc. the same
way regardless of OS or distro

28 of 77
Providers
Providers fulfill resources
Example: package management
Both yum and apt are valid package managers

Puppet uses providers to abstract package
management away from the user

29 of 77
Facter
How does Puppet know about your system?
Using the Ruby library Facter
Facter supports a large number of predefined facts
Custom facts can be defined

-bash-3.2$ facter
architecture => x86_64
domain => nics.utk.edu
facterversion => 1.5.7
fqdn => example.nics.utk.edu
hardwareisa => x86_64
hardwaremodel => x86_64
hostname => example
id => jones
interfaces => eth0,eth0_kraken_una,eth1
.and more
30 of 77
Linux
Centos
Debian 3.1 and later
Fedora Core 2-6
Fedora 7 and later
Gentoo Linux
Mandriva Corporate Server 4
RHEL 3 and Later
Oracle Linux
SuSe Linux 8 and later
Ubuntu 7.04 and later

31 of 77
BSD/*nix/Windows
FreeBSD 4.7 and later
OpenBSD 4.1 and later
Mac OS X
Solaris 2.6
Solaris 7 and later
Windows Support (in beta)

32 of 77
Installing Puppet
Table credit: Turnbull, Pulling Strings with Puppet
33 of 77
Installing Puppet
34 of 77
Installing Puppet
35 of 77
Puppet Installation
and Configuration
37 of 77
Manual Installation of Puppet
Puppet is a client / server based application

Puppet clients are often referred to as nodes, clients, or hosts

The Puppet server is often referred to as the puppetmaster

Not:

38 of 77

Currently Reductive Labs offers support for the following operating systems:

Linux: CentOS, Debian, Fedora, Gentoo, Mandriva, RHEL, Oracle Linux, SUSE,
and Ubuntu

BSD: FreeBSD, and OpenBSD

Other Unix: Mac OS-X, and Sun Solaris

Windows: None currently

Source: http://docs.puppetlabs.com/guides/platforms.html
39 of 77
The Puppet client must be installed on every system even
your Puppetmaster

Most platforms will use the default package manager to install
Puppet

If you use a package management system (i.e. - yum) you will
automatically get most prerequisite libraries

Facter is not installed by default when using the package
manager

Reductive Labs provides source tar balls in case you want to
torture yourself

40 of 77
Installing Puppet on CentOS / RHEL
Ensure that your package manager (yum) is configured to
communicate with the EPEL repo.

# rpm -Uvh
http://download.fedora.redhat.com/pub/epel/5/x86_64/epel-release-5-
3.noarch.rpm

# yum repolist

Should list epel in the left hand column
41 of 77
Installing Puppet on CentOS / RHEL
# yum install -y puppet (installs the client)

# yum install -y puppet-server (installs the server)

# yum install -y facter

# yum install -y ruby-doc (optional if you want help to work with ruby
commands)

These installs will also process other dependencies such as ruby and
ruby-libs

42 of 77
Configure Puppet on CentOS / RHEL
Once yum installation is complete you must configure Puppet

The /etc/puppet directory will be created after installation

Create a simple manifest in /etc/puppet/manifests/ called
site.pp

The puppetmaster daemon needs a syntactically correct file to
run

Example:

node default {
}

43 of 77
The default config file is located at /etc/puppet/puppet.conf

It is initially created with a basic set of options

These options control the behavior of the Puppet suite

44 of 77
Sample puppet.conf file (to see all values use # puppet
genconfig):

# cat /etc/puppet/puppet.conf
[main]
vardir = /var/lib/puppet
logdir = /var/log/puppet
rundir = /var/run/puppet
ssldir = $vardir/ssl
tagmap = /etc/puppet/tagmail.conf
reportfrom = puppet@sample.domain.com
[puppetd]
classfile = $vardir/classes.txt
localconfig = $vardir/localconfig
report = true
[puppetmasterd]
reports = tagmail
autosign = /etc/puppet/autosign.conf

45 of 77
The puppet client config looks for a system named puppet

Add an entry to DNS for the puppet master (recommended)
puppet IN CNAME puppetmaster

You could also change the config to point to a specific system
(not recommended)

Start the puppetmaster daemon
# /etc/init.d/puppetmaster start && tail f /var/log/messages

Ensure the puppetmaster daemon starts on boot
# /sbin/chkconfig puppetmaster on 2345

Basic puppetmaster installation is complete
46 of 77
To complete the configuration we must configure the client to
run on the puppetmaster

Manually start the puppet client once to test
# /etc/init.d/puppet once -v

This will generate a certificate request to the puppetmaster
# puppetca --list (will show the waiting certificate requests)

Multiple ways to resolve this
1. Setup puppetmaster to automatically sign all certificates
2. Setup puppetmaster to only sign specific certificates
3. Perform manual certificate signing each time

We use option #2

47 of 77
Setup Certificate Autosign Option #1
To setup automatic certificate signing you must specify so in
the /etc/puppet/autosign.conf file

An example of autosign.conf file that accepts every new
certificate request

# cat /etc/puppet/autosign.conf

*.sample.domain.com

Pros will automatically sign certs, when reinstalling a system
the autosign.conf file doesnt need to be updated

Cons security risk, not good to automate the certificate
signing mechanism

48 of 77
To setup restrictive automatic signing we will use the
/etc/puppet/autosign.conf file

Example autosign.conf file

# cat /etc/puppet/autosign.conf

server1.sample.domain.com

Pros adds additional control to certificate signing, provides
greater security than option #1, is a good balance of security
and automation

Cons the autosign.conf file must be kept up to date
49 of 77
Manual certificate signing doesnt require the autosign.conf file

Once the certificate request has been made you should be
able to see a list of the waiting requests on the puppetmaster by
using:
# puppetca --list

To sign a specific request run the following:
# puppetca --sign server1.sample.domain.com
50 of 77
You may verify the signed cert by running:
# puppetca --list --all
+ server1.sample.domain.com

The + sign denotes a signed certificate

Pros most secure way to sign certificates
Cons can get cumbersome when scaling your puppet
installation
51 of 77
After the certificate is accepted you can retest by starting the
puppet client
# /etc/init.d/puppet once v

In syslog you will see entries similar to the following:

puppetd[18704]: Starting Puppet client version 0.25.4
puppetmasterd[18047]: Compiled catalog for server1.sample.domain.com in
0.25 seconds
puppetd[18704]: Finished catalog run in 2.96 seconds

Puppet client should start and should finish the catalog run
without any errors

52 of 77
Installation is complete!
Basic Puppet installation is complete!
(seriously, we arent joking)

53 of 77
BREAK

Managing Your
Infrastructure with
Puppet
55 of 77
NICS Puppet Infrastructure
/etc/puppet
files/
manifests/ modules/
auth.conf
autosign.conf
fileserver.conf
puppet.conf
tagmail.conf
byhost/
classes/
nodes.pp
site.pp host1/
host2/
host3/
class1.pp
class2.pp
mod1/
manifests/
files/
templates/
init.pp
Files Folders
Placeholder Names
56 of 77
Sample site.pp

There are many ways to configure your Puppet environment

Heres one way to setup your /etc/puppet/manifests/site.pp file:

#site.pp

import classes/*.pp #This will import every .pp
#file in the classes directory

import nodes #This will import the nodes.pp file that #lives in
/etc/puppet/manifests

You can also set enterprise wide environment variables here
instead of specifically defining them within each node

Environment variables can be used in templates
57 of 77
Classes vs. Modules

Why use the classes directory and the modules directory?

Classes are more global and usually contain many different
modules

Modules are the smallest unit of measure that Puppet builds
from

Here are some sample classes that we use:

badservices.pp, cluster.pp, disable_accounts.pp, diskcheck.pp,
homedirs.pp, hpss.pp, infrastructure.pp, ipmi.pp, packages.pp,
python_env.pp, rootmail.pp, security.pp, snmp.pp,
subversion.pp, yumconfig.pp, yumreposerver.pp
58 of 77
Building Puppet Modules

We store all of our modules in /etc/puppet/modules

This is referenced in our /etc/puppet/puppet.conf file under the
puppetmaster section

Here are some of the modules we have created over time:

accounts, iptables, named, oncalldb, postfix, rt, syslogng, amie_gold, cron,
openssl, postgresql, dhcpd, grid_tools, lustre, moab, networking, otp, psacct,
subversion, drupal, nfs, passwd, puppet, sudo, cacti, httpd, mysql, pbstools,
splunk, sysctl, console, infiniband, ldap, mailman, nagios, ntp, php, ssh,
syslog

You dont have to create all of your modules up front

Work on one application (module) at a time until everything is
in Puppet

59 of 77
Building Puppet Modules

This method of storing modules separately in
/etc/puppet/modules assists in management

We can store module specific files within the module instead of
all together

If youre working on a module, you can stay in that module until
you are finished and ready to test

Inside each module, we have two main directories: manifests
and files

The file directory houses the module specific files

The manifest is where the modules definition lives (init.pp)
60 of 77
Sample module init.pp

Below is a simple example of the named (DNS) class:

class named {
package {
bind: ensure => installed;
bind-chroot: ensure => installed;
}
service { "named":
enable => true,
ensure => running,
require => [ Package["bind"], Package["bind-chroot"] ];
}
file { "/var/named/chroot/etc/rndc.key":
path => "/var/named/chroot/etc/rndc.key",
source => "puppet:///named/var/named/chroot/etc/rndc.key",
require => [ Package["bind"], Package["bind-chroot"] ];
}
}
61 of 77
Module Files

What goes in the files directory?

These files can be anything from configuration files to custom
scripts

Gives the system administrator the ability to control any file

This doubles as a security measure and as a standardization
practice

We store our files in the same directory structure as the final
configuration

The puppet user must be able to read the files

Simplifies system administration

62 of 77
Sample Infrastructure Class

Our infrastructure.pp class uses several different modules

class infrastructure {
include puppet
include yumconfig
include syslog
include security
if $ntp_server {
include ntp::server
} else {
include ntp
}
include ldap
include console
include badservices
}
63 of 77
What is a template?

Puppet templates are flat files containing Embedded Ruby
(ERB) variables

An example ssh_config template follows:

Protocol <%= ssh_Protocol %>
<% if ssh_ListenAddress != "" %>
ListenAddress <%= ssh_ListenAddress %>
<% end -%>

SyslogFacility <%= ssh_SyslogFacility %>
RSAAuthentication <%= ssh_RSAAuthentication %>
PubkeyAuthentication <%= ssh_PubkeyAuthentication %>
UsePAM <%= ssh_UsePAM %>
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL
X11Forwarding <%= ssh_X11Forwarding %>
NoneEnabled no
64 of 77
ERB variable declaration

Ruby ERB variables can be set in different places

They can be specified in the class that calls them:

class resolv {
$searchpath = "testing.com"
$nameservers = ["192.168.0.1", "192.168.0.2"]

file { "resolv.conf":
name => "/etc/resolv.conf",
content => template("resolv-template.erb")
}
}
65 of 77
ERB variable declaration

Or you can set them for each node in
/etc/puppet/manifests/nodes.pp

node server1 {
$nameservers = ["192.168.0.1", "192.168.0.2"]
include resolv
}
node server2 {
$nameservers = ["192.168.0.1", "192.168.0.2"]
include resolv
}

66 of 77
Hands on Exercises

Install and configure Puppet client/server

Create a basic MySQL client module

Create a basic MySQL server module
67 of 77

Install and Configure Puppet client /
server
Using the handouts install and configure the Puppetmaster and
Puppet client

These systems are CentOS 5.5 virtual machines

Login and pw will be given
68 of 77

Create a module
Now that we have Puppet up and running, create a MySQL
client module using the handout

Once you have created the MySQL client module, we would
then like to work through creating a MySQL server module
Advanced Puppet
Topics
70 of 77
Module Dependency

Module dependencies can serialize module installation

Handy when an application needs to have certain files in place
before installing the rest

Puppet runs in a shotgun manner

Different pieces of each class are installed a completely
different times

This is where module dependency comes in

Example: YUM repositories and application specific installs
71 of 77
Intra-module dependencies

Most of the time dependencies are in the context of the same
module

The application must be installed before it is configured

class rt {
package {"rt3":
ensure => installed,
}
file {
"/etc/rt3/RT_SiteConfig.pm":
source => "puppet:///rt/etc/rt3/RT_SiteConfig.pm",
ensure => present,
owner => apache, group => apache, mode => 640,
require => Package[rt3];
}
}
72 of 77
Inter-module dependencies

The more complex your Puppet environment becomes the
greater the need for inter-module dependencies are.

This is where a specific resource defined in module A is
dependent on a resource in module B

class rt {
package {
"rt3":
ensure => installed,
require => Yumrepo[epel];
"mod_fcgid":
ensure => installed;
"perl-Authen-PAM":
ensure => installed;
}
}

73 of 77 73
Module Inheritance

Inherit statements do exactly what they sound like they do

Come in handy when you only need to change something
specific for a node or two

class ntp::verne inherits ntp {

file { "/etc/sysconfig/ntpd":
ensure => file,
source => "puppet:///verne/etc/sysconfig/ntpd",
owner => root,
group => root,
mode => 0644,
notify => [ Service[ntpd], ],
}
}
74 of 77 74
Optimize your MySQL modules for
use with dependencies

HINT - The MySQL server module should have an inheritance
from the MySQL client module
75 of 77 75
Proper Cluster Management

Clusters should be managed with classes not modules

Heres an example of our /etc/puppet/manifests/cluster.pp

class cluster {
include puppet
include yumconfig
include syslog
include sysctl
include ntp
include homedirs
include ldap
include console
include psacct
include security
}
76 of 77 76
Proper Cluster Management

More recent version of Puppet support ruby regex

Heres an example of how to efficiently manage a cluster of 10

node /^clusternode[0-9]+/ {
$disable_ipv6 = False
include cluster
include kernel-utils
boot_kernel { "2.6.18-164.15.1.el5": }
}

To test out your ruby regex use:
http://rubular.com/
77 of 77 77
Questions??

Nick Jones jones@nics.utk.edu

Stephen McNally smcnally@utk.edu

PuppetLabs Primer

Transféré par

Informations du document

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

PuppetLabs Primer

Transféré par

Droits d'auteur :

Formats disponibles

Scalable Systems

Vous aimerez peut-être aussi