Systemimager & puppet

I've recently read this article on puppet wiki describing how to deploy systems for puppet. The method uses kickstart and cobbler. Here I describe how to provision systems for puppet with systemimager.

The goal
Do a bare metal provisioning of a huge number of servers with systemimager.
Have them automagically registered in puppet.

Why I do it
Systemimager has certaing advantages over deploying servers with kickstart or preseed:
-> It is distro-independent
-> It scales (installations can be done with torrent which enables you to deploy several hundreds nodes in around 10 minutes - there is a paper on this on systemimager website)
-> Deployed image is always the same as opposed to kicstart. In kickstart the resulting OS is based on the state (version of packages) in repos from which it has been installed. So unless you keep a local mirror on which you control package versions, you cannot really ensure that OSes you are deploying are equal. So if you decide to re-deploy a system a month later you might find that it's different from your previous installations.
-> Images can be easily modified further on and changes can be put to the clients without interrupting their work (if you want to update a client image, just chroot into it and do apt-get update or yum update - as simple as that ;-). Then populate the change to clients with systemimager tools. They will sync the changes you've done in your image to clients.
-> You can easily monitor progress/errors of your installation with systemimager-monitor
-> With systemimager you get not only a deployment solution, but also cluster management tools like parallel shell, file syncing and - most important - si_updateclient utility. Suppose you have deployed your image to servers and forgot to put your software in /opt. You chroot into your reference image on systemimager-server and untar your software. Then you run si_updateclient on the client and voila changes are synced - your package is installled on the client. This finely complements puppet which is not designed to transfer large data with profiles.

Assumptions
-> I assume that a flavour of linux is to be deployed (any distro *NOT* using grub2 will do). Examples are based on centos.
-> You have a systemimager-server installed and running. This requires dhcp, pxe-boot and storage for images, all of them set up for systemimager. Systemimager has a set of wizards for it.
-> You have puppetmaster instance already in place.

Procedure overwiev
-> Manually install basic, mini linux on one of your new servers. You only install basic release of your linux flavour (just like "Base installation" in centos). This speeds up deployments as the image hass less files.
-> Prepare it for puppet
-> Have its image retrieved by systemimager
-> Deploy the image to other systems
-> Register all systems in puppet

Step 1: Install linux reference image
No comments here - just use your distro iso ;-)

Step 2: Modify the OS to operate with puppet
First install ntpd and configure it. Puppet uses certificates for security. It's likely that hwclock on new servers does not show correct time. So the csr generated by puppet might be valid somewhere in the future or far in the past. Even if it is signed by the puppetmaster, It will not be valid at the time of deployment.
Install puppet client and configure it to point to your puppetmaster & start at boottime.
Install systemimager-client.
Edit /etc/systemimager/updateclient.local.exclude and add /var/lib/puppet/ (if you do further management using systemimager suite, contents of this directory will be left untouched).
Configure passwordless ssh to your clients from systemimager-server. Generate ssh-keys without passphrase (or have a passphrase and further use ssh-agent to cache it) on systemimager-server. Copy ./root/ssh/id-rsa.pub to your clients to /root/.ssh/authorized_keys
Do further modifications as you like.

Step 3: Retrieve golden client image with systemimager
Please, see systemimager manual for details. This is general procedure:

On the systemimager-server:
/etc/init.d/systemimager-server-rsyncd start

On the "golden client":
si_prepareclient --server systemimager-server-ip

On the systemimager-server:
si_getimage --image img-name --golden-client client-ip-addr

The image is stored in a plain dir in /var/lib/systemimager/images/. You can chroot into it and adjust if you forgot something in step 2.

Step 4: Deploy the image to other systems
On the systemimager-server prepare other clients to pxe-boot:
si_mkclientnetboot --netboot --clients ip-list-of-nodes --image img-name

This command generates dhcp,pxe,tftp configuration for your clients so that they install the image next time they boot.

Reboot your new servers and watch them deploying the image ;-) (You might have time for a cup of coffe here unless you are using torrents for deployment which is extremely fast ;-)

After the last node is deployed, run:
si_mkclientnetboot --localboot --clients ip-list-of-nodes
This makes nodes boot from local hdd instead of pxe.

Step 5: Register new clients with puppet
After reboot, all you need to do is to sign new nodes' certificates as they appear. They are ready for puppet configuration. If you have problems at this stage (not all clients appear in puppet etc.), you may use parallel shell from systemimager to troubleshoot (just like: si_psh --hosts 'host_list' 'puppetd --verbose -o'). For this stuff you enabled passwordless login in your image.


Summary
I think the procedure described here is a more versatile replacement for kickstart and preseed instalations. Besides, systmemimager a great cluster management tool by itself.
-> It's faster and less complicated.
-> You don't need a local copy of your repo.
-> Easy to fine tune your images (no scripting for this as it is with kickstart).
-> Systemimager configures pxe, dhcp, tftp stuff for you.
-> If you have a homogenic cluster (HPC worker nodes are a good example) not so big, you may find that you don't even need puppet to manage it. Systemimager will do.

I mention systemimager in some of my posts. Please, check them out on the tag cloud.
Comments are very welcome as usual ;-)