Degrees of control

This post was inspired by what happened to me lately at work. A guy from security came in and told it would be great if we could allow only certain packages to be installed on our linux boxes. Everything what is not specified on the machine's profile would be automatically erased.

When I look at this situation, I come to think that there are times and setups when you want to control every change that happens on your server farm and sometimes you only want to control some parameters of your machines.

So there are basically two approaches:
-> "God-mode": you have a reference server image to which you introduce changes and then sync your servers to this image (changes entered manually on your servers are overwritten)
-> "modelling-mode": you say: this server must have an httpd & postfix running, also group apache needs to be present, etc. . You care only about httpd, postfix and apache group - the rest can be modified freely.

Approach 1. you can use if you run a homogenic server farm, just like an HPC cluster where you have a headnode and a number of similar worker nodes. This approach does not deal well with situations where you have a mixture of different OSes, hardware and machine types. On the upside - you always know what you are running. The security guy is always happy ;-) Also - tools used here are quite simple. All you gotta do is to sync your cients with reference image.

Approach 2. you use if you run more diverse environment (who'd suppose ;-). I mean here a bunch of large websites, serving different domains, several database configurations, proxies etc. - see here you might easily have over 10 installation types, each of them possibly running different OS, hardware etc. When you think about it, it is easy to realize that controlling this mess with approach 1 is impossible. Especially when there are several admins, each controlling his domain of expertise. It's likely that your database admins don't know your configuration tools . Also they know databases better than you. So it's reasonable only to assure that package postgres or mysql is installed on their machines and leave other system tuning up to your fellows.

Some words about tools that can be used here:

For approach 1:
-> systemimager - a cluster deployment and management suite. You store images of your servers in a central repository. They are plain directories so you can chroot into them, install some software, add users, etc. and then propagate changes to your clients. All of this is done with rsync so you don't interrupt your farm members' work.

-> startng machines with common nfs root - machines mount a common root filesystem from a NFS server. What you change on the nfs share is immediately propagated to clients

For approach 2:
I recommend running puppet + nagios. With puppet you ensure that certain aspects of your servers are the way you want them (i.e. apache installed, user apache present etc.). However puppet fails on reporting, so you need to monitor how puppet imposes your configuration with nagios checks. All the rest is in the hands of your fellow admins.

Comments and suggestions higly welcome ;-)