Thursday, May 6, 2010

Consistency in the datacenter

One of the main challenges of system administration is dealing with chaos.
This is in fact one of the features of the world we live in. Every construction degrades over time. Every organized structure turns into chaos if not maintained approprietaly.

At its birth, the server is just like an innocent child. Clean, lean,configured, doing what's it designed for. Over time, developers, sysadmins, etc.
(whoever has access to it) introduce changes to this setup. Some of them are made by you and hopefully are recorded somewhere. Some of them you probably don't authorize and don't even know they were performed. One day you find yourself saying "Where the fuck this load is comming from?" And you discover that something that was once designed to be a plain mysql server now runs apache, nfs and a whole bunch of other stuff.

Of course life of a sysadmin is hard. You got lots of machines and hardly have time to record every single thing you've done on each of them. You want certain aspects of your systems to be the same on every single server. For example you want every machine to get its time from your company's ntp server, log to a remote logging server, get its hostname from local dns, etc. etc..

This can be called configuration consistency management. Usually when talking
about large server farms people focus on rapid deployment (1000 machines in an
hour - large numbers are impressive). But in fact this is not the deployment itself which poses the greatest challenge. Maintaining configuration over time is much harder.

So how you do it? Depends on what farm you are running. If you run a homogenic
computational cluster, you probably have just one type setup that every worker node in the cluster should have. If you work for a dotcom, you are probably dealing with larger number of configs (database servers, www servers, cache servers etc.). Also if the dotcom runs more than one website, the number aforementioned server types is multiplied. Add different linux distributions, bsd, solaris to the mix and you find yourself in the middle of chaos.

In case of homogeneous server farm, you are likely to have just one configuration to maintain. Your focus is not on imposing server configuration, but rather on maintaining a consisitent server image over time. When you deal with different configurations, different websites, you probably want to control only certain aspects of every server (ntp for example), while others are left untouched and can be modified directly by people working on a certain website (users, groups, etc).

The tool I recommend for running a homogenic system farm is systemimager (http://systemimager.sourceforge.net). It is a suite which combines automated deployment (via pxe & rsync/torrent), and further server management. You install one cluster node, get its image to the systemimager server. The image is located in a directory into which you can chroot and make changes. You deploy the image on the nodes with pxe. A small linux distro loads itself into a ramdisk first. The distro creates filesystems and rsync-s the image from systemimager server. And
here comes the most interesting part about consistency. After you have deployed all your systems and you need to update them, there's no need to go to all servers,run scripts, etc. You just chroot into your image, intall software, add users, whatever. Then you rsync your image to the nodes. And voila - all your nodes are upgraded and in consistent state. Of course systemimager is smart enough and will not overwrite anything in /home, /tmp, /varetc.. Downside - AFAIK it now supports fully only linux

If you run lots of different system types, systemimager is probably not the best
choice for you, as you would need to have a separate image for every single node
type. Puppet (http://reductivelabs.com) is probably better here. Puppet is a language to describe system configuration. You can specify which packages must be present, users to be added, services to be running. A process "puppetd" runs on every node and and applies this configuration periodically. You can have
classes of nodes for different hardware types, different services. What's important - you can model only certain aspects of your configuration and leave others untouched.

No comments:

Post a Comment