About half a year ago I started using proxmox with KVM for virtualizing my systems. The product itself is brilliant in my opinion, but still suffers from kvm bugs.
I found that some guests would freeze after live migration from one physical host to another. Dmesg on the VMs showed soft lockups on cpus:
BUG: soft lockup - CPU#3 stuck for 10s
Having googled the problem, I found that the problem appears on hosts with virtio drivers (virtio-net and virtio-disk). The probability is even better if you run an smp guest.
To resolve this problem, just switch back to ide emulation and e1000 for network.
Showing posts with label kvm. Show all posts
Showing posts with label kvm. Show all posts
Saturday, November 6, 2010
Thursday, May 20, 2010
Systemimager and KVM guests - tips
Recently I've been figuring out how to deploy kvm guests with systemimager and pxe. This is a continuation of my previous story.
If you go for kvm you will probably end up using virtio network and virtio-hdd drivers with your guests as they provide best performance. Using these drivers implies some strange hdd naming. You place your instalation od /dev/vda instead of /dev/sda. Unfortunately systemimager (or better - systeminstaller - one of its components), will not recognize vda disks properly which leads to some complications.
Normally, when you run si_prepareclient on your golden-client, partitioning configuration is placed in file /etc/systemimager/autoinstallscript.conf in section <config><disk> (this is done by systeminstaller in fact). Then this section is used by si_mkautoinstallscript to generate parted commands in your <imagename>.master script. In case of /dev/vda disk, the <config><disk> section is empty, therefore you don't get the partitioning section in .master script. Due to this, deployment of your kvm guest image will fail as no partitions will be created. You may either want to debug systeminstaler (try do tebug perl code ;-) or just work it around, which is what I chose.
What you need to do
First, manually add the <disk> section on the golden client to /etc/systemimager/autoinstallscript.conf after you run si_prepareclient. The file might now be looking like this. The <disk> section is in italics.
This one is taken from redhat with no LVM, the hdd size is 12gb, and 2gb go for swap:
<config>
<disk dev="/dev/vda" label_type="msdos" unit_of_measurement="MB">
<!--
This disk's output was brought to you by the partition tool "parted",
and by the numbers 4 and 5 and the letter Q.
-->
<part num="1" size="101" p_type="primary" p_name="-" flags="boot" />
<part num="2" size="10676" p_type="primary" p_name="-" flags="-" />
<part num="3" size="2097" p_type="primary" p_name="-" flags="-" />
</disk>
<fsinfo line="10" real_dev="/dev/vda2" mount_dev="LABEL=/" mp="/" fs="ext3" options="defaults" dump="1" pass="1" />
<fsinfo line="20" real_dev="/dev/vda1" mount_dev="LABEL=/boot" mp="/boot" fs="ext3" options="defaults" dump="1" pass="2" />
<fsinfo line="30" real_dev="tmpfs" mp="/dev/shm" fs="tmpfs" options="defaults" dump="0" pass="0" />
<fsinfo line="40" real_dev="devpts" mp="/dev/pts" fs="devpts" options="gid=5,mode=620" dump="0" pass="0" />
<fsinfo line="50" real_dev="sysfs" mp="/sys" fs="sysfs" options="defaults" dump="0" pass="0" />
<fsinfo line="60" real_dev="proc" mp="/proc" fs="proc" options="defaults" dump="0" pass="0" />
<fsinfo line="70" real_dev="/dev/vda3" mount_dev="LABEL=SWAP-vda3" mp="swap" fs="swap" options="defaults" dump="0" pass="0" />
<boel devstyle="udev"/>
</config>
Second: pull the image from the golden-client (si_getimage).
Third: check your /var/lib/systemimager/scripts/<imagename>.master if it contains "parted" section (cat <imagename>.master |grep parted or something like this should verify this).
Fourth: have your image deployed to a new vm:
si_mkclientnetboot --image yourimage --flavor yourimage
The --flavor is needed to use a kernel with virtio drivers for deployment - it needs to recognize your /dev/vda and virtio ethernet.
That's all. Reboot your new vm with pxe boot and watch your image being deployed.
Comments welcome ;-)
If you go for kvm you will probably end up using virtio network and virtio-hdd drivers with your guests as they provide best performance. Using these drivers implies some strange hdd naming. You place your instalation od /dev/vda instead of /dev/sda. Unfortunately systemimager (or better - systeminstaller - one of its components), will not recognize vda disks properly which leads to some complications.
Normally, when you run si_prepareclient on your golden-client, partitioning configuration is placed in file /etc/systemimager/autoinstallscript.conf in section <config><disk> (this is done by systeminstaller in fact). Then this section is used by si_mkautoinstallscript to generate parted commands in your <imagename>.master script. In case of /dev/vda disk, the <config><disk> section is empty, therefore you don't get the partitioning section in .master script. Due to this, deployment of your kvm guest image will fail as no partitions will be created. You may either want to debug systeminstaler (try do tebug perl code ;-) or just work it around, which is what I chose.
What you need to do
First, manually add the <disk> section on the golden client to /etc/systemimager/autoinstallscript.conf after you run si_prepareclient. The file might now be looking like this. The <disk> section is in italics.
This one is taken from redhat with no LVM, the hdd size is 12gb, and 2gb go for swap:
<config>
<disk dev="/dev/vda" label_type="msdos" unit_of_measurement="MB">
<!--
This disk's output was brought to you by the partition tool "parted",
and by the numbers 4 and 5 and the letter Q.
-->
<part num="1" size="101" p_type="primary" p_name="-" flags="boot" />
<part num="2" size="10676" p_type="primary" p_name="-" flags="-" />
<part num="3" size="2097" p_type="primary" p_name="-" flags="-" />
</disk>
<fsinfo line="10" real_dev="/dev/vda2" mount_dev="LABEL=/" mp="/" fs="ext3" options="defaults" dump="1" pass="1" />
<fsinfo line="20" real_dev="/dev/vda1" mount_dev="LABEL=/boot" mp="/boot" fs="ext3" options="defaults" dump="1" pass="2" />
<fsinfo line="30" real_dev="tmpfs" mp="/dev/shm" fs="tmpfs" options="defaults" dump="0" pass="0" />
<fsinfo line="40" real_dev="devpts" mp="/dev/pts" fs="devpts" options="gid=5,mode=620" dump="0" pass="0" />
<fsinfo line="50" real_dev="sysfs" mp="/sys" fs="sysfs" options="defaults" dump="0" pass="0" />
<fsinfo line="60" real_dev="proc" mp="/proc" fs="proc" options="defaults" dump="0" pass="0" />
<fsinfo line="70" real_dev="/dev/vda3" mount_dev="LABEL=SWAP-vda3" mp="swap" fs="swap" options="defaults" dump="0" pass="0" />
<boel devstyle="udev"/>
</config>
Second: pull the image from the golden-client (si_getimage).
Third: check your /var/lib/systemimager/scripts/<imagename>.master if it contains "parted" section (cat <imagename>.master |grep parted or something like this should verify this).
Fourth: have your image deployed to a new vm:
si_mkclientnetboot --image yourimage --flavor yourimage
The --flavor is needed to use a kernel with virtio drivers for deployment - it needs to recognize your /dev/vda and virtio ethernet.
That's all. Reboot your new vm with pxe boot and watch your image being deployed.
Comments welcome ;-)
Wednesday, May 12, 2010
My virtualization choice for 2010
Recently I've come to the point where I need to switch to a different virtualization platform in my datacenter. I've been using vmware ESX/ESXi but while my infrastructure grew, maintaining a farm of ESX/ESXi servers has become - let's say it shortly - a pain ;-). Lots of competitive platforms have emerged recently which are in my opinion better than ESXi (even if free), so I decided to switch to one of them. There was a lot of hesitation in my choice - so here is a story about it.
In fact, if you don't want to go into vmware, you are left with a choice between XEN and KVM. So the first thing I did was just enabling them on my plain systems. I found Xen to be rock-solid and fast and this was definitely my favourite. I tested some xen guests with my production load and the performance loss was minimal. My production distro was working flawlessly. On the downside - pxe is not supported by xen out-of-the-box (you can have it with pypxeboot package). Also xen seems a bit behind the mainstream with its old kernel (still 2.6.18). KVM was worse on performance, which I think was due to virtio drivers. With my production load I mainly hit limits with virtio-net rendering large number of interrupts - system cpu would jump high. I also needed to switch to newer version of my distro to have virtio support out of the box. On the other hand, KVM receives lots of development and bleeding edge features (e.g. page merging).
Both Xen and KVM share similar downsides:
-> lack of documentation, especially on networking. I have a multiple vlan environment. To get it to work, I spent several hours browsing through forums and putting scraps of information together ;-)
-> lack of fine management interface. Both kvm and xen on ubuntu/redhat, use virt-manager from redhat which I don't like. There is also ovirt but afaik it's not intended for production use now. There are probably some more on sourceforge, but I did not test. If you know any - just comment.
So now when the hypervisor was chosen, I just need to find a distro on which I want to run Xen. I decided to go into Citrix XenServer. It's just ready to run appliance. Multi-vlan environment is handled out of the box and documentation provided. Also console is nice, though windows-based.
...
But finally I ended up with KVM. ;-) Citrix and Redhat are now main contributors to xen. Recently redhat 6 beta has come out. Following was the press release stating that redhat is totally dropping xen in release 6 and goes to kvm instead. So right now I think Citrix being th only developer of xen is in a position similar to vmware. So if I choose Xen, I am going to lock myself into a deal with one company. And now that redhat officially goes into kvm, I suppose they will commit lots of code to it. So I decided I can live for some time with worse performance than Xen and wait for redhat 6 and their stability patches to kvm.
As a platform for KVM I chose proxmox which totally ROCKS in my opinion. It's debian under the hood with pretty recent kernel (2.6.32 AFAIR). When you install it, it's just ready to go. Being debian, proxmox can be deployed in numbers easily with systemimager/preseed. I also expect puppet integration to be a snap. You don't need to download any license keys as it is with XenServer and Vmware. Vlan setup turned out to be easy, gui is web-based and works flawlessly. So right now proxmox is my 100% hit.
In fact, if you don't want to go into vmware, you are left with a choice between XEN and KVM. So the first thing I did was just enabling them on my plain systems. I found Xen to be rock-solid and fast and this was definitely my favourite. I tested some xen guests with my production load and the performance loss was minimal. My production distro was working flawlessly. On the downside - pxe is not supported by xen out-of-the-box (you can have it with pypxeboot package). Also xen seems a bit behind the mainstream with its old kernel (still 2.6.18). KVM was worse on performance, which I think was due to virtio drivers. With my production load I mainly hit limits with virtio-net rendering large number of interrupts - system cpu would jump high. I also needed to switch to newer version of my distro to have virtio support out of the box. On the other hand, KVM receives lots of development and bleeding edge features (e.g. page merging).
Both Xen and KVM share similar downsides:
-> lack of documentation, especially on networking. I have a multiple vlan environment. To get it to work, I spent several hours browsing through forums and putting scraps of information together ;-)
-> lack of fine management interface. Both kvm and xen on ubuntu/redhat, use virt-manager from redhat which I don't like. There is also ovirt but afaik it's not intended for production use now. There are probably some more on sourceforge, but I did not test. If you know any - just comment.
So now when the hypervisor was chosen, I just need to find a distro on which I want to run Xen. I decided to go into Citrix XenServer. It's just ready to run appliance. Multi-vlan environment is handled out of the box and documentation provided. Also console is nice, though windows-based.
...
But finally I ended up with KVM. ;-) Citrix and Redhat are now main contributors to xen. Recently redhat 6 beta has come out. Following was the press release stating that redhat is totally dropping xen in release 6 and goes to kvm instead. So right now I think Citrix being th only developer of xen is in a position similar to vmware. So if I choose Xen, I am going to lock myself into a deal with one company. And now that redhat officially goes into kvm, I suppose they will commit lots of code to it. So I decided I can live for some time with worse performance than Xen and wait for redhat 6 and their stability patches to kvm.
As a platform for KVM I chose proxmox which totally ROCKS in my opinion. It's debian under the hood with pretty recent kernel (2.6.32 AFAIR). When you install it, it's just ready to go. Being debian, proxmox can be deployed in numbers easily with systemimager/preseed. I also expect puppet integration to be a snap. You don't need to download any license keys as it is with XenServer and Vmware. Vlan setup turned out to be easy, gui is web-based and works flawlessly. So right now proxmox is my 100% hit.
Subscribe to:
Posts (Atom)