How to recover a qcow2 file using fsck

I just upgrade Ubuntu on one of my virtual machines and for some reason the file system was corrupted after I rebooted the server. I got errors like:

ext4-fs error ext4_lookup deleted inode referenced

Great. Here is how I fixed the problem.

I found the solution was to attach the qcow2 file as a device on the host and then use fsck. Obviously, you will need to shutdown the virtual machine before doing this.

On the host, enable the nbd (network block device) kernel module. This should be available on all Ubuntu servers after Intrepid.

sudo modprobe nbd max_part=8

You can this use qemu-nbd to connect your qcow2 file as a network block device.

sudo qemu-nbd --connect=/dev/nbd0 /mnt/kvm/wordpress-sites.qcow2

You can then find the partitions using

sudo fdisk /dev/nbd0

On the virtual machine, partition 1 was corrupted, so here is the command I ran:

sudo fsck /dev/nbd0p1

Then disconnect the disk:

sudo qemu-nbd --disconnect /dev/nbd0

Everything seemed to be fine and so I started the domain up again.

KVM: Host CPU does not provide required features

I just exported / copied a virtual machine from one of our servers (a HP Proliant) to my laptop so that I could to mess around with it offline. I used the following commands to export the domain.

On the server:

virsh dumpxml wordpress > /tmp/wordpress.xml

On the laptop:

virsh define wordpress.xml
virsh wordpress start

However, I got this error on the laptop:

error: unsupported configuration: guest and host CPU are not compatible: Host CPU does not provide required features: rdtscp, popcnt, sse4.2, dca

The HP server has xeon processors and my laptop has a little core two duo. Obviously the CPU’s had different capabilities. You can see the cpu definitions in the wordpress.xml file were:

less wordpress.xml
<cpu match='exact'>
    <model>Nehalem</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='tm2'/>
    <feature policy='require' name='est'/>
    <feature policy='require' name='monitor'/>
    <feature policy='require' name='ds'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='rdtscp'/>
    <feature policy='require' name='ht'/>
    <feature policy='require' name='dca'/>
    <feature policy='require' name='pbe'/>
    <feature policy='require' name='tm'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='ds_cpl'/>
    <feature policy='require' name='xtpr'/>
    <feature policy='require' name='acpi'/>
</cpu>

You can also view the same information using virsh capabilities command. It is worth piping it through less because the command produces alot of output and the host cpu information is at the top.

For the laptop:

virsh capabilities | less
<cpu>
    <arch>x86_64</arch>
    <model>Penryn</model>
    <vendor>Intel</vendor>
    <topology sockets='1' cores='2' threads='1'/>
    <feature name='pdcm'/>
    <feature name='xtpr'/>
    <feature name='tm2'/>
    <feature name='est'/>
    <feature name='smx'/>
    <feature name='vmx'/>
    <feature name='ds_cpl'/>
    <feature name='monitor'/>
    <feature name='dtes64'/>
    <feature name='pbe'/>
    <feature name='tm'/>
    <feature name='ht'/>
    <feature name='ss'/>
    <feature name='acpi'/>
    <feature name='ds'/>
    <feature name='vme'/>
</cpu>

So, to fix the issue, I just edited the domains xml definition and put in the correct features.

virsh edit wordpress

Note – if you are using ubuntu on your desktop and have virt-manager installed (which is an awesome gui application which runs on Linux and Macs). You can run virt-manager, open the virtual machine, go to the information tab, select the processor and then “copy host cpu configuration”. I’ve attached a screenshot of how to do this.

virt_manager_copy_configuration_from_host

I hope this helps someone.

Ubuntu 13.04 – kvm binary is deprecated?

I was trying to setup a KVM host with some KVM virtual machines on Ubuntu 13.04 Server.  It was a fresh install on a HP DL360.  However, there seemed to be a fundamental problem with KVM / virsh / virt-install.

When trying to install a virtual machine using virt-install with this command.

virt-install --accelerate -v -w bridge:br0 -m DE:AD:BE:EF:04:7B -n test1 --vcpus=2 -r 4096 --os-variant=ubuntuprecise --vnc --import --disk path=/home/me/vm.qcow2

I get this error:

ERROR internal error process exited while connecting to monitor: W: kvm binary is deprecated, please use qemu-system-x86_64 instead

It seems like a serious problem but it isn’t. I just should have put sudo in the front.

<strong>sudo</strong> virt-install --accelerate -v -w bridge:br0 -m DE:AD:BE:EF:04:7B -n test1 --vcpus=2 -r 4096 --os-variant=ubuntuprecise --vnc --import --disk path=/home/me/vm.qcow2

I hope this helps you because I wasted quite some time reading Ubuntu bug reports. Why can’t the system designers detect that the user didn’t have permissions and throw sensible errors?

MySQL Problems – a complete reinstall

I rebooted an Ubuntu 12.04 server after upgrading the kernel and MySQL.  However, I found that the new MySQL would not start. When I tried:

sudo service start mysql

I got:

start: job failed to start

There were no errors appearing in the /var/log/mysql/error.log and so there was no information to help debug it.  I found this post and tried every single one of the solutions.  When I removed mysql-server-5.5 using:

apt-get –purge remove mysql-server
rm /etc/mysql/ -R

I found the when I reinstalled mysql-server the installation would fail saying:

Unable to set password for the MySQL “root” user  An error occurred while setting the password for the MySQL administrative user. This may have happened because the account already has a password, or because of a communication problem with the MySQL server.

I then got stuck because I could no longer remove MySQL because it was only partially installed.  Every time I tried to remove it, it would ask me to put in a new root password which I obviously could not do.  The solution is to use dpkg to purge the partially installed package:

dpkg –purge mysql-server-5.5

Then I tried reinstalling again and got:

Can’t find file: ‘./mysql/host.frm’ (errno: 13)

Basically, the solution to these problems is a complete purge of MySQL and removing ALL of it and start again.  I found what mysql related packages I have install by running:

dpkg –get-selections | grep mysql

and found I had:

libdbd-mysql-perl
libmysqlclient18
mysql-client-5.5
mysql-client-core-5.5
mysql-common
mysql-server
mysql-server-5.5
mysql-server-core-5.5
mysqltuner
php5-mysql

I removed all them!

apt-get –purge remove mysql-server
apt-get –purge remove mysql-client
apt-get –purge remove mysql-common
…… etc …. etc…

Then cleared the apt-get cache and removed the mysql config and data directories (I have a backup of the data from the night before.)  Note – these commands will remove all your old MySQL data, so don’t run them if you don’t have a backup or you don’t care about the data.

apt-get autoremove
apt-get autoclean
rm /etc/mysql/ -R
rm /var/lib/mysql/ -R

Then reinstalling mysql-server and the other packages.

apt-get install mysql-server
etc….

What a massive waste of time!  Oh well – I hope this post helps someone else out of this pickle.

Monitoring HP Raid Array on Ubuntu Server

I wanted to monitor our HP server raid array on Ubuntu 12.04.  I  am sure that there is a good way of setting this up with Nagios or SMNP – however, I thought I would write a little bash script which emails me if any of the hard drives in the raid array fail.  I assume that you have already installed the HP Proliant Support Pack perhaps using these instructions.

To send an email from command line you will also need mailx.  This is a virtual package on Ubuntu and I tend to install bsd-mailx.  To install it run:

sudo apt-get install bsd-mailx

It will also install Postfix (if this isn’t already setup).  When configuring Postfix, I selected “Internet Site” and entered the hostname of the server.

Here is the bash script that I wrote.  The indentation seems to have got lost but you should be able to recreate it.

#!/bin/bash

# Email settings
email_recipient=”your.email@somewhere.com”

# Current datetime in YYYY-MM-DD_HH-MM-SS
date=`date +”%Y-%m-%d_%H-%M-%S”`

# Make sure the log directory exists.
/bin/mkdir -p /root/harddrive_check/log

log_filename=”/root/harddrive_check/log/$date.log”

hostname=`hostname`

if hpacucli ctrl all show config | grep -q “Failed”; then

echo -e “Harddrive Check Run: `date`” > $log_filename

echo -e “\nUsing Command: hpacucli ctrl all show config” >> $log_filename

hpacucli ctrl all show config >> $log_filename

email_subject=”Harddrive Failed On $hostname”

/usr/bin/mailx -s “$email_subject” “$email_recipient” < $log_filename

fi

I then setup a cronjob on the server.

crontab -e

Adding the line:

0 * * * * /root/harddrive_check.sh

I hope this is helpful to someone.  Let me know if you know a better way of doing this. Thanks for reading.

How to install imapsync on Ubuntu 12.04

Imapsync is a really useful script from syncing imap mail boxes.  I used it when migrating from Zimbra to Google Mail.  I wanted to install imapsync from source on my Ubuntu server but found the instructions a little ridiculous.  The start of the instructions are:

Go into the directory imapsync-x.xx and read the INSTALL file.
You’re already reading the INSTALL file.

Here is what I did it:

1) Download the source from github and extract it.

2) Install a few dependences:

aptitude install libmail-imapclient-perl libdigest-md5-file-perl libterm-readkey-perl libio-socket-ssl-perl libdigest-hmac-perl makepasswd

3) In the source folder run:

make -n install

make install

You should then have a working imapsync.  To test it try:

imapsync -version

Boot partition full of old kernels

I started getting an error message every time I booted into Ubuntu.  “‘The volume “boot” has only ….. bytes disk space remaining.”.   This is caused by old Linux kernels filling up the boot partition on the hard drive.  For most users, you don’t need to keep old kernels and therefore you can remove them to free up some space.   For simplicity sake – it did this using the Synaptic Package Manager.  Here is how:

1) First, you need to find the current kernel that you are was using – so that you don’t remove it!  Open a terminal and type:

uname -a

Linux toshiba 3.5.0-23-generic #35-Ubuntu SMP Thu Jan 24 13:05:29 UTC 2013 i686 i686 i686 GNU/Linux

2) From the output, I was using 3.5.0-23-generic.  Therefore, I could remove all other kernels with version numbers lower than this.   There are three packages that make up the kernel:

  • linux-header-x-x-x-x-generic
  • linux-image-x-x-x-x-generic
  • linux-image-extra-x-x-x-x-generic

Using Synaptic Package Manager search for linux-header, then linux-image and select  the ones with a lower version number than your current kernel.  Mark each package for removal.  The linux-image-extra will automatically be removed when you remove a linux-image.

Then apply the changes and reboot your machine.

p.s. – it strikes me that it would be really good if Ubuntu could automatically do this.  Perhaps this could be built into the automatic update tool.