virsh-sliver: A simple tool for creating slivers of virtual machines on Fedora

Posted on Fri 06 December 2013 in misc • Tagged with Libvirt, Linux, TestingLeave a comment

Basic script to create qcow2 backed kvm domains based on a golden domain. Your golden domain must be using qcow2
To install the the dependencies on fedora 19 run the following:
yum install libvirt-client qemu-img libguestfs-tools xmlstarlet libxml2

The virt-sysprep tool can do much more than this and I could potentially set the hostname and configure the machine to run some scripts when it starts up to check in with puppet which I may do in the future.

Fedora 19 and Gnome 3.8 tweaking

Posted on Sun 07 July 2013 in misc • Tagged with UncategorizedLeave a comment

After a few false starts (Anaconda's disk configuration is still a mess and very unintuitive) I managed to install the latest and greatest Fedora 19 on my home machine. I like the new introduction and things are looking good. I have noticed a few new things in Settings which are good a welcome site for people that like a little bit of choice. Things such as per application notification settings, media sharing, privacy settings and configuration for what appears in the applications search are all new. Also is the inclusion of new choices in the Gnome Online Accounts dialogue such as OwnCloud which is something I am looking at to help me break out of my total reliance on Google for all my information applications. All in all the release is nice and I thank and congratulate all the effort people have put in to create this latest release.

That being said there are a few things that I am disappointed in. I am going to list them and then update with mu fixes in the hopes that what I discover is helpful to someone else.

1. There should be a way to make online accounts be the Settings -> Details -> Default Application for things like email and calendaring

This is the best I have managed so far. This is useful for setting default applications for other things as well

Create a desktop file for Google Calendar at .local/share/applications/google-calendar.desktop with the following content

#!/usr/bin/env xdg-open

[Desktop Entry]
Name=Gnome Google Calendar
GenericName=Google Calendar Web Integration
Comment=Integrate Google Calendar with your desktop
Exec=xdg-open %U

Then create a file at .local/share/applications/defaults.list with the following content

[Default Applications]

You should now be able to change your default Calendar in Gnome Settings to be "Gnome Google Calendar". This will make it the calendar that opens when you click "Open Calendar" in the Gnome Shell clock dropdown.

  1. We have lost the ability to set a default terminal.
I prefer terminator as my default terminal emulator as do many people I know. Previously the way to change the default terminal was a not the most obvious config option and it wasn't where you would expect in Default Applications.
What you did was change it using the gsettings command line tool like this.
gsettings set org.gnome.desktop.default-applications.terminal exec 'terminator'

While this still changes the value it appears that it is deprecated and no longer has any effect. I think this is because the handling of the default terminal is now done using the new GIO library. It appears to be hard coded to use gnome-terminal and then a number of fall backs, none of which are terminator but I don't think this is the case because the change that implemented the hard coding was done back in 2007 so there must be some way of overriding this higher up in the code.

I have given up looking so what I have done is uninstalled gnome-terminal and then created a symlink in its place pointing at terminator

[thughes@titanium: ~]$ sudo yum -y install terminator
[thughes@titanium: ~]$ GT_PATH=$(which gnome-terminal)
[thughes@titanium: ~]$ sudo yum -y remove gnome-terminal
[thughes@titanium: ~]$ sudo ln -s $(which terminator) $GT_PATH
[thughes@titanium: ~]$ which gnome-terminal
[thughes@titanium: ~]$ ll /usr/bin/gnome-terminal
lrwxrwxrwx. 1 root root 15 Jul  7 01:26 /usr/bin/gnome-terminal -> /bin/terminator

3. The menu for things like Files (nautilus) and Empathy is in the top left hand corner next to "Activities".

This one I just don't get and find very unintuitive. I use 2 screens and keep Empathy over on the far right which means that I now have to move the cursor a full 2 screens to be able to access the menu and then back again when I want to use the Contacts list. I don't understand how this one got past the Gnome Human Interface Guidelines guys.I have no answer for this yet and I may raise a bug as it is hard to find and new users will just find it confusing.

Testing Python with Pythonbrew and Virtualenv on Jenkins with ShiningPanda plugin

Posted on Sat 13 April 2013 in misc • Tagged with development, jenkins, Python, shiningpanda, testingLeave a comment

I have finally decided to learn all this stuff about Test Driven Design (TDD). I have a few projects that I go back to every now and then and it would be nice to be able to start hacking away without needing to remember every bit of the project. After looking around and investigating several other Continuous Integration systems I have ended up on Jenkins withShiningPanda plugin. The development team where I work uses Jenkins and it is reasonably easy to set up but I was initially against it because the machine I have to run it on is fairly low spec. It turns out it isn't too much of a resource hog for a JVM based application (only 20% of my memory and 10% constant cpu usage).

Installing Jenkins was easy as they provide rpms for Fedora and its derivatives at with instructions for setting up yum repos.

Installing ShiningPanda was also simple. Go to the *Manage Plugins* section under the *Manage Jenkins* menu item on the left of the Jenkins interface and search for ShiningPanda in the *Available* tab. Select the checkbox beside the plugin and press the *Download now and install after restart*.

The next step was installing multiple python binaries so that I can test that my applications will run on the commonly deployed versions. While looking around to try and work this out I discovered pythonbrew and then a nice description on how to use it with ShiningPanda at which helped me greatly and which I based most of this around.  I needed to make a few tweaks to the script because running Jenkins as a daemon was changing it's current working directory to "/" which was making it fail.

Once Jenkins has restarted after the installation of ShiningPanda go back into the *Manage Jenkins* area and select the Script Console item. This script console allows you to run arbitrary scripts on the Jenkins box. We can use this to install pythonbrew with the following script. This script works on a RPM install of Jenkins which is installed under /var/lib/jenkins. You may need to adjust the paths in the calls to .execute() to suit your system.

def download = """curl -skLO"""
def download_proc = download.execute(null, new File("/var/lib/jenkins"))
def install = """bash pythonbrew-install"""
def install_proc = install.execute(null, new File("/var/lib/jenkins"))


println "return code: ${download_proc.exitValue()}"
println "stderr: ${download_proc.err.text}"
println "stdout: ${}"


println "return code: ${install_proc.exitValue()}"
println "stderr: ${install_proc.err.text}"
println "stdout: ${}"

I unfortunately didn't collect the output of the above script but it has a reasonable amount of output which explains what it did. If it doesn't work then it just prints the output codes of the commands in the stderr parts and something has gone wrong.

Before we continue to setup pythonbrew in Jenkins we need to check that we have the dependencies for building our different versions of python and install if necessary.  There may be more than the following but these were the ones I needed.

rpm -q bzip2-devel gdbm-devel ncurses-devel openssl-devel readline-devel sqlite-devel zlib-devel

You can check later in  /var/lib/jenkins/.pythonbrew/log/build.log once you have attempted to build your first python to see if there are any issues. Look for the bit like this:

Python build finished, but the necessary bits to build these modules were not found:
_bsddb             _tkinter           bsddb185
dl                 imageop            sunaudiodev

Apparently "dl" is for i386, "sunaudiodev" is for Solaris systems, "imageop" and "bsddb" are obsolete and I don't need "tkinter" so I am not worried about that result.

Now that we have pythonbrew installed we need to setup the ShiningPanda plugin to use it to automatically build the versions of python that we want to test against. This is done via the Jenkins web interface. Go to Manage Jenkins -> Configure System and scroll down to the *Python Installations* section. I deleted the system python that was already there and Then selected *Add Python*. Put a name for your python installation in the *Name* section, it needs to have no spaces in it such as "Python-2.6.6" and then select the checkbox beside *Install automatically*. For the *Command* put "$HOME/.pythonbrew/bin/pythonbrew install 2.6.6" and then for *Tool Home* put in "/var/lib/jenkins/.pythonbrew/pythons/Python-2.6.6/". For some reason using "$HOME" didnt work in the *Tool Home* section for me but it worked if I put the full path. Add as many versions of python as you would like by repeating the steps and changing the version number to suit.  Save using the *Save* button at the bottom of the page.

Adding pythonbrew configuration to Jenkins

The next step is to setup a *New Job* using the menu item at the top left from the main page. Add a *Name* for your job and the select*Build multi-configuration project* and hit OK. I use for hosting my repositories so I added a *Git* repository in the *Source Code Management* section the in *Build Triggers* chose *Poll SCM* . There is a help button on the right hand side explaining the format to create a schedule. Now we are to the bit where we configure our python. In the *Configuration matrix* section select *Add Axis* and choose *Python* from the drop down. Now you can select your choice of python installs from the ones you setup earlier. * *

Jenkins Add Axis

The *Build* section is where you can add the commands you would like to run to build and test you project. I run mine in a Virtualenv which is an awesome tool created by Ian Bicking  which creates a isolated python environment for your work. From the *Add build step* drop down select *Virtualenv Builder* and then add your *command*.

add build step

For a simple Django project the commands can be:

pip install Django

Jenkins Build Command

Now *save* it and you are complete. You should now be able to go back to the top of the page and select *Build Now* from the menu at the top left. The first time it runs it will take a while as it needs to build your pythonbrew instances of python but after that it should be much quicker.

KVM Serial console and VNC console

Posted on Tue 19 February 2013 in misc • Tagged with libvirt, kvmLeave a comment

Configuration for libvirt to get serial and graphics working at the same time:

virt-install \
 --name centos6_golden \
 --ram 2048 \
 --arch x86_64 \
 --vcpus 4 \
 --disk path=/var/lib/libvirt/images/centos6_golden.disk,format=qcow2,sparse=true,size=24,bus=virtio \
 --location \
 --hvm \
 --accelerate \
 --nographics \
 --os-type linux \
 --os-variant virtio26 \
 --extra-args 'acpi=force noipv6 console=tty0 console=ttyS0,115200 ks= ksdevice=52:54:00:A8:7A:0A ip= gateway= netmask= dns= ' \
 --network bridge:br250 \
 --mac 52:54:00:A8:7A:0A

Serial console in GRUB:

serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1 terminal --timeout=15 serial console

Change the kernel command line to attach ttys in the right spots. Remove rhgb and quiet and add the following to the kernel command line:

acpi=force console=tty0 console=ttyS0,115200n8

Easy OpenSSH VPN using tunneling

Posted on Thu 08 November 2012 in misc • Tagged with openssh, ssh, tunnel, vpnLeave a comment

This is a simple VPN for those times when you want the ease of use of a VPN but only have a ssh server available.

Both servers need to have ssh configured to allow tunnels.  You need to change the configs for ssh under /etc/ssh

Remote server sshd_config:

PermitTunnel  yes

Local server ssh_config:

Tunnel  yes

Remote server

Setting up the tunnels requires you to use root when sshing. Since I don't open my servers up to remote root logins I work around it by first creating a reverse tunnel from the remote server back to my home machine.  You will need to forward you external ssh port on your broadband router to the ssh on your local machine. Then create the reverse tunnel like this.

ssh -f -N -R 2222:localhost:22 home_user@local-external-address

Local server

Now on your local server you should be able to ssh to the remote machine as root via the reverse tunnel connected to your local port 2222
The -w 0:0 tells ssh to set up a tunX device at both ends and what their number should be. In this case we end up with a tun0 at both ends.
ssh -f -N -w0:0 -p 2222

Next we need to configure the local tun0 device and add in a route to the remote networks

ifconfig tun0 netmask
ip ro add dev tun0

So that we can reference hosts within the remote networks I copy over the resolv.conf from the remote host to the local machine.

cp /etc/resolv.conf /etc/resolv.conf.before_ssh_tunnel
scp -P 2222 /etc/resolv.conf

Remote server

On the remote server,  tell kernel to allow forwarding of traffic then configure the remote tun0 device

echo 1 > /proc/sys/net/ipv4/ip_forward
/sbin/ifconfig tun0 netmask

Still on the remote server we need to configure the firewall to forward the traffic correctly and do some NAT so that the remote server can send packets back to the local machine. Make sure that eth0 is the internal network interface rather than the external facing one as you dont want to vpn into the internet

BIG CAUTION: I need to work out what is blocking in the standard iptables on Fedora. The following will turn off your firewall and only add back in the bits required for this VPN. You may be disabling you external firewall if there isn't another firewall in the way. My particular machine has a hardware firewall in place so I am not too stressed by flushing iptables.

/sbin/iptables -F
/sbin/iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
/sbin/iptables -A FORWARD -i eth0 -o tun0 -m state --state RELATED,ESTABLISHED -j ACCEPT
/sbin/iptables -A FORWARD -i tun0 -o eth0  -j ACCEPT

If you are planning on doing this often then I would recommend something that can automate ssh for you such as wrapping all the commands in Capistrano or Fabric scripts.


I found NetworkManager-ssh-gnome when looking through the list of rpms available in Fedora 19 which automates a lot of the above process.

[thughes@titanium: ~]$ yum info NetworkManager-ssh-gnome.x86_64
Loaded plugins: langpacks, refresh-packagekit
Available Packages
Name        : NetworkManager-ssh-gnome
Arch        : x86_64
Version     : 0.0.3
Release     : 0.8.20130419git3d5321b.fc19
Size        : 32 k
Repo        : fedora/19/x86_64
Summary     : NetworkManager VPN plugin for SSH - GNOME files
URL         :
Licence     : GPLv2+
Description : This package contains software for integrating VPN capabilities with
            : the OpenSSH server with NetworkManager (GNOME files).

The upstream project is available


Posted on Mon 23 July 2012 in misc • Tagged with BashLeave a comment

This is just a simple script for doing a diff of two web pages.



wget -q $1 -O $TMP1
wget -q $2 -O $TMP2

diff $TMP1 $TMP2

rm -f $TMP1 $TMP2

Installing stock CentOS kernel on an OVH or Kimsufi server

Posted on Wed 20 June 2012 in misc • Tagged with Centos, grub.conf, Kimsufi, Linux, OVHLeave a comment

I recently signed up for a small dedicated server at for the grand price of £14 a month which I think is pretty good price for what you get.




Celeron / Atom


1.20+ GHz


64 bits




2 GB

Hard disk

1 TB

FTP Backup

100 GB


Heaps more than I need :-)

The machine itself is nothing special but they do have remote management and you can install a variety of Linux and windows operating systems. I am quite happy with their service and management tools but I wasn't happy with the fact that they run their own custom kernels on them. The kernel that they install is a 'grsec' hardened 3.2 kernel but this caused a variety of issues when doing 'yum update', mainly dependency issues. I Googled around and found a few different solutions on remotely installing CentOS which were all basically variations on the way I describe in a previous post using grub to re-install. Whatever I tried with this including using VNCand attempting a fully automated install with a tested kickstart file it would become unresponsive when rebooting into the OS the first time.

Instead of all these methods I should have been thinking inside the box instead of outside it :-(


yum install kernel

Then add the following stanza to /etc/grub.conf modified to suit the kernel that was installed (Check under /boot/ for the file names).

title CentOS (2.6.32-220.23.1.el6.x86_64)
        root (hd0,0)
        kernel /vmlinuz-2.6.32-220.23.1.el6.x86_64 root=/dev/sda2 ro
        initrd /initramfs-2.6.32-220.23.1.el6.x86_64.img

and change:




After a reboot we end up with the correct stock kernel and the yum issues disappear.

The system is much closer to a normal installation and the only thing I am still unhappy with is the filesystem mounted at / is a physical partition instead of on LVM. This should be easily resolvable by creating a new logical volume and copying everything over and some updates to /etc/fstab and /boot/grub/grub.conf to point at the logical volume and a selinux relabel but I will leave that for another day.

Set txqueuelen on virtual vnetX devices with libvirt

Posted on Tue 03 April 2012 in misc • Tagged with libvirt, Linux, networking, udev, vnetLeave a comment

The txqueuelen is a value in the kernel on network interfaces that sets the transmit queue length. This value can be tuned for different work loads. In the case of modern networking the defaults can sometimes be changed to get better line speeds over ethernet. Most people will do this using a rc.local command to set it on the physical ethX devices like this.

vim /etc/rc.local

Add the following

/sbin/ip link set eth0 txqueuelen 2500

This is a perfectly reasonable way of doing it but what happens when network interfaces appear after boot and the name is unknown before hand? This is exactly what happens with Libvirt vnetX interfaces. Ideally we would be able to get Libvirt to tune these interfaces when it creates them, but that level of control is yet to be implemented BZ#809172. Libvirt >= 0.8.0 has some hooks which enable you to run commands at specific times in the lifecycle of a guest which may be good for this but on RHEL 5 Libvirt is version 0.6.0 so I needed a different solution.

In comes udev. Udev is the device manager for the Linux kernel and you can write rules for when devices are added and removed from the system. A common user visible use of udev is when a camera is plugged in, udev mounts it and informs the desktop environment of its availability. This allows the desktop environment to open a photo import application.

When you start a guest using Libvirt it creates vnetX devices to match the network interfaces seen inside the guest. These vnet devices are the ones I want to change the txqueuelen on. The default txqueuelen in our environment is 500 but with some experimentation we found we could increase the throughput of our network by a factor of 3 by changing it to 2500.

vnet6     Link encap:Ethernet  HWaddr 36:9A:FA:9C:38:0E
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

The X in vnetX is a integer which is allocated on a 'first free' basis so we cannot tell what it will be prior to creation. If you already have a guest running you can look at the information available to udev by running the following command:

#> udevinfo -a -p /sys/class/net/vnet0/
  looking at device '/class/net/vnet0':

These values can be used to match the device in a rule. In this case we want all vnetX devices which we can do using the KERNEL value, this is the name allocated to the device by the kernel. The following will match any vnetX device that gets created


Then we want to run a command to change the txqueuelen when the kernel sees one of those devices change. The RUN value is a list of commands to be run after a device exists. These commands need to be very short lived as they pause udev while they are running.

RUN+="/sbin/ip link set %k txqueuelen 2500"

The final udev rule is:

KERNEL=="vnet[0-9]*", RUN+="/sbin/ip link set %k txqueuelen 2500"

This can be put into a file under /etc/udev/rules.d/ for example /etc/udev/rules.d/60-vnet.rules

Now when you start a guest os the vnet devices will all have a txqueuelen of 2500 (this value suits the specific workload we are running YMMV)

vnet6     Link encap:Ethernet  HWaddr A6:CE:E6:9F:39:FB
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:2500
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

Reverting to a previous snapshot using Linux LVM

Posted on Mon 19 September 2011 in misc • Tagged with LinuxLeave a comment

Reverting to a previous snapshot has been possible for over a year!!!!! How did I miss that ?? This has for a long time been one of my only real criticisms of LVM and I just discovered that it was quietly committed into the kernel back in 2.6.33

The command used to do the revert is contained within lvconvert. From the lvconvert man page:

       Merges a snapshot into its origin volume.  To check if your ker‐
       nel supports this feature, look for 'snapshot-merge' in the out‐
       put  of 'dmsetup targets'.  If both the origin and snapshot vol‐
       ume are not open the merge will start  immediately.   Otherwise,
       the  merge  will start the first time either the origin or snap‐
       shot are activated and both are closed.  Merging a snapshot into
       an  origin that cannot be closed, for example a root filesystem,
       is deferred until the next time the origin volume is  activated.
       When  merging starts, the resulting logical volume will have the
       origin's name, minor number and UUID.  While  the  merge  is  in
       progress,  reads  or  writes  to  the origin appear as they were
       directed to the snapshot being merged.  When the merge finishes,
       the merged snapshot is removed.  Multiple snapshots may be spec‐
       ified on the commandline or a @tag may be used to specify multi‐
       ple snapshots be merged to their respective origin.

A quick check using the command 'dmsetup targets' shows that it is definitely in my kernel so I thought I would give it a quick run through and test it lot. I created a testing logical volume and then put some data on it, took a snapshot, changed the data and then reverted to the snapshot. Here is what I did.

Create a new logical volume with a file system and mount it.

[root@titanium ~]# lvcreate -L1G -n lv_test vg_titanium
  Logical volume "lv_test" created
[root@titanium ~]# mke2fs -j /dev/vg_titanium/lv_test
mke2fs 1.41.14 (22-Dec-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
65536 inodes, 262144 blocks
13107 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376

Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 33 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
[root@titanium ~]# mkdir /mnt/test
[root@titanium ~]# mount /dev/vg_titanium/lv_test /mnt/test

Create some test data then take a snapshot and create some more test data.

[root@titanium ~]# touch /mnt/test/testdata-$(date +%Y%m%d%H%M%S)
[root@titanium ~]# ls -l /mnt/test/
total 20
drwx------. 2 root root 16384 Sep 19 00:39 lost+found
-rw-r--r--. 1 root root     0 Sep 19 00:39 testdata-20110919003936
[root@titanium ~]# SNAP_NAME="lv_test-snapshot-$(date +%Y%m%d%H%M%S)"
[root@titanium ~]# lvcreate -s /dev/vg_titanium/lv_test -L 1G -n $SNAP_NAME
  Logical volume "lv_test-snapshot-20110919003936" created
[root@titanium ~]# touch /mnt/test/testdata-$(date +%Y%m%d%H%M%S)
[root@titanium ~]# ls -l /mnt/test/
total 24
drwx------. 2 root root 16384 Sep 19 00:39 lost+found
-rw-r--r--. 1 root root     0 Sep 19 00:39 testdata-20110919003936
-rw-r--r--. 1 root root     0 Sep 19 00:39 testdata-20110919003937

Here is the actual merge command.

[root@titanium ~]# lvconvert --merge /dev/vg_titanium/$SNAP_NAME
  Can't merge over open origin volume
  Merging of snapshot lv_test-snapshot-20110919003936 will start next activation.

You will need to deactivate and activate to get the merge to start. You can immediately remount the filesystem as your view of it will be that of the snapshot once the merge has started. Once mounted you can check the data.

[root@titanium ~]# umount /dev/vg_titanium/lv_test
[root@titanium ~]# lvchange -an /dev/vg_titanium/lv_test
[root@titanium ~]# lvchange -ay /dev/vg_titanium/lv_test
[root@titanium ~]# mount /dev/vg_titanium/lv_test /mnt/test
[root@titanium ~]# ls -l /mnt/test/
total 20
drwx------. 2 root root 16384 Sep 19 00:39 lost+found
-rw-r--r--. 1 root root     0 Sep 19 00:39 testdata-20110919003936

As we can see we have reverted to the filesystem at the time we took the snapshot.

Clean up :-)

[root@titanium ~]# umount /dev/vg_titanium/lv_test
[root@titanium ~]# lvremove /dev/vg_titanium/lv_test
  Logical volume "lv_test-snapshot-20110919003936" successfully removed
Do you really want to remove active logical volume lv_test? [y/n]: y
  Logical volume "lv_test" successfully removed
rmdir /mnt/test

Turn off the Caps-Lock key

Posted on Tue 13 September 2011 in misc • Tagged with LinuxLeave a comment

I HATE THE CAPS_LOCK KEY!!!! I don't like it when people send me messages with full caps and I don't like accidentally pressing it and then sending messages to other people who then think I am yelling at them. It also wastes too much keyboard space and is in an easy place to accidentally hit. Time to get rid of it.

The following method is for Fedora 15 but will probably work on other Gnome 3 systems.:
Click on the user menu on the top right. Select System Settings
Region and Language -> Layouts -> Options -> Caps Lock key behaviour.

There are several different options but I am going with Disabled for the time being.

Now if I want to yell at someone I need to put in a little more effort but maybe that will make the world a better place :-)