virsh-sliver: A simple tool for creating slivers of virtual machines on Fedora

December 6th, 2013 No comments

Basic script to create qcow2 backed kvm domains based on a golden domain. Your golden domain must be using qcow2
To install the the dependencies on fedora 19 run the following:

yum install libvirt-client qemu-img libguestfs-tools xmlstarlet libxml2

The virt-sysprep tool can do much more than this and I could potentially set the hostname and configure the machine to run some scripts when it starts up to check in with puppet which I may do in the future.

Categories: Libvirt, Linux, Testing Tags:

Fedora 19 and Gnome 3.8 tweaking

July 7th, 2013 3 comments

After a few false starts (Anaconda’s disk configuration is still a mess and very unintuitive) I managed to install the latest and greatest Fedora 19 on my home machine. I like the new introduction and things are looking good. I have noticed a few new things in Settings which are good a welcome site for people that like a little bit of choice. Things such as per application notification settings, media sharing, privacy settings and configuration for what appears in the applications search  are all new. Also is the inclusion of new choices in the Gnome Online Accounts dialogue such as OwnCloud which is something I am looking at to help me break out of my total reliance on Google for all my information applications. All in all the release is nice and I thank and congratulate all the effort people have put in to create this latest release.

That being said there are a few things that I am disappointed in. I am going to list them and then update with mu fixes in the hopes that what I discover is helpful to someone else.

1. There should be a way to make online accounts be the Settings -> Details -> Default Application for things like email and calendaring

This is the best I have managed so far. This is useful for setting default applications for other things as well

Create a desktop file for Google Calendar at  .local/share/applications/google-calendar.desktop with the following content

1
2
3
4
5
6
7
8
9
10
11
12
13
14
#!/usr/bin/env xdg-open
 
[Desktop Entry]
Name=Gnome Google Calendar
GenericName=Google Calendar Web Integration
Comment=Integrate Google Calendar with your desktop
Exec=xdg-open https://www.google.com/calendar %U
Icon=stock_calendar
Terminal=false
Type=Application
Categories=GNOME;GTK;Calendar;Network;
StartupNotify=true
X-Panel-Monitor=true
NoDisplay=true

Then create a file at .local/share/applications/defaults.list with the following content

1
2
[Default Applications]
text/calendar=google-calendar.desktop

You should now be able to change your default Calendar in Gnome Settings to be “Gnome Google Calendar”. This will make it the calendar that opens when you click “Open Calendar” in the Gnome Shell clock dropdown.

2. We have lost the ability to set a default terminal.

I prefer terminator as my default terminal emulator as do many people I know. Previously the way to change the default terminal was a not the most obvious config option and it wasn’t where you would expect in Default Applications.
What you did was change it using the gsettings command line tool like this.

1
gsettings set org.gnome.desktop.default-applications.terminal exec 'terminator'

While this still changes the value it appears that it is deprecated and no longer has any effect. I think this is because the handling of the default terminal is now done using the new GIO library. It appears to be hard coded to use gnome-terminal and then a number of fall backs, none of which are terminator but I don’t think this is the case because the change that implemented the hard coding was done back in 2007 so there must be some way of overriding this higher up in the code.

I have given up looking so what I have done is uninstalled gnome-terminal and then created a symlink in its place pointing at terminator

1
2
3
4
5
6
7
8
[thughes@titanium: ~]$ sudo yum -y install terminator
[thughes@titanium: ~]$ GT_PATH=$(which gnome-terminal)
[thughes@titanium: ~]$ sudo yum -y remove gnome-terminal
[thughes@titanium: ~]$ sudo ln -s $(which terminator) $GT_PATH
[thughes@titanium: ~]$ which gnome-terminal
/usr/bin/gnome-terminal
[thughes@titanium: ~]$ ll /usr/bin/gnome-terminal
lrwxrwxrwx. 1 root root 15 Jul  7 01:26 /usr/bin/gnome-terminal -> /bin/terminator

3. The menu for things like Files (nautilus) and Empathy is in the top left hand corner next to “Activities”.

This one I just don’t get and find very unintuitive. I use 2 screens and keep Empathy over on the far right which means that I now have to move the cursor a full 2 screens to be able to access the menu and then back again when I want to use the Contacts list. I don’t understand how this one got past the Gnome Human Interface Guidelines guys.I have no answer for this yet and I may raise a bug as it is hard to find and new users will just find it confusing.

Categories: Uncategorized Tags:

Testing Python with Pythonbrew and Virtualenv on Jenkins with ShiningPanda plugin

April 13th, 2013 No comments

I have finally decided to learn all this stuff about Test Driven Design (TDD). I have a few projects that I go back to every now and then and it would be nice to be able to start hacking away without needing to remember every bit of the project. After looking around and investigating several other Continuous Integration systems I have ended up on Jenkins with ShiningPanda plugin. The development team where I work uses Jenkins and it is reasonably easy to set up but I was initially against it because the machine I have to run it on is fairly low spec. It turns out it isn’t too much of a resource hog for a JVM based application (only 20% of my memory and 10% constant cpu usage).

Installing Jenkins was easy as they provide rpms for Fedora and its derivatives at http://pkg.jenkins-ci.org/redhat/ with instructions for setting up yum repos.

Installing ShiningPanda was also simple. Go to the Manage Plugins section under the Manage Jenkins menu item on the left of the Jenkins interface and search for ShiningPanda in the Available tab. Select the checkbox beside the plugin and press the Download now and install after restart.

The next step was installing multiple python binaries so that I can test that my applications will run on the commonly deployed versions. While looking around to try and work this out I discovered pythonbrew and then a nice description on how to use it with ShiningPanda at http://ampledata.org/python_version_testing_with_jenkins.html which helped me greatly and which I based most of this around.  I needed to make a few tweaks to the script because running Jenkins as a daemon was changing it’s current working directory to “/” which was making it fail.

Once Jenkins has restarted after the installation of ShiningPanda go back into the Manage Jenkins area and select the Script Console item. This script console allows you to run arbitrary scripts on the Jenkins box. We can use this to install pythonbrew with the following script. This script works on a RPM install of Jenkins which is installed under /var/lib/jenkins. You may need to adjust the paths in the calls to .execute() to suit your system.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
def download = """curl -skLO https://raw.github.com/utahta/pythonbrew/master/pythonbrew-install"""
def download_proc = download.execute(null, new File("/var/lib/jenkins"))
def install = """bash pythonbrew-install"""
def install_proc = install.execute(null, new File("/var/lib/jenkins"))
 
download_proc.waitFor()
 
println "return code: ${download_proc.exitValue()}"
println "stderr: ${download_proc.err.text}"
println "stdout: ${download_proc.in.text}"
 
install_proc.waitFor()
 
println "return code: ${install_proc.exitValue()}"
println "stderr: ${install_proc.err.text}"
println "stdout: ${install_proc.in.text}"

I unfortunately didn’t collect the output of the above script but it has a reasonable amount of output which explains what it did. If it doesn’t work then it just prints the output codes of the commands in the stderr parts and something has gone wrong.

Before we continue to setup pythonbrew in Jenkins we need to check that we have the dependencies for building our different versions of python and install if necessary.  There may be more than the following but these were the ones I needed.

1
rpm -q bzip2-devel gdbm-devel ncurses-devel openssl-devel readline-devel sqlite-devel zlib-devel

You can check later in  /var/lib/jenkins/.pythonbrew/log/build.log once you have attempted to build your first python to see if there are any issues. Look for the bit like this:

Python build finished, but the necessary bits to build these modules were not found:
_bsddb             _tkinter           bsddb185       
dl                 imageop            sunaudiodev

Apparently “dl” is for i386, “sunaudiodev” is for Solaris systems,  “imageop” and “bsddb” are obsolete and I don’t need “tkinter” so I am not worried about that result.

Now that we have pythonbrew installed we need to setup the ShiningPanda plugin to use it to automatically build the versions of python that we want to test against. This is done via the Jenkins web interface. Go to Manage Jenkins -> Configure System and scroll down to the Python  Installations section. I deleted the system python that was already there and Then selected Add Python. Put a name for your python installation in the Name section, it needs to have no spaces in it such as “Python-2.6.6″ and then select the checkbox beside Install automatically. For the Command put “$HOME/.pythonbrew/bin/pythonbrew install 2.6.6″ and then for Tool Home put in “/var/lib/jenkins/.pythonbrew/pythons/Python-2.6.6/”. For some reason using “$HOME” didnt work in the Tool Home section for me but it worked if I put the full path. Add as many versions of python as you would like by repeating the steps and changing the version number to suit.  Save using the Save button at the bottom of the page.

Adding pythonbrew configuration to Jenkins

 

The next step is to setup a New Job using the menu item at the top left from the main page. Add a Name for your job and the select Build multi-configuration project and hit OK. I use Bitbucket.org for hosting my repositories so I added a Git repository in the Source Code Management section the in Build Triggers chose Poll SCM . There is a help button on the right hand side explaining the format to create a schedule. Now we are to the bit where we configure our python. In the Configuration matrix section select Add Axis and choose Python from the drop down. Now you can select your choice of python installs from the ones you setup earlier.

Jenkins Add Axis

The Build section is where you can add the commands you would like to run to build and test you project. I run mine in a Virtualenv which is an awesome tool created by Ian Bicking  which creates a isolated python environment for your work. From the Add build step drop down select Virtualenv Builder and then add your command.

add_build_step

For a simple Django project the commands can be:

1
2
pip install Django
$PYTHON_EXE manage.py test

Jenkins Build Command

Now save it and you are complete. You should now be able to go back to the top of the page and select Build Now from the menu at the top left. The first time it runs it will take a while as it needs to build your pythonbrew instances of python but after that it should be much quicker.

Easy OpenSSH VPN using tunneling

November 8th, 2012 No comments

This is a simple VPN for those times when you want the ease of use of a VPN but only have a ssh server available.

Both servers need to have ssh configured to allow tunnels.  You need to change the configs for ssh under /etc/ssh

Remote server sshd_config:

PermitTunnel  yes

Local server ssh_config:

Tunnel  yes

Remote server

Setting up the tunnels requires you to use root when sshing. Since I don’t open my servers up to remote root logins I work around it by first creating a reverse tunnel from the remote server back to my home machine.  You will need to forward you external ssh port on your broadband router to the ssh on your local machine. Then create the reverse tunnel like this.

ssh -f -N -R 2222:localhost:22 home_user@local-external-address

Local server

Now on your local server you should be able to ssh to the remote machine as root via the reverse tunnel connected to your local port 2222
The -w 0:0 tells ssh to set up a tunX device at both ends and what their number should be. In this case we end up with a tun0 at both ends.

ssh -f -N -w0:0 127.0.0.1 -p 2222

Next we need to configure the local tun0 device and add in a route to the remote networks

ifconfig tun0 10.0.2.1 netmask 255.255.255.252
ip ro add 10.0.0.0/8 dev tun0

So that we can reference hosts within the remote networks I copy over the resolv.conf from the remote host to the local machine.

cp /etc/resolv.conf /etc/resolv.conf.before_ssh_tunnel 
scp -P 2222 127.0.0.1:/etc/resolv.conf /etc/resolv.conf

Remote server

On the remote server,  tell kernel to allow forwarding of traffic then configure the remote tun0 device

echo 1 > /proc/sys/net/ipv4/ip_forward
/sbin/ifconfig tun0 10.0.2.2 netmask 255.255.255.252

Still on the remote server we need to configure the firewall to forward the traffic correctly and do some NAT so that the remote server can send packets back to the local machine. Make sure that eth0 is the internal network interface rather than the external facing one as you dont want to vpn into the internet

BIG CAUTION: I need to work out what is blocking in the standard iptables on Fedora. The following will turn off your firewall and only add back in the bits required for this VPN. You may be disabling you external firewall if there isn’t another firewall in the way. My particular machine has a hardware firewall in place so I am not too stressed by flushing iptables.

/sbin/iptables -F
/sbin/iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
/sbin/iptables -A FORWARD -i eth0 -o tun0 -m state --state RELATED,ESTABLISHED -j ACCEPT
/sbin/iptables -A FORWARD -i tun0 -o eth0  -j ACCEPT

If you are planning on doing this often then I would recommend something that can automate ssh for you such as wrapping all the commands in Capistrano or Fabric scripts.

***[EDIT]***

I found NetworkManager-ssh-gnome when looking through the list of rpms available in Fedora 19 which automates a lot of the above process.

[thughes@titanium: ~]$ yum info NetworkManager-ssh-gnome.x86_64
Loaded plugins: langpacks, refresh-packagekit
Available Packages
Name        : NetworkManager-ssh-gnome
Arch        : x86_64
Version     : 0.0.3
Release     : 0.8.20130419git3d5321b.fc19
Size        : 32 k
Repo        : fedora/19/x86_64
Summary     : NetworkManager VPN plugin for SSH - GNOME files
URL         : https://github.com/danfruehauf/NetworkManager-ssh
Licence     : GPLv2+
Description : This package contains software for integrating VPN capabilities with
            : the OpenSSH server with NetworkManager (GNOME files).

The upstream project is available https://github.com/danfruehauf/NetworkManager-ssh

Categories: Linux, Sysadmin Tags: , , ,

webdiff

July 23rd, 2012 No comments

This is just a simple script for doing a diff of two web pages.

1
2
3
4
5
6
7
8
9
10
11
#!/bin/bash
 
TMP1=$(mktemp)
TMP2=$(mktemp)
 
wget -q $1 -O $TMP1
wget -q $2 -O $TMP2
 
diff $TMP1 $TMP2
 
rm -f $TMP1 $TMP2
Categories: Bash Tags: