All Posts

Running Flask on Spark2

Running Flask on Spark2

The following is a quickstart for running Flask on Spark. Most of the example tutorials I have found are for running a bunch of spark jobs on a spark cluster and returning a result. I was interested in long-running tasks and seeing if I could build a web app that ran on Spark. I thought it would be possible but I didn’t think it would be this easy. Please note that this same procedure will work for lots of python scripts, and I am interested to see what else I can load into Spark.

Gopro Video Stabilization

Gopro Video Stabilization

About a year ago I gave in to my materialistic desires and bought a GoPro Black. It is NOT the cheapest action cam on the market but I kept thinking that if this thing was to record my memories then I wanted them the best they possibly could be. This is one of those purchases that I immediately had buyers remorse for since it is an action cam and those that know me will also know I don’t lead the most action packed life in the world. Nothing has made this more obvious to me than ending up 10 months later and the only things I had used it for was taking fisheye photos of my back garden and filming my feet accidentally.

Nested Virtualization in Centos 6

In the course of testing our puppet manifests with beaker I came across the necessity to run KVM virtual machines inside KVM guests. Normally this has a severe performance penalty as the CPUs need to be fully emulated and cannot take advantage of performance enhancing CPU instructions. Several years ago Intel and AMD added functionality which basically enables CPU instruction passthrough to the guests to get around this limitation. You can see these extensions on a host CPU in /proc/cpuinfo

Openldap fix a incorrect checksum

A while ago we ran into an issue with an incorrect checksum in a openldap ldif file. This means that someone has hand edited the file and caused the checksum to change without updating the checksum itself. There is a nice warning in the file but of course we as sysadmins know better that to read warnings :-/ [[email protected] ~]# service slapd start Checking configuration files for slapd: [WARNING] 57b5d8d1 ldif_read_file: checksum error on "/etc/openldap/slapd.

virsh-sliver: A simple tool for creating slivers of virtual machines on Fedora

Basic script to create qcow2 backed kvm domains based on a golden domain. Your golden domain must be using qcow2 To install the the dependencies on fedora 19 run the following: yum install libvirt-client qemu-img libguestfs-tools xmlstarlet libxml2 The virt-sysprep tool can do much more than this and I could potentially set the hostname and configure the machine to run some scripts when it starts up to check in with puppet which I may do in the future.

virtual-host configuration for Apache HTTP

Posting this literally so I don’t forget it. /etc/httpd/conf.d/vhosts.conf NameVirtualHost * include conf.d/vhosts.d/*.conf mkdir /etc/httpd/conf.d/vhosts.d /etc/httpd/conf.d/vhosts.conf NameVirtualHost * include conf.d/vhosts.d/*.conf /etc/httpd/conf.d/default.conf <VirtualHost _default_:*> ServerAdmin [email protected] DocumentRoot /var/www/vhosts/default/html/ ServerName localhost <VirtualHost> /etc/httpd/conf.d/vhosts.d/ <VirtualHost *:80> ServerName ServerAdmin [email protected] DocumentRoot /var/www/vhosts/ RewriteEngine On RewriteCond %{ENV:REDIRECT_STATUS} !=503 RewriteRule .* - [R=503,L] ErrorDocument 503 /index.html Header always set Retry-After "60" <VirtualHost>

Fedora 19 and Gnome 3.8 tweaking

After a few false starts (Anaconda’s disk configuration is still a mess and very unintuitive) I managed to install the latest and greatest Fedora 19 on my home machine. I like the new introduction and things are looking good. I have noticed a few new things in Settings which are good a welcome site for people that like a little bit of choice. Things such as per application notification settings, media sharing, privacy settings and configuration for what appears in the applications search are all new.

Testing Python with Pythonbrew and Virtualenv on Jenkins with ShiningPanda plugin

I have finally decided to learn all this stuff about Test Driven Design (TDD). I have a few projects that I go back to every now and then and it would be nice to be able to start hacking away without needing to remember every bit of the project. After looking around and investigating several other Continuous Integration systems I have ended up on Jenkins withShiningPanda plugin. The development team where I work uses Jenkins and it is reasonably easy to set up but I was initially against it because the machine I have to run it on is fairly low spec.

Apache HTTPD configuration to allow easy upgrading of OS

mkdir /mnt/media/www/conf ln -s /mnt/media/www/conf /etc/httpd/conf.d/local /etc/httpd/conf.d/local.conf include conf.d/local/*.conf semanage fcontext -a -t httpd_sys_content_t "/mnt/media/www(/.*)?" semanage fcontext -a -t httpd_config_t "/mnt/media/www/conf(/.*)?" restorecon -Rv /mnt/media/www /mnt/media/www/conf/vhost_default.conf <VirtualHost _default_:*> ServerAdmin [email protected] DocumentRoot /mnt/media/www/vhosts/default/html/ ErrorLog logs/default-error_log TransferLog logs/default-access_log <Directory "/mnt/media/www/vhosts"> AllowOverride None # Allow open access: Require all granted </Directory> <Directory "/mnt/media/www/vhosts/default/html"> Options Indexes FollowSymLinks AllowOverride None Require all granted </Directory> </VirtualHost>

Howto setup a isolated installation network with dnsmasq pxeboot and on Fedora 15

Simple pxeboot environment on a Fedora 15 system. This is useful if machines you are trying to build are on a network that cannot see the internet. My particular case is a home server which I want to setup from a place that doesn’t have cabling. I will move it into the cupboard that does have cabling when done but for the time being it is more convenient to set it up on the desk in my bedroom.