Have you ever needed to run a bunch of long running processes from a bash script and get their return codes ? I come across this issue quite frequently in my line of work. The most common one is where i need to run rsync to collect files from many machines then if successful run some other task. Depending on the amount of servers and data this can take several hours to run sequentially and I don’t really like waiting around to check the output so that I can run the next task.
How to speed it up? The obvious way would to be to background the rsysc commands but then I dont know if they were all successful. What if one fails? How would I know which one? Some how I needed to catch the return codes of all the sub-shells and be able to match them to a command. This is where the bash command wait come into play.
~]$ help wait
wait: wait [id]
Wait for job completion and return exit status.
Waits for the process identified by ID, which may be a process ID or a
job specification, and reports its termination status. If ID is not
given, waits for all currently active child processes, and the return
status is zero. If ID is a a job specification, waits for all processes
in the job’s pipeline.
Exit Status:
Returns the status of ID; fails if ID is invalid or an invalid option is
given.
The idea is to collect the PID of each sub-shell using the $! variable and adding it to a list. Then use the wait command to wait for each sub-shell to finish and exit with the return code of the sub-shell command. By adding these return codes to another list then we can iterate over them and match them up with the original list. We split the original list into an array towards the end so we can reference the individual items by an index.
In the following example I have used wget and also deliberately changed testfile03.txt to testfile06.txt to show an example of a non 0 return code.
testfile01
testfile02
testfile03
testfile04
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
| #!/bin/bash
my_list="
http://www.thegoldfish.org/wp-content/uploads/2010/08/testfile01.txt
http://www.thegoldfish.org/wp-content/uploads/2010/08/testfile02.txt
http://www.thegoldfish.org/wp-content/uploads/2010/08/testfile06.txt
http://www.thegoldfish.org/wp-content/uploads/2010/08/testfile04.txt
"
results=''
pids=''
for X in $my_list ; do
wget -o /dev/null $X &
pid=$!
pids="$pids $pid"
done
for pid in $pids
do
wait $pid
result=$?
results="$results $result"
done
echo $results
i=0
my_array=( $my_list )
for ret_val in $results; do
echo ${my_array[$i]} returned $ret_val
((i++))
done |
Stunnel is a quick way on taking a non ssl connection and being able to wrap it in ssl for security
stunnel version 4 – Fedora 12/RHEL 5.3 /Centos 5.3
vim /etc/stunnel/stunnel.conf
add in
client=yes
[gmail]
accept = 127.0.0.1:50000
connect = mail.google.com:443
then run
stunnel version 3 – Ubuntu 8.10 (I haven’t used newer versions)
Ubuntu 8.10 has 2 versions of stunnnel: stunnel3 and stunnel4. They have created a symbolic link from /usr/bin/stunnel -> /usr/bin/stunnel3
If you would like to use version 4 you can use the command stunnel4 otherwise if you wish to use the default version, you will need to create a self signed certificate
openssl req -new -x509 -days 3650 -nodes -out /etc/stunnel/stunnel.pem -keyout /etc/stunnel/stunnel.pem
Then to start stunnel use the following command
stunnel -c -d localhost:50001 -r mail.google.com:443
I quite often need to remove a single line from a file by its line number. The most common use case for me is the known_hosts file when I have reinstalled a system, I have in the past used vim and navigated to the line then removed it. This is all well and good but it gets to be a pain having to do it repeatedly, especially when you manage around 1000 servers and the get rebuilt frequently. Finally today I had had enough so wrote a little script to do this task easily. Hopefully someone else finds this useful
Its usage is : delline LINE FILE
#!/bin/bash
LINE=$1
FILE=$2
if [ ! -f $FILE ] ; then
echo "can't read $FILE: No such file or directory"
exit 1
fi
if [ `expr $LINE + 1 2> /dev/null` ] ; then
sed -i "${LINE}d" $FILE
else
echo $LINE is not numeric
exit 1
fi
This post is here mainly because I always forget how to do it. This is one of the simplest ways to reinstall a Centos (will probably work for RHEL and maybe even Fedora) system without needing PXE or physical access to the machine. Make sure that that you have tested you kickstart before you use it and don’t blame me if anything goes wrong.
Save the following script and make it executable then run it. It will ask some questions about networking and hostname and then write a new grub stanza to you grub.conf. It will also download the correct kernel and initrd from the information you have given it and put them in the correct position for grub to find them when it boots.
When you reboot you should be able to select Kickstart Centos and it will boot off the new kernels and pull down the kickstart then reinstall.
#!/bin/bash -x
echo -n "Enter kickstart url: "
read -e ksurl
echo -n "Enter Hostname: "
read -e hostname
echo -n "Enter IP Address: "
read -e ipaddr
echo -n "Enter Gateway: "
read -e gateway
echo -n "Enter Netmask: "
read -e netmask
echo -n "Enter Nameservers: "
read -e nameservers
repourl=$(curl $ksurl 2>/dev/null | sed -n 's/.*\(http\)/\1/ p')
#echo $repourl
vmlinuz_url="${repourl}/isolinux/vmlinuz"
initrd_url="${repourl}/isolinux/initrd.img"
date_now=$(date +%Y%m%d%H%M%S)
grub_stanza="
title Kickstart Centos 5 ${date_now}
root (hd0,0)
kernel /reinstall/vmlinuz ksdevice=eth0 load_ramdisk=1 prompt_ramdisk=0 ramdisk_size=16384 serial hostname=${hostname} ip=${ipaddr} gateway=${gateway} netmask=${netmask} dns=${nameservers} noipv6 ks=${ksurl}
initrd /reinstall/initrd.img
"
echo "$grub_stanza"
echo -n "Please check the grub stanza above and enter 'y' if it is correct: "
read -e confirmed
if [ $confirmed == 'y' ]; then
echo "Downloading kernel and initrd..."
mkdir -p /boot/reinstall
(cd /boot/reinstall/;/usr/bin/urlgrabber $vmlinuz_url )
(cd /boot/reinstall/;/usr/bin/urlgrabber $initrd_url )
cp /boot/grub/grub.conf /boot/grub/grub.conf.bak_`date +%Y%m%d%H%M%S`
echo "$grub_stanza" >> /etc/grub.conf
fi