Limiting bandwidht of Rapidshare

August 17th, 2010

Rapidshare is a p2p technology that uses standard http protocol to transfer information between users. This can become a major issue for some ISP’s as http traffic is normally included into a high priority protocol classification when it comes to QoS.

With Mikrotik it makes it quite easy to limit connectivity to their servers by following these simple procedures:

Create a new script file and call it what you like. Paste the following lines of code

:foreach i in=\
 [ :toarray "62.67.46.0/24,62.67.57.0/24,64.214.225.0/24,64.215.245.0/24,80.129.35.0/24, \
 80.231.56.0/24,80.239.151.0/24,80.239.159.0/24,80.239.226.0/24,80.239.236.0/24, \
 82.129.35.0/24,82.129.36.0/24,82.129.39.0/24,195.122.131.0/24,195.219.1.0/24, \
 206.57.14.0/24,207.138.168.0/24,212.162.2.0/24,212.162.63.0/24"] \
 do={ /ip firewall address-list add address=$i list=rapidshare }

That will create address lists for all their subnets. Now we need to create connection / packet marking rules for these lists, which can be done as follows:

/ip firewall mangle add chain=prerouting action=mark-connection new-connection-mark=rapidshare passthrough=yes dst-address-list=rapidshare connection-mark=!rapidshare 
 
/ip firewall mangle add chain=prerouting action=mark-packet new-packet-mark=rapidshare passthrough=no connection-mark=rapidshare

That will mark all the connections to the subnets created in the list earlier.

Now, all you need to do is add a simple queue for these marked packets, and voilla!

/queue simple add name="Rapidshare" dst-address=0.0.0.0/0 interface=all parent=none packet-marks=rapidshare direction=both priority=8 queue=default-small/default-small limit-at=32k/32k max-limit=32k/32k burst-limit=0/0 burst-threshold=0/0 burst-time=0s/0s total-queue=default-small

That will limit all up and download to Rapidshare to 32k.

Adding more swap space on Linux (CentOS)

August 13th, 2010

Linux is really a beautiful operating system. I recently had to change my firewall from CentOS to Mikrotik (I am not explaining that whole scenario in this post), but I had a lot of stuff on the old CentOS box. It was running apache, mysql, exim, spamassassin, clamav, squid and a whole lot more stuff, and was mainly used for my development.

So, after installing Mikrotik onto it I moved all the other services to my old Dell file server at home. This machine only has 256MB RAM and 128MB swap space. (I don’t know why I installed with only 128MB, must have been very hammered at the time).

Anyways, soon after mysteriously, services would stop working. Services like exim, bind, spamassassin etc. Upon further investigation,  I found that the machine ran out of memory. Here is a snippet from my /var/log/message log file:

Aug 13 07:12:36 libra kernel: Out of memory: Killed process 10995, UID 93, (exim).
Aug 13 07:12:36 libra kernel: crond invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Aug 13 07:12:36 libra kernel:  [] out_of_memory+0x72/0x1a3
Aug 13 07:12:36 libra kernel:  [] __alloc_pages+0x24e/0x2cf
Aug 13 07:12:37 libra kernel:  [] page_cache_read+0x3f/0x98
Aug 13 07:12:37 libra kernel:  [] filemap_nopage+0x205/0x34a
Aug 13 07:12:37 libra kernel:  [] __handle_mm_fault+0x178/0xa25
Aug 13 07:12:37 libra kernel:  [] do_page_fault+0x23a/0x52d
Aug 13 07:12:37 libra kernel:  [] do_page_fault+0x0/0x52d
Aug 13 07:12:37 libra kernel:  [] error_code+0x39/0x40
Aug 13 07:12:37 libra kernel:  =======================
 
...
 
Aug 13 07:12:37 libra kernel: spamd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Aug 13 07:12:38 libra kernel: crond invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Aug 13 07:12:40 libra kernel: exim invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Aug 13 07:12:40 libra kernel: dovecot invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0

Which means the server was starting to kill memory hungry processes, not good at all :(

Luckily Linux has a command called swapon (use man swapon to see more details).

So using, swapon and dd I was able to quickly add another 1GB of swap to server to alleviate the low memory issue:

Create an empty 1GB file we are going to be using for swap

dd if=/dev/zero of=/swapfile bs=1024 count=1024000; chmod 600 /swapfile

Set the correct permissions on it, set it as a swap area and use it as swap space

chmod 600 /swapfile ; mkswap /swapfile ; swapon /swapfile
 
...
Setting up swapspace version 1, size = 1048571 kB

Bam!

I know have 1GB extra swap space on the machine.

               total         used          free     shared    buffers     cached
Mem:        253452     248016        5436          0       2636     114804
-/+ buffers/cache:     130576     122876
Swap:      1152504     120920    1031584

Encrypting directories with ecryptfs

August 11th, 2010

eCryptfs is a POSIX-compliant enterprise-class stacked cryptographic filesystem for Linux.

I use it to securely store my passwords and other sensitive information on Linux. Here is a small howto for installing and using it on CentOS .

First of install the ecryptfs rpm’s using yum. (It ships as part of the base repository)

yum install -y ecryptfs-utils

It will also install the keyutils and trouser dependencies.

Just as an extra precaution I always chmod 700 the Private directory in my home directory and chown root:root the Directory too. So only root can mount and edit it.

Now to start using it, creat a small shell script in your home directory and call it mount.sh with the following content:

#!/bin/sh
mount -t ecryptfs ~your_user/Private ~your_user/Private -o key=passphrase,ecryptfs_cipher=des3_ede,ecryptfs_key_bytes=24,ecryptfs_passthrough=n

If this is the first time you are mounting the directory it will promp you for some options, use the details below as a guid, but read through the options:

Unable to find a list of options to parse, defaulting to interactive mount
Select key type to use for newly created files:
 1) openssl
 2) tspi
 3) passphrase
 
Selection: 3
Passphrase:
Select cipher:
 1) aes: blocksize = 16; min keysize = 16; max keysize = 32 (not loaded)
 2) blowfish: blocksize = 16; min keysize = 16; max keysize = 32 (not loaded)
 3) des3_ede: blocksize = 8; min keysize = 24; max keysize = 24 (not loaded)
 4) twofish: blocksize = 16; min keysize = 16; max keysize = 32 (not loaded)
 5) cast6: blocksize = 16; min keysize = 16; max keysize = 32 (not loaded)
 6) cast5: blocksize = 8; min keysize = 5; max keysize = 16 (not loaded)
Selection [aes]: 3
Select key bytes:
 1) 24
Selection [24]:
Enable plaintext passthrough (y/n) [n]:
Unable to find a list of options to parse, defaulting to interactive mount
Attempting to mount with the following options:
  ecryptfs_unlink_sigs
  ecryptfs_key_bytes=24
  ecryptfs_cipher=des3_ede
  ecryptfs_sig=7c91c94bb99b6313
WARNING: Based on the contents of [/root/.ecryptfs/sig-cache.txt],
it looks like you have never mounted with this key
before. This could mean that you have typed your
passphrase wrong.
 
Would you like to proceed with the mount (yes/no)? : yes
Would you like to append sig [7c91c94bb99b6313] to
[/root/.ecryptfs/sig-cache.txt]
in order to avoid this warning in the future (yes/no)? : yes
Successfully appended new sig to user sig cache file

Once the directory is mounted it is unencrypted and you can start editing the files that are in the directory. To encrpyt it again simply type:

umount -f ~your_user/Private

Since the contents of the directory will be unencrypted when it is mounted, I have this simple shell script I run in a cronjob every 10 minutes that will send me an email should I forget it is mounted:

#!/bin/sh
 
check=`mount |grep ecryptfs |wc -l`
email="you@yourdomain.com"
 
if [ $check == '1' ]
then
        echo "WARNING: An encrypted filesystem is mounted on `hostname`" | mail -s "Warning" $email
fi

The Dude – Ping probes fail on Windows 7 default install

August 3rd, 2010

The dude is at this present moment in time my favorite monitoring tool. I have integrated it with my clickatell account not to long ago – I will put up a whole post on this in the near future – and it works great for monitoring services and devices and sending SMS notifications on state changes.

Today I reinstalled my monitoring machine to Windows 7 and was pleasantly surprised to find that from within The dude I could ping any devices on my network anymore. After a bit of searching on their forums it seems like quite a few people have run into the same issue. Someone suggested that disabling UAC (User Access Control) in Windows 7 resolved the issue for them.

There seems to be some setting in the user access control settings that prevents The Dude from access to the ping command. Disabling it also worked for me.

Here is how you remove it:

  • Open up control panel
  • Go to User Accounts
  • At the bottom click on “Change User Account Control settings”
  • Drag the slide bar all the way to the bottom the setting for “Never notify me”

A reboot was required for the settings to take effect.

Archiving email with exim for only one user

July 27th, 2010

I got a request today from a client who wanted me to archive all email for only one recipient. Basically a copy of everything our user sends and receives has to be copied to another user’s inbox.

I am sure there are hundreds of ways to do this, but this is the one I got working for me. For this example I used two routers, one for inbound emails and one for outbound emails, both use the same transporter. Our user who’s email we will be snooping on uses the address victim@domain.com.

Routers:

 outbound_archive:
  condition = ${if eq {$sender_address}{victim@domain.com}{1}{0} }
  no_expn
  no_verify
  transport = mailarchive_transport
  driver = accept
  unseen
 
inbound_archive:
  condition = ${if eq {$local_part@$domain}{victim@domain.com}{1}{0} }
  no_expn
  no_verify
  transport = mailarchive_transport
  driver = accept
  unseen

And use the following code in your transporter config:

  mailarchive_transport:
  driver = appendfile
  file = /var/mail/virtual/domain.com/THE_BOSS_INBOX
  delivery_date_add
  envelope_to_add
  return_path_add
  group = mail
  user = mailnull
  mode = 0660
  no_mode_fail_narrower

This example is made for mbox format, but it should work fine for Maildir by just adding the neccessary config to the transporter config for that.

Shell script for automating The Dude backups

July 22nd, 2010

Now that I have a better understanding of how The Dude handles parent / sibling relationships I can vouch for what a great tool it is for monitoring. This freeware application written by the guys from Mikrotik is a very potent and feature rich monitoring tool.

This morning I was looking at automating its backups and found this script on the forums. I did change one or two small things in the script, mostly just two rm -d commands with rm -rf.

#!/bin/sh
 
# this is script for remote backup MukroTik Dude database
# requires: sh, awk, wget
#
# result XML file will be placed in "dude-backup-files" directory (if you don't change this value)
#
# by mr.Z (mr.Z@inbox.ru)
# ver 1.1p, 2009
 
##############################################################################################
 
# Edit 5 lines below for access to your server and set work/backup directory. USE ABSOLUTLEY PATH FOR DIRECTORIES
 
server=x.x.x.x                          # enter server name or IP address
user=admin                              # enter admin user name
password=                               # enter admin password
backupdir=/home/wayne/dude-backup/data          # set directoryfor backup files
workdir=/home/wayne/dude-backup/work            # set directory for temporary files (ATTENTION! AFTER WORK IT WILL BE DELETED)
 
# OPTIONALLY you can set server port and log file
serverport=80
logfile=$backupdir/log.txt
 
##############################################################################################
 
# DO NOT EDIT ANY LINES BELOW
 
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin
# creating directories for work and backup file
if [ ! -e "$workdir" ]
    then
    mkdir -p "$workdir"
fi
if [ ! -e "$backupdir" ]
    then
    mkdir -p "$backupdir"
fi
 
if [ ! -e "$backupdir/old" ]
    then
    mkdir -p "$backupdir/old"
fi
 
today=`date +%Y.%m.%d`  #setting today date (need for getting file from server)
serverfilename=backupbackup-$today.xml  #setting file name on server
backupname=dude-backup-$today.xml
echo "-------------------------------------------------------
Starting new backup procedure at $today
" >> $logfile
 
 
# moving last backup to old directory
mv $backupdir/*backup* $backupdir/old 2> /dev/null
 
# getting file from dude server
echo "Authorizing..."
wget --cookies=on --keep-session-cookies --save-cookies=$workdir/cookie.txt --progress=dot:mega "http://$server:$serverport/dude/main.html?process=login&user=$user&password=$password" -O $workdir/page1.html 2>> $logfile
echo "Please wait, downloading backup XML file. This may take long time..."
wget --cookies=on --load-cookies=$workdir/cookie.txt --progress=dot:mega "http://$server:$serverport/dude/$serverfilename?page=savefile&download=yes" -O $backupdir/$backupname 2>> $logfile
 
 
# cleaning
echo "Cleaning..."
rm -rf $workdir/*
#rm -d $workdir
 
# checking for new backup file and cleaning old
 
if [ `du $backupdir/$backupname | awk -F" " '{print($1)}'` -gt 0 ]
    then
    rm -rf $backupdir/old/*
    #rm -d $backupdir/old
    echo "All done."
    else
    rm -f $backupdir/$backupname
    echo "Backup failed! (see log.txt in $backupdir direectory)"
    echo "Backup failed!" >> $logfile
fi
 
echo "
Backup procedure finished.
-------------------------------------------------------
" >> $logfile
exit

Getting the local SA ip block list

July 20th, 2010

So, a while back I was asking around on some local forums where I could get a list of all the local IP blocks for South Africa, so I could split my local and International traffic. Someone pointed me to Internet Solutions’ public router. Then later on I discovered I had this gem of a shell script that I have forgotten about. This also outputs the subnets in CIDR notation making it easy to use with something like Mikrotik RouterOS.

Thanks to whoever wrote this shell script if I knew who it was or where I got it I would give you the credit you deserve.

#!/bin/sh
echo -e "terminal length 0\nshow ip bgp\nexit" | nc -w 10 route-server.is.co.za 23 | dos2unix | awk '/^*/ {print $2}' | \
       awk -F'[./]' '{ printf $0;
               if (NF==5) {
                       print ""
               } else {
                       if ($1 < 128) {
                               print "/8"
                       } else if ($1 < 192) {
                               print "/16"
                       } else {
                               print "/24" }
                       }
               }'

Proud to be South African

June 23rd, 2010

Today I can say I am more proud to be a South African than ever before, and this in the light of having to deal with the fact that Bafana Bafana is no longer in the running for the world cup trophy.

I have to admit that I wasn’t always optimistic about us hosting a successful world cup. Looking at the crime situation in the country and threats of terrorism and other anarchical acts threatening to derail the world cup it is easy to be pessimistic.

Still, like so many fellow South Africans, after deciding that I will stand by my country representatives in this world cup and support them all the way, and after watching two live games – Brazil vs Ivory Coast and Brazil vs Korea, and seeing how well it all was organised, my mind set started to change dramatically.

The pride I felt yesterday after Bongani Khumalo scored the second goal for South Africa can not be put into words. The glimmer of hope that we might actually be able to pull it off. Unfortunately all didn’t pan out as everyone had hoped in the end. Never the less, I think many many South Africans look at our soccer team in a completely different light now. We can beat some of the best teams in the world and we have proven it in this world cup.

Congratulations to everyone involved in the organising of the soccer world cup, and congratulations to Bafana Bafana for uniting our country in such a great way. Just keep doing what you are doing and keep making us proud.

Moving to WordPress

June 22nd, 2010

So, here we are again. A new day, a new blog. This time powered by WordPress.

I have to admit I am pleasantly surprised at how awesome WordPress really is.
Very easy to install and customize (the little I have done). Awesome admin backend, easy themeable and configrable.

Meteor Hits South African Skyline

June 22nd, 2010

This is rather old I know, but about 10 days ago a meteor entered the earth’s atmosphere somewhere over the South African / Botswana skyline.

Here is what people saw, pretty amazing stuff. As far as I have heard on the radio a group of astronomers and other scientists have been looking for it in Botwsana. Pretty cool stuff.