AWS volume snapshots across multiple regions

So I needed a script to backup volumes each day from multiple regions. I’m sure there are lots of scripts out there but why not add another.

By default this script sets a UTC expiry date tag on snapshots. After the expiry is reached it removes old snapshots.
1st day of month = default expiry 90 days
Sunday = default expiry 21 days
Others = default expiry 1 day

Pre-requisites

Install python 2.7 and python boto library

$ sudo apt-get install python python-pip
$ sudo pip install boto

IAM Policy

Setup a user in AWS IAM with the following policy, keep a copy of credentials you’ll need that for script

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateSnapshot",
                "ec2:CreateTags",
                "ec2:DeleteSnapshot",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeRegions",
                "ec2:DescribeSnapshots",
                "ec2:DescribeVolumeAttribute",
                "ec2:DescribeVolumeStatus",
                "ec2:DescribeVolumes"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Snapshot script

Don’t forget to replace the credentials in the script with your own, you may also want to specify different regions.

#!/usr/bin/env python
from datetime import datetime, timedelta
import boto.ec2, sys

# snapshot.py used to backup volumes across AWS regions
# author: Matt Weston

# Using backup-account in IAM
aws_key = 'AWS_ACCESS_KEY'
aws_secret = 'AWS_SECRET_KEY'
regions = ['us-east-1','us-west-2','ap-southeast-2']

# snaphot date information
current_time = datetime.utcnow()
day_of_month = current_time.day
day_of_week = current_time.weekday()
week_of_year = current_time.isocalendar()[1]
month_of_year = current_time.month
snapshot_date = current_time.strftime('%Y-%m-%dT%H:%M:%S.000Z')

# determine type and expiry based on current day, week or month
snapshot_type = 'daily'
snapshot_expires = current_time + timedelta(days=1)
if day_of_week == 6:
  snapshot_type = 'weekly'
  snapshot_expires = current_time + timedelta(days=21)
if day_of_month == 1:
  snapshot_type = 'monthly'
  snapshot_expires = current_time + timedelta(days=90)
snapshot_expiry = snapshot_expires.strftime('%Y-%m-%dT%H:%M:%S.000Z')

# Get all Regions
for region in regions:
  print "connecting to", region
  try:
    connection = boto.ec2.connect_to_region(region, aws_access_key_id=aws_key, aws_secret_access_key=aws_secret)
    volumes = connection.get_all_volumes()
    print 'creating snapshots for all attached volumes'
    for volume in volumes:
      attached = volume.attachment_state()
      if attached:
        # create snapshots
        attach_data = volume.attach_data
        snapshot_name = 'snapshot: '+attach_data.instance_id+":"+attach_data.device
        snapshot = volume.create_snapshot(snapshot_name)
        snapshot.add_tag("snapshot-by", 'snapshot.py')
        snapshot.add_tag("snapshot-type", snapshot_type)
        snapshot.add_tag("snapshot-expiry", snapshot_expiry)
        snapshot.add_tag("snapshot-instance-id", attach_data.instance_id)
        snapshot.add_tag("snapshot-device", attach_data.device)
        print 'created', snapshot 
     
    print 'deleting expired snapshots for all attached volumes'
    volumes = connection.get_all_volumes()
    for volume in volumes:
      attached = volume.attachment_state()
      if attached:
        # cleanup snapshots
        existing =  volume.snapshots()
        for snapshot in existing:
          if snapshot.status == 'completed' and 'snapshot-expiry' in snapshot.tags:
            snapshot_expiry = snapshot.tags['snapshot-expiry']
            expiry_time = datetime.strptime(snapshot_expiry, '%Y-%m-%dT%H:%M:%S.000Z')
            if expiry_time < current_time:
              print 'expired snapshot', snapshot.id, snapshot.status, snapshot.description

  except Exception, e:
    print "Unexpected error:", sys.exc_info()[0]

Schedule the script using cron

Easy enough to run it as often as needed via cron.

$ chmod +x /path/to/script/snapshot.py
$ crontab -e
# Snapshot attached volumes each day and cleanup expired
30 01 * * * /path/to/script/snapshot.py > /path/to/script/snapshot.log 2>&1

Ratpoison and Screen Blanking

We needed a little kiosk style machine for our ‘company’ dashboard and have opted for a little pi running python and midori. And it works like a treat.

# Install Raspbian and the following
$ ssh pi@raspberrypi
$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get install ratpoison midori

# We also needed for our python web app:
$ sudo apt-get install python python-pip mysql-client libmysqlclient-dev python-dev
$ sudo pip install MySQL-python boto web.py

After that configure your window manager to use ratpoison:

$ sudo update-alternatives --config x-window-manager
There are 2 choices for the alternative x-window-manager (providing /usr/bin/x-window-manager).

  Selection    Path                Priority   Status
------------------------------------------------------------
  0            /usr/bin/openbox     90        auto mode
  1            /usr/bin/openbox     90        manual mode
* 2            /usr/bin/ratpoison   20        manual mode

Setup your xinitrc file to disable screen saver/dpms and start ratpoison

$ sudo vi ~/.xinitrc
xset s off     # don't activate screensaver
xset -dpms     # disable DPMS (Energy Star) features.
xset s noblank # don't blank the video device
exec /etc/alternatives/x-window-manager

Setup the pi to ‘auto login’

# Edit inittab comment and add the following new line
$ sudo vi /etc/inittab
#1:2345:respawn:/sbin/getty --noclear 38400 tty1 
1:2345:respawn:/bin/login -f pi tty1 </dev/tty1 >/dev/tty1 2>&1

Setup startx on login
(todo: probably could do this via debian x-session-manager/lightdm conf)

$ vi ~/.bash_profile 
if [ -z "$DISPLAY" ] && [ $(tty) == /dev/tty1 ]; then
    startx
fi

To confirm Screen saver settings run the following

$ export DISPLAY=:0.0
$ xset q
Keyboard Control:
  auto repeat:  on    key click percent:  0    LED mask:  00000000
  XKB indicators:
    00: Caps Lock:   off    01: Num Lock:    off    02: Scroll Lock: off
    03: Compose:     off    04: Kana:        off    05: Sleep:       off
    06: Suspend:     off    07: Mute:        off    08: Misc:        off
    09: Mail:        off    10: Charging:    off    11: Shift Lock:  off
    12: Group 2:     off    13: Mouse Keys:  off
  auto repeat delay:  660    repeat rate:  25
  auto repeating keys:  00ffffffdffffbbf
                        fadfffefffedffff
                        9fffffffffffffff
                        fff7ffffffffffff
  bell percent:  50    bell pitch:  400    bell duration:  100
Pointer Control:
  acceleration:  2/1    threshold:  4
Screen Saver:
  prefer blanking:  no    allow exposures:  yes
  timeout:  0    cycle:  600
Colors:
  default colormap:  0x20    BlackPixel:  0x0    WhitePixel:  0xffff
Font Path:
  /usr/share/fonts/X11/Type1,built-ins
DPMS (Energy Star):
  Standby: 600    Suspend: 600    Off: 600
  DPMS is Disabled

Using rc.local to start our python script and turn off terminal blanking
(todo: probably could just launch our python app using init.d script… meh)

$ sudo vi /etc/rc.local
#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.

# Print the IP address
_IP=$(hostname -I) || true
if [ "$_IP" ]; then
  printf "My IP address is %s\n" "$_IP"
fi

#Starts the python dashboard
/home/pi/dashboard/simplePython/startdash.sh &

setterm -blank 0

exit 0

Mercurial Contribution Charts

One of the things I’d really like to see is Bitbucket to have Github style contribution graphs and after waiting a while to see if that feature request #4307 was ever going to get a green light I decided to see how hard it would be to build it myself.

Yes I am sure there are 100 people who have already done this… but I didn’t find any that I particularly liked, I wanted the distraction this weekend and it was surprisingly easy to do.

After building a simple hg log parser in python I was to convert the data to json and wrap it with highcharts to produce a couple of really nice looking interactive contribution graphs.

commits

diffs

You can grab the code and take a look yourself: mercurial-charts bitbucket

If you want to contribute, have some ideas or feedback hit me up on Google+

Enabling SSL and Best Practice Ciphers

Get a free SSL certificate from StartSSL which is surprisingly easy, even if their website is a little awkward to navigate.

After creating a private key, obtaining the certificate and intermediate certificate (sub.class1.server.ca.pem) you can setup your apache/nginx server.

Use the modern compatibility ciphers listed on the Mozilla wiki and the sections for both apache and nginx for details on how to configure your individual server or use their handy online tool ssl config generator

Tip for nginx you will want to create a chained certificate, and use that your ssl_certificate file.

$ cat yourdomain.crt intermediate.pem > yourdomain.crt.chained

Tip for apache enable ssl and headers modules

$ sudo a2enmod ssl headers

Tip for wordpress to force administration pages to use SSL add near the top of the wp-config.php

define('FORCE_SSL_ADMIN', true);

Now you can visit my site via https://mattyboy.net

Sendmail Masquerade Using Generic Map

Changes to configure sendmail to masquerade from address allowing me to replace local system email address user@server.localhost.localdomain with another email address like noreply@domain.com or username@domain.com

Create the following files:

root noreply@domain.com
username username@domain.com
example example@domain.com
...
FEATURE(genericstable, hash /etc/mail/generics)dnl
GENERICS_DOMAIN(servername.localhost.localdomain)dnl
GENERICS_DOMAIN(domain.com)dnl
FEATURE(masquerade_envelope)dnl
FEATURE(allmasquerade)dnl
MAILER(smtp)dnl
MAILER(procmail)dnl
...

Run the following commands:

$ makemap hash /etc/mail/generics &lt; /etc/mail/generics
$ m4 /etc/mail/sendmail.mc &gt; /etc/mail/sendmail.cf
$ /etc/init.d/sendmail restart

Test using the following command:

echo &quot;Checking if masquerade worked&quot; | mailx -s &quot;Masquerade Email Test&quot; user@domain.com

Disclaimer: I would not rely on this information in a production environment.

HMAC SHA1 – Java, Python and PHP

Generating a consistent HMAC SHA1 across multiple languages and mirror the java byte array (byte[]) mechanism.

Java Code

import org.apache.commons.codec.binary.Base64;
import org.apache.commons.codec.binary.Hex;
import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;

public class Signer {
    public static void main(String[] args) throws Exception {
        String key = "0f2100e34b54d50fd0138f300d3497579dae5279";
        String data = "secret-message";
        byte[] decodedKey = Hex.decodeHex(key.toCharArray());
        SecretKeySpec keySpec = new SecretKeySpec(decodedKey, "HmacSHA1");
        Mac mac = Mac.getInstance("HmacSHA1");
        mac.init(keySpec);
        byte[] dataBytes = data.getBytes("UTF-8");
        byte[] signatureBytes = mac.doFinal(dataBytes);
        String signature = new String(Base64.encodeBase64(signatureBytes), "UTF-8");
        System.out.println("key = " + key);
        System.out.println("data = " + data);
        System.out.println("signature = " + signature);
    }
}

PHP Equivalent

<?php
  $key = "0f2100e34b54d50fd0138f300d3497579dae5279";
  $data = "secret-message";
  $decodedKey = pack("H*", $key);
  $hmac = hash_hmac("sha1", $data, $decodedKey, TRUE);
  $signature = base64_encode($hmac);
  echo "key = $key\n";
  echo "data = $data\n";
  echo "signature = $signature";
?>

Python Equivalent

import hmac
import hashlib
key = "0f2100e34b54d50fd0138f300d3497579dae5279"
data = "secret-message"
decodedKey = key.decode("hex")
hmac = hmac.new(decodedKey, data.encode('UTF-8'), hashlib.sha1)
signature = hmac.digest().encode('base64')
print "key =", key
print "data =", data
print "signature =", signature

Expected Output

key = 0f2100e34b54d50fd0138f300d3497579dae5279
data = secret-message
signature = MhzlxM4a2htRhimCU/fXNBDBWaU=

Heartbleed nginx check

Few quick commands to check nginx has been patched succesfully for heartbleed.

# check nginx compile config
$ /opt/nginx/sbin/nginx -V
nginx version: nginx/1.4.0
built by gcc 4.7.2 (Debian 4.7.2-5) 
TLS SNI support enabled
configure arguments: --prefix=/opt/nginx --with-http_ssl_module --with-pcre=/opt/nginx/pcre-8.32 --with-zlib=/opt/nginx/zlib-1.2.8

# check which ssl library
$ ldd /opt/nginx/sbin/nginx | grep ssl
	libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f489cd5a000)

$ strings /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 | grep "^OpenSSL "
OpenSSL 1.0.1e 11 Feb 2013

# check full ssl version (including when it was built)
$ openssl version -a
OpenSSL 1.0.1e 11 Feb 2013
built on: Tue Apr  8 08:49:19 UTC 2014
platform: debian-amd64
options:  bn(64,64) rc4(16x,int) des(idx,cisc,16,int) blowfish(idx) 
compiler: gcc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -m64 -DL_ENDIAN -DTERMIO -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -Wl,-z,relro -Wa,--noexecstack -Wall -DMD32_REG_T=int -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM
OPENSSLDIR: "/usr/lib/ssl"

# check changelog for note regarding CVE-2014-0160 patch
$ aptitude changelog openssl
openssl (1.0.1e-2+deb7u5) wheezy-security; urgency=high

  * Non-maintainer upload by the Security Team.
  * Add CVE-2014-0160.patch patch.
    CVE-2014-0160: Fix TLS/DTLS hearbeat information disclosure.
    A missing bounds check in the handling of the TLS heartbeat extension
    can be used to reveal up to 64k of memory to a connected client or
    server.

Download script

Just a little script to download files from my remote server and if need be unrar the files.

#!/bin/bash
LOCAL="/opt/files"
UNRAR="$LOCAL/extract"
LOG="$LOCAL/download.log"
REMOTE="server.com.au:path/to/files"
INPUT=$2

function init() {
  # create required directories
  [ ! -d "$LOCAL" ] && mkdir $LOCAL && echo "created $LOCAL"
  [ ! -d "$UNRAR" ] && mkdir $UNRAR && echo "created $UNRAR"
}

function download() {
  # append wildcard to input
  [ -n "$INPUT" ] && INPUT="${INPUT}*" && echo "Downloading $INPUT"
  [ -z "$INPUT" ] && INPUT="*" && echo "Downloading all files"
  # rsync files from remote server to local folder
  rsync -a --quiet --compress --log-file="$LOG" $REMOTE/$INPUT $LOCAL/ >> $LOG 2>&1 &
}

function unrar() {
  echo "Extracting *.rar files to $UNRAR"
  # unrar convert filenames to lower case, exclude paths and do not overwrite
  find $LOCAL -name "*.rar" -exec unrar x -cl -ep -o- {} $UNRAR \; >> $LOG 2>&1 &
}

init
case "$1" in
  download|d)
    download
    ;;
  unrar|u)
    unrar
    ;;
  both|b)
    download
    unrar
    ;;
  *)
    echo "Usage: $0 {download|unrar|both}"
esac

Tethering to Android Phone for Internet via Bluetooth

As mentioned earlier until I purchase a usb dongle I am running without a network. Just for kicks though I decided to setup tethering via bluetooth with my android phone. Which allows me to update packages etc without having to plug the raspberry pi into a network point.

Bluetooth Pairing

To make it a little easier I paired to raspberry pi from my phone.

# to check the raspberry pi bluetooth name
$ sudo hciconfig hci0 name

# to set the bluetooth name
$ sudo hciconfig hci0 name raspberrypi

# make the raspberry pi discoverable
$ sudo hciconfig hci0 piscan

# setup agent to listen for pairing request
$ sudo bluetooth-agent 12345
# from my phone I connected to raspberrpi-0 and confirmed code was displayed
Confirmation request of 7654321 fir device /org/bluez/1936/hci0/dev_0E_C9_35_42_E6_E9
[Ctrl-C]

# turn of discoverable
$ sudo hciconfig hci0 noscan

# check connected devices
# list currently connected bluetooth devices
$ sudo bluez-test-device list
BC:B0:0B:47:57:A8 matt’s keyboard
FA:C7:48:46:47:33 Logitech Bluetooth Mouse M555b
0E:C9:35:42:E6:E9 Galaxy Nexus

Setup Tethering

To tether to android phone for internet we need a little more configuration

# package required for 'pand' command
$ sudo apt-get install bluez-compat

# connect to phone and sets up network interface
$ sudo pand --connect 0E:C9:35:42:E6:E9 -n

# show networks
$ ifconfig
bnep0     Link encap:Ethernet  HWaddr 00:4d:9c:2a:47:94  
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:12 (12.0 B)  TX bytes:60 (60.0 B)
...

# set dhclient to use bnep0 
$ sudo dhclient bnep0

# test internet connectivity
$ ping www.google.com
PING www.google.com (74.125.237.145) 56(84) bytes of data.
64 bytes from syd01s13-in-f17.1e100.net (74.125.237.145): icmp_req=1 ttl=52 time=26.3 ms
64 bytes from syd01s13-in-f17.1e100.net (74.125.237.145): icmp_req=2 ttl=52 time=26.6 ms

Turning off tethering

To turn of tethering and unconfigure the PAN network

# release bnep0 dhclient
$ sudo dhclient -r bnep0

# disconnect pan
$sudo pand -K

Create an alias to make this tethering easy peasy

As I’ll probably turn tethering on and off lots of times I put all this into an alias

$ vi ~/.bash_aliases
alias leash='sudo pand --connect 0E:C9:35:42:E6:E9 -n && sudo dhclient bnep0'
alias unleash='sudo dhclient -r bnep0 && sudo pand -K'

After reloading bash I can simply type the following to tether or untether

$ leash
$ unleash

If you haven’t already got it installed you should also put Power Toggles on your phone so you can quickly turn wifi / bluetooth tethering on and off.

USB Bluetooth dongle and keyboard

As I am running my raspberry pi without a network (until I can grab a usb wifi dongle) I’m using a spare usb bluetooth dongle and bluetooth keyboard I had lying around.

Bluetooth Installation

$ sudo apt-get install bluez python-gobject

# check bluetooth dongle it recognised
$ hcitool dev
Devices:
	hci0	7B:13:75:FD:11:EB

# scan for discoverable devices
$ hcitool scan
Scanning ...
	BC:B0:0B:47:57:A8	matt’s keyboard
	FA:C7:48:46:47:33	Logitech Bluetooth Mouse M555b

# pair with keyboard
$ sudo bluez-simple-agent hci0 BC:B0:0B:47:57:A8
RequestPinCode (/org/bluez/3465/hci0/dev_BC_B0_0B_47_57_A8)
Enter PIN Code: 12345
# type pin + enter on keyboard
Release
New device (/org/bluez/3465/hci0/dev_BC_B0_0B_47_57_A8)

# pair with mouse
$ sudo bluez-simple-agent hci0 FA:C7:48:46:47:33
RequestPinCode (/org/bluez/2015/hci0/dev_FA_C7_48_46_47_33)
Enter PIN Code: 0000
Release
New device (/org/bluez/2015/hci0/dev_FA_C7_48_46_47_33)

# add devices to trusted list and connect
$ sudo bluez-test-device trusted BC:B0:0B:47:57:A8 yes
$ sudo bluez-test-device trusted FA:C7:48:46:47:33 yes
$ sudo bluez-test-input connect BC:B0:0B:47:57:A8 yes
$ sudo bluez-test-input connect FA:C7:48:46:47:33 yes

# reboot and test if keyboard connects
$ sudo shutdown -r now

# list currently connected bluetooth devices
$ sudo bluez-test-device list
BC:B0:0B:47:57:A8 matt’s keyboard
FA:C7:48:46:47:33 Logitech Bluetooth Mouse M555b

As I’m using an apple wireless keyboard I needed to jump into raspi-config and reconfigure my keyboard settings

$ sudo raspi-config
# 3 - Internationalisation Options
# I3 - Change Keyboard Layout (wait a little for it to load)
# set keyboard to Apple Aluminium Keyboard (ANSI) and default options for rest of prompts