I decided to switch to a low-power Mini-ITX based design rather than replacing the broken motherboard with a comparable one, and after a quick bit of research decided upon an Intel D945GCLF2 motherboard – which is based around the Atom 330 (dual core 1.6 GHz) processor, 2 Gb of RAM and a Morex Venux 668 case from ITX Warehouse. A complete PC (excepting disks) with change to spare from £150 – pretty impressive!

The parts arrived two days later (I ordered after the 5pm cutoff), and within a few hours everything was installed in the new case and ready to go. I was amazed as Debian booted off of the old disk without any issues (try changing the motherboard architecture on Windows without reinstalling from scratch!). After some slight issues with the network I eventually realised the new network card was coming up as eth1 rather than eth0. A quick check of google revealed that the old network card needed to be removed from /etc/udev/rules.d/70-persistent-net.rules. Following this everything was up and running as before.

The fan on the D945GCLF2 is apparently notorious for becoming very noisy after a few weeks, and the fans in the case are of a similar diameter, so I am expecting my next project will be to replace these with either low-noise fans or passive cooling.

I have personally been following a similar system also based upon Lifehacker’s trusted trio, but with “@” prefixes to the labels so they appear at the top of the list, filters so I can email todo items directly into my @action list and using the google labs Multiple Inboxes feature to display action/deferred/waiting emails alongside my inbox.

Somewhat loathe to do this, at a colleague’s suggestion I decided to instead try connecting via a VM running under VirtualBox. This has been my first foray into virtualisation (other than Xen) since trying out VMWare about a decade ago. VirtualBox seems to have all of the features I remember from VMWare (with the added advantage of being free), and it was fairly quick to set up an Ubuntu VM. It also seems that network support has vastly improved from my early memories and the VM had internet access without requiring any manual intervention. And of course there is now support for hardware virtualisation (AMD-V/VT-x) which should in theory make the VM run faster.

Unfortunately stability seems to be somewhat lacking, the VM seems to hang from time to time, requiring it to be “powered off” and restarted (as a virtual reset does not seem to work). Also the “Seamless Mode” support doesn’t seem to be as seamless as I had hoped – although it does remove the guest OS’s desktop, when windows are moved or resized it seems that the “hole” in the host desktop through which they are visible does not resize with them, so the window ends up being cut off when enlarged in size or part of the guest desktop becomes visible when reduced in size.

Still, not bad for a free virtual machine…

Even better, Western Digital have an Advance Replacement RMA program where the replacement drive is shipped immediately without needing to first return the faulty drive. They reserve the value of the replacement from your credit card, and this is not deducted as long as the faulty drive is returned within 30 days.

Since then things have gone pretty smoothly. Finally having the ability to copy and paste is a godsend, as are basic things one would expect from a phone like sending contacts to other people via MMS/email (unfortunately no SMS), receiving MMSes (although it took 12 hours for O2 to respond to my activation SMS) etc.

Spotlight (the search feature on the home screen) seems useful, although emails not downloaded from the server will not be searched. In order search mails on the server, the search function inside Mail has to be used – a search box has been added which appears when scrolling all the way to the top of a mailbox in email, and after searching emails on the phone, a “Continue Search on Server” option is presented which uses the IMAP SEARCH command to search mail on the server.

The phone screen also seems to have been pimped somewhat with more information visible on the recent calls listing to show which of a contact’s numbers (home/mobile/work/etc) the entry refers to. Had a problem for a few hours after the upgrade where the names of my favourite numbers had been replaced by their telephone numbers, but that seems to have gone away on its own.

The maps application looks very slightly different, with slightly different graphic for the GPS location. Also the iPod application now supports more precise scrubbing to find a position in a track – by dragging your finger down the resolution increases. Also there are a bunch more tweaks I haven’t yet tried…

Calendar Entries

1. On GMail calendar, goto Settings -> Calendars -> Export Calendars to download a ZIP of your calendars in ICAL format.
2. On Google Apps calendar, goto Settings -> Calendars -> Import calendar and import the ICAL entries to your calendar.

Contacts

To preserve contact groups, each group needs to be migrated individually.

1. On GMail contacts, use the Export option to export each group in Google’s CSV format. Then export one more CSV of “Everyone (All Contacts)”
2. On Google Apps contacts:
   1. Use the Import option to import the CSV of Everyone, leaving the  “Add these imported contacts to: ” option unchecked.
   2. Then use the Import option to import each group in turn, checking the “Add these imported contacts to: New Group… ” option, and enter the name of the group.

Google Sync

At this point you might want to switch Google Sync over to your Google Apps account. Follow the steps here, but enter your Google Apps email address instead of your GMail address. You then need to go through the steps here and sign in with your Google Apps account to select which calendars to sync. Remember to click the “Google Apps User?” link rather than trying to go through the normal GMail login page – your Google Apps email will not work there!

Email Filters

If you have any rules defined to filter/label your incoming mail, you can export these using  “Filter import/export” in GMail labs:

1. Go to Settings -> Labs and check “Enable” for the “Filter import/export” option on both your GMail and Google Apps account.
2. In your GMail account, go to Settings -> Filters, click Select All and then Export, and save the XML document somewhere.
3. In your Gogle Apps account, go to Settings -> Filters and then “Import filters”, select the file you saved in the previous step, and press “Open file”.

Email Delivery

You should now be ready to switch over your email. Hopefully your MX records already point at Google Apps and you have email forwarding set up from Google Apps to your GMail. If you don’t, you should set this up before you start trying to move your email.

Now is when you must commit to migrating by having all your new mail delivered to Google Apps instead of GMail!

1. Login here, select “Manage this domain”, and ensure that in Users and Groups you have a user set up for the Google Apps email address you want to migrate your GMail to. If you created a group to forward the email address to GMail you will need to delete this group before you can create the user.
2. Use the Inbox link at the top right to log in to the mailbox for the user (signing out if you are not currently logged in as that user).
3. Send a test email to your Google Apps email address to ensure that you are receiving email correctly.
4. Log in to your GMail account, go to Settings -> Forwarding and POP/IMAP and enable forwarding to your Google Apps address.

You should now be receiving all your email in your Google Apps mailbox. You may also need to update POP3/IMAP settings for any mail clients you are using.

Old Emails

The hardest part is now to migrate your old emails from your GMail mailbox to Google Apps. Google provide a number of officially supported options, however none of these are ideal – Mail Fetcher is only intended for new emails rather than migrating old emails, Email Uploader requires you to have all your email in Outlook, and IMAP Mail Migration requires you to upgrade to Google Apps Pro (but looks like a pretty good option other than that).

I decided to go for a free approach, using imapsync based upon these instructions. However I found that imapsync would frequently give up without copying the entire folder. I initially thought this was because the number of messages was too large, so I used the following script to split the mail into chunks by folder and also by date. I also tweaked the regexes to rewrite To/From/CC/BCC headers, and to be a little more tolerant of malformed headers:

#!/bin/bash

LOGFILE=imapsync.log
FOLDERS=( 'Folder1' 'Folder2' 'Folder3' 'Folder4' '...' 'FolderN' 'INBOX' 'All Mail' 'Bin' 'Drafts' 'Starred' 'Sent Mail' )
TIMES=( '--minage 1090' '--maxage 1091 --minage 999' '--maxage 1000 --minage 908' '--maxage 909 --minage 817' '--maxage 818 --minage 726' '--maxage 727 --minage 635' '--maxage 636 --minage 544' '--maxage 545 --minage 453' '--maxage 454 --minage 362' '--maxage 363 --minage 271' '--maxage 272 --minage 180' '--maxage 181 --minage 89' '--maxage 90' )

echo "Starting" > $LOGFILE

for FOLDER in "$@"
do
 for ((j=0;j<${#TIMES};j++))
 do
 TIME=${TIMES[${j}]}

 echo "" >> $LOGFILE
 echo "*** $FOLDER $TIME ***" >> $LOGFILE
 echo "" >> $LOGFILE

 while ! ~/imapsync-1.267/imapsync --host1 imap.gmail.com \
 --port1 993 --user1 user@gmail.com \
 --passfile1 ./passfile --ssl1 \
 --host2 imap.gmail.com \
 --port2 993 --user2 user@domain.com \
 --passfile2 ./passfile --ssl2 \
 --syncinternaldates --split1 100 --split2 100 \
 --authmech1 LOGIN --authmech2 LOGIN \
 --include "$FOLDER" \
 $TIME \
 --useheader "Message-ID" \
 --useheader "Date" --skipsize \
 --regexmess 's/Delivered-To: user\@gmail.com/Delivered-To: user\@domain.com/gi' \
 --regexmess 's/<user\@gmail.com>/<user\@domain.com>/gi' \
 --regexmess 's/^((To|From|Cc|Bcc):.*)user\@gmail.com(.*)$/$1user\@domain.com$3/gim' \
 --regexmess 's/Subject:(\s*)\n/Subject: (no--subject)$1\n/ig' \
 --regexmess 's/Subject: ([Rr][Ee]):(\s*)\n/Subject: $1: (no--subject)$2\n/gi' >> $LOGFILE 2>&1; do
 echo "" >> $LOGFILE
 echo "***** NOT COMPLETE - $FOLDER $TIME *****" >> $LOGFILE
 echo "" >> $LOGFILE
 tail -100 $LOGFILE | mail -s "Imapsync Restarting for $FOLDER $TIME" "user@domain.com"

 echo -n "Sleeping..." >> $LOGFILE
 sleep 1m
 echo "Done." >> $LOGFILE
 done

 echo "" >> $LOGFILE
 echo "***** COMPLETE - $FOLDER $TIME*****" >> $LOGFILE
 echo "" >> $LOGFILE
 tail -100 $LOGFILE | mail -s "Imapsync Complete for $FOLDER $TIME" "user@domain.com"
 done
done

echo "**** DONE ****" >> $LOGFILE
tail -100 $LOGFILE | mail -s "Imapsync Complete" -c "user@domain.com"

The script writes to a logfile, and emails the last 100 lines upon completion. I found this useful as some messages could not be copied. After some debugging I was able to locate these by searching for the Message-ID in my GMail to discover that they were all old messages imported from Outlook which had corrupted headers (newlines inside the header, so the Message-ID could not be found by imapsync). Manually deleting the corrupted messages from my GMail allowed the sync process to complete.

A DSL using the “Oslo” toolset that is capable of expressing order generation and responses from a range of market behaviors in terms of FIX messages in a dialect suitable for use by traders and financial engineers