Over the first few months of this year I have been taking part in a mass online learning course in Natural Language Processing (NLP) run by Stanford University.  They publicised a group of eight courses at the end of last year and I didn't hesitate to sign up to the Natural Language Processing course knowing it would fit very well with things I'm working on in my professional role where I'm doing more and more with text analytics and continuing my work in speech to text.  There were others I could easily have signed up for too, things like security or machine learning, more or less all of them are relevant for something I'm doing.  However, given the time commitment required I decided to fully commit to one course and the NLP one was to be it.

I passed the course with a grade of 85% which was well above the required 70% pass mark.  However, the effort and time required to get there was way more than I was expecting and quite a lot more than the expected time the lecturers (Chris Manning and Dan Jurafsky) had said.  From memory it was an 8 week course with 10 hours a week required effort to complete the work. As it went on the amount of time required went up significantly, so rather than the 80 hours total I think I spent more like 1½ times that at over 120 hours!

There were four of us at work (that I know of) who embarked on the course but due to the commitment of time I've mentioned above only myself and Dale finished.  By the way, Dale has written an excellent post on the structure and content of the course so I'd suggest reading his blog for more details on that stuff, there's little point in me re-posting it as he's written such a good summary.

In terms of the participants on the course, it seems to have been quite a success for Stanford University - this is the first time they have run courses in this way it seems.  The lecturers gave us some statistics at a couple of strategic points throughout the course and it seems there were around 40,000 people registering an interest, of which around 5000 were watching the lecture material and around 2000 completed the course having taken part in the homework assignments.

I'm glad I committed as much as I did.  If I were one of the 5000 just watching the lectures and not doing the homework material I don't think I would have got as much out of it, but the added time required to complete the homework was significant so perhaps there's a trade-off here?  It's certainly the first time I've committed this much of my own personal time (it took over the lives of myself and Dale for quite a few weeks) as I was too busy at work to spend many business hours working on the course so it was all done in evenings and weekends.  That's certainly one piece of feedback I gave at the end of the course, Stanford could make the course timing more flexible but also allow more time for the course to be completed.

My experience with the way the assignments were marked was a little different to the way Dale has described in his post.  I was already very familiar with the concepts of test, development and held-out sets (three different sets of data used when training NLP systems) so wasn't surprised to see that the modules in the course didn't necessarily have an exact answer to them or more precisely that the code your wrote to perfectly analyse some data on your local system may not get full marks as it was marked against a different data set.  This may seem unfair but is common practice in all NLP system training that I know of.

All in all, an excellent course that I'm glad I did.  From what I hear of the other courses, they're not as deeply involved as the NLP course so I may well give another one a go in the future but for now I need to get a little of my life back and have a well earned rest from education.

This post documents how I installed Linux and Windows side-by-side in a dual boot configuration.  This isn't something particularly difficult to do but I wanted to note down what I did so I remember in years to come.  Also, my new PC build contains some very up-to-date hardware (such as UEFI and an SSD drive) and with the combination of the many changes and updates to Fedora 17 (F17) and Windows 7 SP1 (Win7) it made the installation a first of a kind for me in quite a few different ways.

This post may well be updated in the future if I do further installation tweaks and optimisations to the system.  There's also clearly a lot more you can do in terms of system set up than I've written here, installation of drivers in Win7 very much being one of those.  This is simply intended as an installation and base optimisation guide.

Create USB Boot Sticks

Due to a fault with the DVD writer I ordered, it's on its way back to the retailer for replacement.  Not one to let this stop an installation going ahead I wrote install images for F17 and Win7 onto USB stick.

F17 is still in beta with the full release delayed until 29th May at the time of writing.  I tried writing and booting from the KDE Spin Beta Live image but it stopped with a kernel panic during the boot process after the grub screen.  I found using a later copy of the boot.iso (which is the same as a netinst.iso) from the distribution nightly builds solved the problem so clearly whatever the bug is on the beta image it's already been solved.

Creating a USB stick for Fedora is pretty easy these days, no messing around on the command line, I decided to use the liveusb-creator utility.  It fires up a GUI from which you basically just select the iso image you want to write and the USB device to write to, then hit "go" and it does the magic for you.

For me, writing the Windows stick was a little more complicated as it requires a Windows system to run the stick writing utility with the somewhat strange and long winded name of the Windows 7 USB/DVD Download Tool.  Fortunately, I had a Windows 7 VM laying around so I used that and writing the image to USB stick was similarly easy as it was for Fedora.  Simply choose the image file to write, the target device to write it on, then hit "go" and it does the magic for you.

Install Fedora 17

Once I worked around the kernel panic bug I mentioned above and booted with a later F17 image, the installation process was a fairly familiar affair.

With the new PC being a UEFI box there was no need to add the GPT partition when partitioning the SSD. I simply created one large root partition and a 4GB swap partition, leaving half the disk unallocated for the Win7 install later.

Since I didn't boot from the KDE spin image, I swapped out the default Gnome desktop (I've tried hard for a long time to Like Gnome 3 but I just don't so I'm moving back to KDE) and did a lot of package selection using the installer to save me messing around later but also to optimise the amount of data downloaded since I would be installing over my ADSL connection.

Once tip when installing Windows after Linux these days is to ensure the os-prober RPM gets installed.  This was done by default for me.  This package allows grub to detect the presence of other operating systems and add boot entries for them in the grub menu.  It'll come in very handy later on...!

SSD Optimisations for Linux

Even with a distribution as current as Fedora 17, the settings chosen for SSD usage are really rather sub-standard.  There are a lot of handy tips and guides out there for which settings you should change or add to the system to enhance both the performance and longevity of your SSD.  I decided to go with the rather comprehensive information provided for the Arch Linux distribution as they have a great wiki page on SSD optimisation.

Update /etc/fstab

Add the "noatime" and "discard" options to SSD partitions.  The discard option is the really super-important one as it turns on TRIM.

Mount /tmp as tmpfs by adding a line similar to the following:

Doing this allows the system to write temporary files to RAM instead of the SSD.  This will improve performance (RAM is faster than SSD) and reduce writes to the SSD (improving the life of the SSD).  I've added an option to increase the allowable size of the tmpfs file system to 75% of RAM from the default of 50%.  This means I can run compilations or other intensive tasks in RAM without ruining the SSD and get the performance benefits too.  With 16GB main RAM, this will allocate up to 12GB for my /tmp directory.  In normal usage I wouldn't expect to use anywhere near this but it's nice to have it there for when I'm not running too many apps but doing something else such as compiling RPMs.

Change the Scheduler

Optimise Kernel Parameters

With 16GB main RAM I'm not expecting to do much in the way of swapping.  However, I did add a couple of lines to /etc/sysctl.conf to make the system less likely to do so:
  # make the system less likely to swap
  vm.swapiness=1
  vm.vfs_cache_pressure=50

Install Windows 7

Similarly to the F17 install, the Win7 install proceeded with little in the way of drama.  I did select the advanced install option when given the chance but that was just to ensure Win7 installed into the free space I had left on the SSD rather than rudely splat my shiny new F17 installation.

And yes, for those of you wondering why I'm installing Win7 at all, especially dual boot rather than in a VM, it's simply for those times when only Windows will do... so gaming mostly I should think.

There's not much else to do with Windows after installation.  Unlike current Linux distributions it's said to  detect the presence of an SSD drive and apply the appropriate optimisations automatically.  Whether this is entirely true or not I suspect I'll never be any the wiser to.

Update the Boot Loader

It's been quite some time (about eight years) since I last built a PC for myself and I've been promising to build a new one for quite a while, now I've finally got round to ordering some parts.  Not that I need to, but I've justified the expense as a treat to myself after our annual bonus came through, having never really spent the bonus on anything particularly exciting before.

This is a short post detailing what I've gone for and why, not to show off, but so in years to come I can look back (as I have before) on what was available at the time and I'll have a record I can get to should I forget the detail of which parts are in my current PC.

I'm still the sort of geek that likes to build my own computers.  I like to do the research and put them together but it's still definitely the best way to get a good deal and the only way to get each component to be the exact one you want.  I also still like to have a PC for some reason, I have a laptop for work so I've already got something mobile and a PC just feels like the right solution for home use.

So with no further ado, on order are:

Case: I'm sticking with my old Antec SLK3700 case, okay so strictly speaking this isn't on order.

Processor: Intel Core i5-3570K (£175)
Sandybridge was a real game changer when it came out and firmly handed back the power to the hands of Intel  in the processor market.  I've been waiting for Ivybridge to come out for a while, either so I could buy a cheaper Sandybridge or plump for the Ivybridge if the price difference wasn't too great.  Well, for those who know their processors you'll see I've gone for the Ivybridge option.  Currently it's only 20-odd quid more than the equivalent Sandybridge processor and I think the extra expense is worth it to get the better on-chip graphics capabilities and minor improvements in speed and energy efficiency.

Motherboard: Asus P8Z77-V LX (£94)
I'm a sucker for a good Asus board, I've used them in most of my PC builds so that's where I start looking when I want a new machine.  I nearly went for the LE version of this board, it was another 20 quid, but I decided I wouldn't be using the extra features it provides (surround-sound audio and extra sata sockets) so I pocketed the difference and went for this one instead.

Memory: Kingston (4x4GB) DDR3 1600MHz XMP HyperX (£65)
I very nearly went for some Corsair low profile DIMMs but was swayed by the vendor support list for DIMMs for the motherboard chosen above.  I'm still in shock that you can get 16GB RAM for 65 quid!

Storage: Intel 120GB 520 Series SSD (£140)
I've got 4 hard disks in my PC at the moment, one of which is not connected, another is hardly used and the remaining 2 are coupled together in a striped RAID array in an attempt to get some sort of speed out of them.  Hard disks really are the bottleneck in your PC these days so I've decided to shove in a top notch SSD from Intel.  They're the only manufacturer to offer a 5 year warranty and also came highly recommended (under the Hitachi brand, Intel and Hitachi work together on their SSDs) from one of my respected colleagues in the storage department at work.  120GB should be ample for my storage requirements on the PC, the disk will be split to dual boot both Windows 7 64-bit (for those occasions when only Windows will do and a bit of gaming) and the main stay of Fedora x86_64.  My data other than the operating system already lives on a NAS.

PSU: CoolerMaster 600W Silent Pro Modular (£63)
Unfortunately my current 620W ATX power supply only has a 4 pin CPU connector rather than the currently common 8 pin EPS so I've begrudgingly had to fork out for a new PSU.  This CoolerMaster one gets some decent reviews for being quiet, efficient and delivering good consistent power within the tolerances required by ATX.  The 600W rating will also leave me with some overhead should I decide to whack in a high-end GPU at some point in the future.

Thermal Paste: Arctic Silver 5 (£5)
I've got some really old unbranded thermal paste knocking around somewhere but decided to invest in some decent stuff for this build so the Arctic Silver was the way to go.

DVD Writer: Samsung S222AL 22x with Lightscribe (£15)
I've got 2 optical drives at the moment, both are IDE and with the new breed of motherboards (or not necessarily even the ones that are particularly new) IDE is long since dead so I've opted to get a Sata DVD writer for this build.  Similar to the memory, I'm amazed you can pick something like this up for 15 quid!

Front Case Ports: Akasa USB 3.0 Card Reader (£20)
My current case has a couple of USB ports on the front, I thought it would be useful to throw a couple of USB 3.0 ports to the front of the case too.  This unit also has a built-in multi-card reader so I'll no longer have to hunt for my USB SDHC reader every time I want to copy pictures from my camera.

Case Fan: Antec TrueQuiet 120 (£7)
A 12cm fan for the front of my case, I've got a slot to fit another one in so I thought why not given the heat output I'd expect from this build.

Keyboard and Mouse: Logitech Desktop MK120 (£13)
My current keyboard is PS/2 and has seen better days.  I still like a simple keyboard with none of these funny curves or multimedia keys you can get these days so went for this cheap set from Logitech.

The obvious note in this build is a lack of a GPU.  As I mentioned above, I've left overhead in the power supply to put in a GPU in the future should I choose to do so.  I'm going to do this build and run on the GPU built into the processor.  I'll be interested to see what the performance of the HD 4000 is for my needs, if it's sufficient then great, otherwise I'll be tempted towards an NVidia GTX-560 card.  I guess it all depends on if I do a little more gaming then I do currently (which is next to none on the PC) and whether the HD 4000 is up to the job.

The other thing I've got my eye on is an up-rated cooler from the stock cooler supplied with the 3570K.  I quite like the look of the Corsair H60 Hydro should the need arise.

Photo by Beth

We've just finished a fairly major project at home in doing up our living room (right).  It's been fairly major in that the room is quite large and has an open stair well so we've had to include the stairs and landing too.  It's been a complete overhaul with the fire you see in the picture having been fitted, the upstairs ceiling plastered, new carpets, a little wall paper hung, painted and thoroughly decorated.  However, one thing I wanted to do as part of the redecoration was to run Ethernet cable between the TV point in the right of the picture all the way around the room past the door on the left to the computer point in the kitchen.  Basically, I've been long since fed up of wireless being slow, if not dropping out entirely.  So here follows a few notes to remind myself what I did with the hope it might be useful to someone else out there too.

I started out by planning the route for the cable and researching the kit I would need to buy in order to complete the job.  I decided to channel the cable into the wall where the fire is, using a channel I had the fire fitter make behind the fire when it was fitted.  For the rest of the run along the green wall and into the kitchen the cables are sunk into the back of the skirting board. Having replaced the skirting, I routed out a channel just big enough for 2 Ethernet cables and re-hung the skirting back in place.

The next decision was what type of cable to buy, Cat 5e or Cat6, Solid Core or not?  I already knew I wanted 2 cables, driven mainly by the fact it seemed pointless just to run one and a lot of Ethernet wall face plates have 2 sockets in them so that's what I decided on.  Then I went shopping.

I already had a couple of single metal backing boxes to mount face plates on, the rest of my shopping list consisted of the following for the sum of just under 40 quid:

• 2 x 20M Cat6 Shielded Network Cable, 20 Meters, FTP, Grey Ethernet Patch Lead
• 2 x RJ45 Face Plate Wall Sockets Cat6 Double 2 Port with Keystones Jacks
• TeckNet RJ45 and RJ11 N21CL Ethernet cable tester
• KRONE Type IDC Punch Down Tool Telephone/Cat5 Insertion
• RJ45 Ethernet Cable Connectors Cat6, UTP - Pack of 10 pieces
• RJ45 CRIMPER TOOLS
• Ethernet Network Cable kink protection hoods - Pack of 10 pieces

As you can tell from that little list, I decided to go with shielded Cat 6 and bought 2 cables with ends already attached.  I found it was cheaper to do this than to buy a reel of cable!  It was simple enough to cut the ends off and as you can tell from the rest of the list I decided to use whatever length was remaining to make up some Ethernet cables using the ends and crimper tool.

Having been with Vodafone for the best part of 10 years I decided it was time to move on and go with another network offering much cheaper rates, no idea why it's taken me this long to be honest.  I decided to go with Talkmobile as they came recommended from a colleague, have great PAYG and monthly rates and are based on the Vodafone network so I'll get the same coverage I'm already used to.

That all sounds great, except for the fact that moving to Talkmobile has turned into a nightmare of sorts.  You'd think it would be simple to switch network operators.  Basically, you get a Sim card from the new provider, ask your old provider for a PAC, then call the new provider and they arrange to transfer your number over.  Not so.  Here's my story.

Thu 29th March
Ordered Sim from Talkmobile.

Fri 30th March
Talkmobile Sim arrived.

Sat 31st March
Called Vodafone to get my PAC which was given with much less hassle than I was expecting, basically I just told them I found a better deal on another network and they gave me the PAC with no further questions or persuasion to stay with them.

Called Talkmobile support to arrange for my number to be ported to the new Sim.  I was informed because it was a Saturday the request would be queued and completed on Tuesday, no later than 5pm - 6pm.  No problem for me, I still had some Vodafone credit to use up as I was on PAYG with them.  I was told to call back on Wednesday after the porting was complete in order to register and set up the top up card.

Tue 3rd April
Called Talkmobile support at around 7pm and explained my phone was still responding to the temporary number and that my number porting had not yet completed.  The number I want to use is now disconnected and not responding at all so I'm now disconnected from anyone calling or sending me texts, either with the Talkmobile or Vodafone Sims.

After some time spent with them on the phone and going through the phone numbers and Sim numbers for both the Talkmobile Sim and the Vodafone Sim they concluded that the porting had been performed incorrectly and that the problem was with Vodafone as Talkmobile don't do their own number porting but use Vodafone to do it for them.  The fix for this is to do the porting again so another request was put in to port my number, but being late on Tuesday this would be queued on Wednesday for completion on Thursday.

Thu 5th April
The porting was successful, my number is now reconnected and people can call/text me once again.  I set about explaining to those people who had contacted me via other means that my number was now working again - I have no idea if anybody else tried contacting me during this downtime, there's no way of knowing.

I figured out I was now on a different price plan to the one I had originally purchased the Sim to use.  I had sent some texts to people to tell them my number was working again but realised I was over-charged compared to the text price I thought I should be paying.  I also noticed that when I checked my balance I was informed I had "3 minutes before lower rate" which wasn't something I would expect to see of the plan I wanted to use, the PAYG Essentials plan.

I called Talkmobile support again to get them to switch me back to the Essentials plan, figuring that during porting something must have got switched over.  The guy on the phone was very confused and told me the price plan was wrong (he easily switched me over to the one I wanted) and that mobile Internet connection, Voice Mail and MMS were not enabled for my Sim which he said he'd never seen before.  He was also easily able to add those to my Sim card so I could use those services.

I took the opportunity to register at this point and to connect my top up card to the Sim as I'd been informed I should do when I first called Talkmobile.

Fri 6th April
I decided to test the mobile Internet connection, not something I use a huge amount as I'm normally connected to a wireless network and hence why I don't bother with a monthly contract and go with PAYG.  The connection didn't work.  After a bit of searching around the Internet I found out that I have to manually configure the Access Point Name (APN) on my HTC Desire Android phone.

This was an entirely new concept to me as in all my years on the Vodafone network I'd never had to manually configure anything at all, it always "just worked" and was configured over the air to my phone.  The best Talkmobile can do here, however, is to provide a page of settings for Android users to manually configure their phones.  I followed the instructions to set up the APN on my phone and was unable to connect, getting a "connection failed" message.

I was going away for the weekend to see family and decided to wait until Tuesday after I could check my APN settings with my friend at work who is also with Talkmobile.

Tue 10th April
I checked the APN settings with my friend at work and was still unable to connect to the Internet using the mobile network, still getting a "connection failed" message.  Time to phone Talkmobile once again!

I spoke to the assistant (who are all very good, clear English speakers by the way - I'm getting a lot of experience with them) who directed me to the web page for Android settings.  In informed him that I'd already entered the settings and needed some further assistance.  As you would expect he double-checked the settings I entered with me and we decided they matched what was required.  He did some further debugging and got me to re-register my phone onto the Vodafone network and to reboot the phone but nothing appeared to work.  I was put on hold while he did some further checks, he came back to me and informed me there was nothing wrong with my phone but the Sim card is faulty.

The resolution this time is that I'm being sent a new Talkmobile Sim which I should receive in 2-3 days time and will have to go through a number transfer process once again when it has been received as I still want to keep my old number.

Wed 11th April
The replacement Sim arrived in the post so I called Talkmobile to get the number swapped between the original Talkmobile Sim and the replacement.  As I've become used to, the chap on the end of the phone was very helpful and arranged for the number transfer to take place.  It is, however, going to take another 24-48 hours to transfer the number again due to the fact they have to send spreadsheets of porting information over to Vodafone by email (so he said) for the number swaps to be done.

Fri 13th April
Checked the new Sim and my original Talkmobile Sim and the number transfer has not yet completed by 7pm - it was due by 6pm at the latest today.  I'm now wondering whether I should give Talkmobile more time or if it's time to consider looking at another network entirely as this really is becoming a shambles.

Called Talkmobile yet again.  I was informed that there has been an issue (which was not specified to me) with the number transfer and that this issue has been escalated but they are not able to give me more accurate information than that at this time.  I enquired when the transfer might happen and if I should wait indefinitely for the transfer to happen.  The guy on the end of the phone really had no idea and seemed to think there was an outside chance the transfer might happen on the weekend but thought Monday was more likely.  I was advised to call back on Monday for further status.

Tue 17th April
Called Talkmobile for a status update of the number transfer as it's still not completed 1 week after I initiated the process.  I'm told now that there is a backlog of number transfers outstanding and that my number swap is in the queue awaiting processing and there is some hope that it might be done by tomorrow.  Talkmobile offered a £5 credit to my PAYG account for the inconvenience.  I'm assured that one of the managers has my issue in hand and is going to encourage the number swapping team to complete the request as quickly as possible.

Wed 18th April
The good, my number has finally been moved across to the new Sim.  The bad, I still can't connect to the Internet - the very problem the new Sim was supposed to solve!  So, guess what?

A familiar call to Talkmobile to ask what's going on.  I'm told that now my number has been moved across I no longer have GPRS or MMS facilities enabled on my Sim card, so I'm back to the same place I was at on Thursday 5th/Friday 6th April when I couldn't connect to the Internet because my Sim is not enabled.  Talkmobile have to once again escalate the problem back to Vodafone, so it's yet another 24-48 hour wait for the turnaround of my issue.  At least this time the Talkmobile contact on the phone appeared to be taking some ownership of the issue and has said she will contact me either tomorrow or on Saturday with an update on the issue.

I feel sorry for Talkmobile in a way, they appear to be attempting to do everything they can to help and get the issue(s) sorted, but on the other hand I really have no idea where the incompetence lies throughout this whole story.  Once again, I'm left wondering how long it will be before I'll be asking Talkmobile for my PAC?  I've come this far so figure I may as well stick with it a little longer, but if it's not sorted fairly quick-smart from now I think I'll be taking another close look at GiffGaff.

Thu 19th April
I received a call from Talkmobile tonight.  Yes, it seems someone really has taken ownership of my problem and I'm finally getting some good customer service in addition to the polite customer service I've been receiving all along.  I was told the problem(s) should now all be fixed and I should try a manual roam on my phone to disconnect from the network and reconnect again before I should attempt a GPRS connection.  I was informed if that didn't work I should try a couple of times just to be sure.  I confirmed with the person on the phone just what that meant and what I should do on my phone.  We agreed to part company there and I would attempt the instructions with another call-back expected on Saturday to check how I got on.

So, I set about having dinner first of all (I was in the middle of cooking when the call came through).  But not entirely eagerly after that I turned on mobile data on my phone, no worky.  I did a manual roam, still no worky.  I did another manual roam, still no worky.  By this time I'm thinking there's still an issue but decided to allow a little time and try again later.

A couple of hours later and I've just done another couple of manual roam cycles and still got nothing.  I decided a to do a full reboot of the phone and when it came back up, bingo.... IT'S FIXED!!!

I'd like to thank the unnamed (unnamed on this post anyway) person on the end of the phone for finally taking some ownership and getting things sorted out.  I'm sure I'll find the network perfectly ok now I've moved across and you've saved your company a disgruntled customer - I'm merely just an annoyed customer now for the length of time and number of calls I've spent speaking to your service personnel.

In conclusion then, it took me eight calls to Talkmobile, one call-back and exactly 3 weeks to the day to transfer satisfactorily from Vodafone while keeping my number.

Fri 20th April
It's not over, until... well no fat ladies here I don't think!  However, with the saga behind us (I say "us" because I'm certainly glad it's sorted and think Talkmobile are too) I received a call this afternoon from one of the managers.  He was very gracious and first of all apologised for all the troubles I've documented above.  We went on to have a nice chat about the situation, whether there was anything else he could do for me, thanked me for my feedback both here and on Twitter, and he was appreciative of both my patience and for the lack of ranting (ranting is not my style) and good humour on this post.  I gratefully accepted a further £20 credit to my account in addition to the £5 I was offered on the 17th.  We parted company on good terms at the end of the call and I pointed out that all my calls have been courteous (as I've documented) and while my problems weren't always sorted out I appreciate having someone easy to understand (UK call centre) and polite on the other end of the phone, and I asked him to thank his staff.

Going forwards, as I've said before I'm really not expecting to have any problems.  My better half has initiated her transfer to Talkmobile, so unknown to them until now, they were in danger of losing 2 customers.  I wish her better luck than I had, but I really suspect my experience is a rare one-off kind of a situation.

So, the big final question is really whether I would recommend Talkmobile.  I'm hesitant either way at the time of writing.  Based purely on my experience joining it would probably be a negative, however, given the way the situation has been turned around, the likelihood it's a one-off, and assuming normal life with my mobile is resumed over the next month or two I would gladly make a positive recommendation.  So essentially it's a "well done" to Talkmobile, you could have properly screwed up and lost custom, but you didn't and I'm grateful for that.

I have a friend at work who occasionally writes in Perl and so has been learning the language over quite a while, picking up new bits when he needs to write something else. I'm generally his Perl sounding-board (among other topics) and he's getting pretty good at writing things from scratch himself now, even asking questions on things I've not done with Perl, not that I'm a massively advanced Perl programmer myself.

One of the things my friend regularly needs help with and I think confuses a lot of people with Perl is with variables, and in particular hashes. People often say to me they're not sure when to use dollar, percent or at-sign in their Perl variables. I knocked together some pretty noddy code to illustrate the various ways a hash can be used in Perl code as an example for him and thought it might be useful to a wider audience so posting it here.

The source code is below with the output from running this code presented afterwards. Syntax highlighting was done courtesy of perltidy.

#!/usr/bin/perl -w
#
# Some noddy hash examples
# Note: the syntactical differences can be quite subtle
# Note: hashes don't have guaranteed ordering (run this code)

use strict;

################################################################################
# lets play with a hash
################################################################################

# define a variable as a hash with some content
my %hash = (
    'a' => 1,
    'b' => 2,
    'c' => 3
);

print "print the hash value assigned to the key 'a'\n";
print $hash{"a"}."\n\n";

print "print all of the hash values\n";
foreach my $key (keys %hash) {
  print $hash{$key}."\n";
}

print "\nprint all of the hash values in order\n";
foreach my $key (sort keys %hash) {
  print $hash{$key}."\n";
}

print "print all of the hash values with associated keys\n";
foreach my $key (keys %hash) {
  print $key . " is " . $hash{$key}."\n";
}

print "\nprint all of the hash values with associated keys in order\n";
foreach my $key (sort keys %hash) {
  print $key . " is " . $hash{$key}."\n";
}

# get rid of the hash so we're 100% sure we're not using it below!
undef %hash;


################################################################################
# lets play with a hash reference
################################################################################

# define a variable as a hash reference with some content
my $hashref = {
    'a' => 1,
    'b' => 2,
    'c' => 3
};

print "\n\nprint the hash reference value assigned to the key 'a'\n";
print $hashref->{"a"}."\n\n";

print "print all of the hash reference values\n";
foreach my $key (keys %{$hashref}) {
  print $hashref->{$key}."\n";
}

print "\nprint all of the hash reference values in order\n";
foreach my $key (sort keys %{$hashref}) {
  print $hashref->{$key}."\n";
}

print "print all of the hash reference values with associated keys\n";
foreach my $key (keys %{$hashref}) {
  print $key . " is " . $hashref->{$key}."\n";
}

print "\nprint all of the hash reference values with associated keys in order\n";
foreach my $key (sort keys %{$hashref}) {
  print $key . " is " . $hashref->{$key}."\n";
}

The output from running this code is below:

print the hash value assigned to the key 'a'
1

print all of the hash values
3
1
2

print all of the hash values in order
1
2
3
print all of the hash values with associated keys
c is 3
a is 1
b is 2

print all of the hash values with associated keys in order
a is 1
b is 2
c is 3


print the hash reference value assigned to the key 'a'
1

print all of the hash reference values
3
1
2

print all of the hash reference values in order
1
2
3
print all of the hash reference values with associated keys
c is 3
a is 1
b is 2

print all of the hash reference values with associated keys in order
a is 1
b is 2
c is 3

We IBM employees are encouraged, indeed incented, to be innovative and to invent.  This is particularly poignant for people like myself working on the leading edge of the latest technologies.  I work in IBM emerging technologies which is all about taking the latest available technology to our customers.  We do this in a number of different ways but that's a blog post in itself.  Innovation is often confused for or used interchangeably with invention but they are different, invention for IBM means patents, patenting and the patent process.  That is, if I come up with something inventive I'm very much encouraged to protect that idea using patents and there are processes and help available to allow me to do that.


This comic strip really sums up what can often happen when you investigate protecting one of your ideas with a patent.  It struck me recently while out to dinner with friends that there's nothing wrong with failing to invent as the cartoon above says Leibniz did.  It's the innovation that's important here and unlucky for Leibniz that he wasn't seen to be inventing.  It can be quite difficult to think of something sufficiently new that it is patent-worthy and this often happens to me and those I work with while trying to protect our own ideas.

The example I was drawing upon on this occasion was an idea I was discussing at work with some colleagues about a certain usage of your mobile phone [I'm being intentionally vague here].  After thinking it all through we came to the realisation that while the idea was good and the solution innovative, all the technology was already known available and assembled in the way we were proposing, but used somewhere completely different.

So, failing to invent is no bad thing.  We tried and on this particular occasion decided we could innovate but not invent.  Next time things could be the other way around but according to these definitions we shouldn't be afraid to innovate at the price of invention anyway.

A nice little piece of code I came across quite a while ago and have been meaning to blog about ever since I wrote some code to extend it properly to the Rhythmbox music player is a generic cross-player autoqueue project.

The idea of the project is to generate interesting tracks to queue up automatically in your player while you're listening to music based on what you're currently listening to.  Put simply, seed your playlist with a few tracks of the sort you want to listen to, turn the plugin on and it will continue to populate your queue with similar tracks.  It's great when all you want to do is have some non-particular music in the background (say when you're coding for example) and you don't want to bother with managing what's playing right now.  If you're in the mood for some acoustic through to heavy metal, just seed with what you want and let autoqueue do the rest.

The Rhythmbox plugin for this project wasn't particularly mature when I picked it up so I've modified it so now I've got a nice little button in my player that I can use to switch the plugin on or off depending on whether I want to choose the tracks or allow the plugin to do it for me.  Autoqueue attempts to be generic about the players it supports by essentially providing a library that player plugins can talk to over a dbus interface.

The autoqueue project itself currently supports 2 mechanisms to determine track similarity.  There's what the project seems to hold as the primary method which is comparing the audio signal of the track you're listening to with those in your music library.  It uses an external library called mirage for this which was originally written for the Banshee player.  Then there's the method I prefer to use which is to call out to last.fm to ask for the similar tracks to the current one playing and queue up one from the list you already have in your library.  Eventually, I'd like to extend this so it takes into account your last.fm profile such that the similar tracks picked are ones you prefer to listen to.

While it's a great little project and I love this type of idea for generating dynamic playlists, the project is fairly stale, not much activity in development and the code repository is way more up-to-date than any released zip files.  However, if you're interested it's easy to get going so long as you know how to write a plugin for your player.

<edit>See comments from the project maintainer below for latest updates on project status at the time of writing.</edit>

Hint to where my first PC came from.

I was digging around clearing out a load of old paperwork from the house the other day and came across a load of old receipts and invoices.  I nearly threw them straight in the recycle pile but looked through and thought they actually contained quite an insight into the history of computer progress and pricing.

Not necessarily wanting to hang on to the paper I thought I'd make the info available here for future reference so here follows my computer history from way-back-when.  There's a chance not all of it was mine since some of it was delivered to friends houses and I've always regularly ordered kit on behalf of parents (who doesn't?), etc.

1995 June - My first PC system, we actually bought it from a shop in Basingstoke!

Pentium 60 PC with 540MB hard disk, 14" CRT monitor, keyboard, mouse, speakers, joystick, mouse mat, DOS 6.2, Windows for Workgroups 3.1.1, OS/2 Warp.  £1229.

1995 September - Wow, these things really were quite expensive!

10 Floppy disks.  £18.60.

1995 November - Replaced my Epson dot matrix (used with my Acorn A3010).

Canon BJC 4000 Printer.  £280.85.

1996 January - No idea why I kept the receipt for this. 

Worms.  £27.99.

1999 August/September -  I didn't invest in computer kit in the previous years (during university).

Borland Delphi 4 for Windows 95.  £89.30.

2x 16MB SIMM.  £47.00.

Maxtor Quantum Fireball Plus 13.6GB UDMA66 Hard Disk £138.65

120MB LS-120 Floppy disk driver.  £58.75

1x LS-120 120MB Floppy disk.  £10.57. 

Abit BE6 Motherboard.  £88.12

Pentium III 450Mhz Slot 1 Processor.  £88.12

32MB PC100 DIMM.  £31.00

2000 February - This was actually for some university work!

Creative Sound Blaster Live Player 1024 PCI sound card.  £56.40.

2000 August - Probably my first investment in photography as a hobby.

Agfa Snapscan e40 Flatbed Scanner.  £146.86. 

2001 January - Interesting hard disk, small (by today's standards) but fast.

46GB IBM PATA100 hard disk, 2MB cache, 7200 RPM.  £101.47.

Guillemot Force Feedback racing wheel.  £92.81. 

2001 June - First home network.

Dlink 8-Port 10Base-T Ethernet Hub.  £34.07

8cm cooling fan.  £5.87.

5 metre CAT5 patch cable.  £7.05.

2001 August - What on earth did I do with all these hard disks?

IBM Deskstar 20.5GB UDMA100 hard disk £69.32

Maxtor Fireball Plus 10.2GB UDMA100 hard disk £64.62

128MB PC100 168 pin CL2 DIMM model £18.08

2001 September - Not yet writing CDs or DVDs.

Pioneer Slot-in IDE DVD ROM, 16xDVD 40x CD £56.40

2002 May - I'm still using both of these today!

Asus 16xDVD 40xCD Player IDE

Epson Stylus Photo 790

 

Today sees the end of one of my major work streams for 2011 with a presentation of some research to our sponsors.  I've been working for a good chunk of the year researching text analysis, specifically, the automated expression of facts in controlled natural language.  It's always nice to see some work come to fruition, well not quite fruition in this case since it's research but at least it's reached an agreed stopping point - for now.

I haven't often been involved with relatively pure research in my day job so that coupled with leading the project presented a few challenges in itself which was most enjoyable.  While I can't give away the details, I wanted to express the areas this research concerned here.

The project was a text analytics project, not a new field in itself and a subject on which IBM and my local department (Emerging Technologies) contains many well read and respected experts.  For those of you not familiar, text analytics is essentially applying computer systems to text documents such that some sort of processing can be performed e.g. (simple example) the analysis of pages from news web sites to infer what the current news stories are.

One of the complexities we were investigating was natural language processing.  This is a major area of research for computer systems at the moment and presents one of the biggest problems of applying computer systems to human written documents.  Our brain is able to parse language in ways we've not yet managed to teach computers to do, taking into account context, slang, unknown terms and all sorts of other subtle nuances that make it a hard problem to crack for computers.

My recent work has been investigating how we can express things found in documents in the form of controlled natural language which leads to the question of what on earth is that?  Simply put, it's an expression made using normal words but using more rigid semantics than are found in pure natural language.  This makes it possible to parse it using a computer but it still feels fairly natural to the human reader as well.  This sounds great as you get computers talking a language that feels very usable to humans but with all the added power of memory and processing provided by the computer.  It seems to me this approach might only be a stop-gap solution until computers (inevitably it'll happen some time) eventually understand full natural language.

While having a discussion last night with my wife over dinner she expressed a sometimes-heard opinion from her that I occasionally "speak funny".  This came to light recently when on holiday in Ireland, I suspect it's a combination of both this type of research seeping into my use of language but also my semi-conscious approach to trialling these techniques in the real world and what better opportunity than when immersed in another English speaking culture.

So, as this article is published I'll be standing at the front of a room of people talking about the details of our work with my colleagues.  Wish us luck!