Speech to Text

Apologies to the tl;dr brigade, this is going to be a long one... 

For a number of years I've been quietly working away with IBM research on our speech to text programme. That is, working with a set of algorithms that ultimately produce a system capable of listening to human speech and transcribing it into text. The concept is simple, train a system for speech to text - speech goes in, text comes out. However, the process and algorithms to do this are extremely complicated from just about every way you look at it – computationally, mathematically, operationally, evaluationally, time and cost. This is a completely separate topic and area of research from the similar sounding text to speech systems that take text (such as this blog) and read it aloud in a computerised voice.

Whenever I talk to people about it they always appear fascinated and want to know more. The same questions often come up. I'm going to address some of these here in a generic way and leaving out those that I'm unable to talk about here. I should also point out that I'm by no means a speech expert or linguist but have developed enough of an understanding to be dangerous in the subject matter and that (I hope) allows me to explain things in a way that others not familiar with the field are able to understand. I'm deliberately not linking out to the various research topics that come into play during this post as the list would become lengthy very quickly and this isn't a formal paper after all, Internet searches are your friend if you want to know more.

I didn't know IBM did that?

OK so not strictly a question but the answer is yes, we do. We happen to be pretty good at it as well. However, we typically use a company called Nuance as our preferred partner.

People have often heard of IBM's former product in this area called Via Voice for their desktop PCs which was available until the early 2000's. This sort of technology allowed a single user to speak to their computer for various different purposes and required the user to spend some time training the software before it would understand their particular voice. Today's speech software has progressed beyond this to systems that don't require any training by the user before they use it. Current systems are trained in advance in order to attempt to understand any voice.

What's required?

Assuming you have the appropriate software and the hardware required to run it on then you need three more things to build a speech to text system: audio, transcripts and a phonetic dictionary of pronunciations. This sounds quite simple but when you dig under the covers a little you realise it's much more complicated (not to mention expensive) and the devil is very much in the detail.

On the audial side you'll need a set of speech recordings. If you want to evaluate your system after it has been trained then a small sample of these should be kept to one side and not used during the training process. This set of audio used for evaluation is usually termed the held out set. It's considered cheating if you later evaluate the system using audio that was included in the training process – since the system has already “heard” this audio before it would have a higher chance of accurately reproducing it later. The creation of the held out set leads to two sets of audio files, the held out set and the majority of the audio that remains which is called the training set.

The audio can be in any format your training software is compatible with but wave files are commonly used. The quality of the audio both in terms of the digital quality (e.g. sample rate) as well as the quality of the speaker(s) and the equipment used for the recordings will have a direct bearing on the resulting accuracy of the system being trained. Simply put, the better quality you can make the input, the more accurate the output will be. This leads to another bunch of questions such as but not limited to “What quality is optimal?”, “What should I get the speakers to say?”, “How should I capture the recordings?” - all of which are research topics in their own right and for which there is no one-size-fits-all answer.

Capturing the audio is one half of the battle. The next piece in the puzzle is obtaining well transcribed textual copies of that audio. The transcripts should consist of a set of text representing what was said in the audio as well as some sort of indication of when during the audio a speaker starts speaking and when they stop. This is usually done on a sentence by sentence basis, or for each utterance as they are known. These transcripts may have a certain amount of subjectivity associated with them in terms of where the sentence boundaries are and potentially exactly what was said if the audio wasn't clear or slang terms were used. They can be formatted in a variety of different ways and there are various standard formats for this purpose from an XML DTD through to CSV.

If it has not already become clear, creating the transcription files can be quite a skilled and time consuming job. A typical industry expectation is that it takes approximately 10 man-hours for a skilled transcriber to produce 1 hour of well formatted audio transcription. This time plus the cost of collecting the audio in the first place is one of the factors making speech to text a long, hard and expensive process. This is particularly the case when put into context that most current commercial speech systems are trained on at least 2000+ hours of audio with the minimum recommended amount being somewhere in the region of 500+ hours.

Finally, a phonetic dictionary must either be obtained or produced that contains at least one pronunciation variant for each word said across the entire corpus of audio input. Even for a minimal system this will run into tens of thousands of words. There are of course, already phonetic dictionaries available such as the Oxford English Dictionary that contains a pronunciation for each word it contains. However, this would only be appropriate for one regional accent or dialect without variation. Hence, producing the dictionary can also be a long and skilled manual task.

What does the software do?

The simple answer is that it takes audio and transcript files and passes them through a set of really rather complicated mathematical algorithms to produce a model that is particular to the input received. This is the training process. Once system has been trained the model it generates can be used to take speech input and produce text output. This is the decoding process. The training process requires lots of data and is computationally expensive but the model it produces is very small and computationally much less expensive to run. Today's models are typically able to perform real-time (or faster) speech to text conversion on a single core of a modern CPU. It is the model and software surrounding the model that is the piece exposed to users of the system.

Various different steps are used during the training process to iterate through the different modelling techniques across the entire set of training audio provided to the trainer. When the process first starts the software knows nothing of the audio, there are no clever boot strapping techniques used to kick-start the system in a certain direction or pre-load it in any way. This allows the software to be entirely generic and work for all sorts of different languages and quality of material. Starting in this way is known as a flat start or context independent training. The software simply chops up the audio into regular segments to start with and then performs several iterations where these boundaries are shifted slightly to match the boundaries of the speech in the audio more closely.

The next phase is context dependent training. This phase starts to make the model a little more specific and tailored to the input being given to the trainer. The pronunciation dictionary is used to refine the model to produce an initial system that could be used to decode speech into text in its own right at this early stage. Typically, context dependent training, while an iterative process in itself, can also be run multiple times in order to hone the model still further.

Another optimisation that can be made to the model after context dependent training is to apply vocal tract length normalisation. This works on the theory that the audibility of human speech correlates to the pitch of the voice, and the pitch of the voice correlates to the vocal tract length of the speaker. Put simply, it's a theory that says men have low voices and women have high voices and if we normalise the wave form for all voices in the training material to have the same pitch (i.e. same vocal tract length) then audibility improves. To do this an estimation of the vocal tract length must first be made for each speaker in the training data such that a normalisation factor can be applied to that material and the model updated to reflect the change.

The model can be thought of as a tree although it's actually a large multi-dimensional matrix. By reducing the number of dimensions in the matrix and applying various other mathematical operations to reduce the search space the model can be further improved upon both in terms of accuracy, speed and size. This is generally done after vocal tract length normalisation has taken place.

Another tweak that can be made to improve the model is to apply what we call discriminative training. For this step the theory goes along the lines that all of the training material is decoded using the current best model produced from the previous step. This produces a set of text files. These text files can be compared with those produced by the human transcribers and given to the system as training material. The comparison can be used to inform where the model can be improved and these improvements applied to the model. It's a step that can probably be best summarised by learning from its mistakes, clever!

Finally, once the model has been completed it can be used with a decoder that knows how to understand that model to produce text given an audio input. In reality, the decoders tend to operate on two different models. The audio model for which the process of creation has just been roughly explained; and a language model. The language model is simply a description of how language is used in the specific context of the training material. It would, for example, attempt to provide insight into which words typically follow which other words via the use of what natural language processing experts call n-grams. Obtaining information to produce the language model is much easier and does not necessarily have to come entirely from the transcripts used during the training process. Any text data that is considered representative of the speech being decoded could be useful. For example, in an application targeted at decoding BBC News readers then articles from the BBC news web site would likely prove a useful addition to the language model.

How accurate is it?

This is probably the most common question about these systems and one of the most complex to answer. As with most things in the world of high technology it's not simple, so the answer is the infamous “it depends”. The short answer is that in ideal circumstances the software can perform at near human levels of accuracy which equates to in excess of 90% accuracy levels. Pretty good you'd think. It has been shown that human performance is somewhere in excess of 90% and is almost never 100% accuracy. The test for this is quite simple, you get two (or more) people to independently transcribe some speech and compare the results from each speaker, almost always there will be a disagreement about some part of the speech (if there's enough speech that is).

It's not often that ideal circumstances are present or can even realistically be achieved. Ideal would be transcribing a speaker with a similar voice and accent to those which have been trained into the model and they would speak at the right speed (not too fast and not too slowly) and they would use a directional microphone that didn't do any fancy noise cancellation, etc. What people are generally interested in is the real-world situation, something along the lines of “if I speak to my phone, will it understand me?”. This sort of real-world environment often includes background noise and a very wide variety of speakers potentially speaking into a non-optimal recording device. Even this can be a complicated answer for the purposes of accuracy. We're talking about free, conversational style, speech in this blog post and there's a huge different in recognising any and all words versus recognising a small set of command and control words for if you wanted your phone to perform a specific action. In conclusion then, we can only really speak about the art of the possible and what has been achieved before. If you want to know about accuracy for your particular situation and your particular voice on your particular device then you'd have to test it!

What words can it understand? What about slang?

The range of understanding of a speech to text system is dependent on the training material. At present, the state of the art systems are based on dictionaries of words and don't generally attempt to recognise new words for which an entry in the dictionary has not been found (although these types of systems are available separately and could be combined into a speech to text solution if necessary). So the number and range of words understood by a speech to text system is currently (and I'm generalising here) a function of the number and range of words used in the training material. It doesn't really matter what these words are, whether they're conversational and slang terms or proper dictionary terms, so long as the system was trained on those then it should be able to recognise them again during a decode.

Updates and Maintenance

For the more discerning reader, you'll have realised by now a fundamental flaw in the plan laid out thus far. Language changes over time, people use new words and the meaning of words changes within the language we use. Text-speak is one of the new kids on the block in this area. It would be extremely cumbersome to need to train an entire new model each time you wished to update your previous one in order to include some set of new language capability. The models produced are able to be modified and updated with these changes without the need to go back to a full standing start and training from scratch all over again. It's possible to take your existing model built from the set of data you had available at a particular point in time and use this to bootstrap the creation of a new model which will be enhanced with the new materials that you've gathered since training the first model. Of course, you'll want to test and compare both models to check that you have in fact enhanced performance as you were expecting. This type of maintenance and update to the model will be required to any and all of these types of systems as they're currently designed as the structure and usage of our languages evolve.

Conclusion

OK, so not necessarily a blog post that was ever designed to draw a conclusion but I wanted to wrap up by saying that this is an area of technology that is still very much in active research and development, and has been so for at least 40-50 years or more! There's a really interesting statistic I've seen in the field that says if you ask a range of people involved in this topic the answer to the question “when will speech to text become a reality” then the answer generally comes out at “in ten years time”. This question has been asked consistently over time and the answer has remained the same. It seems then, that either this is a really hard nut to crack or that our expectations of such a system move on over time. Either way, it seems there will always be something new just around the corner to advance us to the next stage of speech technologies.

Killer Android Apps

This is my second blog post with this title with the first one having appeared in December 2010.  I thought it would be good to look over which apps I was using back then and of those which I'm still using now but also what new apps I'm using.  It feels like Android and the apps available for it advance at quite a slow pace so it'll be interesting to see the differences between the two blog posts and see what's changed in the last 16 months or so.  I've also updated from Android 2.2 to android 4.2.2 in that time.

Back then I was using the following list of apps regularly:

• Angry Birds (Game)
• Barcode Scanner (Bar Code Scanner)
• eBuddy (Instant Messenger)
• ES File Explorer (File Manager)
• Google Reader (Feed Reader)
• Maps (Navigation)
• RAC Traffic (Traffic/Navigation)
• Scrobble Droid (Social Music)
• Skype (Instant Messenger and VOIP)
• Todo List Widget (Productivity)
• TweetDeck (Social Client)
• Youtube (Video)

AccuWeather (Weather)

Back in the old days of Android 2.2 I was running an HTC Desire with Sense and there was a really good weather widget in sense which meant I didn't bother using a weather app.  Now I'm using Android 4.2.2 on a Nexus 4 and running the stock Google image that widget is no longer available.  I find the AccuWeather app to be a good alternative.  I also keep a mobile bookmark handy for the BBC weather page for my local area.

Amazon (Shopping)

I'm quite surprised that I've started using more dedicated apps.  I suppose they've got better over the years and there is a wider selection available.  If I used Amazon on my HTC Desire I would have done so via the mobile web browser interface but these days I tend to use the app instead.  That's probably got a lot to do with the fact I've got loads more storage space for apps on the new phone though.

Angry Birds (and Bad Piggies)

So I still play Angry Birds which is either a testament to the game or a worrying sign I'm slightly addicted.  A bit of both perhaps.  Now though, there's a whole series of these games to wade through as well as the spin-off Bad Piggies game which I also quite like.

BBC iPlayer and Media Player (TV)

It's not very often I use it but it's still handy to have installed.  It might get even more useful if I ever get round to buying one of the over-priced slimport adapters so I'd be able to pump out HD content from my phone over HDMI.

Bubble UPnP (DLNA Client)

This can come in really handy from time to time either to stream content from my NAS onto the phone but more often to use it as a DLNA controller for other devices in the house.

Business Calendar Free (Calendaring)

The standard Google calendaring app and widget leaves quite a lot of room for improvement in my opinion, hence I find this little free app to be a better alternative.

Chrome (Web Browser)

A default app on the Google Android install but I thought I'd mention it here anyway since it's a huge improvement over any other browser I've used on a mobile device (I've got Firefox installed as well but it's not as good in my opinion) and much better than the previous Android browser I was using.

Chrome to Phone (Link Sharing)

This is a great little app that allows you to ping links from your Chrome web browser (with an installed extension) straight over to your mobile phone.  Sadly, it only works in one direction though so you can't go from phone to desktop although there are third-party scripts that allow you to do that as well.

Dropbox (Cloud Storage)

Another app I don't use a huge amount but it's useful to be able to access my dropbox files on my phone if I need to.

ES File Explorer (File Management)

An app I've been using more or less since day 1 when I got my first Android device.  I think it's still the best file manager on the app store, look no further.

Facebook (Social Networking)

I've started to use the dedicated Facebook app a little more recently than I ever did before.  That's partly because it offers more functionality than I had available before (via either the web interface or Tweetdeck) and partly because Tweetdeck is going away soon and will no longer be available on Android.

Feedly (RSS Reader)

An app that Google forced me to discover recently because they appear to have got fed up with not making any money from Google Reader.  This appears right now to be the best alternative solution available on both Android and the web.  The Feedly producers currently use the Google API and back end for RSS reading but they are in the process of writing their own and are promising a seamless move when Google finally pull Reader in a few months time.  I'm sure more of these apps will pop up in the future now the market has opened up so it'll be interesting to see what I'm using next time I get around to writing one of these blog posts.

Flickr (Photography)

I use this app to upload the occasional photo I take on my mobile.  However, it's also one of those go to apps that I tend to read through more or less every day to see what's new on there.  Much like the Flickr web site itself, it's not exactly radical these days and passes for just about usable but if you're a Flickr user it's the best option I've found on Android if you want an app.

Hotmail (Email)

Previously I always used to use the in-built email app from HTC sense which was compatible with hotmail.  However, that's no longer available to me these days so I had to look around for an alternative when I got my new phone.  The official Hotmail app seems to do the job quite nicely.

LoveFilm (Movie Rental)

Again, another app that I don't use often and would previously have used the mobile web version instead.  However, this is handy to have sitting around for those times when someone recommends you a film and you want to shove it straight onto your rental list.

National Rail (Railways)

This can be incredibly handy for me in two situations, there's the obvious one where I'm using the rail network and want to plan routes, work out costs, see when trains are running and get live departure updates.  However, as the hubby of someone who uses the train network every day it can be really quite useful to spot when late arrivals home might occur or be armed with the latest information if the phone rings.

RAC (Traffic)

I'm still using this and finding it useful to check for traffic jams when I go on a long trip.  It's still not that great an app though so if there are better alternatives out there I'd love to hear about them.

Reader (RSS Reading from Google)

An app that I use every day, quite probably multiple times a day along with the Reader interface in my web browser.  Really very unfortunately indeed, Google have decided to think better of the Reader app and it's going away this year.

Rocket Player (Music)

I don't use Spotify, Google Play Music or other streaming services as I have a preference to copy my own music onto my device and listen to it locally.  I'd be doing this anyway if I were on a plane or something so I don't really see the point in streaming.  Rocket Player seems to be quite a nice app to play your music through.  It comes in a few different forms from a free version (which I use) to unlocking more features with paid versions.  It's simple yet functional and with their equaliser turned on it makes your music sound pretty decent (as far as phone + headphones go) too.

 Scrobble Droid (Social Music)

Another app I'm still using from the old days and more or less since day 1 on my first Android phone.  This simple little app works with music players (Rocket Player is compatible with it) and allows you to scrobble the tracks you're playing on your phone.  If you're not connected to a network then it'll save them up and scrobble when you make a connection instead.

Squeezer (Logitech Squeezebox Controller)

Unfortunately Logitech have neglected the Squeezebox brand since buying them a few years ago and that seems to be pervasive throughout everything Squeezebox including the Android app.  I'm unable to install the official app on the recent version of Android I have but fortunately someone has written an alternative in Squeezer.  I'm glad they did, I think they've made a better job of it than the official app.  Sure, it doesn't have half of the whizzy fancy features you get on the official app but given I don't use the vast majority of those I really don't care and would recommend anyone with a Squeezebox to give Squeezer a try instead.

Skype (Instance Messaging and VOIP)

Another app I've been using for ages on my phone.  It's great to finally be able to do video messaging now I have a front-facing camera too.  Also, the introduction of logging into Skype via your MSN account and compatibility with Messenger means I've stopped using the eBuddy app.

Stickman Games (Gaming)

Most of these are quite fun and generally absolutely hilarious.  I'd recommend anyone with a bit of time to kill to check out Play for the various stickman games, they range from golf to cliff diving with things like skiing, wingsuits and base jumping included in there too.

Talk

As a result of ceasing use of eBuddy I've started using the official Google Talk app instead which seems to be much better all-round than eBuddy for things like battery usage when staying logged into the app permanently.

Todo List Widget (Productivity)

This widget (or set of widgets) is still perfect for a really simple to do list on your phone.  It allows you to add different sized widgets to the desktop (which is a bit superfluous these days since you can resize widgets) and then simply add an item, remove an item and give the list a title.  All very easy.  All very simple.  Job done, don't need any more than that. 

Tweetdeck (Social Client)

Much like Google Reader this is an app I use every day or multiple times a day for reading and managing Twitter and to a lesser extent Facebook.  Another similarity is the announcement of the impending death of Tweetdeck on Android.  A couple of months ago I was looking around for Twitter clients as I know there's a huge range out there and was wondering if there was anything better than Tweetdeck.  I ended up installing 10 different apps to check them all out individually.  However, there wasn't anything I liked nearly as much as Tweetdeck (although one or two came close) because they all seem to rely so much on a huge amount of navigation and taps on the screen to see any content vs Tweetdeck's simple swipe-column interface.  I'm really apprehensive about what's next for Twitter on my phone.

YouTube (Video)

Another standard app, but it's still there on my phone.

So it seems in general I'm using a mixture of the same old apps I've always used, mostly because nothing better has come along rather than not trying to find new apps but I guess I'm probably a little stuck in my ways with the ones I'm using too.  There's definitely a theme of using more apps rather than mobile versions of web sites on my phone now which I put down to having loads more storage space available for installing them.  The list of apps is clearly quite a lot longer though so I'm using a lot more apps than I ever did as I've become a little more comfortable with using a touch screen mobile device and dipping in and out for content more regularly than I used to before my first smart phone.

Nexus 4

This post could as easily have been titled "buying my first mobile phone".  Yes, it's 2013 and I've just shelled out for my first ever new mobile phone, buying the Google/LG Nexus 4 so I thought I'd put something down about switching phones.  Prior to this as regular readers will know, I had an HTC Desire (which was actually a work phone), before that a Nokia N73 (which I bought second hand from eBay).  So I don't have a particularly prolific phone history, I tend to wait for what I want after researching long and hard and buying something that will last me several years (I'd say I change every 3 years or so).  With the price and feature set of the Nexus 4 as they are, it was a no-brainer next move for me so after the early rush on stock saw them getting sold out everywhere I waited and ordered when they came back in stock (on Feb 4th) and took delivery the very next day in spite of the 2 week wait Google were advertising for delivery.  Having had the phone a couple of months now, I'd say I've got used to it so now seems like the right time to talk about it.

So why the move to a new phone this time?

Basically, it wasn't at all driven by the Nexus 4, it was all about the HTC Desire.  I had been running non-standard firmware versions on it for quite a while and got fed up of the various instabilities in them, or out of date Android versions (for the more stable versions) but the real sticking point of the HTC Desire is that 150MB user space for apps.  That was OK back in the day, when apps were small and relatively few and far between in Android.  However, it quickly became far too little and I was either constantly battling to have the minimal set of apps I needed installed or copying them out to SD card in a custom firmware and suffering the slow-down consequences of doing that.

OK, so I decided I needed a new phone but why the Nexus 4?

Price.  Simple.  End of.  Having been lucky enough to have had the HTC Desire since it first came out, I've been used to a high end smart phone for quite a while.  A lot of the cheaper phones on the market today aren't really much better than the HTC Desire even now, so I needed to look high end.  Obvious choices were other top notch phones from HTC or perhaps a move to Samsung for the Galaxy S3.  However, looking at the high end market as it is right now, you just can't beat the Nexus 4 for "bang for your buck".  It's a high end phone offering all the features of the S3 and other similar phones but at a whisker over half of the price, so no contest.

What do I like about the Nexus 4?

I'm going to split this into two parts.  Software and then Hardware.

Well it's Android, and being a Google phone it's a bang up to date version too.    I can still use all the same apps I know and love from the HTC Desire (with the odd change of widget here and there as I've moved away from HTC Sense, obviously) but now I can have them all installed at the same time and running and comparative speed.  There are three things that come to mind when talking about the software differences.  First, the gesture typing keyboard seems really quite adequate to me; I was a Swype user before now but I've not even been remotely tempted to install Swype on the Nexus 4 as the gesture typing from Google more or less exactly replicates the experience with a few subtle differences.  Second, Google Now really does seem very clever indeed; within a couple of days it had worked out where I live and where I work so I get traffic updates before I make the journey; I get similar information if I've travelled somewhere I don't usually go; recently on holiday in Spain, Now welcomed me with local weather information, an estimate of the time back to the airport, a list of local restaurants and attractions, etc.  Third, the camera software completely blows me away; yes there's the photosphere camera which is a nice toy but the in-built ability to do time lapse videos, HDR pictures, panoramic pictures (which get stitched automatically) as well as all the various scene modes, editing facilities, location and social functions they've put in the app make it absolutely first class - my two cameras (a compact that is a few years old and an SLR) get absolutely nowhere near this level of functionality.

Hardware wise, the tech specs speak for themselves being dual core, 2GB RAM, nice screen, camera, etc.  Speaking of camera, I've been particularly impressed by the quality of the pictures and video from it against what I had before with the HTC Desire in addition to the camera software I've already talked about.  I could go on and on about what I like, the list is really long since it's more or less bigger, better and more powerful than the HTC Desire as you'd expect being released 3 years later.

So what's not to like?

If I was an Apple fan-boy I'd say "it's not an iPhone" but more seriously there are probably two downsides that I can think of.  You know it's coming... battery life.  Until humankind invents a more compact version of storing more electricity then battery life is always going to suck.  I can get 2 days from the phone with light usage but under heavier load I can get 1 day out of it.  The other thing is the size.  Preferably, I'd like to be able to break the laws of physics somehow and have a phone with a gorgeous massive screen that is absolutely tiny, unrealistic at the moment.  It is bigger than the HTC Desire and I would say, while thin, is still quite a large phone.

Going Back to University

A couple of weeks ago I had the enormous pleasure of returning to Exeter University where I studied for my degree more years ago than seems possible.  Getting involved with the uni again has been something I've long since wanted to do in an attempt to give back something to the institution to which I owe so much having been there to get good qualifications and not least met my wife there too!  I think early on in a career it's not necessarily something I would have been particularly useful for since I was closer to the university than my working life in age, mentality and a bunch of other factors I'm sure.  However, getting a bit older makes me feel readier to provide something tangibly useful in terms of giving something back both to the university and to the current students.  I hope that having been there recently with work it's a relationship I can start to build up.

I should probably steer clear of saying exactly why we were there but there was a small team from work some of which I knew well such as @madieq and @andysc and one or two I hadn't come across before.  Our job was to work with some academic staff for a couple of days and so it was a bit of a departure from my normal work with corporate customers.  It's fantastic to see the university from the other side of the fence (i.e. not being a student) and hearing about some of the things going on there and seeing a university every bit as vibrant and ambitious as the one I left in 2000. Of course, there was the obligatory wining and dining in the evening which just went to make the experience all the more pleasurable.

I really hope to be able to talk a lot more about things we're doing with the university in the future.  Until then, I'm looking forward to going back a little more often and potentially imparting some words (of wisdom?) to some students too.

New Thinkpad W530

It's been quite a while since I got my last laptop upgrade at work, coming up to 5 years in fact.  We have a 4 year refresh programme so I'm a little overdue but have just been given a shiny new Thinkpad W530 from Lenovo.  This seems to be our current standard issue machine for "developers" which is our way of saying "power users".  I'm part of the software development business and hence get one of these.  The up side of course is the latest and most powerful technology at a reasonably high specification, the downside is they're really quite big and heavy and the power brick - well it really is a brick.  I'll spare giving a full review of the laptop itself as there are plenty of them out there already and you'll know how to find them, however, there are one or two things I wanted to say about the machine and in particular regarding my preferred use of Linux rather than the software it comes pre-installed with.

Here's the specification highlights of the machine I've got (there is a bit of variation available with the W530):

• Intel(R) Core(TM) i7-3720QM CPU @ 2.60GHz (5187.87 bogomips in Linux)
• 16GB DDR3 1600MHz
• 500GB (7200rpm) HDD
• 15.6" 1920x1080
• 9 Cell Battery
• Wireless N
• NVidia Quadro K1000M
• Front Facing Web Cam, Mini Display Port, VGA Out, Headphone, 2x USB3, 2xUSB2.0, Smart Card Reader, Firewire,DVD Burner, GBit Ethernet

It came with firmware version 2.07 which was only 3 months old but had already been superseded by two newer versions when I got it earlier this week (there is a firmware readme available).  The newer versions fixed a couple of well known issues with screen corruption under Linux and the fan always running at full speed (and hence being noisy).  So I downloaded and applied the updated version before I did anything else.


The next thing I did was tweak a few settings in the BIOS to my liking and install Fedora 18 with the KDE desktop.  The installation went very smoothly using the Integrated graphics card on the Ivy Bridge CPU.  The W530 has Optimus built in for efficient switching between the integrated card and discrete NVidia card for a great combo of power/performance, it is however, designed for Windows and Linux support hasn't quite caught up yet although there is an open source option available - which I'm yet to try.  Post installation I installed the latest NVidia drivers available from the RPM Fusion repository (304.64) ready to switch to using the graphics subsystem in discrete only mode.  The advantage of this is greater graphical processing power and also the ability to use external display devices.  The integrated graphics card is only able to drive the laptop screen and doesn't output via the VGA port or display port.  The down side to the NVidia card is a greater power draw so reduced battery life.  Also, at the time of writing the Nouveau driver doesn't support the Quadro K1000M card so you're forced into using the proprietary driver.  This situation can only improve over time and hopefully Optimus support will grow in Linux too but I'm not holding my breath on that one given NVidia's attempt to put support into the Linux kernel was refuted by the kernel developers last year due to it not being GPL code.

Away from the graphics subsystem which was always going to be the most difficult thing under Linux on this machine, the rest of  it appears to be very well supported.  There are a few bits and pieces I haven't quite got around to trying in the couple of days since I got it but my impression is generally quite good.  Speed, as you would expect is very good although nowhere near my home machine which is a similar specification but contains an SSD instead of HDD.  Consequently, I put the speed boost I see at home down to this more or less entirely.

I've also moved away from Gnome (I don't get on with Gnome 3) and gone back to using KDE once again which I had moved away from 5 years ago when I installed my previous laptop as KDE 4 was pretty shocking at the time as well.  I've used KDE a lot more than I have Gnome in terms of years of elapsed usage but I did get on very well with Gnome 2 for the past 5 years and I'm sure I'll miss it.  That said, I can't see myself ever moving to Gnome 3 unless the developers go back on their current manifesto of treating users like idiots.  It'll be interesting to see how the Mate desktop progresses and whether XFCE picks up as well given they both have benefited from Gnome 3's unfortunate design decisions and have a much smaller community of users and developers than either Gnome or KDE.

In general then, I'm pleased with the new machine.  It's up and running to my liking in a very short period of time.  The graphics are bound to be a pain until I get used to relying on the nvidia-settings utility once again.  However, the other benefits it brings in terms of larger memory and greater processing power over my old machine are probably worth it.

Upgrading Fedora 17 to 18

It's been a while since I talked about upgrading Fedora but back in late 2009 and early 2010 I wrote about upgrading my home PC from one version of Fedora to the latest version at the time:

• http://gibbalog.blogspot.co.uk/2009/11/upgrading-fedora-10-to-11.html
• http://gibbalog.blogspot.co.uk/2009/12/upgrading-fedora-11-to-12.html
• http://gibbalog.blogspot.co.uk/2010/05/upgrading-fedora-12-to-13.html

These were about the time (shortly after) the Fedora project introduced its latest piece of tooling, used for upgrading between distribution versions, called preupgrade.  Several years down the line and my PC and the more complex configuration than average that caused a few issues at the time have changed.  Preupgrade has been changed too, dropped in fact.  The Fedora project have moved away from it in favour of producing a new tool called Fedup (yes, yes, nice name).

It really was about time that preupgrade was replaced, as far as I know it was only ever produced as a short-term solution to upgrading Fedora without the need for booting from CD/DVD/USB/PXE and going through the Anaconda upgrade process.  It was a proof of concept if you like.  It's also been openly described within the project as "just a set of hacks" and other such non-complementary terms.  So why the need for something new?  Well I think that's already been best described elsewhere buy the guys behind Fedup so I'll just point you at a little background information on Fedup instead.

Having tried it first at work to upgrade a virtual machine from F17 to F18 I found that Fedup actually works pretty well, in fact to the user it doesn't really appear to do much differently to preupgrade.  You simply start the tool, tell it what to upgrade to, wait while it downloads the packages, then reboot to perform the upgrade, all done.  I recently tried it at home as well and it worked perfectly, no issues at all, which seems to be the experience of all the others users I've talked to about using it as well.  So far then it's a bit thumbs-up and thanks to the Fedup and Fedora teams for making it all so easy for us.

The one pitfall is that you can't upgrade straight from F16 to F18.  Traditionally with Fedora you can jump a version so you've always been able to upgrade directly from any current Fedora version that hasn't gone end of life yet to the very latest without going through intermediate versions.  Between F16 and F18 that wasn't possible though so if you're still on F16 you'll need to preupgrade your way through F17 before doing a Fedup to F18 if that's where you want to get to.  However, from F17 onwards Fedup should support the same version jumping as preupgrade did, it's simply that Fedup isn't available until F17 that causes this particular issue.

Broadband Woes

At the end of November 2012 my until then rock-solid reliable Netgear DG834PN ADSL router started reporting being disconnected from the Internet. I know this because being a decent router it has the facility to email its logs to me periodically so I have a record of everything that goes on with my Internet connection going back for years. I eventually worked out that the disconnections were being caused by phone calls connecting to the house which can happen even when we're out because we have an answering machine. So then to work, on diagnosing whether the fault lies with my equipment or with one of the service providers (Plusnet using BT line).

First I tried connecting my Netgear to the master phone socket since usually it is connected to an extension running around the house. No joy. The Netgear exhibited exactly the same behaviour from both sockets and disconnected when phone calls came in or went out of the house (only on connection, not when the phone was ringing).

Next is to try some different micro filters. No joy. I was still getting disconnected in spite of trying a couple of different ones while connected to the master socket including a genuine BT micro filter I purchased specifically to get a decent quality one for this test.

Now I've got this far I'm starting to think it's a line fault and prepare to do battle with Plusnet to get them to assess my line and perhaps send out a BT engineer to fix the problem. I go on-line to their help system. Their phone system is OK being in the UK but you have to go through a million-and-one press this to do that options. In order to raise a fault you have to answer a bunch of questions to confirm you've made all the checks that I would take for granted. Master phone socket, tick. New micro filter, tick. New router, now hang on a minute here. To the best of my (comparatively extensive) knowledge this question isn't right, routers aren't effected by phone calls, the entire point of an ADSL micro filter is to filter the line so you don't get noise during phone calls.

It seems Plusnet would not allow me to raise a line fault with them unless I had gone through all of these questions. I thought I couldn't honestly answer that I had tried a new router in spite of the fact I couldn't see how a new one would solve the problem. I decided to humour the web page and try a different router. I put the call out on social media and quickly tracked down someone a few streets away from us who had a spare I could try.

The router I borrowed was a cheap and nasty get the job done type router sent out by TalkTalk to their customers when they sign up. No wonder the guy I borrowed it from didn't use it and preferred his trusty old Netgear much like myself. However, it worked! No more disconnections when the phone connected and that was the case for whichever micro filter I used and whether I was connected to the master phone socket or the extension. So then, problem solved, it's my router after all.

Unable to keep the router I had on loan I did a bit of research into which router I should buy next to replace my Netgear. I'm a bit of a Netgear fan-boy, I have a ReadyNAS as well so it was hard to admit, based on hundreds of reviews from dissatisfied customers, their routers are now crap. I thought about going straight to an FTTC connection but wanted to get something fixed in the short term so a cheap router to keep us going for a while is the order of the day. I couldn't decide what to get and noticed Plusnet “give” you a router for a small fee to cover the postage. I decided to go with one of these.

A couple of weeks later, after I had reminded Plusnet to send me the router, it seems they received the order and took my money but somehow forgot to put it in the post, I was the proud new owner of a router made by Technicolor (or Thompson) called the TG852n. Wow wow wow, it really is quite the most appalling box of circuits, wires and firmware I've ever had the displeasure of being insulted by. It's the sort of thing I imagine an electronics company something akin to Frugal Electronics Inc. would produce, corners cut everywhere which can only be to save development cost, and credits the user with absolutely intelligence whatsoever.

I tweeted about what a bunch of crap itis and Plusnet responded wondering what I thought the issues were with it. It was difficult to explain in a 160 character response so I picked a couple of annoyances amongst plethora of wrongness that is the TG852n and sent them back. Here's my top 5:

1. The user interface is restricted, very restricted. You cannot change anything but the very simplest of options. The first thing I wanted to do was switch it over to using my usual subnet range, nope, can't do that, you're fixed at the address range you're given. That is, unless you use the command-line interface. Most people reading this will know that I'm a fan of the command-line more than most being Mr Linux, but I really don't want to learn another one just to configure my router at home, give me a user interface that works. This brings me onto...
2. You can telnet to the router. I didn't have to do anything to do this, you can just log in with the admin user name and password. I don't want that port open on my router thanks very much although it is comforting to know the router is a capable device scuppered only by an incapable user interface. What if I were to accidentally route port 23 from the Internet? No, just no.
3. On the subject of open ports, port 80 appears to be open from the Internet. I don't know why, I didn't tell it to do that. Stop it. I haven't checked what other ports might have been left conveniently hanging for someone to try and get in. Remember I said I have logs going back years, I know how often my IP address is port scanned and password cracked. It happens most days.
4. Firmware updates are more or less not supported. You certainly can't update the firmware via the user interface (surprise surprise) and if you do want to manually update it then you have to do it via TFTP (presumably via PXE boot) from a machine on your home network. Nope again, no thanks, that's rubbish. It seems, however, that Plusnet can push firmware updates to your router. I don't want that either, get off my network[1], I'll manage my end and you stick to managing yours. Finding firmware updates is nearly impossible too, they're not supplied by the manufacturer so you're left to trawl the Plusnet forums looking for posts where someone has linked to a more up-to-date version than you currently have.
5. It's a four port Ethernet router, it has one Ethernet light. Apparently, I'm too dumb to need to see whether traffic is being routed to a certain port on the router via the convenient mechanism of a green light provided by almost every router on the planet[2]. I'm left with one light that might wave about a bit if some traffic happens to pass through the router to the Ethernet ports at some point. Oh, and they're 100MB/s ports too, but what do you expect for a free router I guess?

The final straw for me in the TG832n story and why I've decided to stop using it is that it also disconnected when the phone rang the other day. It reconnected shortly afterwards and everything was fine. However, it's now in a state where sometimes it will disconnect if the phone rings and sometimes it will not. Any which way, it always shows a red “Internet” light and a green “Broadband” light which implies that I have an active ADSL connection but that I am not dialled through to Plusnet properly. Another victory for this router then, with the red light on and glowing brightly I've got full Internet access from any of my wired or wireless devices in the house. So the router is lying to me as well[3]. I could try a firmware update, but that would be difficult.

I appreciate I'm probably not the target audience for this router. Its cut down and simplified user interface that buries what it considers “advanced” options in completely unfathomable places is probably targeted more at the likes of the silver surfer, someone who just wants to plug it in and get connected to the Internet without worrying about it any more than that.  However, the TalkTalk supplied router I have on loan (a Huawei HG532) is also a little confusing to use but it blows the Plusnet one out of the water in terms of the configuration options offered via the web GUI and would surely confuse the heck out of all but the most technical users.

Before I decided to try the Plusnet standard router my research was leading me towards Asus and (as I said before) away from Netgear. I've not tried any Asus networking kit before but I've like their motherboards for years in my home builds and we have a Transformer Prime tablet at home which is excellent. I've yet to receive this one but hope it solves the disconnection problem as well as being something that doesn't exhibit quite so many of the things I've been ranting about above. If it doesn't solve my connection issues then there may be more to say here yet.

[1] I don't really expect that Plusnet would be able to get onto my network but it's a distinct possibility.

[2] Yes, I know it's just a hub and isn't switching the traffic so is effectively broadcasting all the traffic anyway in which case one light is sufficient but I'd rather have 4.

[3] Another possibility is that the light stays red to suggest there has previously been a problem. The wafer-thin documentation actually says when the Internet light is red “Connection to Internet failed. Restart your router and see page 7. if the problem persists, open your web browser and see page 12.”

<update 5th Feb 2013>
I'm now using an Asus DSL-N12U that I purchased and reviewed at Amazon.
</update>

Combining Different Versions of Dojo

During a project I've been working on for the past couple of months my office mate and I needed to work out how to combine a recent version of Dojo and use it on a page where an old version of Dojo was already present.  Our reasons were simple, we were writing a Dojo Mobile app and hence required the use of a new version of Dojo.  However, we were displaying content within our app from a product that embeds an older version of Dojo.

With not much in the way of documentation on how to do this available, I asked our local Dojo guru for some advice and he pointed me towards the "noGlobals" option in dojoConfig.  This is the key to getting things working as it stops the new version of Dojo from writing to the global name space i.e. when you include the old version of Dojo it sets up a global variable called "dojo" if you simply include the new version of Dojo it will also write to the "dojo" global variable.  That means which ever version of Dojo you include last will always be the one you're using when you reference the "dojo" variable.  With the "noGlboals" option in place (which is adhered to by Dojo 1.7 and above I believe) you can continue to use old versions of Dojo from the global name space in conjunction with locally scoped newer versions of Dojo.

It appears there's still not a great deal of examples out there on how to do this so this is my attempt to put together a simple example to show how it's done by combining the ancient Dojo 1.3.3 with the latest at the time of writing Dojo 1.8.1:

<html>
<head>
<title>Dojo Mix-Up</title>

<!-- 
  lets start by including a really old version of Dojo 
-->
<script type="text/javascript" 
  data-dojo-config="isDebug: false, parseOnLoad: false" 
  src="dojo-release-1.3.3/dojo/dojo.js">
</script>

<!-- 
  configure Dojo to load without trampling the global namespace
-->
<script language="JavaScript" type="text/javascript">
  var dojoConfig = {
    noGlobals : true,
  };
</script>

<!-- 
  load the latest version of dojo
-->
<script type="text/javascript" 
  data-dojo-config="isDebug: false, parseOnLoad: false"
  src="dojo-release-1.8.1/dojo/dojo.js">
</script>

</head>

<body>

<!--
  Shove in a couple of divs to write to
-->
<div id="oldDojo"></div>
<div id="newDojo"></div>

<script type="text/javascript">
  // output our global namespace Dojo version (1.3.3)
  dojo.byId("oldDojo").innerHTML = dojo.version;
  
  // output our new Dojo version (1.8.1)
  require(["dojo/dom", "dojo"], function(dom,dojo){
    dom.byId("newDojo").innerHTML = dojo.version;
  });
</script>

</body>
</html>

HTC Desire Cyanogen Mod

I have the HTC Desire phone that to all intents and purposes is a decent phone even by todays standards and was an excellent phone when it was released some 2½ years ago at the time of writing.  That is, with the rather fundamental flaw for a phone that boasts an App Store - a complete lack of internal storage space.  It was built with 512MB flash storage which after HTC had muddled with it was cut to a mere 150MB for users to install applications.  That wasn't so bad back in the day, but these days you just have to install a standard set of Google apps and a few other utility applications and you're done, no more room.

The plight of the HTC Desire was giving me enough grief that I decided to do something about it by flashing CyanogenMod on it.  This like any third-party phone hack is a risky business but I figured if it failed miserably I could always restore the original HTC firmware or more likely get rid of the phone and move to something like the Samsung Galaxy S3 instead.  It turns out Cyanogen is quite the saviour of the HTC Desire and since the instructions for installing it were somewhat less than clear I wrote down what I did based on a Cyanogen forum post and a wiki page, so here it is...

I re-iterate what the forum post states:

#include <std_disclaimer.h>
/*
* Your warranty is now void.
*
* I am not responsible for bricked devices, dead SD cards,
* thermonuclear war, or you getting fired because the alarm app failed. Please
* do some research if you have any concerns about features included in this ROM
* before flashing it! YOU are choosing to make these modifications, and if
* you point the finger at me for messing up your device, I will laugh at you.
*/

Backup
I found it really useful to take a backup of things that were on the phone and put them to one side for safe-keeping both in case something went a little wrong but also to help with restoring the phone when running the new OS.  There are utilities you can use to backup your contacts (or sync them with your Google account), backup your SMS messages, even backup your saved games if you're that worried about carrying over your Angry Birds records to the new phone.  I also took a complete copy of the SD card so I had all the pictures and other things that had accumulated on there as well.

My Phone
BRAVO PVT1 SHIP S-ON
HBOOT-0.93.0001
MICROP-031d
TOUCH PANEL-SYNT0101
RADIO-5.11.05.27
Aug 10 2010,1752:18

Why don't you just S-OFF?
All the instructions talk about S-ON and how to go about gaining S-OFF without really saying anything about what this is.  It turns out that HTC implement a self-protection feature on the phone that is designed to prevent phone hacks and third-party installations that they call S-ON (presumably security on?).  With this switch set in the "on" position your phone cannot be modified.  Unfortunately for HTC they left a security flaw on some of their devices that means this switch can be altered to the "off" position.  Hence, the first thing that must be done to an HTC phone is use some software to take advantage of this and set S-OFF.

Use Revolutionary to set S-OFF
You need to find out some information about your phone similar to the info I've put above.  This will be different between models but for the HTC Desire you simply turn the phone off, then turn it on again while keeping the volume-down key depressed until the information is displayed.

The guys responsible for the software at revolutionary.io insist that you give them some information on your phone which turns out to be a sensible idea since they can attempt to ensure that this beta software doesn't screw up your device or that if it does they can prevent it being installed on similar devices in the future.  You simply go to their web site, fill out a form and download the software.

Put the phone into USB Debugging mode (Settings -> Applications -> Development -> USB debugging) and run the software.  It needs to be run as root under Linux.  It does it's thing and then reboots the phone, you'll now see you've got S-OFF set instead of S-ON if you view your phone information with the volume-down button trick once again.

One other note about revolutionary, being an open source enthusiast, is that they don't provide the source code.  Doing so would expose the security flaw that the software takes advantage of and thus would allow HTC to patch their phones accordingly and stopping this sort of modification instantly.

Use ClockworkMod to Back Up your current HTC firmware
Revolutionary gives you the option of installing ClockworkMod which is definitely worth doing.  ClockworkMod is an improved ROM manager and boot loader with some more advanced features over the stock HTC loader that comes with the phone.  One of these advantages is the ease of which a full backup of the phone's flash memory can be taken.  This appears to be called a NANDroid backup if you come across that term.  I assume that it's essentially just booting into a small program from the boot manager without accessing/mounting the flash device such that the entire flash device can be consistently backed up - a bit like a partimage or dd copy of an unmounted hard disk or partition under Linux.

Boot the phone into the bootloader (this is the same volume-down while holding power when the phone is turned off trick).  Use the volume keys to navigate up/down the menus and the power button to select the desired menu item.  You need to head for the "recovery" option which will take you through to ClockworkMod.  Select the "backup and restore" option to create a backup of the phone as it stands right now - this will give you a fighting chance of restoring the phone as-is should something go pear-shaped or for any other reason you decide to return to the stock HTC firmware.

The backup is placed on the SD card so you need to boot the phone again, access the SD card from your computer with a USB cable and store the backup image safely away somewhere - this is really important since the SD card will be wiped later along with your backup unless you copy it somewhere else first!

Get CyanogenMod Binaries onto the Phone
I grabbed the following files but there may be later versions now:

• http://download.cyanogenmod.com/get/jenkins/3075/cm-7.2.0.1-bravo.zip
• http://cmw.22aaf3.com/gapps/gapps-gb-20110828-signed.zip
• http://cmw.22aaf3.com/gapps/gapps-gb-20110828-newtalk-signed.zip

The first of these is the CyanogenMod package that will be your new operating system on the phone.  The other two packages are Google apps installed on top of the new OS.  I forget why you need the Google ones, but presume you can't get them from the Play Store on Android.

You'll need to copy these into the root directory of the SD card, still using the stock HTC firmware.

Install CyanogenMod
Boot the phone back into the recovery mode with the volume-down trick and selecting the "recovery" option to access ClockworkMod once again.  This time select the "Wipe data/factory reset" option and let that do its thing.  Then select the "Wipe cache partition" and let that complete as well.  Then you need to select the "Choose Zip from SD card" option and select the CyonagenMod zip file.  This will install the OS onto your phone after which you can choose the "Choose Zip from SD card" option again to select the Google apps zip and once again for the other Google apps.

During the installation you should create an ext data partition on the SD card that will be used a little later after the installation has completed (keep reading).  I have the stock 4GB class 2 micro SD card that came with the HTC Desire still in my phone so I opted to carve that up to use 1GB for data and keep 3GB free for use as a normal Android SD card partition.

Reboot
Reboot the phone and simple as that, you've got rid of HTC and you're booted into something that feels a lot closer to what you get on a Samsung phone i.e. a fairly stock version of Android which in this case is is 2.3 Gingerbread.

Install and Configure S2E
This app is the primary reason for installing Cyanogen, it's not the OS that's so much the saviour of the HTC Desire but the fact that using an app such as S2E (simple2ext) you can format part of your SD card for use as internal storage and thus completely resolving the tiny storage problem.  S2E is only compatible with Cyanogen and as far as I'm aware you can't do anything equivalent in the HTC stock firmware.

Once the phone has booted into Cyanogen for the first time, just fire up the Google Play store and find the S2E app.  Install it to the phone and run the configuration part of the app.  This lets you pick which parts of your installation you want to move to the ext partition you created on the SD card.  You can move over whatever you think you need to and when you're done just reboot the phone once again.

The S2E configuration will have installed a little script into the boot sequence of the phone such that on this reboot (i.e. the next reboot after running the S2E configuration) the script run and move the data you have selected onto the SD card.  It will only do this each time after you've run the S2E configuration and made changes.  These reboots can take quite a while since you could be moving quite a lot of data from your internal storage onto the SD card so be patient while it completes and boots back into Cyanogen.  Once you've done that you're good to go with using the phone again.

Final Thoughts
Having worked through all that, you'll need to tailor your phone to how you like it once again.  Restore the backed up SMS messages, contacts, files, etc.  You'll need to completely re-download any apps you liked on the HTC, configure those, set up widgets, etc, etc.  All standard stuff really, except this time you'll actually have some space in which to install things.

I'm currently using Cyanogen 7.2.0.1 which seems stable enough, it has crashed the phone a couple of times, just after I installed it but now it seems to have settled down and is running absolutely fine.  Running apps from the SD card using S2E can seem to make the phone rather laggy too so tweaking the settings in the S2E configuration to give a good balance of data stored on the SD and on the internal memory seems to be wise, along with tweaking other settings such as the read buffer.  I dropped the read buffer from the standard 2MB down to 512KB and that seems to have removed the lag.  Another way of reducing lag would be to update the SD card, say to one of the latest class 10 cards in which case it'll probably be quicker than the internal memory anyway!  I've not done that as yet and seem to be running just fine as I am for now.

Building a New PC

This post is related to a couple of my recent posts, the New PC Build post that explains the kit I ordered for this build with a little reasoning thrown in for good measure, and the New PC Install Notes post where I dumped a few of my thoughts about the software installation.  However, this post is specifically about the hardware build.

This is what I started out with, a set of shiny new boxes and an old PC which would be gutted back to the case and have only the case reused.

It's a nice Antec case and I couldn't find anything much better around today so didn't see the point in buying another one even if it did mean I had to strip it out before I started.  This is the shot with the case stripped back and all the stuff I used to have in there strewn across the bed I was doing the build on.

I only had 1 case fan in my previous build and decided to upgrade to include 2 fans in this build by including this Antec fan.  It seems to me that a lot of the new cases you buy these days have moved away from the ATX specification for front-to-back airflow as they have all sorts of fans and vents all over the place.  I quite like the idea of front-to-back airflow to give a nice continuous stream of air to the parts inside.

Here's the fan mounted in the front of my old case.  It was supplied with some little rubber grommets for screwless fixing into any case.  However, my case already had the purple clips you can see in this picture in place so I had no need for the grommets and the fan just clipped into place very quickly and easily.

Next up it was the power supply.  Here's the one I chose while still shrink-wrapped.

It turns out this is probably the most over-packaged PSU in the world.  The box is designed to give the wow factor when you open it, the likes of which are usually reserved for high-end consumer products such as a squeezebox, it has a fold-out design in a glossy box to reveal a very neatly packed PSU with all the cables hidden underneath.

The fan fitted pretty quickly and easily as well, the build was going well so far.

One of the things I use my PC for most is for photos.  I decided when putting together a spec for the new machine to include an internal card reader for convenience.  I also wanted to mount a USB 3 socket on the front of the box so this little reader from Akasa seems to fit the bill perfectly.

Here's the card reader after being unpacked and fitted.  It's much like fitting an old floppy drive, being a 3½" device and you simply connect it up to power and in this case the USB 3 motherboard header and it's job done.  I've not included a floppy drive in this build so it also makes the front of the case look less bare.

One of the indulgences of this build, given they're expensive against a traditional hard disk, is the inclusion of an SSD drive.

Here's the content of the box, you get a molex to SATA converter (presumably in case you're fitting it to some ancient power supply) as well as the fixing screws, a SATA cable and conveniently a 2½" to 3½" mounting bracket - which I needed as my case doesn't have any 2½" mounts in it.  You also get the manual and CD but those are still sealed ;-)

This is round the back of the SSD after fitting, inside the case.  You can see it's a pretty simple affair as you'd expect with just a SATA and power connectors on the back.  You can also see how the 2½" to 3½" mounting bracket works from this picture.  The bundle of wires at the top of the picture is coming from the back of the Akasa USB 3 card reader and slot.

There's plenty of better pictures of the Asus P8Z77-V LX board out there, but this is my board fresh out of the box and ready to be fitted into my case.

This is the board in place in my old case.  It was really very easy to fit as I'd removed an old Asus board before fitting this one and all the mounting holes were in exactly the same place.

Here's the new Ivy Bridge 3570K CPU out of the box with some Arcitc Silver 5 thermal compound.  Interestingly the stock cooler doesn't look too bad and is certainly much easier to fit than previous coolers I've worked with.

As I said above, I was pleasantly surprised to see the stock cooler being so easy to fit.  It was (as has been usual for coolers) a screwless installation with the four plastic fasteners secured in place with a simple firm (but not too firm) press towards the motherboard.

Here's the memory in box, ready to be fitted.  As usual for DIMMs it was a pretty undramatic installation process as they slotted in very nicely next to the CPU.  These don't have any huge fins on them for cooling and have a very simple heat sink arrangement making them low profile so I had no worries about overhanging coolers interrupting the fitting of the DIMMs since I would be using all four slots in the above picture.

All in all it was a pretty painless installation process and I've been using the machine for a couple of weeks already without any real issues.

Natural Language Processing Course

Over the first few months of this year I have been taking part in a mass online learning course in Natural Language Processing (NLP) run by Stanford University.  They publicised a group of eight courses at the end of last year and I didn't hesitate to sign up to the Natural Language Processing course knowing it would fit very well with things I'm working on in my professional role where I'm doing more and more with text analytics and continuing my work in speech to text.  There were others I could easily have signed up for too, things like security or machine learning, more or less all of them are relevant for something I'm doing.  However, given the time commitment required I decided to fully commit to one course and the NLP one was to be it.

I passed the course with a grade of 85% which was well above the required 70% pass mark.  However, the effort and time required to get there was way more than I was expecting and quite a lot more than the expected time the lecturers (Chris Manning and Dan Jurafsky) had said.  From memory it was an 8 week course with 10 hours a week required effort to complete the work. As it went on the amount of time required went up significantly, so rather than the 80 hours total I think I spent more like 1½ times that at over 120 hours!

There were four of us at work (that I know of) who embarked on the course but due to the commitment of time I've mentioned above only myself and Dale finished.  By the way, Dale has written an excellent post on the structure and content of the course so I'd suggest reading his blog for more details on that stuff, there's little point in me re-posting it as he's written such a good summary.

In terms of the participants on the course, it seems to have been quite a success for Stanford University - this is the first time they have run courses in this way it seems.  The lecturers gave us some statistics at a couple of strategic points throughout the course and it seems there were around 40,000 people registering an interest, of which around 5000 were watching the lecture material and around 2000 completed the course having taken part in the homework assignments.

I'm glad I committed as much as I did.  If I were one of the 5000 just watching the lectures and not doing the homework material I don't think I would have got as much out of it, but the added time required to complete the homework was significant so perhaps there's a trade-off here?  It's certainly the first time I've committed this much of my own personal time (it took over the lives of myself and Dale for quite a few weeks) as I was too busy at work to spend many business hours working on the course so it was all done in evenings and weekends.  That's certainly one piece of feedback I gave at the end of the course, Stanford could make the course timing more flexible but also allow more time for the course to be completed.

My experience with the way the assignments were marked was a little different to the way Dale has described in his post.  I was already very familiar with the concepts of test, development and held-out sets (three different sets of data used when training NLP systems) so wasn't surprised to see that the modules in the course didn't necessarily have an exact answer to them or more precisely that the code your wrote to perfectly analyse some data on your local system may not get full marks as it was marked against a different data set.  This may seem unfair but is common practice in all NLP system training that I know of.

All in all, an excellent course that I'm glad I did.  From what I hear of the other courses, they're not as deeply involved as the NLP course so I may well give another one a go in the future but for now I need to get a little of my life back and have a well earned rest from education.

New PC Install Notes

This post documents how I installed Linux and Windows side-by-side in a dual boot configuration.  This isn't something particularly difficult to do but I wanted to note down what I did so I remember in years to come.  Also, my new PC build contains some very up-to-date hardware (such as UEFI and an SSD drive) and with the combination of the many changes and updates to Fedora 17 (F17) and Windows 7 SP1 (Win7) it made the installation a first of a kind for me in quite a few different ways.

This post may well be updated in the future if I do further installation tweaks and optimisations to the system.  There's also clearly a lot more you can do in terms of system set up than I've written here, installation of drivers in Win7 very much being one of those.  This is simply intended as an installation and base optimisation guide.

Create USB Boot Sticks

Due to a fault with the DVD writer I ordered, it's on its way back to the retailer for replacement.  Not one to let this stop an installation going ahead I wrote install images for F17 and Win7 onto USB stick.

F17 is still in beta with the full release delayed until 29th May at the time of writing.  I tried writing and booting from the KDE Spin Beta Live image but it stopped with a kernel panic during the boot process after the grub screen.  I found using a later copy of the boot.iso (which is the same as a netinst.iso) from the distribution nightly builds solved the problem so clearly whatever the bug is on the beta image it's already been solved.

Creating a USB stick for Fedora is pretty easy these days, no messing around on the command line, I decided to use the liveusb-creator utility.  It fires up a GUI from which you basically just select the iso image you want to write and the USB device to write to, then hit "go" and it does the magic for you.

For me, writing the Windows stick was a little more complicated as it requires a Windows system to run the stick writing utility with the somewhat strange and long winded name of the Windows 7 USB/DVD Download Tool.  Fortunately, I had a Windows 7 VM laying around so I used that and writing the image to USB stick was similarly easy as it was for Fedora.  Simply choose the image file to write, the target device to write it on, then hit "go" and it does the magic for you.

Install Fedora 17

Once I worked around the kernel panic bug I mentioned above and booted with a later F17 image, the installation process was a fairly familiar affair.

With the new PC being a UEFI box there was no need to add the GPT partition when partitioning the SSD. I simply created one large root partition and a 4GB swap partition, leaving half the disk unallocated for the Win7 install later.

Since I didn't boot from the KDE spin image, I swapped out the default Gnome desktop (I've tried hard for a long time to Like Gnome 3 but I just don't so I'm moving back to KDE) and did a lot of package selection using the installer to save me messing around later but also to optimise the amount of data downloaded since I would be installing over my ADSL connection.

Once tip when installing Windows after Linux these days is to ensure the os-prober RPM gets installed.  This was done by default for me.  This package allows grub to detect the presence of other operating systems and add boot entries for them in the grub menu.  It'll come in very handy later on...!

SSD Optimisations for Linux

Even with a distribution as current as Fedora 17, the settings chosen for SSD usage are really rather sub-standard.  There are a lot of handy tips and guides out there for which settings you should change or add to the system to enhance both the performance and longevity of your SSD.  I decided to go with the rather comprehensive information provided for the Arch Linux distribution as they have a great wiki page on SSD optimisation.

Update /etc/fstab

I've only got one SSD in my system at the moment, I've removed all my old hard disks with no intention to use them right now as all my data is stored on my NAS.

Add the "noatime" and "discard" options to SSD partitions.  The discard option is the really super-important one as it turns on TRIM.

Mount /tmp as tmpfs by adding a line similar to the following:

  tmpfs /tmp tmpfs nodev,nosuid,size=75% 0 0

Doing this allows the system to write temporary files to RAM instead of the SSD.  This will improve performance (RAM is faster than SSD) and reduce writes to the SSD (improving the life of the SSD).  I've added an option to increase the allowable size of the tmpfs file system to 75% of RAM from the default of 50%.  This means I can run compilations or other intensive tasks in RAM without ruining the SSD and get the performance benefits too.  With 16GB main RAM, this will allocate up to 12GB for my /tmp directory.  In normal usage I wouldn't expect to use anywhere near this but it's nice to have it there for when I'm not running too many apps but doing something else such as compiling RPMs.

Change the Scheduler

The recommendation for an SSD is to move away from the default cfq IO scheduler and switch over to the noop scheduler.  This can be done in a variety of ways that are documented in the Arch Linux wiki page I've linked above.  Since I've only got 1 disk in my machine and it's an SSD I changed the scheduler option using grub.

For Fedora 17 this consisted of an edit to the /etc/default/grub file and adding elevator=noop to the existing GRUB_CMDLINE_LINUX stanza.  Then rebuilding the grub configuration with the command:

grub2-mkconfig > /boot/grub2/grub.cfg

Optimise Kernel Parameters

With 16GB main RAM I'm not expecting to do much in the way of swapping.  However, I did add a couple of lines to /etc/sysctl.conf to make the system less likely to do so:
  # make the system less likely to swap
  vm.swapiness=1
  vm.vfs_cache_pressure=50

Install Windows 7

Similarly to the F17 install, the Win7 install proceeded with little in the way of drama.  I did select the advanced install option when given the chance but that was just to ensure Win7 installed into the free space I had left on the SSD rather than rudely splat my shiny new F17 installation.

And yes, for those of you wondering why I'm installing Win7 at all, especially dual boot rather than in a VM, it's simply for those times when only Windows will do... so gaming mostly I should think.

There's not much else to do with Windows after installation.  Unlike current Linux distributions it's said to  detect the presence of an SSD drive and apply the appropriate optimisations automatically.  Whether this is entirely true or not I suspect I'll never be any the wiser to.

Update the Boot Loader

So Win7 didn't kill off my F17 installation which is always a bonus, but it does assume divine rights over the entire system so writes its boot loader all over the existing grub installation allowing you only to boot windows with no menu options for anything else - not ideal.  Now I need to re-install grub which I've always found easiest to do by booting a rescue Linux CD and using a chroot to the installed OS.  This has changed slightly with the inclusion of grub 2 so here's what I did.

Boot the F17 USB stick once again but this time choose the "troubleshooting" boot option and select to boot the "rescue environment".  Red Hat based systems have always done a great job of rescue environments so you'll boot into a text based system that asks you if you want various things turned on in the rescue environment such as networking (although they assume you want that these days) and if you want to attempt detection of installed Linux systems (which you do).

Drop out to the shell prompt and chroot to /mnt/sysimage.  Then rebuild the grub config (this is where os prober installed above comes in very handy) to include an entry for booting Win7.  Then re-install grub.  Job done.  Once you've booted to the rescue shell, the commands are something along the lines of:

  chroot /mnt/sysimage

  grub2-mkconfig > /boot/grub2/grub.cfg

  grub2-install /dev/sda

  exit

  exit

  reboot

This assumes you want to install grub to /dev/sda of course.  You'll need to exit (or ctrl+d) twice, once to get out of the chroot and once more to return to the rescue interface.  Then choose to reboot from there as it'll cleanly unmount your file systems and do a better job of clearing up than if you simply rebooted the system yourself.