As you might imagine, that many meter readings accumulate into a dataset which is pretty big, and it’s been a lot of fun trying to wrangle it into shape. The sensors have come in all shapes and sizes, and all levels of reliability, so (as always) a major part of the work has been in cleaning and making sense of the data, before actually being able to do a lot with it. The good news is that the results are great – an average reduction in emissions of over 60%, with some homes reducing by more than 80% (fivefold!). There’s also a load more we could do with the same dataset – we haven’t even begun to dig into the behavioural and other factors that are revealed by this very fine-grained information.

So for now, take a look at the report, and you can register to gain your own access to the raw data at https://est.amee.com/. We’ll be releasing the analysis code shortly, as well. 

A couple of recent ones that are up online:

Using data for EVIL! at the Strata Data Warfare conference

Big Data comes to the NHS at ODI Fridays lunchtime lecture

More public talks coming soon:

Open Data for CSR, Intelligent Futures, 12th Feb

R and Shiny at Data Science London, 21st Feb

R and NHS data, LondonR, 19th March

Devoxx, 26th-27th March

It’s very exciting to see something we’ve made be covered so extensively – it was on the front page of the Independent, on the PM programme on Radio 4 (47 mins in), the Economist, the FT, and even in the Revolution Analytics R blog as well as various medical and pharma blogs.

It’s been interesting the different ways this piece of work has been interpreted by different journalists. In the text on the site, and when being interviewed, we were very careful to emphasise the context of the data and that NHS prescribing advisers already do a lot of work on this issue. Most of the responses we had seemed to understand this very well, and it’s been great to see how many NHS employees have been in touch both to say that they enjoy it and to suggest future developments that would be helpful. We did get a few sensational headlines, but most of the coverage was brilliantly balanced, with Radio 4 and the Economist being especially impressive in the care they took to ensure they portrayed the issue fairly.

Anyway, the amount of value that this small project has been able to create makes it pretty clear that there’s a lot more to be done with public health data. It also makes me realise that people loooove maps as a data visualisation. Lesson for every project there, fit in a map if you can – it’s fun and interesting to play with, and people pay much more attention to the resulting data as a result.

It covers prescribing variation of statins in the UK last year, making it really easy to explore variation which could be costing extra money. I definitely recommend taking a look – it’s fascinating to explore.

It got some great coverage too – the Financial Times, the Economist, and Huffington Post all had approving mentions.

Last week, the NHS Information Centre released another quarter of a billion rows of data, so there’s a lot more to be done here, and this kind of big data is exactly our bag. Watch this space!

The full details will all be published in due course, but actually doing the project has reminded me of a few basic principles that come up again and again:

1. Good domain knowledge usually beats super-smart algorithms. Working with qualified doctors and using their understanding of the ins and outs of prescribing behaviour was massively more important to this project than the mathematical and analytical aspects.

2. The effort of getting data doesn’t necessarily correlate with its importance. This whole project was based on open data from the NHS Information Centre, free for anyone to download – acquisition was easy, it’s actually doing something with it that was the harder part.

In October you’ll have a few chances to see the very rare and shy Team Mastodon in public. 

On October 2nd, Fran will be talking at O’Reilly Strata London on How to Make Big Data Massively Greener (contact us for discounted tickets for the whole event)

On October 4th, we’ll be showing off all the awesome stuff Mastodon C do at the Bethnal Green Ventures Demo Day at Google Campus (free tickets, contact us for an invitation).

On October 16th, Bruce will be talking at Big Data Con London (unfortunate name there) on Big Data in the Cloud with Clojure

See you there!

You might want to skip this post if you’re already way ahead on this topic, but for everyone else, I thought I’d try and clarify…

A Data Warehouse is a structured relational database, where you aim to collect together all the interesting data from multiple systems. When putting data into a warehouse, you need to clean and structure it at the point of insertion. This cleaning and structuring process is usually called ETL – Extract, Transform, and Load. The data warehouse approach is helpful because then your data looks clean and simple, but it is also very limiting: you’re now unable to change the questions you want to ask later since the data’s already been pre-processed, or to correct for errors in the ETL process. It can also get very expensive, as enterprise data warehouses are typically built on specialised infrastructure which becomes very pricey for large datasets. There’s a nice diagram of a full data warehouse structure on the Wikipedia article here.

The Hadoop ecosystem (could also be called the ‘Big Data’ approach) starts from the same aim of wanting to collect together as much interesting data as possible from different systems, but approaches it in a radically better way. With this approach, you dump all data of interest into a big data store (usually HDFS – Hadoop Distributed File System). This is often in cloud storage – cloud storage is good for the task, because it’s cheap and flexible, and because it puts the data close to cheap cloud computing power. You can still then do ETL and create a data warehouse using tools like Hive if you want, but more importantly you also still have all of the raw data available so you can also define new questions and do complex analyses over all of the raw historical data if you wish. The Hadoop toolset allows great flexibility and power of analysis, since it does big computation by splitting a task over large numbers of cheap commodity machines, letting you perform much more powerful, speculative, and rapid analyses than is possible in a traditional warehouse.

One of the most confusing things in this field is naming – one man’s data warehouse is another man’s data store – so I’m sure I’ll get some corrections from people who use different terminology from that above. However, I hope this does untangle things at least a little bit.

I went to a great talk recently by Philip Tromans at the London Hive meetup which covered some very useful Hive Optimisation tips. His full deck is here, but I’ve shamelessly recopied a couple of the most useful points here:

1. Avoid NULLs in join fields. The field that you’re joining on gets hashed and then the hash mod (some value) determines which reducers the job goes to. NULL values all get sent to the same reducer, which is usually pointless and is time consuming if there are a lot of NULL values. Push a non-null condition into the join to make things go much faster, i.e. instead of “A join B on (A.id=B.id)”, do “A join B on (A.id=B.id and B.id IS NOT NULL)”
2. Use map-side joins and order join tables well. There is a default which is set to ‘off’ which selects the optimal join. If you precede your query with ‘SET hive.auto.convert.join=TRUE;’ then it will do a map-side join where appropriate, which can make things much faster. Also, joins happen one at a time, left to right, and joining to a big table is expensive, so use sub-joins to minimise size of join.
3. There are a bunch of other useful defaults in conf/hive-default.xml.template. Taking a look at these, and setting the ones you like to TRUE, will save an awful lot of unnecessary woe.

I’d recommend reading his whole deck to get some more insight.