Democratising attention data at theguardian.com – by Graham Tackley

This was a talk I was looking forward to, because it was built on a story from the real world, around subjects that hit hime with me.

Graham Tackley gave us his story of how The Guardian has become one of the largest newspapers with 4-5 million users a day, by embracing the web, principles of open data and small engaged teams. At first skeptics feared publishing all of their articles online would undermine the business, but they have moved on to become one of the most dominating newspapers because of this decision.

Today everyone at the guardian has access to a dashboard with live data and a range of visualisations about traffic allowing them to make informed decisions about which articles to keep and which themes are trending. Visualisations include everything from a top 20 of what people have read in the last 3 minutes to graphs that can be broken down and filtered.

An important part of their success is the focus on choosing data that can be understood by non-technical people, and data that is clear, relevant and actionable. With this in mind a key metric is pageviews – because it is extremely relevant and there is not much chance of misunderstanding what it means. By following pageviews, how they change over time and what referrals they come from the editors have a strong decision making tool.

From hack to cloud based solution

A 4 hour delay on numbers from analysis tools was an issue they encountered early on, which was recognised when they thought about it in the same way as the developers that had long release cycles did before implementing continuous delivery. So Graham took it upon him to get this data faster by using log aggregation. He needed to show this could be done quickly so he set the goal of getting it done in 24 hours. Good news was that he only needed the last 3 minutes worth of data, so he did it in memory using a scala list which was filtered continuously. Naturally this solution had its problems because measuring requests leave you with the challence of bot filtering and requests not being equal to user interactions. However it was still a revalation and the users were delighted, so they begged him not to turn it off after the hack-day ended. This showed how a pretty big problem could be solved by applying a simple solution and still bring tons of value.

Next step was that they wanted to move to javascript based tracking to solve the before mentioned issues. So without rewriting the original dashboard they built a system om AWS using SNS for notifications and SQS as a queue. When this was ready they simply changed the dashboard to receive events from javascript that were gathered on the new AWS backend. This gave them the more precise data they were looking for.

At this point everything was still processed in memory, which was a pain when they wanted to change parts out. They looked around and found elasticsearch. They hoocked on another listener to SNS and put data into elasticsearch. Then they changed the dashboard to read from elasticsearch. This was not a rewrite of the dashboard either but just changing parts out. They could simply flip the DNS and move to the new datasource without any downtime. To Grahams surprise their ops team loved them for this cowboy approach because they could do this themselves, without any consequences for the ops team.

After moving to AWS autoscaling was used as a way of describing a box and creating a number of them as needed making a whole class of problems disappear. As Graham said “basically solving infrastructure the same way do in code – describe a class and create instances”.

So through this whole process they evolved a hack into an autoscaling cloud solution. What Graham learned was that “architectures can and must evolve rapidly, so responsibility for software and infrastructure cannot be split”. The tooling around this has evolved greatly in the last period of time, making this kind of approach possible.

Designing with openness in mind

Frictionless access was an important aspect of the design process, so they made sure that everyone would have access to the data by using gmail for authentication. With this in place and realizing that it was important to be able to display data to non-technical people, they engaged with their users to find out what was the most interesting data. A good example of this was a simple view of how many readers go to see another article, how many go to another page on the site and how many leave the website. Clear, actual and actionable data.

Actually most features people ask for they do not do, but when they do settle on a feature they build it fast. They are two people building it, doing QA, testing etc. Spending about half their time on the platform. Later they have included one more person to do facebook activities, but that is the entire team.

Today the dashboard is a key part of the future for the guardian and everyone is still passionate about using and extending the tool. All this came to be because they started out creating a minimum viable product and let it evolve from there.

Finishing off the talk Graham concluded that the key lesson for him is that small teams can have a massive impact with:

• Rapid evolving software
• Close collaboration of users
• Clear relevant and actionable data