SDMX

SDMX (Statistical data and metadata exchange) is a widely accepted standard for communicating data  in the realm of official statistics.

‘SDMX is an initiative to foster standards for the exchange of statistical information. Sponsored by BIS – ECB – EUROSTAT – IMF – OECD – UN – World Bank.’

More and more applications accept and use this standard for data input as well as for data output: So for instance the ECB Inflation dashboard or EUROSTATS plans to offer sdmxed data out of its database or the activities in PC-Axis tabulation software.

RDF, scovo

But official statistics are not the only ones who discuss and conceptualise statistical data exchange.  scovo (‘The Statistical Core Vocabulary’) is another attempt, it ‘ specifies an [RDF-Schema] vocabulary for representing statistical data on the Web’.

Behind this we find – among others – the people who launched RIESE (RDFizing and Interlinking the EuroStat Data Set Effort).

Planetary conjunction

SDMX and RDF are beginning to approach each other.  And this may lead to a new kind of open data: Already open statistical data become accessible as linked open data.

Such activities can be seen in the UK where the UK Government Data Developers (a Google Group, ONS is participating) are working on this topic.

Jenni Tennison describes this in his blog: ‘One of the things that we’ve been discussing over on the UK Government Data Developers mailing list is how best to represent the vast quantities of statistical data that the government produces, in RDF. This is what we’ve come up with: 1. We’ll use SCOVO (scv: <http://purl.org/NET/scovo#>)as our main vocabulary. ….’
and later on:  ‘One source of sub-properties of scv:dimension (and subtypes of scv:Dimension) is SDMX (Statistical Data and Metadata eXchange). This provides standard ways of indicating things like the area and time that a statistic applies to…. ‘.

The work goes on and I hope that more of the specialists in statistical offices will enter the discussion.

At the IMAODBC Conference 2009 in Toronto Gareth Mc Guinness from the IMF presented a paper about SDMX and Web 3.0:  That’s just semantics.

See also my paper at the Prague Conference 2009,  ‘Official statistics beyond Web 2.0: Challenges, rewards and risks to come‘, especially pages 5ff.

World bank launches an interactive visualizing tool resembling gapminder, it’s called Data Visualizer.

‘The time series used in Data Visualizer is a subset of 2009 World Development Indicators database. It contains 49 indicators for 209 countries and 18 aggregates from 1960-2007. Data includes social, economic, financial, information & technology, and environmental indicators.’

Tools allow to change the title and to export the graph as a jpeg. See this example:

A video presentation explains how to use the new tool.

Google’s interest in statistics is steadily growing. And this not least because Ola Rosling now works with Google and pushes visualization tools like motion chart and the public-data project.

In his presentation at the Gov 2.0 in Washington (September 2009) Ola Rosling presents theese tools: Ola Rosling, “Seeing Data as Change Over Time”

Accessing the data

How will Google access public data? The preference is bulk downloads:

Modes of Data Access

Really? We all know the risks of this: It’s manual work (isn’t it?), it needs resources, it’s repetitive … . With APIs and a standardized format like RDF and open silos (!)  the future could begin … . See the remarks in this article ‘Official statistics beyond Web 2.0: Challenges, rewards and risks to come’.

Sources for Google’s public data are the U.S Census Bureau and the Bureau of Labor Statistics, also EUROSTAT is mentioned by Ola Rosling. What’s next? OECD and UN for much more country data? Will Google with its excellent internet presence become an important provider of international public data? So welcome Stats Google in the family….

More about Gov 2.0 in this blogstats post.

And about the public-data project there is a good description by Ola Rosling in the Googleblog ‘Adding search power to public data’

In addition, there is Google Internet Stats

On September 9th 2009 Google UK  quietly launched a microsite with statistics about the internet from a variety of sources. The statistics are divided into five sections: Technology, Macro Economic Trends, Media Landscape, Media Consumption and Consumer Trends.

Google-Internet-Stats

There are various sources: B2Bonline.com, BARB, BusinessWeek, Coke, Commission of the European Communities, Comscore, Core Metrics, Datamonitor, Deloitte, The Economist, eMarketer, Enders Analysis, Eurostat, Film Distributors Association, Financial Times, Forrester, GFK, Google Insights for Search, Greenbee.com, Guardian, HarvardBusiness.org, Hitwise, IAB, IFPI, IMF, Internet Retailing, Internetworldstats.com, JP Morgan, KMPG, Media & Marketing, Mediascope Europe, Mindshare, Motorola, Net Imperative, New York Magazine, Nielsen, NMA, Ofcom, Ipsos MediaCT, PriceWaterhouseCoopers, QuickPlay Media Inc., Retail Week, Reuters, TGI Net, Times Online, TNS, Verdict Research, Wall Street Journal, WARC, YouTube, ZenithOptimedia, GM

In a interview given at a technology symposium at the Embassy of Finland in Washington (15 October 2009), ‘Berners-Lee speaks about the importance for governments to place great amounts of data on the Web and the emergence of the semantic Web. He cites successful examples in Britain.’

And he repeats what he already said in one of his blog posts in 2008:

‘In all this Semantic Web news, though, the proof of the pudding is in the eating. The benefit of the Semantic Web is that data may be re-used in ways unexpected by the original publisher. That is the value added. So when a Semantic Web start-up either feeds data to others who reuse it in interesting ways, or itself uses data produced by others, then we start to see the value of each bit increased through the network effect.’

In another interview with NIGHTLY BUSINESS REPORT CORRESPONDENT Darren Gersh he looks into the crystal ball:

‘DARREN GERSH: So it’s been 20 years since you invented the worldwide web. What is the biggest challenge for the Internet in the next 20 years?

BERNERS-LEE: Well, I think looking forward the most difficult thing that we’re not doing is we’re not actually studying the web. The web’s huge. It’s a huge system. There are actually more web pages out there than there are neurons in a person’s brain. So there are a lot of nerve cells (ph) in a person’s brain, but we are starting to figure out how the brain works but we really don’t study how the web works. It’s humanity connected, all these people making links, following links, exchanging ideas, trying to put together new forms of democracy, new social networking systems. We don’t really understand what is going to work and what isn’t. We don’t really know what are the dangers. Could it become unstable? What are the really huge opportunities? So studying the web is really important. I think the danger is we don’t study it and then suddenly, something happens like the financial downturn, like spam coming along, like one of these things where whoa! Oops, didn’t plan for that. Now what went wrong? So we shouldn’t be looking back and thinking what went wrong? We should be looking forward and thinking, OK, what are the things that could happen? What would be likely to happen? How can we tweak the web? After all the web, unlike the brain, is something that we designed. It’s an engineering thing.’

And he confesses that // in the http adress is in fact unnecessary .

There are specialized search engines like sig.ma using semantic technology (but working in niches), and there are the big search machines slowly starting to use such an approach. Yahoo does it for some time already and Google goes RDFa and microformats.

Google looks for markup formats (microformats and RDFa) and displays reviews and ratings and also information about persons in Rich Snippets. New types of data beyond reviews and people will follow.

Googles Webmaster Blog describes this:

‘Imagine that you have a review of a restaurant on your page. In your HTML, you show the name of the restaurant, the address and phone number, the number of users who have provided reviews, and the average rating. People can read and understand this information, but to a computer it is nothing but strings of unstructured text. With microformats or RDFa, you can label each piece of text to make it clear that it represents a certain type of data: for example, a restaurant name, an address, or a rating. This is done by providing additional HTML tags that computers understand. These don’t affect the appearance of your pages, but Google and any other services that look at the HTML can use the tags to better understand your information, and display it in useful ways—for example, in search results.’

The conference “Statistics – Investment in the future 2″ took place in Prague from 14 to 15 September 2009.

More than 100 participants attended 22 sessions.

-_-_-_-_-_-_-_-_-_-_-_

The session ‘Statistics and the Internet’ saw 4 presentations:

Managing the Internet in a statistical institution
Leon Oestergaard, Statistics Denmark

Influence of the Internet on dissemination of official statistics
Tomaz Smrekar, Statistical Office of the Republic of Slovenia

Official statistics beyond Web 2.0: Challenges, rewards and risks to come
Armin Grossenbacher, Swiss Federal Statistical Office, Switzerland

Evaluating the public’s perception of  NSO websites using a non- standard mixed methodological approach
Derek Bond and Elaine Ramsey, University of Ulster at Coleraine, UK

Abstracts, presentations and papers of the session ‘Statistics and the Internet’ can be found on CSU-Website.

Gov 2.0 Summit Videos (from Google Public Sector Blog)

Monday, September 14, 2009 | 11:45 AM

If you weren’t able to make the Gov 2.0 Summit last week in DC, you’re in luck – videos of most presentations are now online.

We’ll post an update when Ola Rosling’s presentation on public data search and visualization is online.

Tim O’Reilly interviewed Google’s Chief Economist Hal Varian about how government can take advantage of real time data and economic indicators.

From Space Time Research website:

Twitter – ABS launches TableBuilder

Use TableBuilder for ad hoc analysis of Australia’s 2006 Census data, and create tables of up to 5 million cells! http://bit.ly/zzIWR

TableBuilder also works on the iPhone http://twitpic.com/fjire

Follow Space-Time Research on Twitter spacetimeresrch

On this blog we usually showcase best practices of how to communicate statistics and keep the technological aspects of it in the background – which is the right way to do. But we also never get tired of mentioning how statistics is a basis for informed decision making and therefore a foundation for democracy. To live up to these standards we make sure our methods are well documented and I would argue we should also give some thoughts on what technology we use.

There are two reasons for this: We should allow our users to learn from our applications on the web, build upon them, mash them up with other stuff we didn’t imagine or even improve them. And secondly when we talk about archiving in the digital age, we are well advised to use open standards. Everyone of us who recently tried to open some old Word 2.0 documents will understand what I mean.
The topic comes up as several statistical offices have moved from the SVG format for interactive statistical graphics to Flash. See the latest population pyramid from ONS or the election atlas in Germany. While SVG is an open graphics standard just like HTML, you can think of Flash more like Word documents, those are closed binary files. Users cannot look behind the scenes and if the source code gets forgotten or the technology changes dramatically, all is lost.
Now there is a flipside to it: Just like Microsoft Word, Flash is ubiquitous, works really well and works the same way across all supported platforms. SVG on the other hand had its ups and downs. Since 2008 it is very well supported on modern browsers such as Opera, Firefox, Safari and Google Chrome but even Internet Explorer 8 doesn’t handle it at all. There was a plugin for Internet Explorer, but that never had the significant market share that Flash enjoyed (no YouTube without Flash!) and was “end of lifed” in January 2009, meaning SVG support on Internet Explorer was gone at the beginning of this year. So why all the bemoaning, free and open doesn’t allways win.
Well, things are changing right now. With the help of Google an open source project aims at adding SVG support to Internet Explorer through the Flash plugin. And they are aiming high, want to implement it in Wikipedia, which uses SVG as a base format for all their graphics and maps. It’s called the SVG Web project, and already it works well enough to support our use-cases. I’ve put up an animated population pyramid and an interactive map with it and couldn’t be happier.
Even in Internet Explorer you can right click in the graphics and “view source”, see how everything was done, adapt it, improve it … And when your are using Firefox or Safari you can print these graphics into PDF and get print quality vector graphics.
Give it a look, talk about it with your tech people and let me know what you think. We have comments here for a reason.