‘To promote continued job growth, Government efficiency, and the social good that can be gained from opening Government data to the public, the default state of new and modernized Government information resources shall be open and machine readable. Government information shall be managed as an asset throughout its life cycle to promote interoperability and openness, and, wherever possible and legally permissible, to ensure that data are released to the public in ways that make the data easy to find, accessible, and usable. In making this the new default state, executive departments and agencies (agencies) shall ensure that they safeguard individual privacy, confidentiality, and national security.’ (Source: Executive Order)

Comments

There Is More To The US Open Data Policy Than Meets The Eye

by Andrea Di Maio

Daily Happiness Averages for Twitter, September 2008 to present

The happiness rank (whatever this is) for ‘happy’: ok!

The happiness rank for ‘sadness’: ok!

And this, the happiness rank for ‘statistics’ …. below the average! Unbelievable

… See also OECD’s Better Life Index

Luckily data visualization on the web has become less than a rocket science and especially the ONS Data Visualisation Center has shown excellent work in this space, that was done in-house by statisticians.

Therefore I feel it being appropriate to invite the followers of this blog to The Graphical Web 2013 Conference, which takes place October 21.-23. in San Francisco. The conference may be known to you as SVGopen, which it was called until 2012 but the name-change reflects the wider use of different graphical technologies such as Canvas or WebGL. However they all have in common that they are open standards and don’t require any plugins.

Conference submissions are welcome until June 9th and National Statistical Institutions have certainly a lot of material to showcase in this realm. The conference will give you the opportunity to get in contact with developers from your favorite Library (D3), browser vendors and w3c experts. Also expect a lot of representatives from Adobe to attend.

Observation is the first step, simulation and exploration follow.

FuturICT – New Science  and Technology to Manage Our Complex, Connected World

Knowledge Accelerator

‘The FuturICT Knowledge Accelerator wants science to catch up with the speed at which new problems are arising in our changing world as a consequence of globalization, technological, demographic and environmental change. The ultimate goal of the project is to understand and manage complex, global, socially interactive systems. FuturICT will create the scientific methods and ICT platforms needed to address planetary-scale challenges and opportunities in the 21st century and thus contribute to make our societies more adaptive, resilient and sustainable.

FuturICT as a whole will act as a Knowledge Accelerator, turning massive data into knowledge and technological progress. Specifically, FuturICT will build a sophisticated simulation, visualization and participation platform, called the Living Earth Platform. This platform will power Interactive Observatories, to detect and mitigate crises, and Participatory Platforms, to support the decision-making of policy-makers, business people and citizens, and to facilitate a better social, economic and political participation.’ From http://www.futurict.ch/ and http://www.futurict.eu/

prevent our society from running into a Dark Age of Information. A Background article:

‘FuturICT’s Dirk Helbing wrote an essay entitled Google as God? Opportunities and Risks of the Information Age. The essay “Google as God?” analyzes the Challenges and Opportunities of information society. Everything might happen to us, from a Big Brother Society to a Participatory Market Society. We will have to take the right decisions – but our society is not well prepared for these choices. To prevent our society from running into a Dark Age of Information, a public debate is urgently needed.’

-.-.-.-.-.-.-

 http://www.scoop.it/t/futurict-in-the-news

So it’s not surprising that these two find together, more and more.
A special workshop will be organized during the The 12th International Semantic Web Conference ISWC, 21-25 October 2013, Sydney, Australia.

It is the 1st International Workshop on Semantic Statistics (SemStats 2013) organized by Raphaël Troncy (EURECOM), Franck Cotton (INSEE), Richard Cyganiak (DERI), Armin Haller(CSIRO) and Alistair Hamilton (ABS).

‘ISWC 2013 is the premier international forum for the Semantic Web / Linked Data Community. Here, scientists, industry specialists, and practitioners meet to discuss the future of practical, scalable, user-friendly, and game changing solutions.’

The workshop summary

How to publish linked statistics? And: How to use linked data for statistics? These are the key questions of this workshop.

‘The goal of this workshop is to explore and strengthen the relationship between the Semantic Web and statistical communities, to provide better access to the data held by statistical offices. It will focus on ways in which statisticians can use Semantic Web technologies and standards in order to formalize, publish, document and link their data and metadata.

The statistics community faces sometimes challenges when trying to adopt Semantic Web technologies, in particular:

• difficulty to create and publish linked data: this can be alleviated by providing methods, tools, lessons learned and best practices, by publicizing successful examples and by providing support.
• difficulty to see the purpose of publishing linked data: we must develop end-user tools leveraging statistical linked data, provide convincing examples of real use in applications or mashups, so that the end-user value of statistical linked data and metadata appears more clearly.
• difficulty to use external linked data in their daily activity: it is important do develop statistical methods and tools especially tailored for linked data, so that statisticians can get accustomed to using them and get convinced of their specific utility.’

A tradition

RDF, Triples, Linked Data … these are topics statisticians already treated and adapted. But rather on an individual track and not as an organization.

This blog has a lot of information about Semantic Web and Official Statistics, about 40 posts since 2007.

See this post (2012) with a recent paper from Statistics Switzerland (where a study on publishing linked data has just been finished in collaboration with the Bern University of Applied Sciences): http://blogstats.wordpress.com/2012/10/15/imaodbc-2012-and-the-winner-is/

Or this (2009) about SDMX and RDF http://blogstats.wordpress.com/2009/10/27/sdmx-and-rdf-getting-acquainted/ or about LOD activities in 2009: http://blogstats.wordpress.com/2009/04/25/semantic-web-and-official-statistics/

.

On the input side it’s Big Data

‘“Big Data” is a term used to describe massive information stores – generally measured in petabytes and exabytes – and also refers to the methods and technologies used to analyze these large data volumes.  The core principles of Big Data (data mining, analytics) have been around for some time, but recent technology has enabled the collection and analysis of previously unimaginable data volumes at extremely high speeds.’ So says for example SAP and gives some examples how  Big Data will change your life (big words and they show how big software and hardware players begin to occupy the field).

Official Statistics has already put this on the agenda! And so has the in United Nations Statistics Division’s (UNSD) Friday Seminar on Emerging Issues, 22 February 2013.

Some papers from this Seminar:

Gosse van der Veen Statistics Netherlands. High Level Group for the Modernization of Statistical Products and Services. Big Data: Big Opportunity!

The High-Level Group for the Modernisation of Statistical Production and Services (HLG) established an informal Task Team of national and international experts, coordinated by the UNECE Secretariat. The Paper of this group gives an excellent overview of the topic: What Does “Big Data” mean for Official Statistics.

.

Andrew Wyckoff, Big Data for Policy,Development and Official Statistics, Directorate for Science, Technology & Industry. Organisation for Economic Co-operation and Development OECD (personal opinion).

.

Aspects of Big Data and real-time analytics are provided in another paper by Global Pulse (an innovation initiative launched by the Executive Office of the United Nations Secretary-General): Big Data for Development: Opportunities & Challenges

.

The discussion is launched and as mentions the HLG  paper: ‘To use Big data, statisticians are needed with a different mind-set and new skills. The processing of more and more data for official statistics requires statistically aware people with an analytical mind-set, an affinity for IT (e.g. programming skills) and a determination to extract valuable ‘knowledge’ from data. These so-called “data scientists” can be derived from various scientific disciplines.’

.

On the output side it’s (Linked) Open Data in combination with APIs

Open Data is not at all a new topic for Official Statistics. National Statistical Institutes were forerunners in openly providing data; organizations like UN or EUROSTAT went this way as well.

Several Open Data initiatives (USA, UK, France, EU …) consist mostly of data catalogues, and are in that sense also public relations initiatives. A large part of the data so provided consists of statistical data already available, often, on the website of the National Statistical Institute concerned. The EU portal, for instance, offers 5716 datasets  of statistical data from a total of 5893 (as of April 2013).

Further central questios are the licensing of data, as well as their availability in machine-readable formats.

Machine-readable statistical data, Application Programming Interfaces (APIs) to the data and especially Linked Open Data LOD (–> essentials, –>tutorial) open the way to creative applications and new models of presenting information.

An Europe-wide Linked Open Data (LOD2) project ‘was launched in September 2010 and will run for four years. It addresses exploitation of the web as a platform for data and information integration, and the use of semantic technologies to make government data more useable.’

Looking for third-party APPs

Data Providers are looking at applications or mashups made with their data  with much interest, and they are even sponsoring competitions and hack days (like Apps4EU) to stimulate the reuse of open data, especially from the public sector.

The most popular APP creator and statistical storyteller is Hans Roslings  with Gapminder. Rosling himself is a pioneer in fighting for open data.

http://www.youtube.com/watch?feature=player_embedded&v=jbkSRLYSojo

Changing paradigms

Open Data, Linked Open Data and APIs are changing the dissemination paradigm of statistical agencies. More people with new skills will do new things. Coding is becoming the new literacy, says i.e. Garrett Heath in his advice for his unborn daughter: ‘I was blown away that the buzz is not around mobile apps, but rather around using APIs. Ten years ago saw the creation of the social networking platforms. The past five years has been about accumulating the data. The next five years and beyond will be about interpreting that data. [My daughter will have access to] a boatload of interesting data sitting in accessible databases that is waiting to be exposed and interpreted with her [the programmer's]) creativity.’

Storytelling with data

Storytelling based on data is less and less the domain of statistical agencies. Storytelling can access multiple (new) resources and take on new forms.  To satisfy the basic idea of an easily understandable and appealing presentation of statistical content, statistical institutions cannot avoid taking certain measures to improve their content and presentation. The ”composer” must know how the music is to be played, that is as a quick, competent, qualitatively unique, reliable and indispensable data source.
But this presentation job can no longer be done on one’s own: cooperative partnerships are necessary and have already begun to some extent, both with partners outside statistical institutions and between such institutions. This discussion has been launched.

Statistical Storytelling revisited! More in a paper from IMAODBC Vilnius 2010:

.

And this: Many small open data give big data insights

FORGET BIG DATA, SMALL DATA IS THE REAL REVOLUTION says Rufus Pollock co-Director of the Open Knowledge Foundation : ‘… the discussions around big data miss a much bigger and more important picture: the real opportunity is not big data, but small data. Not centralized “big iron”, but decentralized data wrangling. Not “one ring to rule them all” but “small pieces loosely joined”.’

Professor Ron Wasserstein, Executive Director of the American Statistical Association and member of the Statistics2013 Steering Committee, describes some of the highlights of The International Year of Statistics (Statistics2013) on research-europe.com:

‘Statisticians work in a vast array of fields: advancing science; creating new methodology; providing information that shapes public policy; and making our medicines more effective, our food safer and our planet better. They come from many cultures, speak many languages and arrive at statistics careers via many paths. But at least one aspect is recognised by virtually every statistician – our discipline and profession are simply not visible enough. It is an invisible science that is having an enormous impact. …

‘The more statistically literate people are, the better it will be for advancing frontiers of science and helping set policies guided by data and observation. We also hope activities that make statistics come alive for people continue well past 2013.’

Census at School Program

Statistics2013 promotes school programs and with this statistical literacy: ‘Census at School is a free, web-based classroom project that engages primary and secondary school students in statistical problem solving using their own data. This international educational initiative, launched in 2000 in the United Kingdom by the Royal Statistical Society, has programs operating in several countries.’

The Day I Started ‘Liking’ Statistics

… thanks to Hans Rosling – once again! See NobyLeong’s Blog post

All this is provided by a newly designed application on Statistic Switzerland’s portal.

Embed

And download all countries as a brochure

Open Data

The Source Data (from Eurostat and Swiss Statistics) are available as an EXCEL file: So data are open and the app made from these data is open, too. It provides selecting and embedding and also the output of all indicators as a PDF file. It may also be embedded into third party websites or other apps can be written by other people.

App made with a CMS

This Portrait-App is one of several Apps of the same flavour. There are also portraits of the 26 Swiss Cantons, the biggest Cities and and the (more than)  2500 Communes.

A Content Management System helps building these Portrait-Apps once the data are in correct shape. And this in a very short time (hours).