There’s more

State of Open Data in Europe

The European Commission (Directorate General for Communications Networks, Content and Technology) just published the second Open Data Maturity Report.

‘Open Data refers to the information collected, produced or paid for by public bodies which can be freely used, modified, and shared by anyone for any purpose.’ (all pages from the report, p.6)

‘The two key indicators used to measure Open Data Maturity are Open Data Readiness and Portal Maturity.
— The first key indicator, Open Data Readiness, assesses to what extent countries have an Open Data policy in place, licensing norms and the extent of national coordination regarding guidelines and setting common approaches.
— The second key indicator, Portal Maturity, explores the usability of the portal regarding the availability of functionalities, the overall re-usability of data such as machine readability and accessibility of data sets, for example, as well as the spread of data across domains.
The two key indicators as well as the sub-indicators are shown in the table below.

(pp.6 and 10)

Open Data Maturity in Europe 2016: The results

Overview by countries:

Results from the Open Data Maturity Assessment for the EU28+ countries 
for 2016 (p.59)

From European Data Portal, all the details here.

(p.59)

And one more result blog about stats enjoys:

About Re-Usability

One of the criteria for open data maturity is the re-usability of data and especially machine readability of data. Six questions focus on this item:

‘When looking at the data on the European Data Portal, over 49 different file formats are used. The most used data formats are CSV, HTML and WMS. The fourth most used data format is PDF. PDF is one of the few data formats that is not machine-readable. The following most frequent distributions are ZIP, JSON, XLS and XLSX, followed by WFS and XML. Numbers range from nearly 49,000 CSV formats to just over 23,000 JSON formats to the least used 263 shape formats. Most data formats are or are related to a spreadsheet, which enables to analyse the data more swiftly.’ (p. 49).

That is not enough and the report recommends:

‘On the more technical side, some improvements are still necessary. To further develop automated processes each national portal should have an API in combination with a complete metadata profile. This allows a portal to share the data with data users more easily. This can for instance enable harvesting data directly from public administrations in an automated fashion, saving efforts in manual uploading of data and limiting errors when editing data and metadata manually.  … Typos or different spellings can limit the discovery of data. Here activities conducted at EU level on controlled vocabularies can be of interest to learn from in order to increase semantic interoperability. ‘  (p.63)

There is more

PDF is poor, XLS is better, CSV even better, also JSON and APIs; and metadata are of crucial importance. The European Data Portal gives a good example: it organizes the datasets in the triple format (RDF) and offers an SPARQL search.

But there is more.

Not only the datasets could foster semantic interoperability but also the data in these sets. Linked data and adequate formats can assure this interoperability and with this extended machine readability and use of data. So why not add this criterion to the questionnaire (Question 7.6 +) and lead the national portals in this direction?

Linked data? Tim Berners-Lee explains

And related: https://blogstats.wordpress.com/2016/02/03/open-data-portals-news/

20 Years Ago

1996

On the 2nd of September 1996, Statistics Switzerland published its brand-new website, www.bfs.admin.ch. It was one of the first (if not the first) of the Swiss Administration (www.admin.ch).

In three languages…

… and already with quite rich structure and content.

The Wayback Machine …

… shows the developments since 1996

https://archive.org/web/.

1996:
Handmade with Frontpage as editing software

https://web.archive.org/web/19970502093430/http://www.admin.ch/bfs/stat_ch/eber_m.htm

November 2004:
New layout, made with Day Communiqué as Content Management System and a database for file download

December 2007 ff
Layout adapted to the general layout of admin.ch. The same CMS now bought by Adobe and renamed Adobe Experience Manager AEM

The next one …

… must be based on a new technology:

• CMS remains state of the art for content presentation
• Assets come from databases
• Web services (via a web service platform) deliver assets from databases to the presentation platform.

And with such a three-layer architecture the website will be able to display data from ubiquitous databases and also offering data to third parties via web services: Open data compatible.

From Quantity to Quality

Open Data is a much-debated topic and – since the Obama administration launched Data.gov on May 21, 2009 – an international competition, too. Nearly 400 Open-Data Portals emerged meanwhile. But very often there is more concern about the number of published data than about the content of these data.

GODI

This issue has been addressed by Open Knowledge (okfn) with its Global Open Data Index (Global Open Data Index). ‘ …simply putting a few spreadsheets online under an open license is obviously not enough. Doing open government data well depends on releasing key datasets in the right way.
Moreover, with the proliferation of sites it has become increasingly hard to track what is happening: which countries, or municipalities, are actually releasing open data and which aren’t? Which countries are releasing data that matters? Which countries are releasing data in the right way and in a timely way?
The Global Open Data Index was created to answer these sorts of questions, providing an up-to-date and reliable guide to the state of global open data for policy-makers, researchers, journalists, activists and citizens.’

.

The Challenge: Be more than a simple measurement tool

The Open Knowledge Community just started ‘a discussion with the open data community and our partners in civil society to help us determine which datasets are of high social and democratic value and should be assessed in the 2015 Index. We believe that by making the choice of datasets a collaborative decision, we will be able to raise awareness of and start a conversation around the datasets required for the Index to truly become a civil society audit of the open data revolution. – See more at http://blog.okfn.org/2015/08/20/the-2015-global-open-data-index-is-around-the-corner-these-are-the-new-datasets-we-are-adding-to-it/

The result is a list of datasets that can be found on Google docs.

Statistics

For National Statistics (in okfn’s definition), these are the (few) chosen sets:

‘Key national statistics such as demographic and economic indicators (GDP, unemployment, population, etc).
To satisfy this category, the following minimum criteria must be met:
– GDP for the whole country updated at least quarterly
– Unemployment statistics updated at least monthly
– Population updated at least once a year’

Open, Useful, Reusable

In OECD’s brand new publication ‘Government at a Glance 2015’ we can find a new indicator: The OUR Index. It stands for ‘Open, Useful, Reusable Government Data’.

‘The new OECD OURdata Index reveals that many countries have made progress in making public data more available and accessible, but large variations remain, not least with respect to the quality of data provided. Governments need to make participation initiatives more accessible, targeted, relevant and appealing.’ (p.8)

Method

‘The data come from the 2014 OECD Survey on Open Government Data. Survey respondents were predominantly chief information officers in OECD countries and two candidate countries (Colombia and Latvia). Responses represent countries’ own assessments of current practices and procedures regarding open government data. Data refer only to central/federal governments and exclude open government data practices at the state/local levels.’ (p.150)

.

Based on G8 recommendations

‘The OECD OURdata Index measures government efforts to implement the G8 Open Data charter based on the availability, accessibility and government support to promote the reuse of data, focusing on the central OGD portal in each country'( p.33)

‘The G8 Open Data Charter defines a series of five principles: 1) open data by default; 2) quality and quantity data; 3) usable by all; 4) releasing data for improved governance and; 5) releasing data for innovation, as well as three collective actions to guide the implementation of those principles.’
‘As a first step in producing a comprehensive measure of the level of implementation of the G8 Open Data Charter, the OECD pilot Index on Open government data assesses governments’ efforts to implement open data in three dimensions:
1. Data availability on the national portal (based on principle 1 and collective action 2);
2. Data accessibility on the national portal (based on principle 3) and
3. Governments’ support to innovative re-use and stakeholder engagement (principle 5).
The only principle not covered in this year’s index is Principle 4: Releasing Data for improved governance value (e.g. transparency) as existing measurement efforts have focused primarily on socio economic value creation’ (p.150)

.

And here comes the ranking

Data for this chart: http://dx.doi.org/10.1787/888933249180

Detailed data for the countries: http://dx.doi.org/10.1787/888933249175

The publication

The publication: OECD (2015), Government at a Glance 2015, OECD Publishing, Paris. http://dx.doi.org/10.1787/gov_glance-2015-en

The Cui-bono Approach to Open Data

What’s the problem? Which data are needed to solve it? Who gets an advantage of it?

These few questions are valuable key for implementing the open data culture. Open data not as ‘l’art pour l’art’ but in a pragmatic approach, demonstrating that the ‘proof of the pudding is in the eating’.

It seems to work very well as Ton Zijlstra showed in his presentation at the Swiss Opendata Conference 2015.

He gives some examples of situations where open data helped to provide a solution to a problem and where stakeholders got an answer to their issues.

‘Planning for socio-economic impact. open data as a policy instrument.’

link ti

Country Portraits – Open and Embedabble

Looking for important statistical indicators of European countries? Comparing these countries? Taking the application to your own website? Making a brochure of it?

All this is provided by a newly designed application on Statistic Switzerland’s portal.

Embed

And download all countries as a brochure

Open Data

The Source Data (from Eurostat and Swiss Statistics) are available as an EXCEL file: So data are open and the app made from these data is open, too. It provides selecting and embedding and also the output of all indicators as a PDF file. It may also be embedded into third party websites or other apps can be written by other people.

App made with a CMS

This Portrait-App is one of several Apps of the same flavour. There are also portraits of the 26 Swiss Cantons, the biggest Cities and and the (more than)  2500 Communes.

A Content Management System helps building these Portrait-Apps once the data are in correct shape. And this in a very short time (hours).

Europe has an Open Data Portal, too

The European Commission

opened its Open Data Portal some days ago.
Powered by CKAN.

.

Most of the 5811 datasets (97%) are statistical ones provided by Eurostat.

.

‘Top Publishers
Eurostat (5634 datasets)
European Environment Agency (106 datasets)
Joint Research Centre (37 datasets)
Directorate-General for Health and Consumers (12 datasets)
Publications Office (11 datasets)
Directorate-General for Education and Culture (3 datasets)
Directorate-General for Communications Networks, Content and Technology (2 datasets)
Directorate-General for Employment, Social Affairs and Inclusion (1 datasets)
Directorate-General for Enterprise and Industry (1 datasets)
Directorate-General for Regional and Urban Policy (1 datasets)‘.

Linked Open Data are provided

An important step! ‘The European Commission Open Data Portal is well aligned with the initiatives of linked data and semantic web technologies. The dataset metadata is available as triples on triple store and attached to the dataset records.’

One App for the moment…

… but an interesting one creating visualisations based on RDF input, with javascript based output (Highcharts charting library).

.

‘CubeViz is a facetted browser for statistical data utilizing the RDF Data Cube vocabulary which is the state-of-the-art in representing statistical data in RDF. This vocabulary is compatible with SDMX and increasingly being adopted.’

open data and official statistics: how and where?

There’s a hot topic concerning official statistics as well as many governmental bodies owning data: open government data OGD.

Official statistics in most countries offer open data for years now. But mostly in their own silos and in various flavors: From PDF to XLS and CSV to RDF LOD, in interactive databases like px-web, oecd.stat, genesis-online etc., with visualization tools and various search facilities.

On the other hand there are initiatives to provide software, create standards and build platforms in order to bring together myriads of data sets and making these accessible in one place.

http://thedatahub.org/  is such a platform assembling more than 3800 datasets (June 2012).

Thedatahub.org is powerd by a software called CKAN

… claiming to be …

.

What about official statistics?

the Data Hub contains a lot of data sets from official statistics

Among these are simple links to National Statistical Institutes

But also machine readable linked open data

.

Several institutions are using CKAN with their own themes.

UK (in CKAN). The Office for National Statistics ONS shows a lot of data (847 data sets) in the CKAN powered catalogue, with rich metadata and links to the ONS website.

EU (in CKAN). Europe’s Public Data aggregates datasets from several sources,

.

CKAN Data Store

As far as I can see statistical institutions use CKAN as a catalogue linking to the sources on the institution’s websites. But CKAN could also be a host for the data itself:

‘As well as holding metadata and links to the offsite data, CKAN can provide secure storage for the data itself. When creating the dataset or resource, you can either link to data hosted elsewhere, or upload it in the same action as registering it on CKAN.’ (link to CKAN).

It would be very interesting to know who in official statistics already envisages such a solution for it’s data ….!

10 billion

Hans Rosling is an early fighter for open data and one of the best, no: the best in presenting insights from these data.

His last example comes from TED (Technology, Entertainment, Design) in Doha, Qatar (April 16-20, 2012) and demonstrates that ‘religion has very little to do with the number of babies per woman’.

.

Some more examples of what has been done with open data are given by  publicdata.eu

With an overview of Apps:

Some of these won prizes in the http://opendatachallenge.org/

And now a new competition sponsored by The Guardian and Google is under way,

‘The Guardian Datastore and Google have teamed up to see who can help visualise the data which will show which governments are adopting the economic policies that will facilitate job growth and innovation to lead the world out of the economic slump.’

They are giving hints to datasets, called ‘the world’s key economic datasets from the UN, World Trade Organisation, IMF and some of the world’s major economic experts…’

t

Another Marathon

Hackathons with open data are well known. Marathons are coming, this one gives teams the task ‘to design a solution to a real world problem that will be given at the start of the 24 hour Marathon’. Data? Open data!

#vizmarathon

Open data: Waiting ….

UK and US governments support open data … not only in their own countries. In an official letter they ask OECD to join this movement.

‘On behalf of US Secretary of State Hillary Rodham Clinton and UK Foreign Secretary William Hague, the heads of the two countries’ missions to the OECD delivered a letter this week to the Organisation’s Secretary General, Angel Gurría. In it, Mrs Clinton and Mr Hague called on the OECD to commit to the principles of the Open Government Partnership, and make all of its core data freely available online. ‘ https://usoecd.cms.getusinfo.com/data.html

and:

Awaiting an answer ……..

.

Recently

in Warsaw there was held the OGDcamp 2011.
Waiting for the keynotes posted …

#ogdcamp

.

An instructive introduction to Open data.

.

And …

a key message

from Vincenzo Patrunos presentation at ISTAT for the Italian Statistics Day (yes! October 20th !!) where were discussed about Open Data and Open Government  during the workshop “Open Official Statistical Data”.

The same from his presentation at IMAODBC 2011. Have a look at it.

Waiting for the paper …. -;)