There’s more

State of Open Data in Europe

The European Commission (Directorate General for Communications Networks, Content and Technology) just published the second Open Data Maturity Report.

‘Open Data refers to the information collected, produced or paid for by public bodies which can be freely used, modified, and shared by anyone for any purpose.’ (all pages from the report, p.6)

‘The two key indicators used to measure Open Data Maturity are Open Data Readiness and Portal Maturity.
— The first key indicator, Open Data Readiness, assesses to what extent countries have an Open Data policy in place, licensing norms and the extent of national coordination regarding guidelines and setting common approaches.
— The second key indicator, Portal Maturity, explores the usability of the portal regarding the availability of functionalities, the overall re-usability of data such as machine readability and accessibility of data sets, for example, as well as the spread of data across domains.
The two key indicators as well as the sub-indicators are shown in the table below.

(pp.6 and 10)

Open Data Maturity in Europe 2016: The results

Overview by countries:

Results from the Open Data Maturity Assessment for the EU28+ countries 
for 2016 (p.59)

From European Data Portal, all the details here.

(p.59)

And one more result blog about stats enjoys:

About Re-Usability

One of the criteria for open data maturity is the re-usability of data and especially machine readability of data. Six questions focus on this item:

‘When looking at the data on the European Data Portal, over 49 different file formats are used. The most used data formats are CSV, HTML and WMS. The fourth most used data format is PDF. PDF is one of the few data formats that is not machine-readable. The following most frequent distributions are ZIP, JSON, XLS and XLSX, followed by WFS and XML. Numbers range from nearly 49,000 CSV formats to just over 23,000 JSON formats to the least used 263 shape formats. Most data formats are or are related to a spreadsheet, which enables to analyse the data more swiftly.’ (p. 49).

That is not enough and the report recommends:

‘On the more technical side, some improvements are still necessary. To further develop automated processes each national portal should have an API in combination with a complete metadata profile. This allows a portal to share the data with data users more easily. This can for instance enable harvesting data directly from public administrations in an automated fashion, saving efforts in manual uploading of data and limiting errors when editing data and metadata manually.  … Typos or different spellings can limit the discovery of data. Here activities conducted at EU level on controlled vocabularies can be of interest to learn from in order to increase semantic interoperability. ‘  (p.63)

There is more

PDF is poor, XLS is better, CSV even better, also JSON and APIs; and metadata are of crucial importance. The European Data Portal gives a good example: it organizes the datasets in the triple format (RDF) and offers an SPARQL search.

But there is more.

Not only the datasets could foster semantic interoperability but also the data in these sets. Linked data and adequate formats can assure this interoperability and with this extended machine readability and use of data. So why not add this criterion to the questionnaire (Question 7.6 +) and lead the national portals in this direction?

Linked data? Tim Berners-Lee explains

And related: https://blogstats.wordpress.com/2016/02/03/open-data-portals-news/

And now: Semantic Statistics (SemStats)

Official Statistics has a long tradition in creating and providing high-quality metadata. And the Semantic Web needs just this: metadata!

So it’s not surprising that these two find together, more and more.
A special workshop will be organized during the The 12th International Semantic Web Conference ISWC, 21-25 October 2013, Sydney, Australia.

It is the 1st International Workshop on Semantic Statistics (SemStats 2013) organized by Raphaël Troncy (EURECOM), Franck Cotton (INSEE), Richard Cyganiak (DERI), Armin Haller(CSIRO) and Alistair Hamilton (ABS).

‘ISWC 2013 is the premier international forum for the Semantic Web / Linked Data Community. Here, scientists, industry specialists, and practitioners meet to discuss the future of practical, scalable, user-friendly, and game changing solutions.’

The workshop summary

How to publish linked statistics? And: How to use linked data for statistics? These are the key questions of this workshop.

‘The goal of this workshop is to explore and strengthen the relationship between the Semantic Web and statistical communities, to provide better access to the data held by statistical offices. It will focus on ways in which statisticians can use Semantic Web technologies and standards in order to formalize, publish, document and link their data and metadata.

The statistics community faces sometimes challenges when trying to adopt Semantic Web technologies, in particular:

• difficulty to create and publish linked data: this can be alleviated by providing methods, tools, lessons learned and best practices, by publicizing successful examples and by providing support.
• difficulty to see the purpose of publishing linked data: we must develop end-user tools leveraging statistical linked data, provide convincing examples of real use in applications or mashups, so that the end-user value of statistical linked data and metadata appears more clearly.
• difficulty to use external linked data in their daily activity: it is important do develop statistical methods and tools especially tailored for linked data, so that statisticians can get accustomed to using them and get convinced of their specific utility.’

A tradition

RDF, Triples, Linked Data … these are topics statisticians already treated and adapted. But rather on an individual track and not as an organization.

This blog has a lot of information about Semantic Web and Official Statistics, about 40 posts since 2007.

See this post (2012) with a recent paper from Statistics Switzerland (where a study on publishing linked data has just been finished in collaboration with the Bern University of Applied Sciences): https://blogstats.wordpress.com/2012/10/15/imaodbc-2012-and-the-winner-is/

Or this (2009) about SDMX and RDF https://blogstats.wordpress.com/2009/10/27/sdmx-and-rdf-getting-acquainted/ or about LOD activities in 2009: https://blogstats.wordpress.com/2009/04/25/semantic-web-and-official-statistics/

Linking UK Government Data, John Sheridan

data.gov in UK has a linked data lead. It’s John Sheridan and he prepared

a short presentation propagating (of course) linked data with a lot of interesting examples!

The summary:

Linked Data: It’s not a top-down system. Berners-Lee and OpenGov

There’s not much noise about Semantic Web these days. But in the fascinating and creative semantic-web niche activities go on.

Once more Tim Berners-Lee explains what Linked data are.

.

The 5-star system helps measuring i.e. how far or near open-gov data  are from being part of the Semantic Web.

★	Available on the web (whatever format), but with an open licence
★★	Available as machine-readable structured data (e.g. excel instead of image scan of a table)
★★★	as (2) plus non-proprietary format (e.g. CSV instead of excel)
★★★★	All the above plus, Use open standards from W3C (RDF and SPARQL) to identify things, so that people can point at your stuff
★★★★★	All the above, plus: Link your data to other people’s data to provide context

.

.

And there is a interesting example how Linked Data are published:

‘Linked data is data in which real-world things are given addresses on the web (URIs), and data is published about them in machine-readable formats at those locations. Other datasets can then point to those things using their URIs, which means that people using the data can find out more about something without that information being copied into the original dataset.

This page lists the sectors for which we currently publish linked data and some additional resources that will help you to use it. Most sectors have one or more SPARQL endpoints, which enable you to perform searches across the data; you can access these interactively on this site‘.

Evaluating

What’s the effect of open data? Some journals (like the Guardian) make ample use of open data, but there is no wide-spread activity or commitment or lots of evaluation studies to be seen. Infoweek just published an article about US open gov and found that there is a lot  to be done as only small groups seem to take notice of this government activity. ‘The most difficult part of open government may be getting the public to participate.   … the “if you build it, they will come” approach simply doesn’t work.’ (InformationWeek, Feb 21, 2011: Open Government Reality Check: Federal agencies are making progress on the Obama administration’s Open Government Directive, but there’s still a long way to go. Here’s our list of top priorities.)

“Journalism Needs Data in 21st Century”

This is an interesting summary from ReadWriteWeb about the role of data in media with several fresh innovative examples coming up 2009.  Some of them have already been presented before as posts in this blog. The conclusion is that journalism is data-driven and demands open access to raw data. Data should be transparent and not be hidden in pdf-files or orther formats without links to raw data (micro data). However, integrity aspects of such transparency are not discussed and they are certainly an important issue among official and public statistics data producers in this context.

Journalism Needs Data in 21st Century

Written by Guest Author / August 5, 2009 2:00 AM / 5 Comments

« Prior Post Next Post »

Journalism has always been about reporting facts and assertions and making sense of world affairs. No news there. But as we move further into the 21st century, we will have to increasingly rely on “data” to feed our stories, to the point that “data-driven reporting” becomes second nature to journalists.

The shift from facts to data is subtle and makes perfect sense. You could that say data are facts, with the difference that they can be computed, analyzed, and made use of in a more abstract way, especially by a computer.

More……

ReadWriteWeb Interview With Tim Berners-Lee, Part 1: Linked Data

Written by Richard MacManus / July 8, 2009 6:00 AM / 8 Comments

—————————————————————

How Linked Data Relates to The Semantic Web

RWW: Earlier this year you gave an inspiring talk at TED about Linked Data. You described Linked Data as a sea change akin to the invention of the WWW itself – i.e. we’ve gone from a Web of documents to a Web of data. Can you please explain though how Linked Data relates to the Semantic Web, is it a subset of it?

TBL: They fit in completely, in that the linked data actually uses a small slice of all the various technologies that people have put together and standardized for the Semantic Web.

Linked Data uses a small slice of the technologies that make up the Semantic Web.

We started off with the Semantic Web roadmap, which had lots of languages that we wanted to create. [However] the community as a whole got a bit distracted from the idea that actually the most important piece is the interoperability of the data. The fact that things are identified with URIs is the key thing.

The Semantic Web and Linked Data connect because when we’ve got this web of linked data, there are already lots of technologies which exist to do fancy things with it. But it’s time now to concentrate on getting the web of linked data out there.

Web inventor Tim Berners-Lee and ReadWriteWeb founder Richard MacManus

————————————————

Linked Data and Governments

RWW: In a recent Design Issues note, you urge governments to put their data online as Linked Data (although you’d also be happy for governments to just make available the raw data – presumably so that others can then structure it). What do you realistically expect, for example, the U.S. or U.K. governments to do over the next year? And in the near future, do you foresee different governments interconnecting their Linked Data sets?

TBL: One can’t generalize, governments are (like most big organizations) fascinatingly diverse inside them. So you’ll find that there are places inside governments where you get a champion who gets linked data and who’s just written a script and produced some linked data. So in the UK government for example, you’ll find there’s RDFa [in the code of its website] for civil service jobs. So if somebody wants to make a database of all the jobs, they can do that very easily.

“The first step of actually putting the data out there is the one that nobody else can do.”

There are other cases where the easiest thing for somebody to do is to just put data up in whatever form it’s available. Comma separated values (CSV) files are remarkably popular. They’re exported sometimes from spreadsheets. It’s remarkable how much information is in spreadsheets. Or sometimes pulled out of a database and then put up on the web. It’s not as good, not as useful to the community, as if Linked Data had been put up there and linked. But the first step of actually putting the data out there is the one that nobody else can do.

Data.gov, a catalog of public data, was launched in May by the U.S. government

The way to go is for government departments to go the extra step and convert [their data] into Linked Data. One of the nice things about Linked Data, when they have a pile of it, is that they could run a SPARQL server on it. SPARQL servers are a commodity product, a solution for all of the people who say ‘but actually I wanted to have XML.’ A SPARQL server will generate an XML file [and] allow somebody to write out, effectively, a URL for the XML file.

“Linked Data is the backplane, it’s the thing that you connect to in both directions.”

In fact, I don’t see why SPARQL servers shouldn’t provide CSV files, something which as far as I know isn’t in the standards. But I’d recommend it, certainly in government context, because CSV files are what people have and what people want.

So the message [for government] is to use RDF. Linked Data is the backplane, it’s the thing that you connect to in both directions. As a [web] producer your job is to make sure that you produce Linked Data one way or another. And as a consumer, there are lots of ways to consume that data once it’s out there as Linked Data.

Read more…..