032 Metadata – Blog about Stats

There’s more

State of Open Data in Europe

The European Commission (Directorate General for Communications Networks, Content and Technology) just published the second Open Data Maturity Report.

‘Open Data refers to the information collected, produced or paid for by public bodies which can be freely used, modified, and shared by anyone for any purpose.’ (all pages from the report, p.6)

‘The two key indicators used to measure Open Data Maturity are Open Data Readiness and Portal Maturity.
— The first key indicator, Open Data Readiness, assesses to what extent countries have an Open Data policy in place, licensing norms and the extent of national coordination regarding guidelines and setting common approaches.
— The second key indicator, Portal Maturity, explores the usability of the portal regarding the availability of functionalities, the overall re-usability of data such as machine readability and accessibility of data sets, for example, as well as the spread of data across domains.
The two key indicators as well as the sub-indicators are shown in the table below.

(pp.6 and 10)

Open Data Maturity in Europe 2016: The results

Overview by countries:

Results from the Open Data Maturity Assessment for the EU28+ countries 
for 2016 (p.59)

From European Data Portal, all the details here.

(p.59)

And one more result blog about stats enjoys:

About Re-Usability

One of the criteria for open data maturity is the re-usability of data and especially machine readability of data. Six questions focus on this item:

‘When looking at the data on the European Data Portal, over 49 different file formats are used. The most used data formats are CSV, HTML and WMS. The fourth most used data format is PDF. PDF is one of the few data formats that is not machine-readable. The following most frequent distributions are ZIP, JSON, XLS and XLSX, followed by WFS and XML. Numbers range from nearly 49,000 CSV formats to just over 23,000 JSON formats to the least used 263 shape formats. Most data formats are or are related to a spreadsheet, which enables to analyse the data more swiftly.’ (p. 49).

That is not enough and the report recommends:

‘On the more technical side, some improvements are still necessary. To further develop automated processes each national portal should have an API in combination with a complete metadata profile. This allows a portal to share the data with data users more easily. This can for instance enable harvesting data directly from public administrations in an automated fashion, saving efforts in manual uploading of data and limiting errors when editing data and metadata manually. … Typos or different spellings can limit the discovery of data. Here activities conducted at EU level on controlled vocabularies can be of interest to learn from in order to increase semantic interoperability. ‘ (p.63)

There is more

PDF is poor, XLS is better, CSV even better, also JSON and APIs; and metadata are of crucial importance. The European Data Portal gives a good example: it organizes the datasets in the triple format (RDF) and offers an SPARQL search.

But there is more.

Not only the datasets could foster semantic interoperability but also the data in these sets. Linked data and adequate formats can assure this interoperability and with this extended machine readability and use of data. So why not add this criterion to the questionnaire (Question 7.6 +) and lead the national portals in this direction?

Linked data? Tim Berners-Lee explains

AoS: The new Atlas of Switzerland … and more

On June 20th, 2016 the moment arrived to unveil the new Atlas of Switzerland (AoS):

Simone Niggli-Luder (orienteering athlete and multiple world champion), Sarah Springman (Rector of the ETH Zurich), Marc Chardonnens (Director of the Federal Office for the Environment). (Foto AG)

AoS Version 4

The new Atlas of Switzerland AoS is (since 2000) already version 4.

“The AoS is mandated by law by the Swiss Federation to visualize themes from different fields such as socio-economy, ecology, history and energy, etc. in an ongoing long-term project. Since its beginning in 1961, the aim of this Swiss national atlas is to offer cartographically sound maps in combination with additional information to the general public in order to visualize visible and hidden structures and processes.” (Source: Atlas of Switzerland goes online and 3D – Concept, Architecture and Visualization. Methods. René Sieber, Marianna Serebryakova, Raimund Schnürer, Lorenz Hurni. EuroCarto 2015)

Version 4 is a step forward. It
– is based on a platform open for third parties’ products,
– is available online and
– presents information on a globe model and in 3D.

Platform

The AtlasPlatformSwitzerland (APS) is the architectural framework for future AoS products. It also offers the possibility to publish affiliated atlases from other institutions; one example is the Hydrological Atlas of the University of Bern.

Online

After downloading the client software (for te moment Windows only), thematic maps can be selected from the online archive and viewed.
Besides the navigation in the Atlas itself an easy access to these maps is also given on AoS’ Website:

Presentation

And here comes the fascinating part of the new Atlas: The presentation based on an earth model and also in 3D.

“The core of this APS – the so-called APSglobe – consists of a 3D virtual globe (osgEarth) with basic tools for navigation, visualization, and querying.” (http://www.atlasderschweiz.ch/portfolio/).

A globalized example shows the flights from Zurich in 3D:

Another map presents population density in two and three-dimensional views:

Drilldowns show settlement structures (here the train station Bern) with the possibility of navigating through cities and of changing perspectives:

Hydrological Atlas

On the same Atlas Platform, the University of Bern publishes its Hydrological Atlas of Switzerland using all the possibilities of navigating and presentation.

More maps: Statistical Atlas of Switzerland

There’s another Atlas with hundreds of thematic maps based on statistics:

More about Statatlas in the –> Leporello

Open Data: Global Goals, Local Impact

Upcoming Open Data Event

IODC16 Madrid. Global goals, local impact

The 4th International Open Data Conference, IODC16 will take place in Madrid. October 6-7, 2016. It will be ‘designed not as a single statement on open data, but rather as a curated record of discussions and debates, providing a snapshot of key issues and setting out a path forward based on the visions, ideas, and agreements explored at IODC 2015′ in Canada.

Reporting IODC 2015

From the 2015 report:

‘More than 1000 participants from 56 different countries took part in the 3rd International Open Data Conference (IODC) in Ottawa from May 28-29, 2015 . Hosted by the Government of Canada, the International Development Research Centre, and the World Bank, IODC brought together open data experts to understand the global impact of open data, coordinate action, and share best practices.’ p.5

‘A few years ago, debates about open data centered on dataset formats and data portals—not any more . Today, the open data debate covers an increasingly broad spectrum of topics: from comprehensive principles for open data and the measurement of impact, to the development of common standards, and critical issues such as privacy, multilingual data communities, and indigenous knowledge.’ p.30

Using Linked Open Data LOD

An example showing how LOD helps break down silos: Florian Bauer’s presentation in IODC 2015 ‘Using Open Data Thesauri to Connect Climate Platforms’.

‘Data + Environment = Mobilizing Knowledge
Addressing climate change and environmental sus-
tainability is a knowledge-intensive enterprise . With
billions of data points from scientific research and mil-
lions of journal articles published in many different
languages over recent decades, organizing all that in-
formation, and getting it to the right people at the right
time requires both active intermediaries and shared
knowledge infrastructures . The Climate Knowledge
Brokers (CKB) group is building on the Renewable En-
ergy and Energy Efficiency Partnership’s (REEEP) mul-
tilingual climate thesaurus to tag and enrich content
from different document repositories, using a shared
Linked Open Data taxonomy . This helps to break down
the silos that keep information apart, building a stron-
ger open data ecosystem.’

Open Data Standards. LOD?!

‘In collaboration with Iniciativa Latinoamericana por los Datos Abiertos (ILDA), Open North interviewed governments and civil society organizations in 10 low- and middle-income countries to get a sense of their progress, challenges, needs and interests with respect to open data standards. After reviewing the previously identified gaps in standardization in light of the interview results, we proposed 32 draft recommendations, which we now invite stakeholders to discuss, comment on, and eventually implement.’ (Guest post on the Blog IODC 2016 from James McKinney of Open North).

The bad news for LOD is: It’s only nice to have in these countries. The good news is: LOD is on the radar even in countries with not so many resources. The five-star step in the WEB needs time.

On page 40/41 the report ‘Identifying recommended standards and best practices for open data’ (By Stéphane Guidoin (Open North), Paulina Marczak (Open North), Juan Pane (ILDA) and James McKinney (Open North).) gives the following recommendation.

‘Implementation
● When possible, government could provide data using linked data format like RDF or JSONLD
● Data tools should support existing ontologies and vocabularies and adopt the relevant ones when possible
Discussion
On one hand, linked data and use of existing vocabularies is perceived a significant aspect of open data. With linked data, it is possible to significantly increase the automation capabilities since data attributes become much less ambiguous.
On the other hand, linked data almost did not appear in the answers of either government or consumer interviewees, and most of the government agencies appeared to already have difficulties to comply with more simple simple recommendations. Finally linked data is more difficult to use and require more advanced tools to be used that average citizens, CSOs and even developer do not always know.
As a consequence, use of linked data is presented as a “nice to have” recommendation and can be added to, but should not replace more accessible formats like tabular data.’

Extended Population Statistics

Providing accurate population stats is a quite difficult task. Censuses, registers, samples – these methods have their strengths and weaknesses; definitions and variables are complex and changing over time. What is a family? What is sex or gender?

And there is a tool that brings a new aspect into the discussion 😉 : people in space.

E-Citizenship and E-Population

Statisticians are working a lot on structuring realty in order to count the right things. But reality is not at all stable, changes are going on all the time and statistics have to decide which changes have to be followed.

So population Statsticians had to adapt to new family models, had to recognize that ‘male’ and ‘female’ is not sufficient and now a new type of people is popping up: e-Citizens.

Estonian e-residency

‘The Republic of Estonia will be the first country to offer e-residency. People from all over the world will have an opportunity to get a digital identity provided by the Estonian government – in order to get secure access to world-leading digital services from wherever you might be.’ From: ‘Become an Estonian E-Resident’

‘An e-resident can use and enjoy the same great digital services that allow Estonians to do anything and everything digitally–sign all documents,launch and
manage companies, do the banking, encrypt files, etc.’

Digital Society – Digital Population?

With a simple ID-Card you become an E-Estonian and a member of the digital society. And who will count this new kind of population?

http://e-estonia.com/

EU Open Data Portal. An Evaluation.

Evaluating the EU Open Data Portal

‘The Department of Library and Information Science at the Complutense University of Madrid (Spain) has published a study that analyses and evaluates the European Open Data Portal.’ . Report presentation: http://www.epsiplatform.eu/content/analysis-and-evaluation-european-union-open-data-portal-0 or direct download: http://revistas.ucm.es/index.php/RGID/article/download/99-118/42697. ____________________________________________________________________________________________________

Some interesting findings summarized below:

Editors

As with most of the other Open Government Data Portals the bulk of data comes from statistical agencies, here from Eurostat: .

Top 10

Number 1 is a translation help for all EU languages (‘translation memory, TM; i.e. sentences and their professionally produced translations’) .

Formats

Evaluation of data formats follows Tim Berners-Lee’s 5star schema. .

5star for metadata queries

Semantic (****) or even Linked (*****) Data are not part of the portal’s data repository. BUT the description of the data with metadata follows 5star requirements and the EU Portal offers a SPARQL endpoint. Great! https://open-data.europa.eu/en/linked-data

Let’s learn SPARQL!

Statistics and Our Understanding of the World

Our understanding of the world around us is shaped by the numbers we use to measure it.

Official statistics’s objective are to measure and to describe this reality – whatever this is – and its transformation. Concepts and survey methods are tools helping to do this.
These tools are themselves changing and developing through time and so is our understanding of the world. This quite complex interrelation is best understood by studying the history of statistics.

Three new books focus on this topic.

Zachary Karabell, The Leading Indicators: A Short History of the Numbers That Rule Our World, Simon&Schuster 2014

A short excerpt from the introduction:

‘For almost a century, people have been inventing statistics to measure our lives, and since the middle of the twentieth century, our understanding of the world has been integrally shaped by those numbers. Our statistical map, however, is showing signs of age. In our desire to have simple numbers to make sense of a complicated world, we forget that our indicators have a history—a reason that they were invented in the first place—and that history reveals their strengths and limitations just as our own personal histories do. Knowing how we came to live in a world defined by a few leading indicators is the first step to assessing whether we are still well served by them. ….. The temptation, then, is to find new formulas, better indicators, new statistics. The search for better numbers, like the quest for new technologies to improve our lives, is certainly worthwhile. But the belief that a few simple numbers, a few basic averages, can capture the multifaceted nature of national and global economic systems is a myth. Rather than seeking new simple numbers to replace our old simple numbers, we need to tap into both the power of our information age and our ability to construct our own maps of the world to answer the questions we need answering.’

Diane Coyle, GDP. A brief but affectionate history, Princeton University Press. 2014

From introduction:

‘GDP is the way we measure and compare how well or badly countries are doing. But this is not a question of measuring a natural phenomenon like land mass or average temperature to varying degrees of accuracy. GDP is a made-up entity. The concept dates back only to the 1940s. ….

Yet the primacy of GDP as the measure of economic success has been increasingly challenged, not so much by politicians or economists as by people who see it as the primary symbol of what’s gone wrong with the capitalist market economy. For example, environmentalists believe it leads to an overemphasis on growth at the expense of the planet, “happiness” advocates think it needs to be replaced with indicators of genuine well-being, and activists such as those in the Occupy movement argue that a focus on GDP has disguised inequality and social disharmony.

There are certainly several reasonable critiques of GDP and the role it has come to play in guiding economic policy. These also include questions about how complicated the statistical construction of GDP has become, and what such a complex abstraction can actually mean. But GDP is also, as this book will show too, an important measure of the freedom and human capability created by the capitalist market economy. GDP indicates, although imperfectly, innovation and human possibility. And it is an important measure of our creativity and care for one another in an economy based more and more on services and intangibles. In 2000, the U.S. Bureau of Economic Analysis declared GDP to be “One of the Great Inventions of the 20th Century.” It is an understandable exaggeration.
This book explains GDP and describes its history, sets out its limitations, and defends it still as a key indicator for economic policy. It is certainly a better indicator than some of the fashionable alternatives (like “happiness”) that have been proposed. I also ask whether GDP alone is still a good enough measure of economic performance—and conclude not. It is a measure designed for the twentieth-century economy of physical mass production, not for the modern economy of rapid innovation and intangible, increasingly digital, services. How well the economy is doing is always going to be an important part of everyday politics, and we’re going to need a better measure of “the economy” than today’s GD.’

Philipp Lepenies, Die Macht der einen Zahl. Eine politische Geschichte des Bruttoinlandsprodukts, Suhrkamp 2013

A political history of GDP …

Aus der Einleitung:

‘Das Bruttoinlandsprodukt (BIP) ist die mächtigste Kennzahl der Menschheitsgeschichte. Keine andere statistische Größe hat jemals eine ähnliche Wirkung entfaltet. Vordergründig ist das BIP nur das Maß der wirtschaftlichen Leistung einer Volkswirtschaft, der in einer Zahl ausgedrückte Wert aller in einer Periode hergestellten Güter und Dienstleistungen. Aber zusammen mit dem Wachstum, das die Veränderungsrate des BIP angibt, ist es viel mehr als bloße Statistik. Das BIP dient als Hauptindikator für Entwicklung und Fortschritt. Positives BIP-Wachstum ist nicht nur erklärtes Ziel fast jeder Regierung, sondern gilt oft auch als der einzig mögliche Ausweg aus einer ökonomischen Krise. Die Wirtschaft und die Politik der Welt definieren sich in hohem Maße über das BIP.

Dabei ist das BIP keine sich selbst erklärende Zahl wie die Temperatur in Celsius, der letztjährige Ausstoß von Treibhausgasen in Tonnen oder der Kaloriengehalt des eigenen Frühstücks. Es ist das Ergebnis einer Berechnungsmethode, die bestimmte wirtschaftliche Aspekte erfasst, andere jedoch nicht. Es beruht auf einer Interpretation dessen, was Leistung und was Volkswirtschaft heißt. …

Die besondere Position des BIP resultiert aus seiner politischen Akzeptanz. Deswegen sind der Zeitpunkt und die Umstände relevant, die dazu geführt haben, dass die Idee des Bruttosozialprodukts als sinnvolle Technologie des Regierens erkannt und seine Berechnung endgültig zur politischen Arithmetik wurde.’