‘Third Wave’ driving latest trends in open-source data sharing

CKAN is an open source community data management system to meet the needs of the open data movement

Forms of data sharing have been around for hundreds of years – since the creation of academic research. As more and more areas of civilised life in the computer age involved the collection of data, it was usually shared on request, and on a right-to-know basis. Different countries’ approach to Freedom of Information legislation improved things in terms of transparent and accessible data as journalists, activists and legal eagles fought for more rights and greater access. One of the biggest revolutions in data sharing came in 2007 when the US Government passed the America COMPETES Act – America Creating Opportunities to Meaningfully Promote Excellence in Technology, Education and Science, promoting the open exchange of data and research between government agencies. Now, we are seeing a “third wave” focusing on partnerships and collaborations across different sectors, differentiating between demand and supply in the data equation. It is one of the latest market trends driving data sharing in the public sector, with the emphasis on good governance, clear regulation and greater accountability, using data sharing and licensing arrangements. CKAN is the “gold standard” in open-source management software and Link Digital’s Steven De Costa spoke about the CKAN project he co-stewards at the latest monthly meeting of the UK Data Vault User Group, on the subject of ‘Data Commons, Data Sharing and Data Marketplaces.’ Steven is leading the project to help government and large enterprises and NGOs (non-governmental organisations) catalogue, store and access datasets with a rich front-end and full application programming interface (API) – utilising visualisation tools and lots more. CKAN makes it easy to publish, share and work with data straight out of the box. Since 2013, Link Digital has contributed to CKAN with its expert consulting background to design and build capability for data repository management. The world’s response to the COVID-19 crisis has been data led. Microsoft has been leading an Open Data campaign over the past couple of years. Sharing and use of growing data sets has dominated the scientific community’s response to climate change, and achieving more sustainable use of the world’s natural resources. So CKAN has become very important in helping societies to understand their problems and seek solutions. It counts the US and Canadian Governments – and the UK’s National Health Service (NHS) – plus public projects in Switzerland and Germany among its high-profile users. The latter also includes departments across state and national government in Australia, where Link Digital is based. CKAN is used extensively to help with digital transformation and open government, as well as helping to create evidence-based policy. CKAN is a data management system (DMS), which can be used to create a variety of solutions, including content management for data rich websites. Firstly, it acts as a data portal – serving as the gateway to an organisation’s data, whether that is publicly, or inside the enterprise’s data platform. Secondly, it works as a data catalogue, creating an inventory of your data, collecting and standardising metadata, so much so that CKAN is a standard choice for enterprise data catalogues. Thirdly, CKAN stores both metadata and data with the capability of unlimited scalability in the Cloud, making it the perfect foundation for data repositories, such as a data lake. The customer gets user and dataset management and the ability to search and discover through metadata, as well as metadata customisation, plus data storage and an API which controls. After explaining the conceptual architecture to show how CKAN would integrate with an organisation’s data sources and platform, Steven gave a demonstration of the software in action – which you can see by watching our video of the presentation. CKAN users include Brisbane City Council, which shares real-time data affecting its services, while New South Wales uses it for its flood protection work and monitoring its climate. One of the biggest data sets, is at

www.pacificdata.org/

which offers a multiple source harvesting and CMS integration across one of the most important regions of the world. The Australian Government’s Geoscience Data Portal has 140,000 data sets covering three millions square kilometres of Northern Australia, looking at chemical, water and mineral resources as part of a $225million ‘Exploring for the Future’ programme. You can see the different CKAN portals around the world by clicking

here