8 Conclusions & Next Steps: Towards a European Music Observatory

In this report, we introduced the Open Music Observatory, a complex data curation and information management system with a novel observatory governance model. The system is technically ready and has been tested with real-life use cases and scenarios: it handles complex data extract-transform-load processes, GDRP and confidentiality problems, and many aspects needed to make a data (sharing) space work. Because it is based on open-source components and on open data standards, it is set on an optimal path to grow as a data ecosystem.

The next development phase of this observatory will be critical: we must persuade music sector stakeholders to trust and use our system for data sharing well beyond the stakeholders of the Open Music Europe project. Once we reach out and onboard external partners, we will certainly have to modify business processes and permissions and provide further technical help, software components, and manuals. Besides these obvious iteration steps, we would also like to agree on a governance model that allows the extensions and the ownership transfer of this Open Music Observatory to a truly European, shared, public-private observatory that can serve the music sector for a long time.

8.1 Co-creating a governance model for the Observatory

Our current roadmap for the institutionalisation of the Observatory is based on the example of Europeana, a joint collection of European libraries and museums. Europeana 1.0 was built over 2.5 years in the European Digital Library Network project. It remained a decentralised network of three organisations: the Europeana Foundation, which is the technical operator of the digital collection; the Europeana Network Association, which is free to join for all interested parties; and the Aggregators’ Forum, which is a technical coordination body to help those technical data providers that send or exchange data with Europeana.

To draw on this analogy, the Open Music Observatory is being created by the Open Music Europe project. As stated in our Grant Agreement and the connecting Consortium Agreement, we treat the duration of the Open Music Europe project as a prototyping and development phase when multiple institutionalisation forms are still possible. Based on our Agreements, we formed the Observatory Stakeholder Network as a stakeholder group to set priorities and express opinions on our work and the potential longer-term institutionalisation alternatives.

We consulted several stakeholders throughout the project, and eventually, following their advice, we decided to form this advisory council when we already had a working data dissemination infrastructure and reviewable data in it to start their work. Not prejudicing any later presented organisational proposals, for the time being, we follow functionally the organisation of Europeana, because we need to deal with similar problems.

Europeana started out as a project of a few national libraries and after 16 years of existence, it aggregates digital collections from more than 3000 organisations of Europe. Such a large-scale but decentralised organisation requires a multi-tier, multi-functional governance structure, where large stakeholders can take ownership, smaller stakeholders can democratically participate and cooperate, and technical providers can work separately on technical-only problems.

The Observatory Stakeholder Network is presented as a consulting body where future founders of the Observatory itself, and a Music Observatory Association of music data users, providers may find their future role. We hope that some of the initial members will become founders in the future European Music Observatory, and others will be forming a democratic Music Observatory Association for grassroot aggregation and dissemination. (Observatory Stakeholder Network↗).
Our Open Music Data Exchange Forum is a consultation body, similar to the Aggregators’ Forum in Europeana, for large data partners.
As stated in our Grant Agreement, we will offer various institutionalisation proposals by the end of our project. If the representative stakeholders of the European music sector will choose not to create a European Music Observatory, we will continue to operate the Open Music Observatory with any interested party. (See project website: openmuse.eu, project data on CORDIS, (Open Music Europe 2023).)

8.1.1 Observatory Stakeholder Network

The Observatory Stakeholder Network is a temporary organisation defined by the Open Music Europe project Consortium Agreement. It is a volunteer advisory body of the observatory.

Tip

The Europeana Network Association (ENA) is a strong and democratic community of experts working in the field of digital cultural heritage. We are united by a shared mission to expand and improve access to Europe’s digital cultural heritage. The Association is free to join and we encourage our members to get involved and benefit from all the ENA has to offer.

We will invite every organisation that has promised data for a future European Music Observatory or has shown interest in the previous CEEMID or Digital Music Observatory collaborations to join our network. In the longer term, we envision some pan-European representative and umbrella organisations becoming founders or board members in the more permanent observatory organisation. For individuals, research groups, and micro-enterprises, we will suggest setting up the European Music Observatory Network Association (or similar entity), learning from the experience of Europeana.

8.1.2 Open Music Data Exchange

Tip

Aggregators work with cultural heritage institutions to gather authentic, trustworthy and robust data and make it accessible through Europeana. All Europeana aggregators are members of the Europeana Aggregators’ Forum (EAF), a network of national, regional, domain and thematic aggregators who - among others - work to exchange the knowledge and best practice that supports aggregation and data sharing with Europeana.

The Europeana Aggregators’ Forum, which will be held in May 2024, comprises 13 cross-national aggregators and 30 national aggregators who help cultural heritage organisations aggregate their collection datasets into the Europeana.

We imagine a similar organisation comprising companies that are not data owners but provide data and IT services for important national or pan-European stakeholders. While they cannot give permission or license data to the Open Music Observatory, their expertise in carrying out data integration will be invaluable in making our data-sharing space behind the observatory work.

8.2 Bottom-up expansion of the Observatory in a federation model

A data federation is a collection of software processes, that allows multiple databases to function as one. This virtual database takes data from a range of sources and converts them all to a common model. This provides a single source of data for front-end applications. Data federation is a key feature of the data (sharing) space. Data federation can only occur after the legal conditions, business process alignment and semantic harmonisation allows the data to be legally and meaningfully integrated.

As explained in Section 2.3, we have started the prototyping of our governance and interoperability model in the Slovak Republic.

The data integration that we started in the Slovak Repubic can be replicated in other parts of the European music sector, among any two or more data sharing partners.

This is an essential aspect of our bottom-up approach: similar data integration can start in other parts of the European music sector. It can take shape in another national (or regional) data-sharing space, a genre-specific one (for contemporary classical music, which has a relatively distinct ecosystem from popular music), or in a business subsector, like publishing. As long as these stakeholder groups use the same standards and architecture, we can gradually connect them into a European music dataspace. We will add such replication cases outside of Slovakia in the Open Music Europe project. Because our system is shaped as a decentralised graph without a central node, it does not matter which countries, sub-sectors, or genres follow Slovakia; the country has no unique role in the eventual Observatory, apart from being a pioneer and knowledge centre in the music data exchange.

The data (sharing) space is not only a national data exchange: it allows us to coordinate data with open knowledge bases and their data graphs internationally; we will demonstrate whom we exchange data with Wikipedia/Wikidata, CIDOC-compliant collections, RiC-compliant archives, or SDMX-compliant statistical services.

8.3 Practical next steps in the project

Task T5.1 developing and validating the Open Music Observatory is running until the end of the Open Music Europe project, and it is connected to almost all other project tasks. Methodological and research tasks of the project identify data gaps and methodologies to fill those gaps, while other tasks provide us with more direct input in the form of data that can be disseminated via the Observatory.

With the current deliverable the Open Music Observatory has been released with a governance model to combine existing open data sources and privately held music sector data with data created within other project tasks. This marks a milestone, linked to MS4 of the project for this system development, as the Observatory is now ready for use by all internal and external stakeholders, enabling open data collection and real-time reporting to begin.

To begin the validation and improvement of the Observatory we have imported data from the project’s background or other sources and encouraged project partners to work with our evolving system already before this milestone. As a result, our observatory systems are not empty at the start, but they also lack the critical mass of data to train data improvement algorithms or provide value-added services. Those functionalities will be utilised as other project tasks report their results and hand over their data to us; and as external stakeholders start to trust us with their data.

Different level of service and help for data exchange with stakeholders

We have done extensive system requirement solicitation and ingested sensitive or complex data as we were developing the system. Based on almost 1.5 years of experience, we believe that we need to further refine the auxiliary tools for three data curation/exchange scenarios.

For data curation excellence centres, like the EU Open Data Portal, Europeana, Wikibase, and Eurostat, we must ensure that our machine-to-machine communication works better and exchanges richer metadata. We improve fully automated and well-standardised workflows.
For larger music sector stakeholders who have an advanced internal IT function and often some library/information science competence but no experience in machine-to-machine communication with standard or open protocols, we need to provide servicing and help with semantic schema adjustments, as they are usually able to export data in tidy and well-organised format. We expect collective rights management organisation, national libraries or larger specialised public music libraries, archives, museums, and larger music information centres to fall into this category.
For organisations that do not have an internal IT/data/library function, or work with a limited, generalist IT function, we need to provide simple tools. For simpler, spreadsheet-based data sources, we would like to improve tutorials and tools for self-servicing. We need to take a cost/value assessment for small organisations with big and complex databases because they will likely need knowledge or data engineering help, which is a limited resource in our project. We expect small enterprises, NGOs, and trade associations or umbrella organisations to fall into this category.

To reach our objectives related to the Open Music Observatory, we will take the following steps after this milestone:

We introduce the system to those organisations that showed a willingness to share data with a future European observatory (See: Section 5.2), and ask for their feedback and at least test data samples;
Based on the external data samples, in our supporting task, we will develop further extract-transform-load components to ensure a smooth data export/import according to the needs of the representative European music stakeholders;
We will disseminate and improve with feedback our data sharing space model, and particularly the accompanying governance model, with experts in the field of data governance, data sharing, and various music industry data and metadata interest groups;
We will finalise the creation of the Slovak Comprehensive Music Database, enabling the MVP demonstrations of several value-added services, as described in Section 7.1. We plan to start disseminating these use cases on large stakeholder forums in the second half of 2025 in the hope of finding additional replication partners and users.
An adequate description of our planned use of AI will be provided for overview, and will be brought to the attention of the Ethics Advisor during the upcoming meeting of the Ethics Board (see Section 7.3.)
We will ingest a large batch of survey data from other work packages in 2025 Q1.