Pilots: Adoption of the WGDC Recommendations

---

The following table shows a list of pilots implementing the RDA WGDC recommendations. Further details can be found by clicking on the name in the table below.

WG Pilots

Name	Data	Type	Status	Notes
CSV-Reference	CSV / SQL	reference	running	Reference implementation
Natural History Museum London	RDBMS	operational	finished
TIMBUS	RDBMS	research	finished	Sensor data
XML-Reference	XML	research	finished	eXist-DB
DEXHELPP	CSV/RDBMS	research	running	Social security data
Git-Reference	ASCII	reference	running	Reference implementation
VAMDC	SQL/NoSQL/ASCII/XML	deployment	running	Distributed data center
CBMI@wustl	RDBMS	deployment	starting	integration into i2b2
CCCA	NetCDF	deployment	finished	climate scenarios data
ACDH	RDBMS, LoD	deployment	starting	thesaurus
ARGO	NetCDF	deployment	planned	ODIP-II
BCO-DMO	CSV	deployment	planned
ENVRIplus		deployment	running
Ocean Networks Canada	Data streams	deployment	starting	Oceanographic data
…	CSV, RDBMS	deployment	planning	Conceptual evaluation, seeking funding

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

---

Short Template

– Pilot name:
– Contact person:
– Type: research pilot / reference implementation / operational system
– Status: finished / active / starting / planned
– Type of data: (RDBMS, XML, CSV, file-based, other)
– Dynamics: (very high-frequency (microsecs) / very frequent (minutes) / frequent (daily/hourly) / sometimes (every other month) / rarely /none)
– Domain:
– Short description:
– Solution / approach:
– Timeline:
– Supplementary material: slides, reports, screenshots, papers, SW, …

Details Template

RDA Data Citation Recommendations and their Application in the CSV Reference Implementation

A.    Preparing the Data and the Query Store

• R1 – Data Versioning: For retrieving earlier states of data sets the data needs to be versioned.
• R2 – Timestamping: Ensure that operations on data are timestamped, i.e. any additions, deletions are marked with a timestamp.
• R3 – Query Store: Provide means to store the queries used to select data and associated metadata.

B.    Persistently Identify Specific Data sets
         When a data set should be persisted, the following steps need to be applied:

• R4 – Query Uniqueness: Re-write the query to a normalised form so that identical queries can be detected. Compute a checksum of the normalized query to efficiently detect identical queries.
• R5 – Stable Sorting: Ensure an unambiguous sorting of the records in the data set.
• R6 – Result Set Verification: Compute a checksum of the query result set to enable verification of the correctness of a result upon re-execution.
• R7 – Query Timestamping: Assign a timestamp to the query either based on the last update to the entire database or the last update to the selection of data affected by the query or the query execution time. This allows retrieving the data as it existed at query time.
• R8 – Query PID: Assign a new PID to the query if either the query is new or if the result set returned from an earlier identical query is different due to changes in the data. Otherwise, return the existing PID.
• R9 – Store Query: Store query and metadata (e.g. PID, original and normalised query, query & result set checksum, timestamp, superset PID,  data set description and other) in the query store.
• R10 – Citation Text: Provide a recommended citation text and the PID to the user.

C.    Upon Request of a PID

• R11 – Landing Page: PIDs should resolve to a human readable landing page of the data set, which provides metadata including a link to the superset (PID of the data source) and citation text snippet.
• R12 – Machine Actionability: the landing page should be machine-actionable and allow retrieving the data set by re-executing the timestamped query.

D.    Upon Modifications to the Data Infrastructure

• R13 – Technology Migration: When data is migrated to a new representation (e.g. new database system, a new schema or a completely different technology), the queries and associated checksums need to be migrated.
• R14 – Migration Verification: Successful query migration should be verified by ensuring that queries can be re-executed correctly.