---
title: "Design Principles & Future Work Semantically Enriched, Standards-Aligned Datasets in R"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteEncoding{UTF-8}
  %\VignetteIndexEntry{Design Principles & Future Work Semantically Enriched, Standards-Aligned Datasets in R}
  %\VignetteEngine{knitr::rmarkdown}
editor_options: 
  markdown: 
    wrap: 80
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```

## Abstract

```{r setup}
library(dataset)
```

> “A dataset is an identifiable collection of data available for access or
> download in one or more formats.” — ISO/IEC 20546

The `dataset` package enriches R data objects with machine-readable metadata by
embedding semantic definitions and provenance at both the variable and dataset
levels. It follows a semantic **early-binding** design: metadata is attached at
creation time, not retrofitted post hoc. This ensures that meaning and context
are preserved throughout the data lifecycle — from exploration to publication —
and enables partial automation of documentation.

This article outlines the design philosophy behind the `dataset` package,
including its theoretical foundations, structure, relationship to other R tools,
and example workflows. It serves as a long-form complement to the package
vignettes.

## Introduction and Motivation

> “The principles of tidy data provide a standard way to organise data values
> within a dataset.”\
> — Wickham (2014)

The `dataset` package extends R’s native data structures by embedding
machine-readable semantics and provenance directly in tidy data objects. It
builds on tidy data principles (Wickham, 2014) but introduces a **semantic
early-binding** approach: metadata is attached when the dataset is created,
ensuring that context and meaning are preserved through all stages of the
workflow — including transformation, validation, serialization, and reuse.

While tidyverse tools enforce structural clarity, they are generally agnostic
about semantics. Variables may be misinterpreted, joined incorrectly, or
published without context. `dataset` addresses this gap by aligning with
international metadata standards, supporting RDF export, and providing an
interface to the W3C Data Cube model.

A tidy dataset, per Wickham’s definition, adheres to three core rules:

-   Each variable forms a column\
-   Each observation forms a row\
-   Each value forms a cell

![Figure 12.1 in R for Data Science: tidy data structure.](images/tidy-1.png)

However, this tidy structure — typically implemented as a `data.frame` or
`tibble` — is not semantically self-describing. In practical workflows, users
often conflate the in-memory structure with the abstract concept of a dataset,
which in metadata terms refers not just to structure but also to definitions,
units, provenance, and contributors.


Several ISO and W3C standards define what constitutes a dataset. According to ISO/IEC 20546, a dataset is an identifiable collection of data available for access or download in one or more formats. The Dublin Core DCMI Metadata Terms define a [dataset](https://www.dublincore.org/specifications/dublin-core/dcmi-terms/dcmitype/Dataset/) as “data encoded in a defined structure.” The W3C’s [Data Cube Vocabulary](https://www.w3.org/TR/vocab-data-cube/#cubes-model-datasets), widely used in official statistics, describes a dataset as a “collection of statistical data that corresponds to a defined structure.” That structure includes observations, metadata about their organisation, *structural metadata* (e.g., units of measure), and *reference metadata* (e.g., creator, publisher).  

This differs from R’s `data.frame` object, which is defined as “tightly coupled collections of variables which share many of the properties of matrices and of lists, used as the fundamental data structure by most of R's modeling software.” In practice, R users often use the terms *data frame* (or tibble) and *dataset* interchangeably. However, even a tidy data frame is underspecified for use in scientific repositories, statistical data exchanges, or many database applications. A `data.frame` exists only in the memory of an R session, limiting its interoperability and reusability. While R can already serialise data frames to formats like `.rds`, `.rda`, or `.csv`, these serialisations by default lack rich, standardised metadata. The `dataset` package bridges that gap by aligning with established metadata standards, producing serialisations that are easier to reuse and interpret.  

The `dataset` package extends R's native data structures with machine-readable metadata. It follows a *semantic early-binding* approach: metadata is embedded as soon as the data is created, making datasets suitable for long-term reuse, FAIR-compliant publishing, and integration into semantic web systems.  

The central innovation of the package is an extended data-frame-like object: a `tibble::tibble()` enhanced with R’s `attributes()` system to store standard metadata from ISO and W3C vocabularies. This `dataset_df` class integrates naturally with tidy data principles (Wickham, 2014), where each variable is a column, each observation is a row, and each type of observational unit forms a table. On top of this tidy structure, `dataset_df` adds a semantic layer so that the meaning of variables and datasets is explicit and machine-readable.  This new class is introduced in  `vignette("dataset_df", package = "dataset").

In research or institutional contexts, a dataset is a form of digital resource,
often archived, cited, or published. Such resources are typically described with
metadata using the Resource Description Framework (RDF), enabling
machine-actionable, language-independent, schema-neutral representation. Our aim is to facilitate the translation or annotation of a tidy R data.frame into such a resource.

RDF also enables description at the level of **elementary statements** — that
is, per-cell metadata combining variable (column) and observation (row). This
allows for fine-grained semantic annotation, supporting full data traceability
and interoperability.

The original tidy workflow was designed for solo, interactive analysis where
analysts had full context. But in collaborative, institutional, or
public-sharing contexts, assumptions must be replaced with formal semantics. Not
only structure, but also clear definitions — of units, classifications, codes,
and contributors — become essential.

Moreover, many statistical data providers follow the **data cube model**, which
resembles tidy data but supports higher dimensionality and more formal metadata.
Examples include SDMX and the W3C Data Cube vocabulary.

Tidy data assumes that column names and structure are sufficient for clarity.
However, ambiguity arises quickly when combining datasets from heterogeneous
sources. A column named `geo` might contain ISO codes in one dataset and
Eurostat codes in another. GDP figures may differ in currency or base year.
These inconsistencies often go unnoticed until late-stage analytical errors.

For example:

```{r dataframe}
data.frame(
  geo = c("LI", "SM"),
  CPI = c("0.8", "0.9"),
  GNI = c("8976", "9672")
)
```

This dataset is tidy, but not self-describing. Is geo using ISO 3166 or Eurostat
codes? Is GNI measured in euros, dollars, or PPP-adjusted values?

The dataset package addresses these challenges by introducing structures for
semantically rich vectors (`defined()`) and annotated tibbles (`dataset_df()`).
It integrates machine-readable metadata directly into R objects and ensures that
labels, units, concept URIs, and provenance are preserved from creation to
publication.

This approach bridges the gap between tidy data and RDF, making formal semantics
part of the tidyverse workflow — without requiring users to leave R or manually
manage external metadata schemas.

## Related Work

Several R packages have offered tools to improve the metadata management of
datasets within the tidyverse ecosystem or its surrounding statistical
traditions.

The `labelled` class in the `labelled` and `haven` packages supports long-form
variable labels and improved handling of value label sets compared to base R’s
`factor` class. This is particularly helpful for variables collected in survey
instruments — a major source of microdata in statistical workflows. However,
real statistical production, as standardized by GSIM (Generic Statistical
Information Model) and DDI (Data Documentation Initiative), involves a far more
complex metadata model. Our contribution builds on these efforts by enabling
users to attach standardized, cross-domain codebook references to such
variables, ensuring that labelling follows recognized metadata vocabularies.

The `dataspice` package allows users to create auxiliary metadata datasets for
publication. Its strength lies in its simplicity: it uses human-readable CSV
files to capture key metadata fields. However, this simplicity introduces
fragility: the metadata may become detached, outdated, or unsynchronised from
the data file, especially in collaborative or iterative settings.

The `rdflib` package, a high-level interface to the RDF library of the same name
in Python, supports RDF serialization and querying. It allows tidy data to be
mapped to RDF triples and serialized into N-Triples, Turtle, RDF/XML, or
JSON-LD. However, it assumes that metadata is retrofitted — applied after the
analytical workflow is complete. While `rdflib` is essential for
interoperability, it requires users to leave the tidyverse workflow or gain RDF
modelling expertise.

Our goal with the `dataset` package is to bridge the semantic and methodological
gap between the `tidyverse` and `rdflib`: to make semantically annotated,
publication-ready datasets part of the R-native workflow from the start.

Another important initiative is the **Frictionless Data** project, which
provides lightweight standards for describing datasets (e.g., via
`datapackage.json`). It enables platform-independent validation and metadata
exchange. In R, the `frictionless` package supports reading, writing, and
validating data packages. However, this system relies heavily on external JSON
schemas and does not integrate metadata into the objects used during analysis —
meaning users must juggle separate metadata files and validation steps,
potentially losing semantic continuity during transformation.

Tierney and Cook (2020), in their paper *Expanding Tidy Data Principles to
Facilitate Missing Data Exploration, Visualization and Assessment of
Imputations*, demonstrate how tidy data can be extended through consistent
metadata structures and function design. Their concept of “nabular data” —
datasets with shadow columns representing missingness — shows how tidy workflows
can accommodate new dimensions of metadata. While their focus is on handling
missing data, their methodological framing is closely aligned with ours:
extending tidy conventions by designing new object classes and verbs, rather
than retrofitting external metadata after the fact.

Collectively, these tools highlight a shared recognition: tidy data principles
offer a strong foundation, but do not, by themselves, guarantee semantic
clarity, interoperability, or reuse. The `dataset` package responds to this need
by embedding standardized metadata directly in R objects, enabling datasets to
remain semantically intact throughout transformation, validation, and
publication.

## Design Principles

-   **Early binding of semantics**: Metadata is attached at the point of dataset
    creation, not after the fact.
-   **Attribute-based, not schema-based**: Metadata lives inside the R object
    itself, not in external schemas or files.
-   **Minimal friction with tidyverse workflows**: Compatible with `dplyr`,
    `tidyr`, `vctrs`, and coercible to `tibble` or `data.frame`.
-   **Persistence across save/load cycles**: Metadata survives R serialization
    (`.rds`, `.rda`).
-   **Tidyverse-like grammar**: Core verbs include `defined()`, `dataset_df()`,
    `provenance()`, `describe()`, `datacite()`, and `dublincore()`.
-   **Full interoperability outside R**: Datasets can be exported as RDF using
    `dataset_to_triples()` and ingested into triple stores.

### Semantic Early Binding

The `dataset` package introduces several new S3 classes that remain fully
compatible with tidyverse idioms and largely interoperable with base R. These
classes rely on R's native attribute system to embed metadata directly within
vectors and tibbles. This enables metadata such as labels, concept URIs,
namespaces, and provenance details to persist during filtering, joining, or
transformation.

The attribute system in R is underused, and most user-friendly packages offer
little support or interface for working directly with object attributes. This
leads to redundancy — with metadata often duplicated within the dataset content
itself.

The `defined()` constructor builds on `labelled::labelled` (originally from
`haven`) and provides a more expressive way to annotate vectors with:

-   A human-readable label (e.g., `"Gross Domestic Product"`)
-   A unit or measurement system (e.g., `"CP_MEUR"`), accessible via
    `var_unit()` and set with `var_unit() <-`
-   A concept URI that uniquely identifies the variable or dimension, handled
    via `var_concept()` and assignment
-   A namespace URI pattern for resolving coded values (e.g., ISO or Eurostat
    country codes), via `var_namespace()`

The `dataset_df()` class extends `tibble` and supports combining enriched
vectors with dataset-level metadata. This includes Dublin Core and DataCite
elements such as title, creator, publisher, subject, and contributors, along
with provenance metadata like creation time or software agent.

### Attribute-Based, Not Schema-Based

The `dataset` package adopts an attribute-based design rather than a
schema-based approach. Metadata is stored directly in R objects using native
attributes, ensuring semantic annotations remain tightly coupled with the data
throughout transformation, saving, and reuse.

This approach eliminates the need for separate schema definitions or JSON
metadata files — lowering the barrier to semantic data publishing within R
workflows.

In R, most objects (especially vectors and data frames) can carry attributes
such as:

-   `names`
-   `class`
-   `label`
-   `unit`
-   `concept`
-   `namespace`

These are lightweight, internal, and flexible. For example:

```{r attributes}
x <- 2457
attr(x, "unit") <- "CP_MEUR"
attr(x, "concept") <- "http://data.europa.eu/83i/aa/GDP"
```

In the dataset package, this metadata is preserved in defined and dataset_df
objects and moves with the data — whether it's saved, joined, subsetted, or
filtered.

By contrast, many CRAN or rOpenSci packages are schema-based: they require
external metadata definitions that describe expected columns, data types, and
semantic rules. While these can support more complex use cases — such as SDMX
structural metadata or JSON Schema validation — they introduce additional
overhead, increase complexity, and risk desynchronisation between data and
metadata.

Schema-based solutions may be more appropriate when data analysts work in teams
alongside research data managers or other documentation specialists. In
contrast, the `dataset` package is designed for individual researchers or small
teams who want to avoid semantic errors when ingesting new data from external
sources — while also enabling standards-compliant data exchange and publication
with minimal additional tooling.

## Persistence Across Save/Load Cycles

Because all metadata is stored as object attributes, it remains intact when
datasets are saved using native R serialization formats like .rds or .rda. These
attributes can be queried, extracted, or exported — but they do not interfere
with regular data manipulation or analysis.

Metadata is added at the time of object creation, in contrast to workflows where
metadata is generated after analysis or stored in sidecar files (e.g., JSON-LD).
This design reduces the risk of metadata being detached, outdated, or
incomplete.

## Base Examples: Using the `dataset` Grammar

This section demonstrates the core grammar of the `dataset` package using
minimal, synthetic examples. These illustrate how to define semantically
enriched vectors, assemble them into annotated datasets, and prepare them for
RDF export or validation.

### Creating Defined Vectors

The `defined()` constructor creates semantically enriched vectors. It extends
`labelled::labelled` with additional attributes such as `unit`, `concept`, and
`namespace`.

```{r}
library(dataset)

gdp <- defined(
  c(2355, 2592, 2884),
  label = "Gross Domestic Product",
  unit = "CP_MEUR",
  concept = "http://data.europa.eu/83i/aa/GDP"
)

geo <- defined(
  rep("AD", 3),
  label = "Geopolitical Entity",
  concept = "http://dd.eionet.europa.eu/vocabulary/eurostat/geo/",
  namespace = "https://www.geonames.org/countries/$1/"
)
```

These vectors behave like regular R vectors but carry internal metadata. This
metadata can be retrieved or reassigned using the accessor and setter functions
provided by the package:

```{r}
var_concept(gdp)
var_unit(gdp)
var_namespace(geo)
```

These attributes are preserved across most data transformations, and persist
when saving to `.rds` or `.rda`.

### Assembling a Dataset with Metadata

Use `dataset_df()` to combine defined vectors into a tibble-like object that
includes dataset-level metadata, such as bibliographic information, identifiers,
and provenance.

```{r smalldataset}
small_dataset <- dataset_df(
  geo = geo,
  gdp = gdp,
  identifier = c(gdp = "http://example.com/dataset#gdp"),
  dataset_bibentry = dublincore(
    title = "Small GDP Dataset",
    creator = person("Jane", "Doe", role = "aut"),
    publisher = "Small Repository",
    subject = "Gross Domestic Product"
  )
)
```

Behind the scenes, the package uses a custom bibrecord class that extends
`utils::bibentry()` to accommodate all metadata fields defined by Dublin Core
and DataCite — two major standards used in repositories, library systems, and
FAIR data infrastructures.

You can review the dataset-level metadata in both formats:

```{r bibrecord}
as_dublincore(small_dataset)
as_datacite(small_dataset)
```

Since these metadata models do not fully overlap, using `dublincore()` will
leave some DataCite-specific fields empty.

### Provenance Tracking

One benefit of early metadata binding is that basic provenance is automatically
tracked. The `provenance()` function returns metadata about when and how the
dataset was created — including the system time and, optionally, the software
environment.

```{r provenance}
provenance(small_dataset)
```

This provenance is also included in the machine-readable metadata that can be
exported using `describe()`, which generates an RDF description of the dataset.

```{r description}
description_nt <- tempfile(pattern = "small_dataset", fileext = ".nt")
describe(small_dataset, description_nt)

# Only a few lines shown:
readLines(description_nt)[5:8]
```

The dataset grammar provides a lightweight but standards-compliant way to attach
metadata during the creation of R objects. Unlike retrofitted metadata tools, it
keeps semantic annotations inside the object throughout filtering, saving, and
publishing. In the next section, we apply this grammar to a real-world scenario
involving statistical datasets with conflicting semantics.

## Applied Example: Joining Data with Semantic Constraints

This example demonstrates how the `dataset` package helps avoid semantic errors
when combining data from heterogeneous sources. We create a small GDP dataset
for three European microstates, measured in millions of euros (CP_MEUR), and
then attempt to append an observation from Tuvalu, measured in US dollars (USD).
The semantic mismatch triggers an error.

### Step 1: Create a Eurostat-Compatible GDP Dataset

```{r eurogpd}
euro_gdp <- defined(
  c(2355, 2592),
  label = "Gross Domestic Product",
  unit = "CP_MEUR",
  concept = "http://data.europa.eu/83i/aa/GDP"
)

geo_europe <- defined(
  c("AD", "LI"),
  label = "Geopolitical Entity",
  concept = "http://dd.eionet.europa.eu/vocabulary/eurostat/geo/",
  namespace = "https://www.geonames.org/countries/$1/"
)

euros_dataset <- dataset_df(
  geo = geo_europe,
  gdp = euro_gdp,
  dataset_bibentry = dublincore(
    title = "European Microstates GDP",
    creator = person("Statistical Unit", role = "aut"),
    publisher = "Eurostat",
    subject = "Gross Domestic Product"
  )
)
```

### Step 2: Create a Dollar-based GDP Dataset

```{r usdgdp}
usd_gdp <- defined(
  56,
  label = "Gross Domestic Product",
  unit = "USD_MILLIONS",
  concept = "http://data.europa.eu/83i/aa/GDP"
)

geo_tuvalu <- defined(
  "TV",
  label = "Geopolitical Entity",
  concept = "http://dd.eionet.europa.eu/vocabulary/eurostat/geo/",
  namespace = "https://www.geonames.org/countries/$1/"
)

tuvalu_dataset <- dataset_df(
  geo = geo_tuvalu,
  gdp = usd_gdp,
  dataset_bibentry = dublincore(
    title = "Tuvalu GDP (USD)",
    creator = person("Island", "Bureau", role = "aut"),
    publisher = "PacificStats",
    subject = "Gross Domestic Product"
  )
)
```

The tidy workflow is based around five operational actions: - Data reshaping
goes from long to wide formats; - sorting arranges rows in a specific order; -
filtering removes rows based on a condition; - transforming, changes existing
variables or adds new ones; - aggregating creates a single value from many
values, say, for example, in computing the minimum, maximum, and mean.

Ideally, each of these steps should be recorded in the metadata. We will only
show data reshaping and transforming, because aggregation can be well described
with defining the new aggregate with `defined()`, and sorting and filtering are
trivial in a format where each observation is uniquely identified.

```{r}
binded <- try(bind_defined_rows(euros_dataset, tuvalu_dataset), silent = TRUE)
```

This will raise an error or warning because the gdp column has inconsistent
units (CP_MEUR vs USD_MILLIONS). The semantic definitions attached to each
vector allow dataset to detect and prevent accidental joins across incompatible
measurement systems.

### Step 3: Transform the Data and Document the Change

```{r mutate}
exchange_rate <- 1.02
eur_tuv_gdp <- defined(
  56 * exchange_rate,
  label = "Gross Domestic Product",
  unit = "CP_MEUR",
  concept = "http://data.europa.eu/83i/aa/GDP"
)

tuvalu_dataset <- dataset_df(
  geo = geo_tuvalu,
  gdp = eur_tuv_gdp,
  dataset_bibentry = dublincore(
    title = "Tuvalu GDP (USD)",
    creator = person("Island", "Bureau", role = "aut"),
    publisher = "PacificStats",
    subject = "Gross Domestic Product"
  )
)
```

In a larger dataset, the user will likely use the tidyverse grammar (or the
grammar of data.table), with mutating the dollar values into euro values. In
this case, the transformation or the mutation should be recorded in the change
of the unit. If you would add population data to the GDP dataset, and compute
GDP/capita, you would also want to add a new long-form variable label, perhaps
change the unit from millions of euros to euros.

```{r}
var_unit(eur_tuv_gdp) <- "M_EUR"
```

The joined dataset needs a new title, and it can be attributed to a new author
and publisher. The vocabulary of the Dublin Core and DataCite metadata standards
used by most repositories and exchanges are covered with convenient helper
functions that retrieve or set the descriptive metadata value. Some of them,
like the title, are protected with explicit overwrite permissions.

```{r metadatachanges}
global_dataset <- bind_defined_rows(euros_dataset, tuvalu_dataset)
dataset_title(global_dataset, overwrite = TRUE) <- "Global Microstates GDP"
publisher(global_dataset) <- "My Research Institute"
creator(global_dataset) <- person("Jane Doe", role = "aut")
language(global_dataset) <- "en"
description(global_dataset) <- "A dataset created from various sources about the GDP of very small states."
global_dataset
```

You can review the descriptive metadata of the dataset with `as_dublincore()` or
`[as_datacite()]` in various formats.

```{r}
as_dublincore(global_dataset)
```

A tidy dataset can be serialised to RDF with dataset_to_triples, which performs
the data reshaping goes from wide to long formats. You can read a lot more in
the vignette-articles of the high-level R-binding to the Python RDFLib library,
rdflib, particularly the *A tidyverse lover's introduction to R* on how to
normalise the data to a format that it can be serialised to a flat RDF file or a
graph database.

```{r}
dataset_to_triples(global_dataset)
```

```{r}
dataset_to_triples(global_dataset, format = "nt")
```

## Full Interoperability

In the semantic web, datasets are often represented as collections of triples:
subject, predicate, and object. The `dataset_to_triples()` function enables this
by converting any dataset_df into a long-form representation where each row
represents a semantically annotated cell.

Unlike tidy datasets that require column-wise joins and reshape operations,
RDF-based datasets eliminate structural joins by relying on identity, context,
and concept URIs. Repeated values are normalized at the semantic level. This
makes triple-based data more flexible for publishing, integration, and querying
across domains.

This design choice affects how we implemented joins and bindings. The package
avoids implementing column-wise joins or wide-format merging because
semantically rich datasets can be recombined or queried directly via SPARQL or
other RDF tooling. Instead, row-wise binding via bind_defined_rows() is
supported, allowing users to append consistent datasets without losing
semantics.

This reflects a deliberate philosophy: rather than duplicate tidyverse
behaviours, dataset encourages upstream semantic modelling and downstream
interoperability.

The dataset_to_triples() function exports a tidy dataset to RDF-style triplets:

triples \<- dataset_to_triples(small_dataset) head(triples)

Each row becomes a triple (subject, predicate, object), typed with XSD and
optionally resolved via URIs. Export is supported through rdflib.

This example illustrates the core design goal of the `dataset` package: to make
semantic metadata first-class citizens of the R data workflow. By embedding
units, concept URIs, and provenance directly in data objects, the package
supports not only reproducible research but also semantically interoperable
publication — all without departing from familiar tidyverse idioms.

The dataset created in this example could be easily validated, documented, and
exported as linked data using standard RDF tooling. This forms the basis for
reproducible, standards-aligned workflows that extend beyond the analyst’s
desktop — into repositories, triple stores, or domain-specific data services.

Yet, the applied example also reveals current limitations and areas for growth
in the `dataset` package, which we now turn to.

## Export and Interoperability

The `dataset` package is designed with FAIR principles in mind, particularly the
goal of enabling machine-actionable data publishing. To support semantic web
compatibility and downstream interoperability, it provides functions that allow
users to convert annotated datasets into RDF-compatible formats.

The key function in this process is:

-   `dataset_to_triples()`: Converts a `dataset_df` into a three-column
    long-form structure—subject, predicate, object—representing each cell as an
    RDF-style triple. These can be exported to tabular or text-based formats, or
    directly ingested by triple stores.

This structure aligns with the W3C's RDF and Data Cube vocabularies, where: -
The **subject** typically encodes an observation or observation unit - The
**predicate** is derived from a concept URI associated with the variable - The
**object** is the value, typed using XML Schema Definitions (e.g.,
`xsd:integer`, `xsd:string`)

These outputs are fully compatible with the
[`rdflib`](https://docs.ropensci.org/rdflib/) package, which can serialize RDF
datasets into: - Turtle (`.ttl`) - RDF/XML (`.rdf`) - N-Triples (`.nt`) -
JSON-LD (`.jsonld`)

This enables dataset publication to: 

- SPARQL endpoints 

- FAIR data repositories

- Wikibase instances (via planned extensions) 

- Semantic web catalogues

Triple-based export promotes structural normalization, eliminates redundancy,
and facilitates data integration across domains and systems.

## Limitations and Future Work

The `dataset` package prioritizes ease of use and integration with existing
tidyverse workflows. It intentionally implements a practical subset of features
drawn from more formal metadata and ontology systems used in statistical
domains, such as SDMX, DDI, and DCAT.

Some features have been deliberately left out to keep the package lightweight
and analyst-friendly:

-   No native support for **data cube slicing** (e.g., filtering all
    observations for a specific dimension level)
-   No column-wise binding (e.g., `bind_cols()`) with semantic integrity checks
-   No built-in **validation against controlled vocabularies** or semantic
    registries
-   Limited UI or interactive support for defining or editing metadata.

One key limitation is the lack of experience with `dataset` in large-scale,
multi-institutional ingestion, exchange, or publication workflows. For example,
it remains unclear whether column-wise binding is necessary in practice, given
that many users will serialize data to RDF triples — where redundancy is
automatically filtered out by triple stores.

Some features could be better developed as stand-alone packages.

-   The
    [`bibrecord()`](https://dataset.dataobservatory.eu/articles/bibrecord.html)
    s3 class with its constructor was created out of necessity, because the
    `utils::bibentry` class and the `utils::person()` do not handle well modern
    library and repository metadata. Most of the work carried out with the
    `bibentry` class to use the `dublincore()` and `datacite()` constructors
    could be easily adopted in the utils package of R, because it does not raise
    backward compatibility problems.

-   The
    [`provenance()`](https://dataset.dataobservatory.eu/reference/provenance.html)
    function could safely be developed into a package of its own, because there
    are countless ways to improve the granularity a dataset provenance
    description.

Several downstream features and companion packages are under development:

-   **`wbdataset`**: export to the Wikibase data model for collaborative
    metadata curation
-   **Observation status flags**: support for tagging cells as "estimated",
    "provisional", or "forecasted"
-   **Validation helpers**: checks for missing concept URIs, unit mismatches, or
    inconsistent namespaces
-   **External thesauri support**: integration with vocabularies such as EuroVoc
    or GEMET
-   **FAIR alignment**: better coverage of 5-star and 8-star FAIR metadata
    criteria

We expect the need for tailored adaptations in specific domains — including
environmental statistics, cultural heritage, and social sciences — where
existing metadata models often deviate from general-purpose ontologies.

The `dataset` package does not aim to replace enterprise-scale metadata
infrastructure (e.g., SDMX registries), but rather to empower individual
researchers and small teams to produce semantically valid, publication-ready
datasets without high setup costs.

## Limitations and Future Work

The `dataset` package prioritizes ease of use and integration with existing
tidyverse workflows. It implements a practical subset of features inspired by
more formal metadata and ontology systems used in statistical domains, such as
SDMX, DDI, and DCAT.

Several features have been deliberately left out to keep the package lightweight
and analyst-friendly:

-   No native support for **data cube slicing** (e.g., filtering all
    observations for a specific dimension level)
-   No column-wise binding (e.g., `bind_cols()`) with semantic integrity checks
-   No built-in validation against **controlled vocabularies** or semantic
    registries
-   Limited UI or interactive tooling for defining or editing metadata

A key limitation is the limited experience with `dataset` in large-scale,
multi-institutional ingestion, exchange, or publication workflows. For example,
it is still unclear whether column-wise semantic binding is necessary in
practice — given that many users export to RDF triples, where redundancy is
naturally eliminated by triple stores.

Some internal components could be better developed as stand-alone packages:

-   The
    [`bibrecord()`](https://dataset.dataobservatory.eu/articles/bibrecord.html)
    S3 class was introduced out of necessity. Base R’s `utils::bibentry` and
    `utils::person()` do not adequately support modern library and repository
    metadata. Much of the work done in the `dublincore()` and `datacite()`
    constructors could be ported upstream to the `utils` package without
    introducing backward compatibility issues.

-   The
    [`provenance()`](https://dataset.dataobservatory.eu/reference/provenance.html)
    function could reasonably be split into a separate package, as there are
    many opportunities to increase the granularity and expressiveness of dataset
    provenance descriptions.

Several downstream features and companion packages are under development:

-   **`wbdataset`**: export to the Wikibase data model for collaborative
    metadata curation
-   **Observation status flags**: tagging of individual cells as "estimated",
    "provisional", or "forecasted"
-   **Validation helpers**: checks for missing concept URIs, unit mismatches, or
    inconsistent namespaces
-   **External thesauri support**: integration with vocabularies such as EuroVoc
    or GEMET
-   **FAIR alignment**: support for 5-star and 8-star FAIR metadata compliance

We anticipate the need for tailored extensions in domain-specific contexts —
including environmental statistics, cultural heritage, and social sciences —
where metadata conventions often deviate from general-purpose ontologies.

The `dataset` package is not intended to replace enterprise-scale metadata
infrastructure (e.g., SDMX registries), but rather to empower individual
researchers and small teams to produce semantically valid, publication-ready
datasets with minimal overhead.
