dataplane

Start here

Most users only need one of these: the PDF manual, the GitHub repo, or the install command.

Package manual (PDF) View repository

Install

Install the current version directly from GitHub:

remotes::install_github("jclark50/dataplane")

Quick examples

Copy/paste runnable mini-recipes. These show the typical pattern: spec → write → read + inspect metadata.

Example 1 — Write one Parquet file that carries its own “data dictionary”

library(data.table) library(dataplane) dt <- data.table(temp_c = 25.12, rh_pct = 55) sp <- dp_spec_default(dt, declared_system = "metric") sp <- dp_set_units(sp, "temp_c", metric_units = "degC") sp <- dp_set_units(sp, "rh_pct", metric_units = "pct") dp_write(dt, "example.parquet", spec = sp) res <- dp_read("example.parquet", attach_units = "declared") res$dt[] res$meta$spec_dt[]

Example 2 — Dataset folder workflow (many files + one sidecar metadata file)

library(arrow) library(data.table) library(dataplane) dt <- data.table(date = as.Date("2026-01-01"), temp_c = 25.1, rh_pct = 55) sp <- dp_spec_default(dt, "metric") sp <- dp_set_units(sp, "temp_c", metric_units = "degC") sp <- dp_set_units(sp, "rh_pct", metric_units = "pct") dir.create("dp_ds", showWarnings = FALSE) dp_write_dataset(arrow::Table$create(dt), path = "dp_ds", spec = sp) opened <- dp_open("dp_ds", metadata = "sidecar") opened$meta$spec_dt[] opened$meta$audit_dt[]

Tip: the PDF manual is the most complete reference. The repository README usually contains the most current end-to-end workflow examples.

What’s inside

A small toolkit for making Parquet datasets more reliable and more portable.

I/O helpers

Read/write datasets cleanly

Metadata

Attach + retrieve dataset info

Scaling

Encode/decode scaled fields

Consistency

Schema + unit conventions

Direct links

https://jclark50.github.io/dataplane/dataplane_1.0.0.pdf