Metadata extraction

dataerai.metaextract reads 44 scientific data formats across 76 file extensions — from array containers (HDF5, NetCDF, Zarr) and imaging (TIFF, FITS, DICOM) to genomics, mass spec and instrument files — and returns plain, JSON-ready metadata dictionaries with physical quantities tagged for unit-aware search. It can also convert supported files to xarray/netCDF, classify the data type, compute checksums, and render interactive Plotly visualizations. It ships inside the dataerai-sdk package — the same library DataErai runs server-side for automatic metadata extraction on upload.

Install

The scientific stack is optional, so a bare pip install dataerai-sdk stays light. Pull in only the capability you need:

pip install "dataerai-sdk[metaextract]"             # core extractors
pip install "dataerai-sdk[metaextract-converters]"  # + xarray / netCDF conversion
pip install "dataerai-sdk[metaextract-viz]"         # + Plotly visualizations
pip install "dataerai-sdk[metaextract-checksums]"   # + fast blake3 checksums
pip install "dataerai-sdk[metaextract-utils]"       # + download / benchmark helpers
pip install "dataerai-sdk[metaextract-all]"         # everything above

The [metaextract] extra pulls compiled scientific dependencies (h5py, hyperspy, lxml, igor2). On Linux you may need a C toolchain (e.g. build-essential) for any wheels that aren’t pre-built for your platform.

Supported formats

dataerai.metaextract extracts metadata from 44 scientific data formats across nine domains — array containers (HDF5, NetCDF, Zarr, MATLAB, Parquet), imaging (TIFF/OME, FITS), microscopy (Nikon ND2, Zeiss CZI, Leica LIF, Olympus), medical & neuro (DICOM, NIfTI, EDF, NWB), genomics (FASTA/FASTQ, VCF, SAM/BAM, GFF, FCS), mass spec (mzML, imzML, Thermo RAW), spectroscopy (Bruker OPUS, Renishaw WDF, JCAMP-DX), crystallography (CIF, VASP) and computational chemistry (Gaussian/ORCA, XYZ, MOL/SDF, PDB/mmCIF) — alongside the original AFM/EM/XRD/RHEED instrument set. See the full Supported formats table for extensions and the reader each one needs. Every physical quantity is emitted in the units engine’s {"value", "unit"} shape, so extracted values are immediately searchable in unit-aware metadata search. Probe the live capability matrix at runtime with get_supported_formats() — entries degrade to extract: false when an optional dependency is missing rather than raising on import.

Convert, classify and visualize

Beyond metadata, the package offers three further capabilities:

Convert to xarray/netCDF — the instrument formats .ibw, .xrdml and .imm carry full converters, so convert_file() returns a labelled xarray.Dataset (and can save it as netCDF). Other formats are metadata-only.
Classify the data type — detect_data_type() matches extracted metadata against built-in definitions (for example Tapping Mode AFM, Single Frequency PFM, XRD Rocking Curve) so you can route a file to the right pipeline. See Data-type detection.
Visualize — ready-made Plotly plots ship for AFM, the three PFM variants (single-frequency, vector, DART) and XRD (rocking curve, 2θ–ω scan, reciprocal space map), plus generic metadata explorers. See Visualization.

20-second example

from dataerai.metaextract import extract_metadata, detect_data_type

metadata = extract_metadata("scan.ibw")     # → dict of instrument settings
data_type = detect_data_type(metadata)      # → {'type': 'Tapping Mode AFM', 'confidence': 0.95, ...}

Next steps

Quickstart

Extract, convert, checksum, and batch-process files.

Data-type detection

Classify metadata and define your own data types.

Visualization

Interactive Plotly views of metadata and data.

Add a format

Write an extractor/converter for a new instrument file.

Command-line

SDKs

Integrations