Add a format

Adding a format is one module + one test + one registry row. See sdk/python/dataerai/metaextract/CONTRIBUTING_EXTRACTORS.md for the full contract.

Piece	Base class / hook	Purpose
Extractor	`MetadataExtractor` (or a `bases.*` base)	metadata → dict
Unit tagging	`units.tag(value, "unit")`	make a value range-searchable
Registry row	`FormatSpec` in `registry.py`	wire extension → extractor
Converter	`Converter` subclass	file → `xarray.Dataset`
Data type	`DataType` + checks	classify metadata
Visualization	`register_visualization()`	plot a data type

Source lives under sdk/python/dataerai/metaextract/ in the monorepo.

1. Write an extractor

Subclass MetadataExtractor and implement _do_extract(), or reuse a structural base from bases.py — Hdf5Extractor, XmlMetadataExtractor, or TextHeaderExtractor. Place it under the matching domain package (containers/, microscopy/, medical/, bio/, compchem/, …, or an instruments/<Category>/<vendor>/). The base class filters out blank-string values automatically. Read headers/attributes only — never load full arrays — and unit-tag every physical quantity with units.tag() so it is searchable in the units engine:

from ..core import MetadataExtractor
from ..units import tag


class MyFileExtractor(MetadataExtractor):
    """Extractor for .myformat files."""

    def _do_extract(self) -> dict:
        with open(self.file_name, "rb") as f:
            metadata = {}
            # metadata["instrument"] = read_instrument(f)
            metadata["exposure_time"] = tag(0.25, "s")    # -> {"value": 0.25, "unit": "s"}
            metadata["pixel_size"] = tag(65.0, "nm")
            return metadata

tag() returns the {"value", "unit"} override the console’s unit-aware search consumes when the unit maps to the catalog, or the bare value otherwise — so it is always safe to call. Calling .extract() (not _do_extract()) runs the extraction and applies filtering:

from dataerai.metaextract.instruments.AFM.oxfordAFM.ibw import IBW

metadata = IBW("data.ibw").extract()

2. (Optional) write a converter

Subclass Converter (from core.py) and implement _do_convert(), which returns an xarray.Dataset. The public convert(output_path=...) wrapper runs it and optionally writes netCDF. Converters typically reuse the matching extractor for the metadata and add the array data on top. Put it under converters/<Category>/<vendor>/<format>.py. Only formats with a converter row appear with convert: true in get_supported_formats().

3. Register the format

Add one FormatSpec row to the declarative SPECS table in registry.py. extract_metadata() / convert_file() / get_supported_formats() pick it up by extension automatically — no edits to api.py.

FormatSpec(
    extensions=(".myext",),
    extractor="dataerai.metaextract.containers.myformat:MyFileExtractor",
    converter=None,                  # or "module:Class"
    aliases=("myformat",),           # for the format= keyword
    label="My instrument format",
    domain="containers",
),

The registry resolves the class lazily: if your extractor’s heavy parser library isn’t installed, the row resolves to None and the format is simply absent in that environment — so import your parser at the top of the module (let it raise ImportError) rather than guarding it yourself. Pure-stdlib/numpy extractors always register. Don’t claim a generic extension (.xml, .out, .raw) that would hijack unrelated files — add a content sniff and lean on the format= alias instead.

4. (Optional) add a data type + visualization

Define a DataType so the format is classified, and register a visualizer so visualize_data() can route to it. register_visualization() is a plain function — pass the data-type name and your plotting function (it is not a decorator):

from dataerai.metaextract.viz.router import register_visualization


def visualize_my_type(dataset, data_var="data", **kwargs):
    # build and return a Plotly Figure
    ...


# The name must match the DataType you defined above
register_visualization("My Data Type", visualize_my_type)

visualize_data(dataset) then detects the data type from the dataset’s metadata and dispatches to your function via the visualization registry.

5. Test it on a real file

Add a test under sdk/python/tests/metaextract/. Test on a real file — generate a genuine, valid file with the format’s own writer (or build a minimal one by hand from the documented layout), then extract and assert known fields, including at least one unit-tagged value. Skip cleanly when the writer/reader library is absent. Run the suite with the full extras installed:

pip install -e "sdk/python[metaextract-all]"
pytest sdk/python/tests/metaextract

Once the format is supported in dataerai.metaextract, DataErai’s server-side automatic extraction picks it up too — the console calls the exact same library.

Command-line

SDKs

Integrations

Metadata extraction

1. Write an extractor

2. (Optional) write a converter

3. Register the format

4. (Optional) add a data type + visualization

5. Test it on a real file

​1. Write an extractor

​2. (Optional) write a converter

​3. Register the format

​4. (Optional) add a data type + visualization

​5. Test it on a real file

1. Write an extractor

2. (Optional) write a converter

3. Register the format

4. (Optional) add a data type + visualization

5. Test it on a real file