> ## Documentation Index
> Fetch the complete documentation index at: https://docs.dataerai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Add a format

> Extend dataerai.metaextract with an extractor, converter, data type, and visualization for a new instrument file format.

Adding a format is **one module + one test + one registry row**. See
[`sdk/python/dataerai/metaextract/CONTRIBUTING_EXTRACTORS.md`](https://gitlab.com/dataerai/datatransfer-dataerai/-/blob/beta/sdk/python/dataerai/metaextract/CONTRIBUTING_EXTRACTORS.md)
for the full contract.

| Piece         | Base class / hook                         | Purpose                       |
| ------------- | ----------------------------------------- | ----------------------------- |
| Extractor     | `MetadataExtractor` (or a `bases.*` base) | metadata → dict               |
| Unit tagging  | `units.tag(value, "unit")`                | make a value range-searchable |
| Registry row  | `FormatSpec` in `registry.py`             | wire extension → extractor    |
| Converter     | `Converter` subclass                      | file → `xarray.Dataset`       |
| Data type     | `DataType` + checks                       | classify metadata             |
| Visualization | `register_visualization()`                | plot a data type              |

Source lives under [`sdk/python/dataerai/metaextract/`](https://gitlab.com/dataerai/datatransfer-dataerai/-/tree/beta/sdk/python/dataerai/metaextract) in the monorepo.

## 1. Write an extractor

Subclass `MetadataExtractor` and implement `_do_extract()`, or reuse a structural
base from [`bases.py`](https://gitlab.com/dataerai/datatransfer-dataerai/-/blob/beta/sdk/python/dataerai/metaextract/bases.py) —
`Hdf5Extractor`, `XmlMetadataExtractor`, or `TextHeaderExtractor`. Place it under
the matching domain package (`containers/`, `microscopy/`, `medical/`, `bio/`,
`compchem/`, …, or an `instruments/<Category>/<vendor>/`). The base class filters
out blank-string values automatically.

Read **headers/attributes only** — never load full arrays — and **unit-tag every
physical quantity** with `units.tag()` so it is searchable in the units engine:

```python theme={null}
from ..core import MetadataExtractor
from ..units import tag


class MyFileExtractor(MetadataExtractor):
    """Extractor for .myformat files."""

    def _do_extract(self) -> dict:
        with open(self.file_name, "rb") as f:
            metadata = {}
            # metadata["instrument"] = read_instrument(f)
            metadata["exposure_time"] = tag(0.25, "s")    # -> {"value": 0.25, "unit": "s"}
            metadata["pixel_size"] = tag(65.0, "nm")
            return metadata
```

`tag()` returns the `{"value", "unit"}` override the console's
[unit-aware search](/data/metadata-extraction) consumes when the unit maps to the
catalog, or the bare value otherwise — so it is always safe to call.

Calling `.extract()` (not `_do_extract()`) runs the extraction and applies filtering:

```python theme={null}
from dataerai.metaextract.instruments.AFM.oxfordAFM.ibw import IBW

metadata = IBW("data.ibw").extract()
```

## 2. (Optional) write a converter

Subclass `Converter` (from [`core.py`](https://gitlab.com/dataerai/datatransfer-dataerai/-/blob/beta/sdk/python/dataerai/metaextract/core.py)) and implement `_do_convert()`, which returns an `xarray.Dataset`. The public `convert(output_path=...)` wrapper runs it and optionally writes netCDF. Converters typically reuse the matching extractor for the metadata and add the array data on top. Put it under `converters/<Category>/<vendor>/<format>.py`. Only formats with a converter row appear with `convert: true` in `get_supported_formats()`.

## 3. Register the format

Add one `FormatSpec` row to the declarative `SPECS` table in
[`registry.py`](https://gitlab.com/dataerai/datatransfer-dataerai/-/blob/beta/sdk/python/dataerai/metaextract/registry.py).
`extract_metadata()` / `convert_file()` / `get_supported_formats()` pick it up by
extension automatically — no edits to `api.py`.

```python theme={null}
FormatSpec(
    extensions=(".myext",),
    extractor="dataerai.metaextract.containers.myformat:MyFileExtractor",
    converter=None,                  # or "module:Class"
    aliases=("myformat",),           # for the format= keyword
    label="My instrument format",
    domain="containers",
),
```

The registry resolves the class **lazily**: if your extractor's heavy parser
library isn't installed, the row resolves to `None` and the format is simply
absent in that environment — so import your parser at the top of the module (let
it raise `ImportError`) rather than guarding it yourself. Pure-stdlib/numpy
extractors always register. Don't claim a generic extension (`.xml`, `.out`,
`.raw`) that would hijack unrelated files — add a content sniff and lean on the
`format=` alias instead.

## 4. (Optional) add a data type + visualization

Define a [`DataType`](/metaextract/data-types) so the format is classified, and register a visualizer so [`visualize_data()`](/metaextract/visualization) can route to it. `register_visualization()` is a plain function — pass the data-type name and your plotting function (it is **not** a decorator):

```python theme={null}
from dataerai.metaextract.viz.router import register_visualization


def visualize_my_type(dataset, data_var="data", **kwargs):
    # build and return a Plotly Figure
    ...


# The name must match the DataType you defined above
register_visualization("My Data Type", visualize_my_type)
```

`visualize_data(dataset)` then detects the data type from the dataset's metadata and dispatches to your function via the visualization registry.

## 5. Test it on a real file

Add a test under `sdk/python/tests/metaextract/`. Test on a **real file** —
generate a genuine, valid file with the format's own writer (or build a minimal
one by hand from the documented layout), then extract and assert known fields,
including at least one unit-tagged value. Skip cleanly when the writer/reader
library is absent. Run the suite with the full extras installed:

```bash theme={null}
pip install -e "sdk/python[metaextract-all]"
pytest sdk/python/tests/metaextract
```

<Tip>
  Once the format is supported in `dataerai.metaextract`, DataErai's server-side [automatic extraction](/data/metadata-extraction) picks it up too — the console calls the exact same library.
</Tip>