Data-type detection

detect_data_type() matches extracted metadata against configurable lookup tables to identify what kind of data a file holds (for example Tapping Mode AFM vs Contact Mode AFM), so you can route it to the right processing or visualization.

Detect a type

from dataerai.metaextract import extract_metadata, detect_data_type

metadata = extract_metadata("afm_scan.ibw")
result = detect_data_type(metadata)

if result:
    print(result["type"])                  # 'Tapping Mode AFM'
    print(f"{result['confidence']:.0%}")    # '95%'
    print(result["matched_checks"], "/", result["total_checks"])
else:
    print("No matching data type found")

detect_data_type() returns None when nothing matches, otherwise a dict:

{
    "type": "Tapping Mode AFM",
    "confidence": 0.95,
    "matched_checks": 4,
    "total_checks": 5,
    "description": "AFM data in tapping mode…",
    "checks": [
        {"field": "ImagingMode",
         "description": "Imaging mode is AC Mode",
         "matched": True},
        # …
    ],
}

Each entry in checks reports the metadata field it looked at, a human-readable description, and whether it matched. confidence is simply matched_checks / total_checks.

detect_data_type() raises InvalidMetadataError if you pass something other than a dict, and NoDataTypeDefinitionsError if no data-type definitions are available. Both are importable from dataerai.metaextract.

Built-in data types

A fresh DataTypeDetector() (and the detect_data_type() convenience function) ships with these built-in definitions:

AFM/PFM — Tapping Mode AFM, Single Frequency PFM, Vector PFM, DART PFM
XRD — XRD Rocking Curve, XRD 2Theta-Omega Scan, XRD RSM (reciprocal space map)
RHEED — kSA RHEED IMM Movie
Other modalities — Mass Spectrometry Run, NIfTI Neuroimaging Volume, Biological Sequence, Flow Cytometry, FITS Astronomical Data

Call detector.list_data_types() to print the live list in your environment.

Use the detector directly

from dataerai.metaextract.data_type import DataTypeDetector

detector = DataTypeDetector()           # built-in data types
result = detector.detect(metadata)
print(detector.list_data_types())

# Add another definition to an existing detector
detector.add_data_type(custom_type)

Define a custom type

from dataerai.metaextract.data_type import (
    DataType, DataTypeCheck, DataTypeDetector, create_custom_check,
)

custom_check = create_custom_check(
    field="CustomField",
    check_func=lambda x: x == "expected_value",
    description="Checks if CustomField equals expected_value",
)

custom_type = DataType(
    name="My Custom Data Type",
    checks=[
        # value can be an exact match, a list (OR logic), or a callable
        DataTypeCheck(field="Format", value="Custom",
                      description="Format must be Custom"),
        DataTypeCheck(field="MicroscopeModel", value=["Vero", "Cypher"],
                      description="Microscope is Vero or Cypher"),
        custom_check,
    ],
    description="A custom data type definition",
    min_matches=1,  # at least one check must match; defaults to all
)

detector = DataTypeDetector(data_types=[custom_type])
result = detector.detect(metadata)

A DataTypeCheck’s value can be an exact value, a list or tuple (matches if the metadata value is any item — OR logic), or a callable predicate. min_matches defaults to “all checks must match” when omitted. check_func can be any predicate, so checks can express ranges or complex validation:

def in_range(value):
    try:
        return 0 <= float(value) <= 100
    except (ValueError, TypeError):
        return False

range_check = create_custom_check(
    field="ScanSize", check_func=in_range,
    description="Scan size must be between 0 and 100 nm",
)

This is the same classifier DataErai runs server-side to tag uploads with a data_type and confidence score — see Automatic metadata extraction.

Command-line

SDKs

Integrations

Metadata extraction

Data-type detection

Detect a type

Built-in data types

Use the detector directly

Define a custom type

​Detect a type

​Built-in data types

​Use the detector directly

​Define a custom type

Detect a type

Built-in data types

Use the detector directly

Define a custom type