> ## Documentation Index
> Fetch the complete documentation index at: https://docs.dataerai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Zenodo bulk imports

> Run capped, resumable Zenodo imports from an operator shell and monitor which records have already landed in a project.

Platform operators can import large Zenodo record sets into a project with a
resumable local runner. Use this workflow when the normal **Import from Zenodo**
screen is too small for the number of records you need to stage.

The import creates metadata-first assets. Dataerai stores the Zenodo metadata
and download links, but it does not copy Zenodo files into the project's
allocation.

<Note>
  Most operators should use the [Zenodo managed import admin
  panel](/data/zenodo-managed-import-admin) instead. It runs and schedules the
  same harvest from the web app — no shell access required — and is the
  recommended path. The resumable local runner described below is an advanced
  option for operators who need to drive a capped, restartable harvest from an
  operator shell.
</Note>

<Note>
  Start every new harvest with a capped scale. Move to a larger scale only after
  the staging, runner, and status output look correct for the target project.
</Note>

## Choose a cap

Each staging or runner command accepts a named scale:

| Scale    | Maximum records | Use it for                                 |
| -------- | --------------- | ------------------------------------------ |
| `smoke`  | 10              | Credential, permission, and resume checks. |
| `small`  | 100             | First operator run against a real project. |
| `medium` | 1,000           | A larger sample before a long import.      |
| `large`  | 10,000          | A controlled production-sized slice.       |
| `full`   | Uncapped        | A deliberate full harvest.                 |

You can also pass `--max-records <count>` to set an exact cap. Use
`--max-records 0` only when you intend to run uncapped.

## Stage records from Zenodo

Run OAI-PMH discovery to create a durable batch of Zenodo record IDs:

```bash theme={null}
python manage.py stage_zenodo_oai_import \
  --name "Zenodo smoke import" \
  --project-id <project-uuid> \
  --creator-email <operator-or-owner@example.org> \
  --scale smoke
```

The command logs the batch ID, pages read, records discovered, records staged,
duplicates skipped, existing project assets skipped, and whether the cap was
reached.

Use optional date filters when you only want a future window:

```bash theme={null}
python manage.py stage_zenodo_oai_import \
  --project-id <project-uuid> \
  --creator-email <operator-or-owner@example.org> \
  --from-date 2026-06-01 \
  --until-date 2026-06-30 \
  --scale small
```

By default, staging skips Zenodo records that the target project already has.
Pass `--include-existing-assets` only when you intentionally want to revisit
records that are already materialized in the project.

## Resume discovery

If discovery stops before it finishes, resume it with the saved batch ID:

```bash theme={null}
python manage.py stage_zenodo_oai_import \
  --resume-batch-id <batch-uuid>
```

The command continues from the saved OAI resumption token and reconciles the
saved counter with the batch's durable record rows before it fetches the next
page. That protects the cap if the previous process stopped after writing rows
but before saving its final progress state.

## Run the import

Drain staged records with the local runner:

```bash theme={null}
python manage.py run_zenodo_bulk_import_batch \
  --batch-id <batch-uuid> \
  --scale smoke \
  --progress-every 10
```

Use `--rate-limit-per-minute <count>` to override the batch's stored rate limit
for one run. Use `--max-records <count>` when you want an exact per-run limit
instead of a named scale.

If a process stopped while records were marked running, reclaim stale work:

```bash theme={null}
python manage.py run_zenodo_bulk_import_batch \
  --batch-id <batch-uuid> \
  --reset-stale-running-minutes 60 \
  --retry-failed \
  --progress-every 25
```

Use `--retry-failed` only when you are ready to retry failed records. Successful
records stay complete and are not run again.

## Monitor progress

Check a batch at any time:

```bash theme={null}
python manage.py zenodo_bulk_import_status --batch-id <batch-uuid>
```

For automation or log collection, request JSON:

```bash theme={null}
python manage.py zenodo_bulk_import_status \
  --batch-id <batch-uuid> \
  --json
```

The status output shows:

* Batch status and processed totals.
* Pending, running, stale-running, failed, imported, updated, unchanged, and
  gone counts.
* How many Zenodo assets and distinct Zenodo record IDs already exist in the
  target project.
* Recent failed records with their last error.
* The saved OAI discovery state.

List recent Zenodo batches when you do not have the batch ID:

```bash theme={null}
python manage.py zenodo_bulk_import_status --list-recent --limit 20
```

## Look for new Zenodo records later

To find records that appeared after your last run, stage a new date window with
`--from-date` and keep the default behavior that skips existing project assets.
Run the status command afterward to confirm the batch only staged records that
still need runner attention.

<Warning>
  A full Zenodo harvest can be large and slow. Use capped runs first, keep the
  rate limit conservative, and monitor failures before increasing scale.
</Warning>