Zenodo bulk imports

Platform operators can import large Zenodo record sets into a project with a resumable local runner. Use this workflow when the normal Import from Zenodo screen is too small for the number of records you need to stage. The import creates metadata-first assets. Dataerai stores the Zenodo metadata and download links, but it does not copy Zenodo files into the project’s allocation.

Most operators should use the Zenodo managed import admin panel instead. It runs and schedules the same harvest from the web app — no shell access required — and is the recommended path. The resumable local runner described below is an advanced option for operators who need to drive a capped, restartable harvest from an operator shell.

Start every new harvest with a capped scale. Move to a larger scale only after the staging, runner, and status output look correct for the target project.

Choose a cap

Each staging or runner command accepts a named scale:

Scale	Maximum records	Use it for
`smoke`	10	Credential, permission, and resume checks.
`small`	100	First operator run against a real project.
`medium`	1,000	A larger sample before a long import.
`large`	10,000	A controlled production-sized slice.
`full`	Uncapped	A deliberate full harvest.

You can also pass --max-records <count> to set an exact cap. Use --max-records 0 only when you intend to run uncapped.

Stage records from Zenodo

Run OAI-PMH discovery to create a durable batch of Zenodo record IDs:

python manage.py stage_zenodo_oai_import \
  --name "Zenodo smoke import" \
  --project-id <project-uuid> \
  --creator-email <operator-or-owner@example.org> \
  --scale smoke

The command logs the batch ID, pages read, records discovered, records staged, duplicates skipped, existing project assets skipped, and whether the cap was reached. Use optional date filters when you only want a future window:

python manage.py stage_zenodo_oai_import \
  --project-id <project-uuid> \
  --creator-email <operator-or-owner@example.org> \
  --from-date 2026-06-01 \
  --until-date 2026-06-30 \
  --scale small

By default, staging skips Zenodo records that the target project already has. Pass --include-existing-assets only when you intentionally want to revisit records that are already materialized in the project.

Resume discovery

If discovery stops before it finishes, resume it with the saved batch ID:

python manage.py stage_zenodo_oai_import \
  --resume-batch-id <batch-uuid>

The command continues from the saved OAI resumption token and reconciles the saved counter with the batch’s durable record rows before it fetches the next page. That protects the cap if the previous process stopped after writing rows but before saving its final progress state.

Run the import

Drain staged records with the local runner:

python manage.py run_zenodo_bulk_import_batch \
  --batch-id <batch-uuid> \
  --scale smoke \
  --progress-every 10

Use --rate-limit-per-minute <count> to override the batch’s stored rate limit for one run. Use --max-records <count> when you want an exact per-run limit instead of a named scale. If a process stopped while records were marked running, reclaim stale work:

python manage.py run_zenodo_bulk_import_batch \
  --batch-id <batch-uuid> \
  --reset-stale-running-minutes 60 \
  --retry-failed \
  --progress-every 25

Use --retry-failed only when you are ready to retry failed records. Successful records stay complete and are not run again.

Monitor progress

Check a batch at any time:

python manage.py zenodo_bulk_import_status --batch-id <batch-uuid>

For automation or log collection, request JSON:

python manage.py zenodo_bulk_import_status \
  --batch-id <batch-uuid> \
  --json

The status output shows:

Batch status and processed totals.
Pending, running, stale-running, failed, imported, updated, unchanged, and gone counts.
How many Zenodo assets and distinct Zenodo record IDs already exist in the target project.
Recent failed records with their last error.
The saved OAI discovery state.

List recent Zenodo batches when you do not have the batch ID:

python manage.py zenodo_bulk_import_status --list-recent --limit 20

Look for new Zenodo records later

To find records that appeared after your last run, stage a new date window with --from-date and keep the default behavior that skips existing project assets. Run the status command afterward to confirm the batch only staged records that still need runner attention.

A full Zenodo harvest can be large and slow. Use capped runs first, keep the rate limit conservative, and monitor failures before increasing scale.

⌘I

Get started

Get data in

Organize & describe

Verifiable identity

Datasets & analysis

Discover

AI & LLM search

Sharing & access

Account & organization

Attribution

Help & reference

Zenodo bulk imports

Choose a cap

Stage records from Zenodo

Resume discovery

Run the import

Monitor progress

Look for new Zenodo records later

​Choose a cap

​Stage records from Zenodo

​Resume discovery

​Run the import

​Monitor progress

​Look for new Zenodo records later

Choose a cap

Stage records from Zenodo

Resume discovery

Run the import

Monitor progress

Look for new Zenodo records later