The import pipeline

An Import (Import) brings external data — Locations, Samples, Lab Reports, Water Levels, Logger Streams — into Erde through a staged, reviewable workflow. You upload files, Erde validates every row, you review and fix problems, and only then do you commit. The commit runs in the background and is reversible: rolling back an import deletes exactly the records it created. This page explains how the pipeline works and how an import moves through its statuses.

The stages

An import follows the same sequence regardless of data type. Each stage is gated by the import's status, and the workflow can move backward (re-validate, edit, reset) at well-defined points. To perform an import step by step, see Import data.

1. Type selection. Each import is scoped to one import type — the kind of data being loaded. Erde lists only the import types whose format is active.
2. Configuration. An import is scoped to a Site (always required) and, for operational data, a Project. The type's format determines how the file is parsed; the source is one or more CSV (.csv, .txt) or Excel (.xlsx, .xls) files.
3. Create. Erde parses the files into staged rows and creates the import in Draft status.
4. Validation. Erde checks every row, marks it Valid or Invalid, and assigns a planned action — Create or Skip. An Update action is reserved for a future import mode. Validation runs in the background, the same way commit does (see Asynchronous commit and validation).
5. Review and edit. On the import's detail page, invalid rows can be fixed in place, rows can be excluded, manual rows added (where the import type allows it), or the data reverted to the original parsed values. Any successful row edit returns the import to Draft, so it must be re-validated after editing.
6. Commit. When the import is Ready to Commit, the commit is confirmed. The request returns immediately and a background worker applies the changes (see Asynchronous commit and validation).
7. Roll back (optional). After a commit, the import can be rolled back to remove everything it created.

Figure: a simplified happy-path overview of the import pipeline; rollback applies only to a committed import — see the status state machine below for the exact transitions.

Asynchronous commit and validation

Validating or committing a large import can touch hundreds of thousands of records — far more than an HTTP request should hold open. So both operations are split into a fast synchronous half and a background half, sharing the same worker, queue, and recovery sweep.

When you validate or commit, the request:

1. Loads the import, checks your permissions, and confirms a processor is registered for the format.
2. Moves the import to Validating (or Committing) status and saves.
3. Places the import on an in-memory work queue.
4. Returns 202 Accepted right away.

A background worker is the sole consumer of that queue. For each import it opens a fresh database scope, restores the identity of the user who started the operation (so audit fields record the real user, not the system account), and runs the format's validation or commit — dispatched by the import's persisted status. On success a validation becomes Ready to Commit, Validation Failed, or Nothing to Commit, and a commit becomes Committed; an unhandled failure becomes Validation Failed or Commit Failed.

The import's detail page replaces the rows grid with a progress notice and polls the import's status every 2 seconds while it is Validating or Committing, then reloads fully once the status changes. When the operation finishes, Erde sends the user a notification — Import "name" validated, or Import "name" completed / failed for a commit — linking back to the import.

info

The queue holds identities only and does not survive a process restart. A startup recovery sweep, described next, is what guarantees no import is left stuck.

Recovery sweep

If the server restarts mid-job, the in-memory queue is gone, but the import row still says Validating or Committing. A recovery sweep resolves these orphans. The worker runs the sweep once at startup, before it drains the queue, and again on a timer every 5 minutes.

The sweep applies a 5-minute grace window: it only acts on imports whose last-modified time is older than five minutes, so an import set to a transient status moments ago is never mistaken for stuck. It also excludes the import currently being processed and any import still waiting in the queue, so a legitimately long job — for example, committing millions of Logger Readings — is never cut short. Stale imports are transitioned and attributed to the system account:

Stuck status	Recovery transition	Message
Validating	Validation Failed	Server restarted during validation. Please re-validate.
Committing	Commit Failed	Server restarted during commit. Please re-validate and retry.
Rolling Back	Rollback Failed	Server restarted during rollback. Manual intervention may be required.

Rollback, by contrast, runs synchronously inside the HTTP request — it is a fast delete-by-import, not per-row processing, so it does not risk a timeout.

The status lifecycle

Every import carries an import status. There are eleven values; the status drives which actions are available and how the status badge reads.

Figure: the import status state machine. Editing a row at any editable status also returns the import to Draft.

Status	Meaning
Draft	Created or edited; not yet validated
Validating	Validation in progress
Validation Failed	One or more rows are invalid; fix and re-validate
Nothing to Commit	Validation passed but every row would be skipped or is excluded
Ready to Commit	Validation passed with rows to create
Committing	Commit accepted; the background worker is applying it
Committed	Commit succeeded
Commit Failed	Commit failed; edit and re-validate to retry
Rolling Back	Rollback in progress
Rolled Back	Rollback succeeded; the import can be reset to Draft or deleted
Rollback Failed	Rollback failed; dismiss the error to return to Committed

A few rules tie the diagram together:

• Validate is allowed from Draft, Validation Failed, Nothing to Commit, and Commit Failed — and only when the import has rows.
• Commit is allowed only from Ready to Commit.
• A Commit Failed import is editable again: fix rows, re-validate, and retry.
• Dismiss returns a Rollback Failed import to Committed. This is safe because a failed rollback runs in a single transaction and therefore deleted nothing.
• Reset returns a Rolled Back import to Draft, clearing commit and rollback metadata while keeping any excluded rows excluded.

Provenance and rollback

Every record an import creates carries the import's identifier — its ImportId. Records the import merely reuses or appends to are left unstamped and are never touched by a rollback.

This stamp is what makes rollback exact and reversible. A rollback deletes every record whose ImportId matches the import, in child-before-parent order, inside a single transaction. Because the whole rollback is atomic, a rollback that fails has deleted nothing — which is why you can dismiss a Rollback Failed import straight back to Committed and try again.

The same stamping keeps a failed commit clean. For formats that commit atomically, a Commit Failed import created nothing — the transaction rolled back. For continuous-data formats that commit in bulk (such as logger telemetry), re-committing first purges any records this import already stamped, so retrying converges on a correct result. Import records themselves are excluded from the audit log; the rollback deletes are deliberately raw and audit-free.

How it fits together

The pipeline is one shared workflow over many data types. The data you can import — and the constraints on each — come from the import type you choose and the rules its processor enforces.

Aspect	Where it is defined	Notes
What you can import	The import type and its format	Site data such as Locations and field artifacts; operational data such as Samples, Lab Reports, Water Levels, Well Purges, Field Parameters, and Loggers
Whether a Project is required	The import type	Operational data requires a Project; site infrastructure does not
Whether a record is touched by rollback	The ImportId stamp	Only import-created records are stamped and deletable
Whether an import is locked out of editing	The owning Site or Project lock	Imports follow the same aggregate locking and auditing as the data they write — see Aggregates, locking, and auditing

Related

• Importing data
• Field Monitoring
• Aggregates, locking, and auditing
• Glossary