When an upload fails

A submission that DataPipe has accepted but not yet written to your storage provider gets HTTP 202 instead of 201. The body carries error: null.

Treat a 202 as success and do not resubmit. DataPipe already has the data and the retry is scheduled; sending it again would store the participant's data twice, under two filenames, or fail the second time on a duplicate name. If your experiment code retries on non-201 responses, change it to treat any 2xx as success.

There are four situations that produce a 202 on /api/data:

• The provider write failed. The most common case: an outage, a rate limit, an expired credential, or a Zenodo record that is already full.
• DataPipe could not check the filename. Its filename record — the list of names this experiment has already used — could not be rebuilt, usually because your Drive folder, Dataverse dataset, or Zenodo deposition is missing or access was revoked. Rather than risk overwriting existing data, the submission is queued.
• That filename record was being rebuilt. Another request was already rebuilding it, which takes at most 60 seconds.
• An archive merge was in progress. A merge is rearranging your Zenodo record, so writes are queued rather than rejected. These entries carry the code CONTENTION and drain in about a minute.

A queued upload still counts toward your session limit — it is a real session, just not stored yet.

A background worker wakes up every five minutes and takes up to 25 queued uploads whose next attempt is due, oldest first. It interleaves them across accounts, one per account per pass, so one researcher's broken connection cannot hold up everybody else's data.

Each queued upload gets five attempts. The wait doubles after each failure, starting at one hour:

So an upload that keeps failing runs out of attempts about 31 hours after it was queued. The doubling is capped at 24 hours, and if your provider sends a Retry-After header DataPipe honors it instead, clamped to the same 24 hours.

Running out of retries is not the same as losing the file. DataPipe keeps the file for seven days from the moment it was queued — or up to fourteen if it could not deliver the failure notification about it — so a permanently failed upload stays downloadable from your dashboard for roughly five and a half more days after the last attempt, longer still if that notification is undelivered.

Failures that are handled faster

Two kinds of failure clear far sooner than an outage does, so they get their own schedules.

• Write contention (CONTENTION) — another write to the same folder, dataset or deposition is already in flight, which clears in seconds. The first attempt is 60 seconds out and the waits are 2, 4, 8, 16 and 30 minutes, so all five attempts happen inside about 31 minutes of queueing.
• Expired credentials and provider blips (AUTH_EXPIRED, UNAVAILABLE) — these cannot be told apart from something that has already healed itself, so DataPipe takes one look after 60 seconds instead of waiting an hour. If that look fails it goes back to the hourly chain above, giving five attempts across about 30 hours from queueing.

Because the worker runs every five minutes, a 60-second wait means the attempt lands at the next five-minute tick.

One case costs nothing at all: if an archive merge is running on your Zenodo record when the retry comes due, the upload is rescheduled 60 seconds later without using up an attempt.

Uploads that never reported back

If a request is cut off before it can finish — a server restart, a memory limit on a very large submission — the participant's data has already been written to DataPipe's storage, so it is still there. A recovery sweep runs every fifteen minutes, picks up anything that has been sitting for more than fifteen minutes, and moves it into the upload queue, where it appears in your dashboard and retries like any other queued file.

One limitation is worth knowing: a recovered session gets its raw data file, but Psych-DS metadata and derived tables are not regenerated for it. The raw file is the source of truth, and the next live submission re-merges the dataset description.

Whenever an experiment has uploads waiting, its dashboard shows a status line reading “N uploads waiting to be stored” — as a warning while every one of them is still retrying, and as an error as soon as any one of them has used up its attempts. The panel below it lists the files.

Each row tells you:

• Filename — the name the submission would be stored under.
• Status — waiting, being uploaded right now, or failed, plus when the next retry is due.
• Reason — a plain-language description of what went wrong.
• Stored for — how much of the seven days since queueing is left (up to fourteen if a failure notification about the file could not be delivered).

The panel also has a “Why did these uploads fail?” section listing the common causes, and the download controls described below.

You never have to wait for a retry to succeed to get your data. Every row in the queued files panel has a download button that gives you that one file, decrypted, under the filename it was submitted with. Text files come back as CSV or JSON; base64 submissions are decoded back to the original binary.

Download all as ZIP collects every queued, in-flight and failed file for the experiment into a single archive named after the experiment ID. Use it as soon as any file shows as failed: it costs nothing, and it means you hold a copy regardless of what the retries do.

If a download fails, nothing has been lost — DataPipe still holds the file for the rest of its retention window (seven days, or up to fourteen if a failure notification about it could not be delivered). Try the single-file buttons if the ZIP will not build.

The endpoint behind these buttons is documented, if you want to script it. Queue status

After the fifth failed attempt the upload is marked failed and is not tried again. Download it from the dashboard and add it to your Drive folder, Dataverse dataset, or Zenodo deposition by hand. It is worth doing promptly: the file is deleted seven days after it was queued — or up to fourteen if DataPipe could not deliver the failure notification about it — failed or not.

Some failures are permanent immediately, because retrying could never help. DataPipe marks these failed on the first pass rather than spending five attempts on them:

• The experiment was finalized while the upload was queued. Finalizing seals your Zenodo record permanently, so the file cannot be added to it. The data is not lost — download it from the queued files panel.
• The experiment or the owning account no longer exists.
• The stored copy could not be read. Rare, and it still appears in the panel with a reason rather than disappearing silently.

If uploads are failing on every submission rather than occasionally, the cause is usually the connection to your storage provider rather than the provider being down.

Expired credentials, and how to restore them, are covered here. When a token expires