Filenames, archives and your storage
How DataPipe checks filenames, what happens if you edit your storage during collection, and why archives appear mid-study.
How filenames are checked
Every file DataPipe stores must have a name no other submission has used. That is what stops a participant who submits twice — or a duplicate request from a flaky connection — from overwriting data you have already collected.
The check is DataPipe's, not your storage provider's. Most providers will happily accept the same name twice, each in its own way, so DataPipe keeps its own filename record — the list of names an experiment has used — and consults that first.
The filename record does not contain your filenames. Each name is hashed with a per-experiment secret and only the hash is stored, so the list of names your participants' files were given is not something DataPipe holds. Entries last 90 days from the submission that created them.
What your experiment sees:
- The name is already taken —
400 OSF_FILE_EXISTS. Nothing is stored, and the participant's submission is rejected. Give each submission a name you know is unique, such as one built from a random ID. - The record is being rebuilt right now —
202. Another request is already rebuilding it, which takes at most 60 seconds; the submission is queued and lands shortly after. - The record cannot be rebuilt —
202. DataPipe could not list your Drive folder, Dataverse dataset, or Zenodo deposition, usually because it was deleted or access was revoked. Rather than risk overwriting real data it queues the submission and retries.
A name reserved by a submission that never completed is released fifteen minutes after that submission arrived, so an interrupted request does not block the name forever.
Don't edit your storage during collection
Adding, renaming, or deleting files there yourself while an experiment is collecting can cost you a participant's data. Download from your storage as much as you like; write to it only once collection is finished.
DataPipe keeps its own filename record so a participant who submits twice cannot overwrite existing data. Files that appear without DataPipe writing them are missing from that record, so the next submission that happens to use the same name may be silently renamed, or may overwrite what you added.
What that looks like differs by provider, and none of them is a clean error:
| Provider | What happens to a name DataPipe does not know about |
|---|---|
| Google Drive | Both files are kept. Drive allows duplicate names in a folder and never reports a conflict, so you are left with two files with the same name and no indication which is which. |
| Dataverse | The new file is silently renamed — README.md becomes README-1.md. Nothing is lost, but the stored name is no longer the one your experiment asked for. |
| Zenodo | The existing file is overwritten. A Zenodo write replaces whatever is at that key and reports no conflict, so the file you added by hand is gone. |
| OSF (legacy) | The write is refused with a real name conflict. OSF is the only provider that reports one. |
Files you added by hand do become visible to DataPipe eventually: if the filename record ever has to be rebuilt from your storage, it reads whatever is there and adopts those names as taken. That is not something you can trigger, so it is not a fix — it only means the window in which a hand-added file is invisible is bounded rather than permanent.
What your files are named
Usually the name your experiment sent. Two rules can change it, and both are worth knowing before you go looking for a file.
Slashes in a filename do not create folders. When Psych-DS metadata — the optional description DataPipe writes alongside your data — is on, a name like condition-A/abc.json is flattened with hyphens before the path is built, so it is stored at data/raw/condition-A-abc~a145753b.json. The prefix is kept rather than discarded, so two submissions that share a name after the last slash still do not collide. The short code after ~ is derived from the name you sent and is stable across resubmissions; it keeps condition-A/abc.json distinct from a file literally named condition-A-abc.json. Names without a slash are stored exactly as sent and get no code.
Zenodo cannot store a slash at all. It holds every file under a flat name, with no folders, so every remaining / becomes an underscore: data/raw/subject-1.json is stored as data_raw_subject-1.json. If you open a metadata-enabled Zenodo deposition mid-study and find a flat list of underscore names instead of a data/raw/ tree, this is why — nothing has gone wrong. The real Psych-DS directory structure exists inside the archives described below, where DataPipe controls the paths.
On Google Drive, slashes do create real folders when metadata is off — condition-A/abc.json lands in a condition-A folder. Two submissions that share a name after the last slash but sit in different folders are different files and both are kept.
What the metadata files contain, and how to turn them on. Psych-DS metadata
Archives during collection
If you open a Zenodo deposition partway through a study and find files named datapipe-batch-0001.zip, nothing has gone wrong and nothing has been lost. Your sessions are inside them.
Zenodo allows 100 files per record. A study that writes several files per session would hit that ceiling long before it finished collecting, so once a record reaches 80 of its 100 files DataPipe merges older sessions into a zip and removes the originals. The five most recent sessions are left loose so you can still open and spot-check recent data in Zenodo's own interface. Each archive holds at most 95 files or 150 MB, whichever comes first.
Two files are never archived: dataset_description.json, which describes your dataset and its variables and should stay visible on the record, and .psychds-ignore, which is rewritten on every submission anyway.
Nothing is deleted until the archive is verified. DataPipe uploads the zip, compares the checksum Zenodo reports back against the one it computed, records the names now held inside the archive, and only then deletes the originals. If the checksum cannot be verified, the originals stay where they are.
There is no schedule for this — a merge is triggered by your record growing, not by a timer, so it runs the moment it is needed. While a merge is running, submissions are queued rather than rejected: your participants still get a success response, and the queued files land about a minute later.
The archives are ordinary zip files. Unzip one and you get the folder tree back, complete with the data/raw/ paths that Zenodo itself cannot represent. Zenodo also previews zip contents on the record page, so you can see what is in an archive without downloading it.
At the end of collection every batch is merged into one final archive. Finishing a study
File count limits
On providers with a limit on how many files one record can hold — Zenodo allows 100 — DataPipe combines older sessions into archives to stay under it. Files added by hand count toward that limit, and can fill the record faster than DataPipe expects.
- Zenodo — 100 files and 50 GB per record. This is the only file-count ceiling DataPipe works around, and the reason archives exist.
- Google Drive — no file-count limit that matters here. The real constraint is your account's storage quota, which on a free Google account is 15 GB shared across Drive, Gmail and Photos.
- Dataverse — file size and storage limits are set by the installation hosting your dataset, and are not readable through its API, so DataPipe does not enforce a number of its own.
If a Zenodo record does fill up completely, DataPipe needs one free slot to upload the archive that would relieve it. It takes that slot by temporarily removing .psychds-ignore — a file whose contents never change — and writes it back afterwards. An experiment with metadata off has no such file to give up, so archiving stops and needs you: remove one file from the record yourself and it will proceed on the next submission.