Psych-DS metadata

With metadata on, DataPipe writes a dataset_description.json alongside your data describing the dataset and every variable in it, in the Psych-DS format, and rewrites it after each session. Turn it on from your experiment dashboard when you plan to share or publish your data: it makes the dataset easier for someone else to read and reuse.

Turning it on also changes where your files land. Instead of one file per session at the top of your Drive folder, Dataverse dataset, or Zenodo deposition, each session produces:

• data/raw/<your filename>

  The submission exactly as your experiment sent it, byte for byte. This is the file that matters; everything else is derived from it.

• data/<name>_data.csv

  The session's main data table. A CSV submission keeps its original bytes here; a JSON submission is written out as a table.

• data/<name>_measure-<column>_data.csv

  One sidecar table per column that held nested objects or arrays — survey responses, mouse-tracking samples and the like — so the main table stays flat.

• dataset_description.json

  At the top level: the description of the dataset and every variable in it, rewritten after each session.

• .psychds-ignore

  At the top level: tells Psych-DS validators to skip data/raw/, which holds your originals rather than Psych-DS tables.

With metadata off, none of this happens: each submission is stored at the top level under the filename you sent, and no other files are created.

Zenodo cannot store folders, so a Zenodo record shows these paths flattened into single names. Filenames, archives and your storage

You can turn this setting on or off freely until the first submission arrives. After that it is locked for the life of the experiment, and the switch on your dashboard becomes read-only.

The reason is that the setting decides where your files are stored — at the top of your dataset with it off, under data/raw/ with it on. Changing it partway through would leave the sessions you already collected in one place and every session after it in another, and DataPipe's duplicate detection would no longer recognise the earlier files — so a participant who resubmitted a filename from before the change would get a second copy rather than being caught as a repeat.

If you need to change it after collecting data, create a new experiment with the setting you want.

You will not get subfolders inside your dataset, whatever filenames you send. If a filename contains a folder — say condition-A/abc.json — DataPipe replaces the slash with a hyphen and appends a short code before building any path, so the file is stored as data/raw/condition-A-abc~a145753b.json and its derived tables follow the same name.

This is deliberate: a Psych-DS dataset keeps a flat data/ folder, and keeping the prefix rather than dropping it stops two participants' files from colliding when they share a leaf name in different folders.

The code after ~ is derived from the name you sent, so it is stable — resend the same filename and you get the same stored name. It exists because the hyphen alone is ambiguous: condition-A/abc.json and condition-A-abc.json are different submissions, and without the code they would flatten to the same path and the second one would be rejected as a duplicate. Filenames with no slash in them are stored exactly as you sent them and never pick up a code.

For each variable, DataPipe records its data type and, when available, a human-readable description from the relevant jsPsych plugin documentation.

“When available” is doing real work in that sentence. DataPipe does not ship a table of descriptions. At the moment a session arrives, it fetches the source of the jsPsych plugin that produced each variable from unpkg.com and reads the descriptions out of that source's documentation comments. So descriptions exist only for plugins and extensions published to npm under the official jsPsych names.

Two consequences follow, and neither announces itself:

• Custom, private, renamed and unpublished plugins have no source to fetch. Their variables are described as "unknown". Your data is stored normally and the variables still appear in the metadata — only the prose descriptions are missing.
• An unpkg outage does the same thing, for that session only. The fetch is made while the submission is being handled; if it fails, that session's new variables come out as "unknown". DataPipe does not go back later to fill them in, and a variable already described keeps its description.

dataset_description.json is an ordinary file in your own storage, so you can write the missing descriptions in yourself. Do it after collection ends: DataPipe rewrites that file from its own copy of the metadata after every session, so an edit made mid-study is overwritten by the next participant.

DataPipe also combines information across sessions, such as observed numeric ranges and categorical values. The description file is never rebuilt from just the newest session — each one is merged into what is already there:

• new values seen for a categorical variable are added to its list of levels;
• a numeric variable's minimum is lowered and its maximum raised to cover the new session;
• variables that appear for the first time are appended, so a condition that only some participants see is still described;
• variables already described are kept — nothing is dropped because a later session did not contain it.

Four jsPsych bookkeeping variables — trial_type, trial_index, time_elapsed and internal_node_id — are written once and then left alone, since their meaning does not change from session to session.

Metadata is a description of your data, and DataPipe treats it that way. The derived tables and .psychds-ignore are uploaded after your raw file has landed, and if one of them fails it is queued and retried on its own while the submission still succeeds. The metadataMessage field on every data response reports what happened; it never decides whether a submission is accepted.

There is one case where metadata does affect the response. If DataPipe cannot produce metadata from a submission at all, that request comes back as a 400 with METADATA_ERROR. The usual cause is a submission that parses as JSON but is not an array of trials — metadata needs the trial array jsPsych produces. Anything that does not parse as JSON is treated as CSV instead.

Even then the data itself is not lost. DataPipe keeps the copy it took when the submission arrived, and a scheduled sweep picks that copy up once it is more than fifteen minutes old and sends it to your storage provider. A session recovered this way arrives as the raw file only — no derived tables are generated for it — so it is worth fixing the shape of your data rather than relying on the sweep.

What DataPipe does with a submission it has accepted but not yet delivered. When an upload fails