External Data and the Transfer Specification

Part of Clinical Data Management and Integrity: External Data and Reconciliation.

About 5 minutes. Free, in your language, no account needed.

Why this matters

A modern trial's data mostly does not come from the case report form. It comes from central laboratories, imaging vendors, electronic outcome systems, wearable devices, interactive response technology and central reading services, each with its own system, format and schedule.

That data is as much the study's data as anything the site enters, and the sponsor is answerable for it. What differs is that nobody at the sponsor watched it being created, which makes the specification of what should arrive the only control that exists.

What the specification must settle

Which fields, named precisely, with their formats and units. A field called "result" with no unit is an invitation to a silent unit mismatch, which is among the hardest defects to detect because every value is plausible.

The identifiers that link a record to a participant and a timepoint. This is where most transfer failures occur. A vendor identifying participants by its own accession number, with a mapping held only by the vendor, creates a dependency nobody can resolve when the mapping is wrong.

The frequency and the mechanism. Weekly, monthly, on completion, by secure transfer to a named location, with a named contact.

What a transfer contains: only new records, or everything to date. Cumulative transfers are safer because they are self-correcting; incremental ones are smaller and lose data permanently if one is missed.

How corrections are conveyed. A vendor that corrects a result must be able to say so in a way the receiving system understands as a correction rather than as a new record. Without this, corrections arrive as duplicates and the database holds both values.

What "no data" looks like. A transfer with no records must be distinguishable from a transfer that did not happen. This single distinction prevents a large class of silent failures.

The reconciliation

The specification says what should arrive. The reconciliation confirms it did.

A reconciliation that compares the received file against itself is not a reconciliation. It must compare against an independent expectation: the visits that occurred, the samples that were drawn, the assessments the schedule required.

The expectation usually comes from the clinical database. Participants who had a visit at which a sample was to be drawn should have a laboratory result; those who do not are a finding, either a missing transfer, a missing sample, or a visit that did not happen as recorded.

Running it in both directions matters. Records received for participants or timepoints where none was expected are as informative as expected records that did not arrive, and they frequently indicate an identifier problem.

Frequency

Reconciliation should run at the frequency of transfer, not at the frequency of convenience.

Monthly transfers reconciled quarterly means a failure can persist for up to three months before detection. Since the data will still exist at the vendor, the loss is rarely permanent, and the harm is the period during which the study was monitoring itself on incomplete data.

In practice

A data manager sets up a central laboratory transfer and writes the specification with the vendor rather than accepting the vendor's standard file layout.

Three things come out of the conversation that the standard layout would not have covered.

The vendor reports results below the limit of quantification as a text string rather than a number, which would break every numeric derivation downstream. Agreed: a separate flag field and a numeric value.

The vendor's correction mechanism resends the whole record with a new version number, and the receiving system was configured to treat each record as new. Agreed: the version field is used to supersede.

The vendor sends no file at all when a period has no results. Agreed: an empty file with a header, so absence is distinguishable from silence.

Three decisions, two hours. Each one, left undecided, produces a defect that would surface months later and be expensive to unpick.

What this looks like when it goes wrong

An imaging vendor transfers tumor assessments monthly. In month seven a configuration change causes one site's records to carry a site identifier the clinical database does not recognize.

The records arrive. They fail to match a participant. They are written to an exception file.

Nobody reads the exception file, because reading it was not assigned to anyone. The transfer itself succeeds every month, the file arrives, the load completes, and every automated status is green.

At the pre-lock reconciliation eleven months later, 48 assessments are missing for that site. They exist at the vendor and are recovered, so no data is permanently lost.

What was lost is eleven months of not knowing. The study's monitoring of its own primary endpoint ran on incomplete data throughout, and an interim review was conducted on it.

The control that would have caught it existed. The exception file was produced faithfully every month and had no owner.

Key takeaways