Project

General

Profile

Metadata format

The metadata format for declaring files must be a valid JSON (http://www.json.org) object.

The key values are case insensitive.

Key Data type Required
key
Restricted
value
Description
file_name String x The name of the file
file_type String x x The type of the file
file_size Integer x The size of the file, in bytes
checksum See below A list of checksums for the file (version 2.0 onwards)
crc See below The file checksum (pre version 2.0)
file_format String x The format of the file. Defaults to "unknown" if not provided.
content_status String x The file content status. Defaults to "good" if not provided.
group String x The SAM work group the file is associated with
data_tier String x The data tier.
event_count Integer The number of events in the file.
first_event Integer The first event number in the file.
last_event Integer The last event number in the file.
start_time Date The start time of the file.
end_time Date The end time of the file.
file_partition Integer The file partition number for the file.
application See below The application used to create the file.
process_id Integer The SAM process ID for the job that created this file. (Only meaningful for files which are produced by a SAM processing job.)
data_stream String x The stream of the file.
runs See below The run numbers associated with this file.
lum_block_ranges See below Pairs of luminosity block numbers for this file.
parents See below x The parent files of this file.
params See below Generic metadata parameters
<category>.<name> See below Generic metadata parameters defined by the experiment

Required keys must be present in the metadata passed to the server. Restricted values are confined to predefined values (customizable by each experiment).

Those familiar with the original SAM metadata should note that there are no longer any file type specific restrictions on the required metadata keys.

Generic Parameters

Experiment defined generic parameters can be supplied as either top level keys of the form <category>.<name>, or a top level key of "params" : { "<category>" : { "<name>" : value } }

Checksum format

The checksum value is a list of strings, where each string consists of "<type>:<value>"; for example "checksum": [ "enstore:2627192881" ]. This form is only supported in version 2 of the server (PostgreSQL based).

In version 1, the CRC can be provided as either a pair of [crc_value, crc_type] or an object of { "crc_value" : crc_value, "crc_type" : crc_type }. The crc_value can be an integer or a string. For backwards compatibility, version 2 will also accept this form for input; the only supported crc_type is adler 32 crc type which is translated to enstore.

Application format

The application can be provided as either a triplet of [ family, name, version ], or an object of { "family" : family, "name" : name, "version" : version }.

Runs

The runs are a list of pairs of [ run_number, run_type ] or [ run_number, subrun_number, run_type ]. The run type must be from a list of preexisting types.

Luminosity block ranges

The luminosity block ranges are a list of pairs of [ min, max ] values.

Parents

The parents must be provided as a list of files. Each file can be a string name, an integer file ID, or an object containing either file_name or file_id keys. If both the name and ID are provided the ID takes priority and the name is ignored.

Dates

Dates can be given in the form "19-Sep-2012", "19-Sep-2012 13:51", "19-Sep-2012 13:51:32", "2012-09-19", "2012-09-19T13:51", or "2012-09-19T13:51:32". They are always interpreted as UTC, not the local timezone.

Examples

NOvA raw data:

{
 "file_name": "ndos_r00015095_s29_t05.raw", 
 "file_id": 3981994, 
 "create_date": "2013-09-20T14:47:05+00:00", 
 "file_type": "importedDetector", 
 "file_format": "raw", 
 "file_size": 49365800, 
 "checksum": [
  "enstore:2830241712" 
 ], 
 "content_status": "good", 
 "group": "nova", 
 "data_tier": "raw", 
 "application": {
  "family": "online", 
  "name": "datalogger", 
  "version": "33" 
 }, 
 "event_count": 2195, 
 "first_event": 665181, 
 "last_event": 689075, 
 "start_time": "2013-09-19T13:07:00+00:00", 
 "end_time": "2013-09-19T13:43:36+00:00", 
 "data_stream": "5", 
 "Online.ConfigIDX": 0, 
 "Online.DataLoggerID": 0, 
 "Online.DataLoggerVersion": 33, 
 "Online.Detector": "ndos", 
 "Online.DetectorID": "3", 
 "Online.Partition": "0", 
 "Online.RunControlID": "0", 
 "Online.RunControlVersion": "0", 
 "Online.RunEndTime": 1379598216, 
 "Online.RunNumber": 15095, 
 "Online.RunSize": 12341450, 
 "Online.RunStartTime": 1379535095, 
 "Online.RunType": "0", 
 "Online.Stream": "5", 
 "Online.Subrun": 29, 
 "Online.SubRunEndTime": 1379598216, 
 "Online.SubRunStartTime": 1379596020, 
 "Online.TotalEvents": 2195, 
 "Online.TriggerCtrlID": "0", 
 "Online.TriggerListIDX": 0, 
 "Online.TriggerPrescaleListIDX": 0, 
 "Online.TriggerVersion": "0", 
 "Online.ValidTriggerTypesHigh": "0", 
 "Online.ValidTriggerTypesHigh2": "0", 
 "Online.ValidTriggerTypesLow": "0", 
 "runs": [
  [
   15095, 
   29, 
   "online" 
  ]
 ], 
 "file_partition": 29
}

Examples from D0:

{
 "file_name": "all_1_0000251225_033.raw", 
 "file_type": "importedDetector", 
 "file_format": "unknown", 
 "file_size": 515401544, 
 "crc": [
  "116146095L", 
  "adler 32 crc type" 
 ], 
 "content_status": "good", 
 "group": "online daq", 
 "data_tier": "raw", 
 "application": {
  "family": "datalogger", 
  "name": "datalogger", 
  "version": "0.0" 
 }, 
 "event_count": 1994, 
 "first_event": 2626636, 
 "last_event": 2699060, 
 "start_time": "2009-04-17T01:00:35", 
 "end_time": "2009-04-17T01:03:22", 
 "data_stream": "all_1", 
 "runs": [
  [
   251225, 
   "physics data taking" 
  ]
 ], 
 "file_partition": 33, 
 "lum_block_ranges": [
  [
   6691069, 
   6691069
  ]
 ]
}
{
 "file_name": "recoT_all_1_0000251225_mrg_033-035.raw_p20.12.05c", 
 "file_type": "derivedDetector", 
 "file_format": "evpack", 
 "file_size": 1181116006, 
 "crc": [
  "1772377263L", 
  "adler 32 crc type" 
 ], 
 "content_status": "good", 
 "group": "dzero", 
 "data_tier": "thumbnail", 
 "application": {
  "family": "reconstruction", 
  "name": "d0reco", 
  "version": "p20.12.05c" 
 }, 
 "process_id": 32689578, 
 "event_count": 6087, 
 "first_event": 2626868, 
 "last_event": 2843492, 
 "start_time": "2009-04-24T02:33:35", 
 "end_time": "2009-04-24T02:33:35", 
 "data_stream": "all_1", 
 "global.facilityname": "d0cabosg2.fnal.gov", 
 "global.jimclusterjobid": "23146.1240161886", 
 "global.jimglobaljobid": "diesburg_d0mino01.fnal.gov_122126_6840", 
 "global.jimlocaljobid": "524673", 
 "thumbnail.appfamily": "reconstruction", 
 "thumbnail.appname": "d0reco", 
 "thumbnail.appversion": "p20.12.05c", 
 "thumbnail.frameworkrcpname": "runD0reco_data.rcp", 
 "thumbnail.frameworkrcppackage": "d0reco", 
 "thumbnail.numfiles": "0", 
 "thumbnail.numrecords": "0", 
 "thumbnail.paramcategory": "reconstructed", 
 "thumbnail.processid": "32689578", 
 "thumbnail.skiprecords": "0", 
 "thumbnail.thumbdatatier": "thumbnail", 
 "thumbnail.usemaxopt": "1", 
 "runs": [
  [
   251225, 
   "physics data taking" 
  ]
 ], 
 "lum_block_ranges": [
  [
   6691069, 
   6691071
  ]
 ], 
 "parents": [
  {
   "file_name": "recoT_all_1_0000251225_033.raw_p20.12.05c_diesburg_d0mino01.fnal.gov_122126_6840_d0cabosg2.fnal.gov", 
   "file_id": 46408205
  }, 
  {
   "file_name": "recoT_all_1_0000251225_034.raw_p20.12.05c_diesburg_d0mino01.fnal.gov_122126_6840_d0cabosg2.fnal.gov", 
   "file_id": 46431636
  }, 
  {
   "file_name": "recoT_all_1_0000251225_035.raw_p20.12.05c_diesburg_d0mino01.fnal.gov_122126_6840_d0cabosg2.fnal.gov", 
   "file_id": 46407745
  }
 ]
}