hexfarm.rutgers.edu Secondary Datasets

Creating Secondary Datasets

Introduction

There is a CreateDatasetDaemon (CRDSD) on the hexfarm so we can make secondary datasets. If presented with a list of files, the daemon will concatenate or copy the files into a buffer area and then store them in SAM, i.e. send them to Fermilab to be put on tape in Enstore. The daemon also takes care of defining the dataset in SAM. A well-understood ac++ job (no crashes) can write directly into the buffer area.

The files of a dataset created this way are available to any SAM job on any CAF/SAM-Station in CDF, including HEXCAF of course.

Defining a new Rutgers dataset

First choose the dataset ID:

Dataset ID's for Rutgers should be in the sequence z8v20z, z8v21z, z8v22z, ... z8v29z, z8v2az, ... z8v2zz, z8v20y, ... z8v2zy, z8v20x, ... z8v2za, z8v209, ... z8v2z0, y8v20z, ... 08v2z0 (Keep it all lower case.) Choose a new one just after the last used one. Look at the link, on the web page above, which lists active Rutgers datasets in SAM, to see used ID's. Better still, use the script new_dataset.csh to find the next in the sequence, see below. 

To find the next Rutgers dataset ID, run script new_dataset.csh:

  hexfm1>cd /cdf/data00/DATA_INFO/RU
  hexfm1>./new_dataset.csh
   /cdf/data00/DATA_INFO/RU
   Count of 8v2 datasets =  67 Potential IDs used =  70
   Next dataset ID = z8v2zy

Make your dataset's control area

To add a new secondary dataset, e.g. z8v2zy,
   hexfm1> cd /cdf/data00/DATA_INFO/RU  # dataset control area
   hexfm1> mkdir _z8v2zy         # your dset's control area
   hexfm1> chmod g+w _z8v2zy     # give access to the daemon
   hexfm1> cd _z8v2zy
   hexfm1> touch .ignore         #suspend daemon action until ready

Describe your dataset

Put .name, .parent, and desc.txt files in your dataset control area to describe the dataset.

Examples are "di-tau_sumr03_topremake.name" and "eexo18.parent" in z8v2az. The dataset name is a short description and the parent is the dataset used as input when creating the dataset. Put "mcmcmc.parent" for a Monte Carlo dataset.

File desc.txt should contain a long description of the fileset. This file may be modified after defining the dataset because it is available to read on the web. Thus you should update desc.txt if more is learned about the dataset.

Your dataset control area can be seen on the web at URL http://hexfarm.rutgers.edu/DATA_INFO/RU/_z8v2az

If you remove the .ignore file, the daemon will enter the definition in Sam, see list of Rutgers datasets. But you may want to wait until you specify the files to be copied into the dataset, see next section.

Putting files in your dataset

Create a file "concat.input" in your dataset's control area with a list of the files to put into your dataset. Make sure there is a .ignore or concat.ignore file in the control area before modifying concat.input. See below for use of the .ignore files.

The format of concat.input is 1 file per line but the usual "ls" wild card characters are allowed. Each line must be an explicit file path or root daemon file path, e.g. 

/scr8/OTHER/amit/remake/summer03/eexo18/eexo18*.root*
and/or
root://rutpc7.fnal.gov:1095//cdf/rutgers/data/amitl/data/kshort/v0strip1.dst*

Standard dh files should be about 1 GB in size, so the CRDSDaemon will concatenate your files if their average size is too small; it uses AC++Dump. If the files are large enough, the files will be copied into Sam.

File names in Sam need to have the standard dh form, i.e. they contain the dataset ID, and the lowest runsection number in the lowest run, in hex. The CRDSDaemon will change the file names to conform.

If the daemon concatenates your files, you should inspect the log file for the concatenate job log/conc_*.log or log/conc_*.tgz in your dataset's control area. If the daemon will need to concatenate your files, do not put more than about 100 Gb of files into concat.input at any one time since the buffer used by the daemon might overflow if the transfer of files to Fermilab is slow. Monitor the buffer here.

Files are entered in Sam using the samStoreCdfFile procedure.

To check for files to arrive in Sam, look at Sam's dataset display (enter the dataset ID in top line and click run); then in the next page, click on the dataset ID link to get a list of files.

Controlling the daemon's actions on your dataset

Files .ignore and concat.ignore in your dataset control area are used to control the action of the CRDSDaemon on your dataset. These control files prevent the daemon from initiating new actions; they do not interrupt ongoing actions.

The daemon has 3 functions:

The .ignore file prevents the daemon from initiating any of these actions. So use "touch .ignore" whenever you make changes to any files in your dataset control area. Then "rm .ignore" when you have finished the changes.

The concat.ignore file prevents the daemon from reading concat.input. It is created by the daemon whenever it finds some new files to process in concat.input. An explicit delete of concat.ignore by the user is required to make the daemon read concat.input again.

Thus it is safe to edit concat.input, or to add files to disk areas satisfying the wild cards in concat.input, while the concat.ignore file exists. Do not delete the concat.ignore file while files are still being written that satisfy your concat.input list.

The concatenate part of the daemon keeps a list of files already processed and compares files specified by concat.input against that list in order to find new ones to concatenate or process.

Reading your new dataset

Read your new dataset the same way that you read other Sam datasets, see the Sam hints page.