Table Of Contents

Previous topic

drslib API HowTo

Next topic

CMIP5 product detection

This Page

drs_tool command-line interface

The drs_tool command can be used to invoke functions from the drslib.drs_tree API. The tool is designed to help ESGF datanode managers to

  1. Prepare incoming data for publication, placing files in the DRS directory structure.
  2. Manage multiple versions of publication-level datasets to minimise disk usage.
  3. Deduce the CMIP5 product into which data should be published according to the rules definied in the CMIP5 experiment definition.

Concepts

DRS root
The root of the published DRS directory tree. This is synonimous with the directory representing the DRS activity component.
Incoming directory
The directory scanned for data files to be added to the DRS directory tree. This directory is scanned recursively to find NetCDF files in the DRS filename encoding. It defaults to <drs-root>/output for compatibility with the output structure of CMOR2. However, the directory names under the incoming directory are not taken into account when deducing the DRS components of files.
DRS pattern
Each invocation of drs_tool is given a set of DRS component values that define the portion of the DRS space on which it acts.

Usage

Usage: drs_tool [command] [options] [drs-pattern]

command:

list list publication-level datasets
todo show file operations pending for the next version
upgrade make changes to the selected datasets to upgrade to the next version
mapfile make a mapfile of the selected dataset
history list all versions of the selected dataset
init initialise CMIP5 product detection data
drs-pattern:
A dataset identifier in ‘.’-separated notation. Use the ‘%’ to indicate unknown DRS components, e.g. cmip5.%.MPI-M will match any product.
Options:
-h, --help show this help message and exit
-R ROOT, --root=ROOT
 Root directory of the DRS tree
-I INCOMING, --incoming=INCOMING
 Incoming directory for DRS files. Defaults to <root>/output
-a ACTIVITY, --activity=ACTIVITY
 Set DRS attribute activity for dataset discovery
-p PRODUCT, --product=PRODUCT
 Set DRS attribute product for dataset discovery
-i INSTITUTE, --institute=INSTITUTE
 Set DRS attribute institute for dataset discovery
-m MODEL, --model=MODEL
 Set DRS attribute model for dataset discovery
-e EXPERIMENT, --experiment=EXPERIMENT
 Set DRS attribute experiment for dataset discovery
-f FREQUENCY, --frequency=FREQUENCY
 Set DRS attribute frequency for dataset discovery
-r REALM, --realm=REALM
 Set DRS attribute realm for dataset discovery
-v VERSION, --version=VERSION
 Force version upgrades to this version
-P FILE, --profile=FILE
 Profile the script exectuion into FILE
--detect-product
 Automatically detect the DRS product of incoming data
-j JSON_DRS, --json-drs=JSON_DRS
 Use the JSON output from the ceda-cc quality control tool to define the incoming set of files and their associated DRS terms.

An Example

This example walks through using drs_tool to prepare datasets for publication. It uses some dummy data files based on test runs of the Met Office Hadley Centre HadGEM2-ES model. You can repeat the example from within the drslib source distribution directory.

First generate a dummy set of incoming data files. In this example we assume all incoming files are in a single flat directory, however drs_tool will recursively travers the incoming directory to find all DRS-compliant NetCDF files.

$ mkdir mohc_eg
$ python test/gen_drs.py test/mohc_delivery.ls mohc_eg/incoming
$ ls mohc_eg/incoming/ | head
baresoilFrac_Lmon_HadGEM2-ES_rcp45_r1i1p1_201512-204011.nc
baresoilFrac_Lmon_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc
baresoilFrac_Lmon_HadGEM2-ES_rcp45_r1i1p1_206512-209011.nc
baresoilFrac_Lmon_HadGEM2-ES_rcp45_r1i1p1_209012-209911.nc
c3PftFrac_Lmon_HadGEM2-ES_rcp45_r1i1p1_201512-204011.nc
c3PftFrac_Lmon_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc
c3PftFrac_Lmon_HadGEM2-ES_rcp45_r1i1p1_206512-209011.nc
c3PftFrac_Lmon_HadGEM2-ES_rcp45_r1i1p1_209012-209911.nc
c4PftFrac_Lmon_HadGEM2-ES_rcp45_r1i1p1_201512-204011.nc
c4PftFrac_Lmon_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc
$ ls mohc_eg/incoming/ | wc -l
494

We now have about 500 dummy NetCDF files in mohc_eg/incoming. You can ask drs_tool to list which publication-level datasets these files would be put in using the drs_tool list subcommand. For this to work drs_tool requires 2 DRS components not decidable from the filenames: activity and product [*]. drs_tool list will list all publication-level datasets with the criteria given, including those that would be created by processing the incoming directory.

[*]later versions of drslib will be able to decide the product component from other components and by inspecting the NetCDF.
$ drs_tool list -R mohc_eg/ -I mohc_eg/incoming/ cmip5.output1
==============================================================================
DRS Tree at mohc_eg/
------------------------------------------------------------------------------
cmip5.output1.MOHC.HadGEM2-ES.rcp45.3hr.atmos.3hr.r1i1p1                *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.3hr.land.3hr.r1i1p1                 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrLev.r1i1p1             *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrPlev.r1i1p1            *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.day.atmos.day.r1i1p1                *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.day.land.day.r1i1p1                 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1             *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.atmos.Amon.r1i1p1               *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.land.Lmon.r1i1p1                *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.landIce.LImon.r1i1p1            *
==============================================================================

The asterisk against each dataset_id indicates there are files in the incoming directory to add to the dataset. In this case all datasets are empty.

We can restrict drs_tool list output by using a dataset_id wildcard. For instance to select only datasets in the atmos realm:

$ drs_tool list -R mohc_eg/ -I mohc_eg/incoming/ cmip5.output1.%.%.%.%.atmos
==============================================================================
DRS Tree at mohc_eg/
------------------------------------------------------------------------------
cmip5.output1.MOHC.HadGEM2-ES.rcp45.3hr.atmos.3hr.r1i1p1                *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrLev.r1i1p1             *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrPlev.r1i1p1            *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.day.atmos.day.r1i1p1                *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.atmos.Amon.r1i1p1               *
==============================================================================

The same effect can be achieved with individual component options:

$ drs_tool list -R mohc_eg/ -I mohc_eg/incoming/ cmip5.output1 --realm=atmos
==============================================================================
DRS Tree at mohc_eg/
------------------------------------------------------------------------------
cmip5.output1.MOHC.HadGEM2-ES.rcp45.3hr.atmos.3hr.r1i1p1                *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrLev.r1i1p1             *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrPlev.r1i1p1            *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.day.atmos.day.r1i1p1                *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.atmos.Amon.r1i1p1               *
==============================================================================

Now we will focus on a single dataset in the aerosol realm and show how to move files into the DRS directory structure ready for publication. We can check what filesystem commands will be done using the drs_tool todo subcommand.

$ drs_tool list -R mohc_eg/ -I mohc_eg/incoming/ cmip5.output1 --realm=aerosol
==============================================================================
DRS Tree at mohc_eg/
------------------------------------------------------------------------------
cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1             *
==============================================================================
$ drs_tool todo -R mohc_eg/ -I mohc_eg/incoming/ cmip5.output1 --realm=aerosol | head
==============================================================================
DRS Tree at mohc_eg/
------------------------------------------------------------------------------
Publisher Tree cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1 todo for version 20100927
------------------------------------------------------------------------------
mv mohc_eg/incoming/emidust_aero_HadGEM2-ES_rcp45_r1i1p1_206512-209011.nc /home/spascoe/git/esgf-drslib/mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/files/emidust_20100927/emidust_aero_HadGEM2-ES_rcp45_r1i1p1_206512-209011.nc
ln -s /home/spascoe/git/esgf-drslib/mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/files/emidust_20100927/emidust_aero_HadGEM2-ES_rcp45_r1i1p1_206512-209011.nc /home/spascoe/git/esgf-drslib/mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/emidust/emidust_aero_HadGEM2-ES_rcp45_r1i1p1_206512-209011.nc
mv mohc_eg/incoming/reffclwtop_aero_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc /home/spascoe/git/esgf-drslib/mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/files/reffclwtop_20100927/reffclwtop_aero_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc
ln -s /home/spascoe/git/esgf-drslib/mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/files/reffclwtop_20100927/reffclwtop_aero_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc /home/spascoe/git/esgf-drslib/mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/reffclwtop/reffclwtop_aero_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc
mv mohc_eg/incoming/dryso2_aero_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc /home/spascoe/git/esgf-drslib/mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/files/dryso2_20100927/dryso2_aero_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc

You can see here that drslib will move files into datestamped directories under <dataset-dir>/files then symbolically link them into the DRS directory structure. To do the actual moving use drs_tool upgrade. Then use drs_tool list to view the result.

$ drs_tool upgrade -R mohc_eg/ -I mohc_eg/incoming/ cmip5.output1 --realm=aerosol
==============================================================================
DRS Tree at mohc_eg/
------------------------------------------------------------------------------
Upgrading cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1 to version 20100927 ... done
==============================================================================
$ drs_tool list -R mohc_eg/ -I mohc_eg/incoming/ cmip5.output1
==============================================================================
DRS Tree at mohc_eg/
------------------------------------------------------------------------------
cmip5.output1.MOHC.HadGEM2-ES.rcp45.3hr.atmos.3hr.r1i1p1                *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.3hr.land.3hr.r1i1p1                 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrLev.r1i1p1             *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrPlev.r1i1p1            *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.day.atmos.day.r1i1p1                *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.day.land.day.r1i1p1                 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1.v20100927   -
cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.atmos.Amon.r1i1p1               *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.land.Lmon.r1i1p1                *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.landIce.LImon.r1i1p1            *
==============================================================================

Using drs_tool‘s criteria options you can upgrade multiple datasets in one command:

$ drs_tool upgrade -R mohc_eg/ -I mohc_eg/incoming/ cmip5.output1 --realm=atmos --frequency=6hr
==============================================================================
DRS Tree at mohc_eg/
------------------------------------------------------------------------------
Upgrading cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrLev.r1i1p1 to version 20100927 ... done
Upgrading cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrPlev.r1i1p1 to version 20100927 ... done
==============================================================================
$ drs_tool list -R mohc_eg/ -I mohc_eg/incoming/ cmip5.output1
==============================================================================
DRS Tree at mohc_eg/
------------------------------------------------------------------------------
cmip5.output1.MOHC.HadGEM2-ES.rcp45.3hr.atmos.3hr.r1i1p1                *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.3hr.land.3hr.r1i1p1                 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrLev.r1i1p1.v20100927   -
cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrPlev.r1i1p1.v20100927  -
cmip5.output1.MOHC.HadGEM2-ES.rcp45.day.atmos.day.r1i1p1                *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.day.land.day.r1i1p1                 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1.v20100927   -
cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.atmos.Amon.r1i1p1               *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.land.Lmon.r1i1p1                *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.landIce.LImon.r1i1p1            *
==============================================================================

Finally you need to send publish the datasets with esgpublish. To make this easier drs_tool can create a mapfile of a dataset:

$ drs_tool mapfile -R mohc_eg/ -I mohc_eg/incoming/ cmip5.output1 --realm=aerosol >rcp45.mon.aerosol.map
$ head rcp45.mon.aerosol.map
mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/loadsoa/loadsoa_aero_HadGEM2-ES_rcp45_r1i1p1_206512-209011.nc | cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1
mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/loadsoa/loadsoa_aero_HadGEM2-ES_rcp45_r1i1p1_209012-209911.nc | cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1
mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/loadsoa/loadsoa_aero_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc | cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1
mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/loadsoa/loadsoa_aero_HadGEM2-ES_rcp45_r1i1p1_201512-204011.nc | cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1
mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/loadbc/loadbc_aero_HadGEM2-ES_rcp45_r1i1p1_201512-204011.nc | cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1
mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/loadbc/loadbc_aero_HadGEM2-ES_rcp45_r1i1p1_206512-209011.nc | cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1
mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/loadbc/loadbc_aero_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc | cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1
mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/loadbc/loadbc_aero_HadGEM2-ES_rcp45_r1i1p1_209012-209911.nc | cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1
mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/wetbc/wetbc_aero_HadGEM2-ES_rcp45_r1i1p1_209012-209911.nc | cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1
mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/wetbc/wetbc_aero_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc | cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1

Some further examples of usage can be found in the doctest file test/test_command.txt.