The design of the library has been driven by the following requirements:
- Deduce the DRS directory structure a DRS-compliant filename.
- Validate filenames and directory paths against the DRS.
- Convert the CMIP3 directory structure into a DRS-compliant form.
- Manage multiple versions of DRS publication-level datasets on the filesystem.
- Detect the CMIP5 product DRS component during data publication.
# Using pip $ pip install drslib # OR using setuptools $ easy_install drslib
If you are upgrading drslib you will need to add the “-U” option to either pip or easy_install.
Note: To install onto an ESG datanode you can use use the easy_install in /usr/local/uv-cdat/bin. This will install drslib into /usr/local/uv-cdat/bin/python and install drs_tool as /usr/local/uv-cdat/bin/drs_tool.
drslib will use CMIP5 MIP tables to deduce realm and other metadata. MIP tables are not required for other projects as selected with the -s scheme switch. You can point to your MIP table location using the environment variable MIP_TABLE_PATH. All tables should be named CMIP5_*.
Alternatively you can use metaconfig to configure the location of your MIP tables. Create the following in your $HOME/.metaconfig.conf file:
[metaconfig] configs = drslib [drslib:tables] path = /path/to/mip/tables model_table = /path/to/model_table
The option model_table is optional and if specified should point to the “CMIP5 Modelling Group” spreadsheet in CSV format. This table is used to map model names to institute names. The drslib package is distributed with a recent version of this file.
Default DRS attributes used by the drs_tool command can be set in the drs section:
[drslib:drs] root = /path/to/drs/activity institute = MOHC model = HadCM3
It is usually convenient to set at least root and activity in the configuration file.
Overriding DRS vocabularies¶
Drslib ships with reasonable defaults for the DRS vocabularies, however sometimes you will want to override or extend the defaults supplied. At this time drslib supports extending the vocabularies of institutes, models and experiments.
To define all CMIP5 experiments including individual decadal experiments you can define the drslib:vocabularies section as follows:
[drslib:vocabularies] experiments = 1pctto2x 2xco2 pdcntrl sresa1b 1pctto4x amip picntrl sresa2 20c3m commit slabcntl sresb1 decadal1960 decadal1961 decadal1962 decadal1963 decadal1964 decadal1965 decadal1966 decadal1967 decadal1968 decadal1969 decadal1970 decadal1971 decadal1972 decadal1973 decadal1974 decadal1975 decadal1976 decadal1977 decadal1978 decadal1979 decadal1980 decadal1981 decadal1982 decadal1983 decadal1984 decadal1985 decadal1986 decadal1987 decadal1988 decadal1989 decadal1990 decadal1991 decadal1992 decadal1993 decadal1994 decadal1995 decadal1996 decadal1997 decadal1998 decadal1999 decadal2000 decadal2001 decadal2002 decadal2003 decadal2004 decadal2005 decadal2006 decadal2007 decadal2008 decadal2009
Institutes and models can be defined with the institutes option. This is interpreted as a newline separated list of lines, each line being the institute name a colon then space separated list of models.
institutes = NOAA-GFDL:GFDL-ESM2G MOHC:HadGEM2-ES HadCM3 HadGEM2-CC
drslib uses Python’s standard logging infrastructure to give details of it’s operation. Messages are sent to loggers under the drslib logger. You can configure logging via metaconfig by pointing to a separate logging configuration file:
[metaconfig] configs = drslib logging = /path/to/logging.conf
The format of logging.conf should conform to the Python logging `configuration file format`__. An example logging configuration is given below which will log product detection decisions separately from general drslib warnings:
# # Basic logging configuration for drs_tool # # This configuration prints product detection decisions to STDERR and logs # warnings to ./drs_tool.log # [loggers] keys=root,drslib,p_cmip5 [handlers] keys=drslib_h,p_cmip5_h [formatters] keys=f1,f2 #--------------------------------------------------------------------------- # Loggers # No catch-all logging [logger_root] handlers= level=NOTSET [logger_drslib] qualname=drslib handlers=drslib_h [logger_p_cmip5] qualname=drslib.p_cmip5 handlers=p_cmip5_h propagate=0 #--------------------------------------------------------------------------- # Handlers & Formatters [handler_drslib_h] class=FileHandler args=('./drs_tool.log', ) formatter=f1 level=INFO [handler_p_cmip5_h] class=StreamHandler args=(sys.stderr, ) formatter=f2 level=INFO [formatter_f1] format=%(asctime)s [%(levelname)s] %(name)s: %(message)s datefmt= [formatter_f2] format=[%(levelname)s] %(name)s: %(message)s
drslib ships with a test suite compliant with nose. The suite can be run in various ways. The test suite uses the extension NoseXUnit to produce XML reports of the test results. NoseXUnit will be automatically installed if you run the tests via setup.py:
$ python setup.py test
Or if the depencencies are satisfied you can run all tests with: