ebas_insert

The program ebas_insert imports EBAS NASA Ames files into the EBAS database.

Synopsis

ebas_insert.py [-h] [--version] [--cfgfile CFGFILE]
               [--loglevconsole LOGLEVCONSOLE]
               [--loglevfile LOGLEVFILE] [--logfile LOGFILE]
               [--profile] [--nodb] [--dbHost DBHOST] [--db DB]
               [--dbUser DBUSER] [--dbPasswd DBPASSWD]
               [--transcomment COMMENT] [--nowrite] [--nocommit]
               [--ignore IGNORE] [--skip-variables SKIP_VARIABLES]
               [--non-interactive] [--nrt] [--email-notify]
               [--email-override EMAIL_OVERRIDE]
               [--email-bcc EMAIL_BCC] [--email-smtp EMAIL_SMTP]
               [filenames [filenames ...]]

Commandline arguments

For information about the general concepts for commandline arguments, please refer to Commandline arguments.

General arguments

-h, --help

Show help text and exit.

-v, --version

Display version information and exit.

Configuration arguments

--cfgfile=CFGFILE

Set the EBAS configuration file to be used. See Configuration file for detailed information.

This argument requires an argument value CFGFILE (the configuration file to be used). CFGFILE is the full path and file name, the file path might be absolute or relative. The current user’s home directory may be specified with a tilde character (~), other user’s home directory may be specified as ~other_user.

Note

There may not be any blank characters in the CFGFILE argument value! If the filename contains blank characters, wrap it into double quotes ("). Example:

--cfgfile="my ebas config.cfg"

Examples:

Specify a config file named ebas.cfg in the current working directory (relative path):

--cfgfile=ebas.cfg

Specify a config file named ebas.cfg in the directory config under the current working directory (relative path):

--cfgfile=config/ebas.cfg

Specify a config file named ebas.cfg in the directory /home/me (absolute path):

--cfgfile=/home/me/ebas.cfg

Specify a config file named ebas.cfg in the current user’s home directory:

--cfgfile=~/ebas.cfg

Specify a config file named ebas.cfg in user collegue’s home directory:

--cfgfile=~collegue/ebas.cfg

Default: ~/ebas.cfg

Logging arguments

--loglevconsole=LOGLEVEL

Set log level for console output.

This argument requires an argument value LOGLEVEL. All messages from EBAS are categorized with logging severities (e.g. error messages are written with a different severity as information messages). Severities are: CRITICAL, ERROR, WARNING, INFO, DEBUG (descending severity)

Setting the LOGLEVEL controls which category of messages will be displayed to the user. Possible values for LOGLEVEL are:

• silent - no messages will be displayed in the console output
• critical - only CRITICAL messages will be displayed in the console output
• errors - only messages with severity CRITICAL or ERROR will be displayed in the console output
• warnings - only messages with severity CRITICAL, ERROR or WARNING will be displayed in the console output
• info - messages with severity CRITICAL, ERROR, WARNING or INFO will be displayed in the console output
• debug - all messages will be displayed in the console output

Default: info

--loglevfile=LOGLEVEL

Set log level for logfile output.

This argument requires an argument value LOGLEVEL. All messages from EBAS are categorized with logging severities (e.g. error messages are written with a different severity as information messages). Severities are: CRITICAL, ERROR, WARNING, INFO, DEBUG (descending severity)

Setting the LOGLEVEL controls which category of messages will be included in the logfile. Possible values for LOGLEVEL are:

• silent - no messages will be written to the logfile and no logfile will be created
• critical - only CRITICAL messages will be written to the logfile
• errors - only messages with severity CRITICAL or ERROR will be written to the logfile
• warnings - only messages with severity CRITICAL, ERROR or WARNING will be written to the logfile
• info - messages with severity CRITICAL, ERROR, WARNING or INFO will be written to the logfile
• debug - all messages will be written to the logfile

Default: silent

--logfile=LOGFILE

Set file name for logfile output (including path).

This argument requires an argument value LOGFILE. LOGFILE is the full path and file name, the file path might be absolute or relative. The current user’s home directory may be specified with a tilde character (~), other user’s home directory may be specified as ~other_user.

Note

There may not be any blank characters in the LOGFILE argument value! If the path or file name contains blank characters, wrap it into double quotes ("). Example:

--logfile="ebas logfiles/logfile.log"

Default: filename constructed of program name and start time in the current working directory (e.g. ./ebas_list_ds_2015-12-27_123147.log)

Changed in version V.3.02.00: the previous default file name included colons (:) as time separator, (e.g. ./ebas_list_ds_2015-12-27_12:31:47.log)

--profile

Activate program profiling.

Note

This option is for EBAS developers only. Code profiling is an analytical technique used in runtime optimization. Using this option has no benefit for a user (it makes the code slower actually).

Default: False

Database arguments

Database user and password can also be fetched from your .netrc file (recommended authentication method).

See Database Authentication for more information on this. If you want to use netrc authentication, you need to leave the --dbUser and --dbPasswd agruments away!

--nodb

Do not connect to the database. Most programs will have limited functionality without database connection (e.g. ebas_insert can still perform file syntax checks and limited semantic checks). Some programs will lose all their functionality, but even in those cases, the nodb option might be useful for tests or other arguments validation, or simply to get a –version and –help output when the database is unreachable.

New in version 3.00.07.

--dbHost=DBHOST

Specify the database host name according to the database client configuration file (sql.ini for sybase drivers, freetds.conf for freetds drivers). If this sounds unfamiliar, ask a system administrator. At NILU, we use two database servers for EBAS, ODIN (database server for the test system) and SLEIPNER (operational database server). The respective names to be used for the argument –dbHost on ratatoskr are:

• ODIN_ASE
• SLEIPNER_ASE

--db=DBNAME

The name of the database to be used. If you don’t know for sure that you want to change this, leave it at the default (ebas_new)

--dbUser=DBUSER

The user name to be used for connecting to the database. See Database Authentication.

--dbPasswd=DBPASSWD

The password to be used for connecting to the database. See Database Authentication.

Warning

Please do NOT use the --dbPasswd argument!

On multi-user UNIX systems, each users is able to see the command used to start any program on the system.

Therefore, including sensitive information in commands is generally strongly discouraged on UNIX systems.

See Database Authentication for alternatives to using the --dbPasswd argument.

Note

There may not be any blank characters in the DBPASSWD argument value! If the password contains blank characters, wrap it into double quotes ("). Example:

--dbPasswd="with blank"

--transcomment=COMMENT

A comment text for the action performed. This comment is archived in database connected to the changes. This comment should be used for documenting the changes and can be very useful for future reference.

Optional, Default: None

Note

in order to write a comment with blank characters, the commet text needs to be quoted with double quotes ("). Example:

--transcomment="deleted obsolete data because of update submission"

versionadded:: 3.03.00

--nowrite

This prevents that the changes are written to the database. Used for tests runs, if you want to check if the operation would succeed, but you don’t want the operation to be actually performed.

--nocommit

This prevents that the changes in the database are committed. In contrast to nowrite, the changes are written to the database, but then rolled back, so that no permanent changes to the database occur. This option is mainly for debugging purpose, for testing the actual database operations but without changing the database.

Arguments specific to ebas_insert

--ignore=IGNORE

ignore certain errors while reading (numformat, dx0, identical-values)

• numformat

  ignore if number format does not match missing value code

• dx0 allow DX=0 with regular data (DX!=0 will always raise an error if not correct)

• identical-values

  ignore errors related to all values of one variable being equal (e.g. all data missing for one variable, all values 0.0, or all values any other constant value). Those cases usually indicate an ERROR in the data. However if this is investigated and those data should be read, use this flag to downgrade the ERROR severity to WARNING. For the opposite (those cases are definitely identified as ERROR should not be read while the rest of the file should be read, consider the –skip-variables parameter)

  New in version 3.00.08.

(default: None)

--condense-messages CONDENSE_MESSAGES

New in version 3.02.00.

Threshold for condensation of multiple occurrences of the same error or warning message. When specifying 0, the condensation is turned off and every single message will be propagated. Default: 0

If more then CONDENSE_MESSAGES identical messages (errors or warnings) are generated, they are condensed. When condensing, the first occurrence of the message is propagated (printed on console log, written into logfile). A second message is generated right after the first one, providing additional information about the repeated (and suppressed) messages.

Example:

ERROR   : line 1000: variable[231]: Missing value for flag column is not allowed.
ERROR   : line 1000: The previous message repeats 9 times for variable 231. First occurrences in lines: 1000, 1001, 1002, 1003, 1004, ...

--skip-variables SKIP_VARIABLES

New in version 3.00.08.

skip the variable(s) referenced by the variable number(s) while reading the file. Specify a single variable number or a comma separated list of variable numbers. Variable numbers start with 1 for the variable succeeding end_time (same numbering as in all WARNING/ERROR messages) start_time and end_time can not be skipped, numflags may not be skipped

(default: None)

--skip-data

New in version 3.02.00.

Do not read data from the file, only header information and metadata.

This option can be handy for testing purposes: Checking only the header for a big file --skip-data --nowrite. This way reading is superfast.

If ebas_insert is used to read a file with --skip-data, and no nowrite is used, The datasets will be created and all metadata information from the file header is written to the database. The datasets will be created without any measurement data. In case, the datasets only match partly, the interactive setup of datasets can be done as usual. This option is useful when a huge datafile should be scheduled for automatic import (e.g. during nighttime). In case the datasets do not exist or do not match exactly, the non-interactive automatic insert would not work. Thus, the datasets and metadata can be prepared interactive beforehand.

--non-interactive

no user interaction if new station or dataset should be created - should be used in automated processes. (default: False, i.e. interactive behaviour)

--nrt

activate NRT operation (DATA WITHOUT HISTORY) (default: False)

--email-notify --no-email-notify

Changed in version 3.02.00: The previos notification functionality (send warnings and errors through email) has been discontinued.

Notify data submitters on successful insert into the database (default: False)

Note

Users who work mainly with operational data management should turn on this option in their ebas config file, thus ensuring an email is sent to all data submitters by default.

--email-override EMAIL_OVERRIDE

New in version 3.02.00.

Override the email recipients for notification emails. By default, the email addresses are extracted from the data submitters specified in file’s metadata. If --email-override is specified, no email will be sent to the data submitters, but instead to the email address(es) specified.

It is possible to specify a single email address or a comma separated list of addresses.

This option can be used instead of totally disabling the notification option. The email can be forwarded manually later, if needed.

(default: None)

--email-bcc EMAIL_BCC

New in version 3.02.00.

If specified, the notification email will be additionally sent in blind copy to the email address(es) specified. This option is very convenient to keep track of all the emails sent to data submitters.

It is possible to specify a single email address or a comma separated list of addresses. (default: <username>@nilu.no)

Note

It’s highly reccommended for all users to set their correct email address in their ebas config file. The default <username>@nilu.no does not work for all users on all machines.

--email-smtp EMAIL_SMTP

New in version 3.02.00.

Change the server for sending emails. The default server should usually work, but in some cases (installations on the internal network or local installation on a user PC), the email server needs to be changed.

filename

input file(s), EBAS NASA-Ames format (default: None)