.. index::
    pair: V.3.00.08; Release Notes

******************************
  EBAS 3.00.08 Release Notes
******************************

:Author: PE/ATMOS <pe@nilu.no>

.. seealso::

   :doc:`index`

Special Note
============

Release 3.00.08 is an untypical release. After :doc:`V.3.00.07<V.3.00.07>`, a
major :doc:`V.3.01.00<V.3.01.00>` was planned and development started as a new
branch.
However 3.01.00 was stuffed
with too many new features and development progressed only very slow.
Over time, many minor change requests were demanded urgently and development
split into
a `post 3.00.07 agile branch` where development cycles were very short and
changes pushed into production without tagging and documenting a proper release.

At the end of this phase, before releasing :doc:`V.3.01.00<V.3.01.00>`, this
agile changes are retrospectively collected into a formal release with proper
documentation.

Features
========

* All ebas commandline programs: new selection criteria.
  :option:`--diff`

  Especially useful for :ref:`differential extract <diffextract>`, but the functionality is available in all commandline programs.

* All ebas commandline programs: new selection criteria
  :option:`--datalevel`

  | Issue Description: `EWB-461 <http://jira.nilu.no/browse/EWB-461>`_
  | Fix: svn r1472

* :ref:`EBASprogram_ebas_insert`: added option
  :option:`--skip-variables <ebas_insert --skip-variables>`

  | Issue Description: `EWB-451 <http://jira.nilu.no/browse/EWB-451>`_
  | Fix: svn r1441

* :ref:`EBASprogram_ebas_insert`: Check for all-missing, all-equal values for
  a variable are now ERROR instead of WARNING

  | Issue Description: `EWB-452 <http://jira.nilu.no/browse/EWB-452>`_
  | Fix: svn r1442 (bugfixes r1444, r1445)

* :ref:`EBASprogram_ebas_insert`: added ``identical-values`` to
  :option:`ebas_insert --ignore=identical-values <ebas_insert --ignore>`

  | Issue Description: `EWB-452 <http://jira.nilu.no/browse/EWB-452>`_
  | Fix: svn r1442 (bugfixes r1444, r1445)

* :ref:`EBASprogram_ebas_insert`: added possibility to automatically ignore
  ``identical-values`` for certain metadata combinations

  | Issue Description: `EWB-469 <http://jira.nilu.no/browse/EWB-469>`_
  | Fix: svn r1535

  currently defined (but may be extended anytime):

  * instrument type ``optical_particle_size_spectrometer``, component
     ``particle_number_size_distribution`` and particle size bin > ``400 nm``

     values may all be 0.00 (no big particles measured)

  * instrument type ``TEOM`` and component ``temperature``

     values may be constant

  * data level ``0``, instrument types ``dmps`` and ``smps``
    
    * component ``status``, ``temperature``, ``flow_rate`` or
       ``particle_diameter``

       values may be constant

    * component ``particle_number_concentration`` and size bin number > ``25``

       values may all be 0.0 (no big particles measured)

* :ref:`EBASprogram_ebas_insert`: Conversion on import now adds a comment to
  the metadata

  | Issue Description: `EWB-454 <http://jira.nilu.no/browse/EWB-454>`_
  | Fix: svn r1452

* :ref:`EBASprogram_ebas_insert` / :ref:`EBASprogram_ebas_extract`: Additional
  components which are converted on import and export:

  * add conversion for component ``ethanal``

    | Issue Description: `EWB-453 <http://jira.nilu.no/browse/EWB-453>`_
    | Fix: svn r1453

  * add conversion for component ``temperature`` matrix ``met`` from ``K`` to
    ``C`` (import only)

    | Issue Description: `EWB-465 <http://jira.nilu.no/browse/EWB-465>`_
    | Fix: svn r1557

  * add conversion for component ``SO2`` matrix ``air`` from ``nmol/mol`` to
    ``ug S/m3``

    | Issue Description: `EWB-466 <http://jira.nilu.no/browse/EWB-466>`_
    | Fix: svn r1551

  * conversion for component ``ozone``: standard temperature changed from
    ``273.15 K`` to ``293.15 K``

    | Issue Description: `EWB-449 <http://jira.nilu.no/browse/EWB-449>`_
    | Fix: svn r1450

* :ref:`EBASprogram_ebas_insert`: added range check for
  ``Volume std. temperature`` and ``Volume std. pressure``

  | Issue Description: `EWB-467 <http://jira.nilu.no/browse/EWB-467>`_
  | Fix: svn r1517

* :ref:`EBASprogram_ebas_insert`: Value/flag consistency: new exceptions for
  auxiliary datasets

  | Issue Description: `EWB-411 <http://jira.nilu.no/browse/EWB-411>`_
  | Fix: svn r1446

* :ref:`EBASprogram_ebas_insert`: all file checks can now make exceptions based
  on the data level. Needed for level 0 file checks - especially in 
  ebas-submit-tool. We can now, based on data level:

   * accept additional, non standard values for all master data (special
     parameters only allowed in certain data levels, additional component
     characteristics, etc.) E.g. component ``NO_#counts`` (NOxy level 0 template)
     or ``equivalent_black_carbon_loading`` (filter_absorption_photometer
     level 0 template)
     
   * accept additional, non standard metadata elements only allowed
     in certain data levels. E.g. metadata element
     ``Duration of stay in converter or bypass line`` which is only allowed in
     the NOxy level 0 template

  Those are exceptions for checking files which are not meant to be stored in
  the database. The exceptions are only used for the file check routines.
  Those data can still not be actually stored in the database unless the
  exceptions are properly implemented in the database.

  | Issue Description: `EWB-479 <http://jira.nilu.no/browse/EWB-479>`_
  | Fix: svn r1563

* :ref:`EBASprogram_ebas_insert`: NasaAmes1001: convert ``no break space``
  automatically to normal blank in all file read operations

  | Issue Description: `EWB-444 <http://jira.nilu.no/browse/EWB-444>`_
  | Fix: svn r1402

* :ref:`EBASprogram_ebas_insert`: prevent that the same instrument is used at
  the same time at different stations

  | Issue Description: `EWB-458 <http://jira.nilu.no/browse/EWB-458>`_
  | Fix: svn r1466
  | Backported to :doc:`V.3.00.07<V.3.00.07>`, svn r1469

.. _diffextract:

* :ref:`EBASprogram_ebas_extract`: added differential extracts using
  :option:`ebas_extract --diff` and :option:`ebas_extract --diffxml`

  | Issue Description: `EWB-384 <http://jira.nilu.no/browse/EWB-384>`_
  | Fix: svn r1181, r1182, r1187, r1198, r1198, r1202, r1203, r1208

* :ref:`EBASprogram_ebas_extract`: added 
  :option:`--metadata_option setkey <ebas_extract --metadata_options>` to add
  :term:`ds_setkey` to the metadata output.

  | Issue Description: `EWB-374 <http://jira.nilu.no/browse/EWB-374>`_
  | Fix: svn r1164

* :ref:`EBASprogram_ebas_extract`: new output file format 
  :option:`NetCDF <ebas_extract --format>` (prototype)

  | Issue Description: `EWB-314 <http://jira.nilu.no/browse/EWB-314>`_
  | Fix: svn r1522

* :ref:`EBASprogram_ebas_list_ds`: Data level is now included in the dataset
  list

  | Issue Description: `EWB-460 <http://jira.nilu.no/browse/EWB-460>`_
  | Fix: svn r1470

* :ref:`EBASprogram_ebas_list_ds`: HFI metadata are now included in
  :option:`ebas_list_ds --verbose` output (instrument manufacturer,
  instrument model, instrument serial number)

  | Issue Description: `EWB-476 <http://jira.nilu.no/browse/EWB-476>`_
  | Fix: svn r1543

* :ref:`EBASprogram_ebas_list_ds`: Submision metadata are now included in 
  :option:`ebas_list_ds --verbose` output (revision, submitter, originator,
  importer)

  | Issue Description: `EWB-477 <http://jira.nilu.no/browse/EWB-477>`_
  | Fix: svn 1559

* new program :ref:`EBASprogram_ebas_plot`

Bugfix:
=======

* All ebas commandline programs: time criteria with additional HVM and HDP
  critera: false positives when HVM and HDP time intervals do not overlap.

  | Issue Description: `EWB-392 <http://jira.nilu.no/browse/EWB-392>`_
  | Fix: svn r1181

* :ref:`EBASprogram_ebas_insert`: bug when inserting NRT data before existing
  data in the data coverage with the same HFI or HVM metadata: HFI and HVM
  intervals are duplicated (and not aggregated as expected). When exporting
  those data, multiple export files are generated for the interval fragments.

  | Issue Description: `EWB-393 <http://jira.nilu.no/browse/EWB-393>`_
  | Fix: svn r1172

* :ref:`EBASprogram_ebas_insert`: check metadata: length check missing for some
  string metadata

  | Issue Description: `EWB-306 <http://jira.nilu.no/browse/EWB-306>`_
  | Fix: svn r1317

* :ref:`EBASprogram_ebas_insert`: check laboratory code

  | Issue Description: `EWB-326 <http://jira.nilu.no/browse/EWB-326>`_
  | Issue Description: `EWB-419 <http://jira.nilu.no/browse/EWB-419>`_
  | Fix: svn r1273

* :ref:`EBASprogram_ebas_insert`: detection limit and uncertainty in scientific
  notation does not work

  | Issue Description: `EWB-425 <http://jira.nilu.no/browse/EWB-425>`_
  | Fix: svn r1348, r1349

* :ref:`EBASprogram_ebas_insert`: some PS (DO and DS) generated twice

  | Issue Description: `EWB-432 <http://jira.nilu.no/browse/EWB-432>`_
  | Fix: svn r1440

* :ref:`EBASprogram_ebas_insert`: finds non-exact matching DS, but does not list
  them

  | Issue Description: `EWB-439 <http://jira.nilu.no/browse/EWB-439>`_
  | Fix: svn r1386

* :ref:`EBASprogram_ebas_insert`: Erroneus boundary check (Missing values are
  wrongly rejected as below lower bound)

  | Issue Description: `EWB-442 <http://jira.nilu.no/browse/EWB-442>`_
  | Fix: svn r1438

* :ref:`EBASprogram_ebas_insert`: crash when reading file without data lines

  | Issue Description: `EWB-462 <http://jira.nilu.no/browse/EWB-462>`_
  | Fix: svn r1476, r1481


* :ref:`EBASprogram_ebas_insert`: crash when a newline is found inside a quoted
  string

  | Issue Description: `EWB-415 <http://jira.nilu.no/browse/EWB-415>`_
  | Fix: svn r1449

* :ref:`EBASprogram_ebas_insert`: check sample times not in the future 

  | Issue Description: `EWB-468 <http://jira.nilu.no/browse/EWB-468>`_
  | Fix: svn r1518

* :ref:`EBASprogram_ebas_insert`: crash when meta data ``Method ref`` is missing

  | Issue Description: `EWB-339 <http://jira.nilu.no/browse/EWB-339>`_
  | Fix: svn r1274
  | Backported to :doc:`V.3.00.07<V.3.00.07>`, svn r1275

* :ref:`EBASprogram_ebas_insert`: file read: flags are rounded in some cases

  | Issue Description: `EWB-416 <http://jira.nilu.no/browse/EWB-416>`_
  | Fix: svn r1264, r1323
  | Backported to :doc:`V.3.00.07<V.3.00.07>`, svn r1265, r1324

* :ref:`EBASprogram_ebas_insert`: unit conversion crashes

  | Issue Description: `EWB-397 <http://jira.nilu.no/browse/EWB-397>`_
  | Issue Description: `EWB-413 <http://jira.nilu.no/browse/EWB-413>`_
  | Fix: svn r1241
  | Backported to :doc:`V.3.00.07<V.3.00.07>`, svn r1242

* :ref:`EBASprogram_ebas_insert`: crash when using 2 different instrument types
  for the same instrument ref

  | Issue Description: `EWB-414 <http://jira.nilu.no/browse/EWB-414>`_
  | Fix: svn r1246
  | Backported to :doc:`V.3.00.07<V.3.00.07>`, svn r1247

* :ref:`EBASprogram_ebas_extract`: extracting NRT data: revision date sometimes
  older than data

  | Issue Description: `EWB-394 <http://jira.nilu.no/browse/EWB-394>`_
  | Fix: svn r1174, r1177

* :ref:`EBASprogram_ebas_extract`: crashes on instrument names containing ``/``
  (slash)

  | Issue Description: `EWB-320 <http://jira.nilu.no/browse/EWB-320>`_
  | Fix: svn r1447

* :ref:`EBASprogram_ebas_extract`: flag error when extracting multicolumn

  | Issue Description: `EWB-440 <http://jira.nilu.no/browse/EWB-440>`_
  | Fix: svn r1406

* :ref:`EBASprogram_ebas_extract`: crash when converting units on export

  | Issue Description: `EWB-418 <http://jira.nilu.no/browse/EWB-418>`_
  | Fix: svn r1270, r1281
  | Backported to :doc:`V.3.00.07<V.3.00.07>`, svn r1271, r1282

* :ref:`EBASprogram_ebas_extract`: unit convert: problems with historic data
  (too little precision)

  | Issue Description: `EWB-455 <http://jira.nilu.no/browse/EWB-455>`_
  | Fix: svn r1451

* :ref:`EBASprogram_ebas_delete`: crash when deleting a whole interval including
  project associations

  | Issue Description: `EWB-441 <http://jira.nilu.no/browse/EWB-441>`_
  | Fix: svn r1392

* :ref:`EBASprogram_ebas_delete`: full delete of NRT datasets crashes

  | Issue Description: `EWB-487 <http://jira.nilu.no/browse/EWB-487>`_
  | Fix: svn r1602, r1603

  not completely fixed (workaround for now, final fix is postponed)


* ebas.io: problems when reading ebas nas files with user libraries (ebas-io)

  | Issue Description: `EWB-396 <http://jira.nilu.no/browse/EWB-396>`_
  | Fix: svn r1279

                

Backported from future releases
===============================

* :ref:`EBASprogram_ebas_extract`: Format XML: gml:id is sometimes not unique

  | Issue Description: `EWB-285 <http://jira.nilu.no/browse/EWB-285>`_
  | Backported from :doc:`V.3.01.01<V.3.01>`, svn r1986, r1988
  | Backport: svn r1987, r1989