Debugging

Overview

Teaching: 30 min
Exercises: 15 min
Compatibility: ESMValTool v2.7.0

Questions

How can I handle errors/warnings?

Objectives

Fix a broken recipe

Every user encounters errors. Once you know why you get certain types of errors, they become much easier to fix. The good news is that ESMValTool creates a record of the output messages and stores them in log files. They can be used for debugging or monitoring the process. This lesson helps you understand the different types of errors and when you are likely to encounter them.

Log files

Each time we run ESMValTool, it will produce a new output directory. This directory should contain the run folder that is automatically generated by ESMValTool. To examine this, we run a recipe_python.yml that can be found in lesson Running your first recipe. Check lesson Configuration on how to set paths.

In a new terminal, run the recipe:

  cd esmvaltool_tutorial
  esmvaltool run examples/recipe_python.yml

esmvaltool: command not found

ESMValTool encounters this error because the conda environment esmvaltool has not been activated. To fix the error, before running the recipe, activate the environment:

conda activate esmvaltool

conda environment

More information about the conda environment can be found at Installation.

Let’s list the files in the run directory:

  ls esmvaltool_output/recipe_python_#_#/run

main_log_debug.txt  main_log.txt  map  recipe_python.yml  resource_usage.txt
timeseries

In the main_log_debug.txt and main_log.txt, ESMValTool writes the output messages, warnings and possible errors that might happen during pre-processings. To inspect them, we can look inside the files. For example:

  cat esmvaltool_output/recipe_python_#_#/run/main_log.txt

Now, let’s have a look inside the folder timeseries/script1:

  ls esmvaltool_output/recipe_python_#_#/run/timeseries/script1/

diagnostic_provenance.yml log.txt  resource_usage.txt  settings.yml

In the log.txt, ESMValTool writes the output messages, warnings and possible errors that are related to the diagnostic script.

If you encounter an error and don’t know what it means, it is important to read the log information. Sometimes knowing where the error occurred is enough to fix it, even if you don’t entirely understand the message. However, note that you may not always be able to find the error or fix it. In that case, ESMValTool community helps you figure out what went wrong.

Different log files

In the run directory, there are two log files main_log_debug.txt and main_log.txt. What are their differences?

Solution

The main_log_debug.txt contains the output messages from the pre-processor whereas the main_log.txt shows general errors and warnings that might happen in running the recipe and diagnostics script.

Let’s change some settings in the recipe to run a regional pre-processor. We use a text editor called nano to open the recipe file:

  nano recipe_python.yml

Text editor side note

No matter what editor you use, you will need to know where it searches for and saves files. If you start it from the shell, it will (probably) use your current working directory as its default location. We use nano in examples here because it is one of the least complex text editors. Press ctrl + O to save the file, and then ctrl + X to exit nano.

See the recipe_python.yml

# ESMValTool
# recipe_python.yml
---
# See https://docs.esmvaltool.org/en/latest/recipes/recipe_examples.html
# for a description of this recipe.

# See https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/overview.html
# for a description of the recipe format.
---
documentation:
  description: |
    Example recipe that plots a map and timeseries of temperature.

  title: Recipe that runs an example diagnostic written in Python.

  authors:
    - andela_bouwe
    - righi_mattia

  maintainer:
    - schlund_manuel

  references:
    - acknow_project

  projects:
    - esmval
    - c3s-magic

datasets:
  - {dataset: BCC-ESM1, project: CMIP6, exp: historical, ensemble: r1i1p1f1, grid: gn}
  - {dataset: bcc-csm1-1, version: v1, project: CMIP5, exp: historical, ensemble: r1i1p1}

preprocessors:
# See https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/preprocessor.html
# for a description of the preprocessor functions.

  to_degrees_c:
    convert_units:
      units: degrees_C

  annual_mean_amsterdam:
    extract_location:
      location: Amsterdam
      scheme: linear
    annual_statistics:
      operator: mean
    multi_model_statistics:
      statistics:
        - mean
      span: overlap
    convert_units:
      units: degrees_C

  annual_mean_global:
    area_statistics:
      operator: mean
    annual_statistics:
      operator: mean
    convert_units:
      units: degrees_C

diagnostics:

  map:
    description: Global map of temperature in January 2000.
    themes:
      - phys
    realms:
      - atmos
    variables:
      tas:
        mip: Amon
        preprocessor: to_degrees_c
        timerange: 2000/P1M
        caption: |
          Global map of {long_name} in January 2000 according to {dataset}.
    scripts:
      script1:
        script: examples/diagnostic.py
        quickplot:
          plot_type: pcolormesh
          cmap: Reds

  timeseries:
    description: Annual mean temperature in Amsterdam and global mean since 1850.
    themes:
      - phys
    realms:
      - atmos
    variables:
      tas_amsterdam:
        short_name: tas
        mip: Amon
        preprocessor: annual_mean_amsterdam
        timerange: 1850/2000
        caption: Annual mean {long_name} in Amsterdam according to {dataset}.
      tas_global:
        short_name: tas
        mip: Amon
        preprocessor: annual_mean_global
        timerange: 1850/2000
        caption: Annual global mean {long_name} according to {dataset}.
    scripts:
      script1:
        script: examples/diagnostic.py
        quickplot:
          plot_type: plot

Keys and values in recipe settings

The ESMValTool pre-processors cover a broad range of operations on the input data, like time manipulation, area manipulation, land-sea masking, variable derivation, etc. Let’s add the preprocessor extract_region to a new section annual_mean_regional:

preprocessors:
  annual_mean_regional:
    annual_statistics:
      operator: mean
    extract_region:
      start_longitude: -10
      end_longitude: 40
      start_latitude: 27
      end_latitude: 70

Also, we change the projects value esmval to tutorial:

projects:
  - tutorial
  - c3s-magic

Then, we save the file and run the recipe:

  esmvaltool run recipe_python.yml

ValueError: Tag 'tutorial' does not exist in section 'projects' of
esmvaltool/config-references.yml 2020-06-29 18:09:56,641 UTC [46055] INFO If you
have a question or need help, please start a new discussion on
https://github.com/ESMValGroup/ESMValTool/discussions If you suspect this is a
bug, please open an issue on https://github.com/ESMValGroup/ESMValTool/issues To
make it easier to find out what the problem is, please consider attaching the
files run/recipe_*.yml and run/main_log_debug.txt from the output directory.

The values for the keys author, maintainer, projects and references in the recipe should be known by ESMValTool:

A list of ESMValTool author, maintainer, and projects can be found in the config-references.yml.
ESMValTool references in BibTeX format can be found in the ESMValTool/esmvaltool/references directory.

ESMValTool can’t locate the data

You are assisting a colleague with ESMValTool. The colleague replaces the CanESM2 entry in dataset: CanESM2, project: CMIP5 to ACCESS1-3 and runs the recipe. However, ESMValTool encounters an error like:
ERROR   No input files found for variable {'short_name': 'tas', 'mip': 'Amon',
'preprocessor': 'annual_mean_amsterdam', 'variable_group': 'tas_amsterdam',
'diagnostic': 'timeseries', 'dataset': 'ACCESS1-3', 'project': 'CMIP5', 'exp':
'historical', 'ensemble': 'r1i1p1', 'recipe_dataset_index': 1, 'institute':
['CSIRO-BOM'], 'product': ['output1', 'output2'], 'timerange': '1850/2000',
'alias': 'CMIP5', 'original_short_name': 'tas', 'standard_name':
'air_temperature', 'long_name': 'Near-Surface Air Temperature', 'units': 'K',
'modeling_realm': ['atmos'], 'frequency': 'mon', 'start_year': 1850,
'end_year': 2000}
ERROR   Looked for files matching
['tas_Amon_ACCESS1-3_historical_r1i1p1*.nc'], but did not find any existing
input directory
ERROR   Set 'log_level' to 'debug' to get more information
What suggestions would you give the researcher for fixing the error?

Solution

Check user-config.yml to see if the correct directory for input data is introduced

Check the available data, regarding exp, mip, ensemble, start_year, and end_year

Check the variable names in the diagnostics section in the recipe

Check pre-processed data

The setting save_intermediary_cubes in the configuration file can be used to save the pre-processed data. More information about this setting can be found at Configuration.

save_intermediary_cubes

Note that this setting should only be used for debugging, as it significantly slows down the recipe and increases disk usage because a lot of output files need to be stored.

Check diagnostic script path

The result of the pre-processor is passed to the examples/diagnostic.py script, that is introduced in the recipe as:

scripts:
  script1:
    script:
      script: examples/diagnostic.py

The diagnostic scripts are located in the folder diag_scripts in the ESMValTool installation directory <path_to_esmvaltool>. To find where ESMValTool is located on your system, see Installation.

Let’s see what happens if we can change the script path as:

scripts:
  script1:
    script:
      script: diag_scripts/ocean/diagnostic_timeseries.py

  esmvaltool run examples/recipe_python.yml

esmvalcore._task.DiagnosticError: Cannot execute script
'diag_scripts/ocean/diagnostic_timeseries.py'
(~/mambaforge/envs/esmvaltool2.6/lib/python3.10/site-packages/esmvaltool/
diag_scripts/diag_scripts/ocean/diagnostic_timeseries.py):
file does not exist. 2022-10-18 11:42:34,136 UTC [39323] INFO If you have a
question or need help, please start a new discussion on
https://github.com/ESMValGroup/ESMValTool/discussions If you suspect this is a
bug, please open an issue on https://github.com/ESMValGroup/ESMValTool/issues To
make it easier to find out what the problem is, please consider attaching the
files run/recipe_*.yml and run/main_log_debug.txt from the output directory.

The script path should be relative to diag_scripts directory. It means that the script diagnostic_timeseries.py is located in <path_to_esmvaltool>/diag_scripts/ocean/. Alternatively, the script path can be an absolute path. To examine this, we can download the script from the ESMValTool repository:

wget https://raw.githubusercontent.com/ESMValGroup/ESMValTool/main/esmvaltool/\
diag_scripts/ocean/diagnostic_timeseries.py

One way to get the absolute path is to run:

readlink -f diagnostic_timeseries.py

Then we can update the script path :

scripts:
  script1:
    script:
      script: <path_to_script>/diagnostic_timeseries.py

Then, run the recipe again and examine the output to see Run was successful!

Available recipe and diagnostic scripts

ESMValTool provides a broad suite of recipes and diagnostic scripts for different disciplines like atmosphere, climate metrics, future projections, IPCC, land, ocean, ….

Re-running a diagnostic

Look at the main_log.txt file and answer the following question: How to re-run the diagnostic script?
Solution

The main_log.txt file contains information on how to re-run the diagnostic script without re-running the pre-processors:
2020-06-29 20:36:32,844 UTC [52810] INFO    To re-run this diagnostic script, run:
If you run the command in a terminal, you will be able to re-run the diagnostic.

Memory issues

If you run out of memory, try setting max_parallel_tasks to 1 in the configuration file. Then, check the amount of memory you need for that by inspecting the file run/resource_usage.txt in the output directory. Using the number, there you can increase the number of parallel tasks again to a reasonable number for the amount of memory available in your system.

Key Points

There are three different kinds of log files: main_log.txt, and main_log_debug.txt and log.txt.

previous episode

ESMValTool Tutorial

lesson home

Debugging

Overview

Log files

conda environment

Different log files

Solution

Text editor side note

See the recipe_python.yml

Keys and values in recipe settings

ESMValTool can’t locate the data

Solution

Check pre-processed data

save_intermediary_cubes

Check diagnostic script path

Available recipe and diagnostic scripts

Re-running a diagnostic

Solution

Memory issues

Key Points

previous episode

lesson home