Running your first recipe
Overview
Teaching: 15 min
Exercises: 15 min
Compatibility: ESMValTool v2.13.0Questions
How to run a recipe?
What happens when I run a recipe?
Objectives
Run an existing ESMValTool recipe
Examine the log information
Navigate the output created by ESMValTool
Make small adjustments to an existing recipe
This episode describes how ESMValTool recipes work, how to run a recipe and how to explore the recipe output. By the end of this episode, you should be able to run your first recipe, look at the recipe output, and make small modifications.
Running an existing recipe
The recipe format has briefly been introduced in the Introduction episode. To see all the recipes that are shipped with ESMValTool, type
esmvaltool recipes list
We will start by running examples/recipe_python.yml
esmvaltool run examples/recipe_python.yml
or if you have the user configuration file in your current directory then
esmvaltool run --config_dir . examples/recipe_python.yml
If everything is okay, you should see that ESMValTool is printing a lot of output to the command line. The final message should be “Run was successful”. The exact output varies depending on your machine, but it should look something like the example log output below.
2025-11-26 09:59:37,412 UTC [3098040] INFO
______________________________________________________________________
_____ ____ __ ____ __ _ _____ _
| ____/ ___|| \/ \ \ / /_ _| |_ _|__ ___ | |
| _| \___ \| |\/| |\ \ / / _` | | | |/ _ \ / _ \| |
| |___ ___) | | | | \ V / (_| | | | | (_) | (_) | |
|_____|____/|_| |_| \_/ \__,_|_| |_|\___/ \___/|_|
______________________________________________________________________
Earth System Model Evaluation Tool
A community tool for the evaluation of Earth system models.
https://esmvaltool.org
The Earth System Model Evaluation Tool (ESMValTool) is a community
diagnostics and performance metrics tool for the evaluation of Earth
System Models (ESMs) that allows for routine comparison of single or
multiple models, either against predecessor versions or against
observations.
Tutorial: https://tutorial.esmvaltool.org
Documentation: https://docs.esmvaltool.org
Contact: esmvaltool-dev@listserv.dfn.de
If you find this software useful for your research, please cite it using
https://doi.org/10.5281/zenodo.3387139 for ESMValCore or
https://doi.org/10.5281/zenodo.3401363 for ESMValTool or
any of the reference papers listed at https://esmvaltool.org/references/.
Have fun!
2025-11-26 09:59:37,413 UTC [3098040] INFO Package versions
2025-11-26 09:59:37,413 UTC [3098040] INFO ----------------
2025-11-26 09:59:37,413 UTC [3098040] INFO ESMValCore: 2.13.0
2025-11-26 09:59:37,413 UTC [3098040] INFO ESMValTool: 2.13.0
2025-11-26 09:59:37,413 UTC [3098040] INFO ----------------
2025-11-26 09:59:37,413 UTC [3098040] INFO Reading configuration files from:
/sw/spack-levante/miniforge3-25.9.1-0-Linux-x86_64-oqcirx/lib/python3.12/site-packages/esmvalcore/config/configurations/defaults (defaults)
/home/b/b383829/.config/esmvaltool (default user configuration directory)
/home/b/b383829/esmvaltool_tutorial (command line argument)
2025-11-26 09:59:37,413 UTC [3098040] INFO Writing program log files to:
/home/b/b383829/esmvaltool_output/recipe_python_20251126_095937/run/main_log.txt
/home/b/b383829/esmvaltool_output/recipe_python_20251126_095937/run/main_log_debug.txt
/home/b/b383829/esmvaltool_output/recipe_python_20251126_095937/run/cmor_log.txt
2025-11-26 09:59:39,256 UTC [3098040] INFO Starting the Earth System Model Evaluation Tool at time: 2025-11-26 09:59:39 UTC
2025-11-26 09:59:39,256 UTC [3098040] INFO ----------------------------------------------------------------------
2025-11-26 09:59:39,256 UTC [3098040] INFO RECIPE = /sw/spack-levante/miniforge3-25.9.1-0-Linux-x86_64-oqcirx/lib/python3.12/site-packages/esmvaltool/recipes/examples/recipe_python.yml
2025-11-26 09:59:39,256 UTC [3098040] INFO RUNDIR = /home/b/b383829/esmvaltool_output/recipe_python_20251126_095937/run
2025-11-26 09:59:39,257 UTC [3098040] INFO WORKDIR = /home/b/b383829/esmvaltool_output/recipe_python_20251126_095937/work
2025-11-26 09:59:39,257 UTC [3098040] INFO PREPROCDIR = /home/b/b383829/esmvaltool_output/recipe_python_20251126_095937/preproc
2025-11-26 09:59:39,257 UTC [3098040] INFO PLOTDIR = /home/b/b383829/esmvaltool_output/recipe_python_20251126_095937/plots
2025-11-26 09:59:39,257 UTC [3098040] INFO ----------------------------------------------------------------------
2025-11-26 09:59:39,257 UTC [3098040] INFO Running tasks using at most 256 processes
2025-11-26 09:59:39,257 UTC [3098040] INFO If your system hangs during execution, it may not have enough memory for keeping this number of tasks in memory.
2025-11-26 09:59:39,257 UTC [3098040] INFO If you experience memory problems, try reducing 'max_parallel_tasks' in your configuration.
2025-11-26 09:59:39,367 UTC [3098040] INFO Creating tasks from recipe
2025-11-26 09:59:39,367 UTC [3098040] INFO Creating tasks for diagnostic map
2025-11-26 09:59:39,367 UTC [3098040] INFO Creating diagnostic task map/script1
2025-11-26 09:59:39,368 UTC [3098040] INFO Creating preprocessor task map/tas
2025-11-26 09:59:39,368 UTC [3098040] INFO Creating preprocessor 'to_degrees_c' task for variable 'tas'
2025-11-26 09:59:39,373 UTC [3098040] INFO Found input files for Dataset: tas, Amon, CMIP6, BCC-ESM1, CMIP, historical, r1i1p1f1, gn, v20181214
2025-11-26 09:59:39,376 UTC [3098040] INFO Found input files for Dataset: tas, Amon, CMIP5, bcc-csm1-1, historical, r1i1p1, v1
2025-11-26 09:59:39,377 UTC [3098040] INFO PreprocessingTask map/tas created.
2025-11-26 09:59:39,378 UTC [3098040] INFO Creating tasks for diagnostic timeseries
2025-11-26 09:59:39,378 UTC [3098040] INFO Creating diagnostic task timeseries/script1
2025-11-26 09:59:39,378 UTC [3098040] INFO Creating preprocessor task timeseries/tas_amsterdam
2025-11-26 09:59:39,378 UTC [3098040] INFO Creating preprocessor 'annual_mean_amsterdam' task for variable 'tas_amsterdam'
2025-11-26 09:59:39,385 UTC [3098040] INFO Found input files for Dataset: tas, Amon, CMIP6, BCC-ESM1, CMIP, historical, r1i1p1f1, gn, v20181214
2025-11-26 09:59:39,390 UTC [3098040] INFO Found input files for Dataset: tas, Amon, CMIP5, bcc-csm1-1, historical, r1i1p1, v1
2025-11-26 09:59:39,391 UTC [3098040] INFO PreprocessingTask timeseries/tas_amsterdam created.
2025-11-26 09:59:39,391 UTC [3098040] INFO Creating preprocessor task timeseries/tas_global
2025-11-26 09:59:39,391 UTC [3098040] INFO Creating preprocessor 'annual_mean_global' task for variable 'tas_global'
2025-11-26 09:59:39,400 UTC [3098040] INFO Found input files for Dataset: tas, Amon, CMIP6, BCC-ESM1, CMIP, historical, r1i1p1f1, gn, v20181214, supplementaries: areacella, fx, 1pctCO2, v20190613
2025-11-26 09:59:39,407 UTC [3098040] INFO Found input files for Dataset: tas, Amon, CMIP5, bcc-csm1-1, historical, r1i1p1, v1, supplementaries: areacella, fx, r0i0p0
2025-11-26 09:59:39,408 UTC [3098040] INFO PreprocessingTask timeseries/tas_global created.
2025-11-26 09:59:39,409 UTC [3098040] INFO These tasks will be executed: timeseries/tas_global, map/tas, map/script1, timeseries/tas_amsterdam, timeseries/script1
2025-11-26 09:59:39,421 UTC [3098040] INFO Wrote recipe with version numbers and wildcards to:
file:///home/b/b383829/esmvaltool_output/recipe_python_20251126_095937/run/recipe_python_filled.yml
2025-11-26 09:59:39,421 UTC [3098040] INFO Using Dask threaded scheduler. The distributed scheduler is recommended, please read https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html#dask-configuration how to use a distributed scheduler.
2025-11-26 09:59:39,421 UTC [3098040] INFO Running 5 tasks using 5 processes
2025-11-26 09:59:39,421 UTC [3098040] INFO Using the threaded Dask scheduler with 85 worker threads per preprocessing task. See https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html#f5 for more information.
2025-11-26 09:59:39,597 UTC [3098174] INFO Starting task map/tas in process [3098174]
2025-11-26 09:59:39,598 UTC [3098175] INFO Starting task timeseries/tas_amsterdam in process [3098175]
2025-11-26 09:59:39,599 UTC [3098176] INFO Starting task timeseries/tas_global in process [3098176]
2025-11-26 09:59:39,693 UTC [3098040] INFO Progress: 3 tasks running, 2 tasks waiting for ancestors, 0/5 done
2025-11-26 09:59:40,241 UTC [3098175] INFO Extracting data for Amsterdam, Noord-Holland, Nederland (52.3730796 °N, 4.8924534 °E)
2025-11-26 09:59:40,410 UTC [3098174] INFO Computing and saving data for preprocessing task map/tas
2025-11-26 09:59:41,229 UTC [3098176] INFO Computing and saving data for preprocessing task timeseries/tas_global
2025-11-26 09:59:41,253 UTC [3098175] INFO Extracting data for Amsterdam, Noord-Holland, Nederland (52.3730796 °N, 4.8924534 °E)
2025-11-26 09:59:41,415 UTC [3098174] INFO map/tas [########################################] | 100% Completed | 1.00 s
2025-11-26 09:59:41,420 UTC [3098174] INFO Successfully completed task map/tas (priority 1) in 0:00:01.822711
2025-11-26 09:59:41,519 UTC [3098175] INFO Generated PreprocessorFile: /home/b/b383829/esmvaltool_output/recipe_python_20251126_095937/preproc/timeseries/tas_amsterdam/MultiModelMean_historical_Amon_tas_1850-2000.nc
2025-11-26 09:59:41,595 UTC [3098040] INFO Progress: 2 tasks running, 2 tasks waiting for ancestors, 1/5 done
2025-11-26 09:59:41,601 UTC [3098177] INFO Starting task map/script1 in process [3098177]
2025-11-26 09:59:41,608 UTC [3098177] INFO Running command ['/sw/spack-levante/miniforge3-25.9.1-0-Linux-x86_64-oqcirx/bin/python', '/sw/spack-levante/miniforge3-25.9.1-0-Linux-x86_64-oqcirx/lib/python3.12/site-packages/esmvaltool/diag_scripts/examples/diagnostic.py', '/home/b/b383829/esmvaltool_output/recipe_python_20251126_095937/run/map/script1/settings.yml']
2025-11-26 09:59:41,609 UTC [3098177] INFO Writing output to /home/b/b383829/esmvaltool_output/recipe_python_20251126_095937/work/map/script1
2025-11-26 09:59:41,609 UTC [3098177] INFO Writing plots to /home/b/b383829/esmvaltool_output/recipe_python_20251126_095937/plots/map/script1
2025-11-26 09:59:41,609 UTC [3098177] INFO Writing log to /home/b/b383829/esmvaltool_output/recipe_python_20251126_095937/run/map/script1/log.txt
2025-11-26 09:59:41,609 UTC [3098177] INFO To re-run this diagnostic script, run:
cd /home/b/b383829/esmvaltool_output/recipe_python_20251126_095937/run/map/script1; MPLBACKEND="Agg" /sw/spack-levante/miniforge3-25.9.1-0-Linux-x86_64-oqcirx/bin/python /sw/spack-levante/miniforge3-25.9.1-0-Linux-x86_64-oqcirx/lib/python3.12/site-packages/esmvaltool/diag_scripts/examples/diagnostic.py /home/b/b383829/esmvaltool_output/recipe_python_20251126_095937/run/map/script1/settings.yml
2025-11-26 09:59:41,695 UTC [3098040] INFO Progress: 3 tasks running, 1 tasks waiting for ancestors, 1/5 done
2025-11-26 09:59:41,780 UTC [3098175] INFO Computing and saving data for preprocessing task timeseries/tas_amsterdam
2025-11-26 09:59:42,293 UTC [3098176] INFO timeseries/tas_global [########################################] | 100% Completed | 1.00 s
2025-11-26 09:59:42,305 UTC [3098176] INFO Successfully completed task timeseries/tas_global (priority 4) in 0:00:02.705760
2025-11-26 09:59:42,496 UTC [3098040] INFO Progress: 2 tasks running, 1 tasks waiting for ancestors, 2/5 done
2025-11-26 09:59:42,849 UTC [3098175] INFO timeseries/tas_amsterdam [########################################] | 100% Completed | 1.00 s
2025-11-26 09:59:42,859 UTC [3098175] INFO Successfully completed task timeseries/tas_amsterdam (priority 3) in 0:00:03.261031
2025-11-26 09:59:43,191 UTC [3098040] INFO Progress: 1 tasks running, 1 tasks waiting for ancestors, 3/5 done
2025-11-26 09:59:43,291 UTC [3098040] INFO Progress: 2 tasks running, 0 tasks waiting for ancestors, 3/5 done
2025-11-26 09:59:43,341 UTC [3098178] INFO Starting task timeseries/script1 in process [3098178]
2025-11-26 09:59:43,348 UTC [3098178] INFO Running command ['/sw/spack-levante/miniforge3-25.9.1-0-Linux-x86_64-oqcirx/bin/python', '/sw/spack-levante/miniforge3-25.9.1-0-Linux-x86_64-oqcirx/lib/python3.12/site-packages/esmvaltool/diag_scripts/examples/diagnostic.py', '/home/b/b383829/esmvaltool_output/recipe_python_20251126_095937/run/timeseries/script1/settings.yml']
2025-11-26 09:59:43,348 UTC [3098178] INFO Writing output to /home/b/b383829/esmvaltool_output/recipe_python_20251126_095937/work/timeseries/script1
2025-11-26 09:59:43,348 UTC [3098178] INFO Writing plots to /home/b/b383829/esmvaltool_output/recipe_python_20251126_095937/plots/timeseries/script1
2025-11-26 09:59:43,348 UTC [3098178] INFO Writing log to /home/b/b383829/esmvaltool_output/recipe_python_20251126_095937/run/timeseries/script1/log.txt
2025-11-26 09:59:43,349 UTC [3098178] INFO To re-run this diagnostic script, run:
cd /home/b/b383829/esmvaltool_output/recipe_python_20251126_095937/run/timeseries/script1; MPLBACKEND="Agg" /sw/spack-levante/miniforge3-25.9.1-0-Linux-x86_64-oqcirx/bin/python /sw/spack-levante/miniforge3-25.9.1-0-Linux-x86_64-oqcirx/lib/python3.12/site-packages/esmvaltool/diag_scripts/examples/diagnostic.py /home/b/b383829/esmvaltool_output/recipe_python_20251126_095937/run/timeseries/script1/settings.yml
2025-11-26 09:59:49,356 UTC [3098177] INFO Successfully completed task map/script1 (priority 0) in 0:00:07.755227
2025-11-26 09:59:49,495 UTC [3098040] INFO Progress: 1 tasks running, 0 tasks waiting for ancestors, 4/5 done
2025-11-26 09:59:51,644 UTC [3098178] INFO Maximum memory used (estimate): 0.5 GB
2025-11-26 09:59:51,645 UTC [3098178] INFO Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur.
2025-11-26 09:59:52,193 UTC [3098178] INFO Successfully completed task timeseries/script1 (priority 2) in 0:00:08.852388
2025-11-26 09:59:52,297 UTC [3098040] INFO Progress: 0 tasks running, 0 tasks waiting for ancestors, 5/5 done
2025-11-26 09:59:52,297 UTC [3098040] INFO Successfully completed all tasks.
2025-11-26 09:59:52,368 UTC [3098040] INFO Wrote recipe with version numbers and wildcards to:
file:///home/b/b383829/esmvaltool_output/recipe_python_20251126_095937/run/recipe_python_filled.yml
2025-11-26 09:59:52,601 UTC [3098040] INFO Wrote recipe output to:
file:///home/b/b383829/esmvaltool_output/recipe_python_20251126_095937/index.html
2025-11-26 09:59:52,601 UTC [3098040] INFO It looks like you are connected to a remote machine via SSH. To show the output html file, you can try the following command on your local machine:
server=10.1.1.50 && port=31415 && ssh -t -L ${port}:localhost:${port} b383829@${server} /sw/spack-levante/miniforge3-25.9.1-0-Linux-x86_64-oqcirx/bin/python -m http.server ${port} -d /home/b/b383829/esmvaltool_output/recipe_python_20251126_095937
Then visit http://localhost:31415 in your browser
2025-11-26 09:59:52,601 UTC [3098040] INFO If the port 31415 is already in use, you can replace it with any other free one (e.g., 12789). If you are connected through a jump host, replace the server IP address 10.1.1.50 with your SSH server name
2025-11-26 09:59:52,601 UTC [3098040] INFO Ending the Earth System Model Evaluation Tool at time: 2025-11-26 09:59:52 UTC
2025-11-26 09:59:52,601 UTC [3098040] INFO Time for running the recipe was: 0:00:13.345216
2025-11-26 09:59:53,210 UTC [3098040] INFO Maximum memory used (estimate): 3.7 GB
2025-11-26 09:59:53,211 UTC [3098040] INFO Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur.
2025-11-26 09:59:53,214 UTC [3098040] WARNING Input data is not (fully) CMOR-compliant, see /home/b/b383829/esmvaltool_output/recipe_python_20251126_095937/run/cmor_log.txt for details
2025-11-26 09:59:53,214 UTC [3098040] INFO Run was successful
Pro tip: ESMValTool search paths
You might wonder how ESMValTool was able find the recipe file, even though it’s not in your working directory. All the recipe paths printed from
esmvaltool recipes listare relative to ESMValTool’s installation location. This is where ESMValTool will look if it cannot find the file by following the path from your working directory.
Investigating the log messages
Let’s dissect what’s happening here.
Output files and directories
After the banner and general information, the output starts with some important locations.
- Did ESMValTool use the right config file?
- What is the path to the example recipe?
- What is the main output folder generated by ESMValTool?
- Can you guess what the different output directories are for?
- ESMValTool creates two log files. What is the difference?
Answers
- The config file should be the one we edited in the previous episode, something like
/home/<username>/.config/esmvaltool/config-user.ymlor~/esmvaltool_tutorial/config-user.yml.- ESMValTool found the recipe in its installation directory, something like
/home/users/username/mambaforge/envs/esmvaltool/bin/esmvaltool/recipes/examples/or if you are using a pre-installed module on a server, something like/apps/jasmin/community/esmvaltool/ESMValTool_<version> /esmvaltool/recipes/examples/recipe_python.yml, where<version>is the latest release.- ESMValTool creates a time-stamped output directory for every run. In this case, it should be something like
recipe_python_YYYYMMDD_HHMMSS. This folder is made inside the output directory specified in the previous episode:~/esmvaltool_tutorial/esmvaltool_output.- There should be four output folders:
plots/: this is where output figures are stored.preproc/: this is where pre-processed data are stored.run/: this is where esmvaltool stores general information about the run, such as log messages and a copy of the recipe file.work/: this is where output files (not figures) are stored.- The log files are:
main_log.txtis a copy of the command-line outputmain_log_debug.txtcontains more detailed information that may be useful for debugging.
Debugging: No ‘preproc’ directory?
If you’re missing the preproc directory, then your
config-user.ymlfile has the valueremove_preproc_dirset totrue(this is used to save disk space). Please set this value tofalseand run the recipe again.
After the output locations, there are two main sections that can be distinguished in the log messages:
- Creating tasks
- Executing tasks
Analyse the tasks
List all the tasks that ESMValTool is executing for this recipe. Can you guess what this recipe does?
Answer
Just after all the ‘creating tasks’ and before ‘executing tasks’, we find the following line in the output:
[3098040] INFO These tasks will be executed: timeseries/tas_global, map/tas, map/script1, timeseries/tas_amsterdam, timeseries/script1So there are three tasks related to timeseries: global temperature, Amsterdam temperature, and a script (tas: near-surface air temperature). And then there are two tasks related to a map: something with temperature, and again a script.
Examining the recipe file
To get more insight into what is happening, we will have a look at the recipe file itself. Use the following command to copy the recipe to your working directory
esmvaltool recipes get examples/recipe_python.yml
Now you should see the recipe file in your working directory (type ls to
verify). Use the nano editor to open this file:
nano recipe_python.yml
For reference, you can also view the recipe by unfolding the box below.
recipe_python.yml
# ESMValTool # recipe_python.yml # # See https://docs.esmvaltool.org/en/latest/recipes/recipe_examples.html # for a description of this recipe. # # See https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/overview.html # for a description of the recipe format. --- documentation: description: | Example recipe that plots a map and timeseries of temperature. title: Recipe that runs an example diagnostic written in Python. authors: - andela_bouwe - righi_mattia maintainer: - schlund_manuel references: - acknow_project projects: - esmval - c3s-magic datasets: - {dataset: BCC-ESM1, project: CMIP6, exp: historical, ensemble: r1i1p1f1, grid: gn} - {dataset: bcc-csm1-1, project: CMIP5, exp: historical, ensemble: r1i1p1} preprocessors: # See https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/preprocessor.html # for a description of the preprocessor functions. to_degrees_c: convert_units: units: degrees_C annual_mean_amsterdam: extract_location: location: Amsterdam scheme: linear annual_statistics: operator: mean multi_model_statistics: statistics: - mean span: overlap convert_units: units: degrees_C annual_mean_global: area_statistics: operator: mean annual_statistics: operator: mean convert_units: units: degrees_C diagnostics: map: description: Global map of temperature in January 2000. themes: - phys realms: - atmos variables: tas: mip: Amon preprocessor: to_degrees_c timerange: 2000/P1M caption: | Global map of {long_name} in January 2000 according to {dataset}. scripts: script1: script: examples/diagnostic.py quickplot: plot_type: pcolormesh cmap: Reds timeseries: description: Annual mean temperature in Amsterdam and global mean since 1850. themes: - phys realms: - atmos variables: tas_amsterdam: short_name: tas mip: Amon preprocessor: annual_mean_amsterdam timerange: 1850/2000 caption: Annual mean {long_name} in Amsterdam according to {dataset}. tas_global: short_name: tas mip: Amon preprocessor: annual_mean_global timerange: 1850/2000 caption: Annual global mean {long_name} according to {dataset}. scripts: script1: script: examples/diagnostic.py quickplot: plot_type: plot
Do you recognize the basic recipe structure that was introduced in episode 1?
- Documentation with relevant (citation) information
- Datasets that should be analysed
- Preprocessors groups of common preprocessing steps
- Diagnostics scripts performing more specific evaluation steps
Analyse the recipe
Try to answer the following questions:
- Who wrote this recipe?
- Who should be approached if there is a problem with this recipe?
- How many datasets are analyzed?
- What does the preprocessor called
annual_mean_globaldo?- Which script is applied for the diagnostic called
map?- Can you link specific lines in the recipe to the tasks that we saw before?
- How is the location of the city specified?
- How is the temporal range of the data specified?
Answers
- The example recipe is written by Bouwe Andela and Mattia Righi.
- Manuel Schlund is listed as the maintainer of this recipe.
- Two datasets are analysed:
- CMIP6 data from the model BCC-ESM1
- CMIP5 data from the model bcc-csm1-1
- The preprocessor
annual_mean_globalcomputes an area mean as well as annual means- The diagnostic called
mapexecutes a script referred to asscript1. This is a python script namedexamples/diagnostic.py- There are two diagnostics:
mapandtimeseries. Under the diagnosticmapwe find two tasks:
- a preprocessor task called
tas, applying the preprocessor calledto_degrees_cto the variabletas.- a diagnostic task called
script1, applying the scriptexamples/diagnostic.pyto the preprocessed data (map/tas).Under the diagnostic
timeserieswe find three tasks:
- a preprocessor task called
tas_amsterdam, applying the preprocessor calledannual_mean_amsterdamto the variabletas.- a preprocessor task called
tas_global, applying the preprocessor calledannual_mean_globalto the variabletas.- a diagnostic task called
script1, applying the scriptexamples/diagnostic.pyto the preprocessed data (timeseries/tas_globalandtimeseries/tas_amsterdam).- The
extract_locationpreprocessor is used to get data for a specific location here. ESMValTool interpolates to the location based on the chosen scheme. Can you tell the scheme used here? For more ways to extract areas, see the Area operations page.- The
timerangetag is used to extract data from a specific time period here. The start time is01/01/2000and the span of time to calculate means is1 Monthgiven byP1M. For more options on how to specify time ranges, see the timerange documentation.
Pro tip: short names and variable groups
The preprocessor tasks in ESMValTool are called ‘variable groups’. For the diagnostic
timeseries, we have two variable groups:tas_amsterdamandtas_global. Both of them operate on the variabletas(as indicated by theshort_name), but they apply different preprocessors. For the diagnosticmapthe variable group itself is namedtas, and you’ll notice that we do not explicitly provide theshort_name. This is a shorthand built into ESMValTool.
Output files
Have another look at the output directory created by the ESMValTool run.
Which files/folders are created by each task?
Answer
- map/tas: creates
/preproc/map/tas, which contains preprocessed data for each of the input datasets, a file calledmetadata.ymldescribing the contents of these datasets and provenance information in the form of.xmlfiles.- timeseries/tas_global: creates
/preproc/timeseries/tas_global, which contains preprocessed data for each of the input datasets, ametadata.ymlfile and provenance information in the form of.xmlfiles.- timeseries/tas_amsterdam: creates
/preproc/timeseries/tas_amsterdam, which contains preprocessed data for each of the input datasets, plus a combinedMultiModelMean, ametadata.ymlfile and provenance files.- map/script1: creates
/run/map/script1with general information and a log of the diagnostic script run. It also creates/plots/map/script1/and/work/map/script1, which contain output figures and output datasets, respectively. For each output file, there is also corresponding provenance information in the form of.xml,.bibtexand.txtfiles.- timeseries/script1: creates
/run/timeseries/script1with general information and a log of the diagnostic script run. It also creates/plots/timeseries/script1and/work/timeseries/script1, which contain output figures and output datasets, respectively. For each output file, there is also corresponding provenance information in the form of.xml,.bibtexand.txtfiles.
Pro tip: diagnostic logs
When you run ESMValTool, any log messages from the diagnostic script are not printed on the terminal. But they are written to the
log.txtfiles in the folder/run/<diag_name>/log.txt.ESMValTool does print a command that can be used to re-run a diagnostic script. When you use this the output will be printed to the command line.
Modifying the example recipe
Let’s make a small modification to the example recipe. Notice that now that you have copied and edited the recipe, you can use
esmvaltool run recipe_python.yml
to refer to your local file rather than the default version shipped with ESMValTool.
Change your location
Modify and run the recipe to analyse the temperature for your own location.
Solution
In principle, you only have to modify the location in the preprocessor called
annual_mean_amsterdam. However, it is good practice to also replace all instances ofamsterdamwith the correct name of your location. Otherwise the log messages and output will be confusing. You are free to modify the names of preprocessors or diagnostics.In the
difffile below you will see the changes we have made to the file. The top 2 lines are the filenames and the lines like@@ -39,9 +39,9 @@represent the line numbers in the original and modified file, respectively. For more info on this format, see here.--- recipe_python.yml +++ recipe_python_london.yml @@ -39,9 +39,9 @@ convert_units: units: degrees_C - annual_mean_amsterdam: + annual_mean_london: extract_location: - location: Amsterdam + location: London scheme: linear annual_statistics: operator: mean @@ -83,7 +83,7 @@ cmap: Reds timeseries: - description: Annual mean temperature in Amsterdam and global mean since 1850. + description: Annual mean temperature in London and global mean since 1850. themes: - phys realms: @@ -92,9 +92,9 @@ tas_amsterdam: short_name: tas mip: Amon - preprocessor: annual_mean_amsterdam + preprocessor: annual_mean_london timerange: 1850/2000 - caption: Annual mean {long_name} in Amsterdam according to {dataset}. + caption: Annual mean {long_name} in London according to {dataset}. tas_global: short_name: tas mip: Amon
Key Points
ESMValTool recipes work ‘out of the box’ (if input data is available)
There are strong links between the recipe, log file, and output folders
Recipes can easily be modified to re-use existing code for your own use case