Snakemake Rules

Snakemake Rules#

Warning

Outdated Documentation

This page is currently not up to date and reflects an older commit (e6f69605067650bd949dfd66ae139d4e2ffa02a0) of the project. The content may be inaccurate or missing recent changes.

Updates are planned and will appear in future releases. Please refer to the project repository for the latest information.

This chapter documents the scripts executed via the snakemake routine.

The workflow management tool snakemake works by defining different rules which require input files and produce outputfiles. Snakemake automatically determines which rules have to be executed to produce a desired output file. In this chapter, the different rules are explained. All rules are defined in the :file:snakefile in the top level of the repository. All rules execute python files. They are documented in the next chapter and linked here.

downloadCordex#

Output: config["data_dir"] + {domain}/original/{model}/{climate\_variable}\_m{model}\_rcp{rcp}\_{year}\_{toTemporal}.nc

Script: download_cordexData.py

  • downloads cordex data using wget scripts

renameCordex#

Output: config["data_dir"] + '/{domain}/original/{model}/{climate\_variable}\_m{model}\_rcp{rcp}\_{year}\_{toTemporal}.nc

Script: rename_cordexData.py

  • only used if downloadCordex is False in config

  • renames existing cordex data to be used in cd2es

downloadERA5#

Output: config["data_dir"] + /{domain}/observed/{climate\_variable\_ERA5}\_ERA5\_{year}.nc

Script: download_era5Data.py

  • downloads ERA5 reanalysis data for regression and bias adaption

  • you need to be registered at the copernicus climate data store and your credits must either be in a .cdsapirc (see Getting started 7.) or you must enable giveCdsCredentialsInConfig in config and provide your data in the config

build_topology#

Output: results/{domain}/topo.nc

Script: build_topo.py

  • build a topology file which is used to map all cordex climate files to

  • borders are chosen via the choice of domain

  • the spatial resolution can be influenced by changing the geography/x_size or y_size values in the config

process_cordex_data#

Input: results/{domain}/topo.nc, config["data_dir"] + {domain}/original/{model}/{climate\_variable}\_m{model}\_rcp{rcp}\_{year}\_{toTemporal}.nc

Output: config["data_dir"]+/{domain}/remap/{model}/{climate\_variable}\_m{model}\_rcp{rcp}\_{year}\_{toTemporal}.nc

Script: process_cordex_data.py

  • takes the topology file and a cordex climate file as input

  • remaps cordex data from rotated coordinate system to rectangular coordinate system, interpolates if necessary and removes the 29.02. in leap years

  • cd2es cannot work with cordex data where time is given in date time 360 * you need to use different models

calculate_quantiles#

Input: config["data_dir"] + {domain}/remap/{model}/{climate_variable}_m{model}_rcp{rcp}_{year}_{toTemporal}.nc

Output: config["data_dir"] + {domain}/remap/{model}/{climate_variable}_m{model}_rcp{rcp}_{toTemporal}_quantiles.nc

Script: calculate_quantiles.py

  • takes historic model data for all years given in the config under bias_adaption between yearHistStart and yearHistEnd as input

  • calculates the 100 quantiles for each grid point for the given climate variable (necessary for bias adaption)

calculate_quantiles_observed#

Input: config["data_dir"] + {domain}/observed/{climate_variable}_observed_{year}_{toTemporal}.nc

Output: config["data_dir"] + {domain}/observed/{climate_variable}_observed_{toTemporal}_quantiles.nc

Script: calculate_quantiles.py

  • same as calculate_quantiles, only for reanalysis data

bias_adopt#

Input: config["data_dir"] + {domain}/remap/{model}/{climate_variable}_m{model}_rcp{rcp}_{toTemporal}_quantiles.nc, config["data_dir"] + {domain}/observed/{climate_variable}_observed_{toTemporal}_quantiles.nc, config["data_dir"]+/{domain}/remap/{model}/{climate\_variable}\_m{model}\_rcp{rcp}\_{year}\_{toTemporal}.nc

Output: results/{domain}/{technology}\_m{model}\_rcp{rcp}\_{year}\_ba.csv

Script: biasAdopt.py

  • takes processed cordex files and historic climate model and reanalysis quantile files as input

  • bias adopts the climate file using quantile delta mapping

  • used to correct biases in climate data

  • more information: https://doi.org/10.1175/JCLI-D-14-00754.1

build_{technology}_profile#

Input: config["data_dir"]+/{domain}/remap/{model}/sfcWind_m{model}_rcp{rcp}_{year}_1h.nc

Output: results/{domain}/{technology}\_m{model}\_rcp{rcp}\_{year}\_notAgg.nc

Script: build_solar_profile.py, build_wind_profile.py, build_csp_profile.py, build_hydro_profile.py, build_demand_profile.py, build_tppOT_profile.py, build_tppCL_profile.py

  • builds capacity factor time series for different technologies ‘csp’, ‘pv’, ‘wind’ (onshore), ‘wind_offshore’, ‘hydro_production’ (only Europe), ‘demand’ (only Europe) and tpp(CoolingType)_p{Plant} with cooling type in [CL, OT] (closed loop, once through * efficiency of thermal power plants)

  • takes a processed cordex (or ERA5) climate data file as input

  • uses parameters given in config

  • hydro requires additional data on power plant location * at the moment only data for Europe is available, therefore it does not work for other continents

  • all rules appear twice, once for cordex data, once for ERA5

demand_regression#

Input: results/{{domain}}/tas_m{{model}}_rcp{{rcp}}_{year}_d_aggregated.csv

Output: results/{domain}/demand_regression_m{model}_rcp{rcp}.csv

Script: demand_regression.py

  • takes already aggregated cordex temperature files as input

  • calculates quadratic regressions for historic demand data vs. historic temperatures

  • requires historical demand data * at the moment only data for Europe is available, therefore it does not work for other continents

  • if demandRegression/makePlots is true in the config, plots of the regression functions are created in the folder :file:demandPlots

calibrate_hydro_power plants#

Input:

config["data_dir"]+/{{domain}}/remap/{{model}}/mrro_m{{model}}_rcp{{rcp}}_{year}_d.nc

Output: results/{domain}/power plant_database_m{model}_rcp{rcp}.csv

Script: hydro_calibration.py

  • takes processed cordex runoff (mrro) files as input

  • uses the power plant database for europe to calculate average run offs in the past and match it to the installed capacity of the existing hydro power plants

calculate_turbine_curve#

Output: results/{domain}/turbine_curve_{onOffshore}.txt

Script: calculate_turbine_curve.py

  • calculated an on* or offshore turbine curve based on the information given in the config below on/offwind

  • the curve is calculated for each wind speed between 0 and 40 m/s in 0.01 m/s steps

  • to simplify later calculations, the resulting curve is fitted with an :math:exp(-a \cdot x^2) \cdot (b \cdot x^3+c \cdot x^2+d \cdot x) function

  • the fit parameters are saved in the output file and used for further calculations

calculate_OT_flow#

Input: config["data_dir"]+/{{domain}}/remap/{{model}}/mrro_m{{model}}_rcp{{rcp}}_{year}_d.nc

Output: results/{domain}/OT_flow_m{model}_rcp{rcp}.nc

Script: calcOTFlow

  • takes processed cordex runoff (mrro) files as input

  • calculates historic water availability for once through power plants

aggregate_files_dataDirect, aggregate_tech_files etc.#

Script: aggregateFiles.py

  • geographically aggregates climate data files or capacity factor files either to countries (if no custom bus map use is specified in config) or to the geography given in the custom bus map

build_backbone_input#

Input: various time series files

Output: results/{domain}/backbone\_m{model}\_rcp{rcp}\_{year}.xlsx

Script: build_backbone_input.py

Contents