Templating with OmegaConfigLoader¶

Introduction to templating¶

Templating, or variable interpolation as it's called in OmegaConf, allows you to reference values from one part of your configuration in another. This helps you avoid duplication and makes your configuration easier to maintain.

When you use templating, you can define a value in one place and reference it multiple times across your configuration files. If you need to change that value, you update it in a single location.

How templating works with the OmegaConfigLoader¶

Parameters¶

Templating for parameters works out of the box if the template values are within the parameter files. It also works if the file containing the template values follows the same config pattern specified for parameters. By default, the config pattern for parameters is: ["parameters*", "parameters*/**", "**/parameters*"]. Suppose you have one parameters file called parameters.yml containing parameters with omegaconf placeholders like this:

model_options:
  test_size: ${data.size}
  random_state: 3

and a file containing the template values called parameters_variables.yml. The file name can be anything as long as it matches the config pattern for parameters:

data:
  size: 0.2

Since both of the file names (parameters.yml and parameters_variables.yml) match the config pattern for parameters, the OmegaConfigLoader will load the files and resolve the placeholders as expected.

Catalog¶

From Kedro 0.18.10 templating also works for catalog files. To enable templating in the catalog, ensure that the template values are within the catalog files, or that the file containing them follows the same config pattern specified for catalogs. By default, the config pattern for catalogs is: ["catalog*", "catalog*/**", "**/catalog*"].

Any template values in the catalog need to start with an underscore _. This is because of how catalog entries are validated. Templated values will neither trigger a key duplication error nor appear in the resulting configuration dictionary.

Suppose you have one catalog file called catalog.yml containing entries with omegaconf placeholders like this:

companies:
  type: ${_pandas.type}
  filepath: data/01_raw/companies.csv

and a file containing the template values called catalog_variables.yml:

_pandas:
  type: pandas.CSVDataset

Since both of the file names (catalog.yml and catalog_variables.yml) match the config pattern for catalogs, the OmegaConfigLoader will load the files and resolve the placeholders as expected.

Other configuration files¶

It's also possible to use variable interpolation in configuration files other than parameters and catalog, such as custom Spark or MLflow configuration. This works in the same way as variable interpolation in parameter files. You can still use the underscore for the templated values if you want, but it's not mandatory like it is for catalog files.

Global variables¶

From Kedro 0.18.13, you can use variable interpolation in your configurations using "globals" with OmegaConfigLoader. The benefit of using globals over regular variable interpolation is that the global variables are shared across different configuration types, such as catalog and parameters. By default, these global variables are assumed to be in files called globals.yml in any of your environments. If you want to configure the naming patterns for the files that contain your global variables, you can do so by overwriting the globals key in config_patterns. You can also bypass the configuration loading to directly set the global variables in OmegaConfigLoader.

Suppose you have global variables located in the file conf/base/globals.yml:

my_global_value: 45
dataset_type:
  csv: pandas.CSVDataset

my_param : "${globals:my_global_value}"

companies:
  filepath: data/01_raw/companies.csv
  type: "${globals:dataset_type.csv}"

my_param: "${globals: nonexistent_global, 23}"

Runtime parameters¶

Kedro allows you to specify runtime parameters for the kedro run command with the --params CLI option. These runtime parameters are added to the KedroContext and merged with parameters from the configuration files to be used in your project's pipelines and nodes. From Kedro 0.18.14, you can use the runtime_params resolver to show that you want to override values of certain keys in your configuration with runtime parameters provided through the CLI option. This resolver can be used across different configuration types, such as parameters, catalog, and more, except for "globals".

Consider this parameters.yml file:

model_options:
  random_state: "${runtime_params:random}"

kedro run --params random=3

companies:
  type: pandas.CSVDataset
  filepath: "${runtime_params:folder, 'data/01_raw'}/companies.csv"

Using globals and runtime_params together¶

As mentioned above, runtime_params are not designed to override globals configuration. This is done to avoid unexplicit overrides and to simplify parameter resolutions. Thus, globals has a single entry point - the yaml file.

You can still use globals and runtime_params by specifying globals as a default value to be used in case the runtime parameter is not passed.

Consider this parameters.yml:

model_options:
  random_state: "${runtime_params:random, ${globals:my_global_value}}"

and this globals.yml file:

my_global_value: 4

This will allow you to pass a runtime parameter named random through the CLI to specify the value of model_options.random_state in your project's parameters:

kedro run --params random=3

If the random parameter is not passed through the CLI --params option with kedro run, then my_global_value from globals.yml is used for the model_options.random_state.

Resolvers¶

Instead of hard-coding values in your configuration files, you can also dynamically compute them using OmegaConf's resolvers functionality. You use resolvers to define custom logic to calculate values of parameters or catalog entries, or inject these values from elsewhere. To use this feature with Kedro, pass a dict of custom resolvers to OmegaConfigLoader through CONFIG_LOADER_ARGS in your project's src/<package_name>/settings.py. The example below illustrates this:

import polars as pl
from datetime import date


def date_today():
    return date.today()


CONFIG_LOADER_ARGS = {
    "custom_resolvers": {
        "add": lambda *my_list: sum(my_list),
        "polars": lambda x: getattr(pl, x),
        "today": date_today,
    }
}

model_options:
  test_size: "${add:1,2,3}"
  random_state: 3

date: "${today:}"

my_polars_dataset:
  type: polars.CSVDataset
  filepath: data/01_raw/my_dataset.csv
  load_args:
    dtypes:
      product_age: "${polars:Float64}"
      group_identifier: "${polars:Utf8}"
    try_parse_dates: true

from omegaconf.resolvers import oc

CONFIG_LOADER_ARGS = {
    "custom_resolvers": {
        "oc.env": oc.env,
    }
}