config

Interface to check & load the configurations

This class provides a lightweight wrapper for OmegaConf and allows checking and loading configurations. It supports loading configurations from various sources such as files, AWS S3, and config strings. The class also provides methods for saving configurations and setting default values for missing config arguments.

classmethod Config.load(config: str | dict | DictConfig | OmegaConf | Path)

Load the configuration for the etl.

Parameters:

config (Union[str, dict, OmegaConf]) –

The configuration for the etl. - str or Path: This could have several cases:

• Path to the config file.

• S3 path to the config file.

• Config string. This is similar to loading a yaml file with open().

• dict: Config dictionary.

• OmegaConf: Config object.

Returns:

The loaded configuration.

Raises:

• ValueError – If the provided config is not a valid path or S3 path.

• TypeError – If the provided config is not of type str, dict, or OmegaConf.

classmethod Config.save(config, path: str | Path)

Saves the configuration to a specified path.

Parameters:

• config – The configuration to be saved.

• path (Union[str, Path]) – The path where the configuration should be saved.

Raises:

ValueError – If the provided path is not a valid S3 path.

classmethod Config.default(emr: bool = False)

Fill the missing config with default values.

Parameters:

emr (bool, optional) – Flag indicating whether the config is for EMR. Defaults to False.

Returns:

Default configuration dictionary.

Return type:

dict

classmethod Config.set_default(config: OmegaConf, emr: bool = False)

Sets the missing config arguments with default values.

Parameters:

• config (OmegaConf) – The configuration object to merge with default values.

• emr (bool, optional) – Whether to use EMR configuration. Defaults to False.

Returns:

The merged configuration object.

Return type:

OmegaConf