Structuring the Pipeline Core

Now that we have the components required for training a model, it’s time to put them together. We combine these components using what we call the core of the pipeline. The core imports the necessary library modules and calls them in the appropriate order.

The pipeline has several functionalities:

• Read command-line user input.

• Read user configuration and build a pipeline DAG.

• Topologically sort the DAG.

• Traverse the DAG and invoke the function corresponding to each vertex.

• Log debug and error messages.

Here’s how the project configuration file looks.