Python Models

Overview of Python models

dbt allows data engineers and analysts to use a common tool and language i.e., SQL. However, since data scientists primarily work with Python to create their machine learning models, dbt introduced a new feature: Python models. They come with some downsides:

• They’re slower and more expensive than SQL models.

• Python comes with many libraries and functions, which have a lot of capabilities but imply a steep learning curve.

• This is still a new and experimental feature. There is not much documentation on Python models, and debugging can be difficult, especially because error messages are not always explicit.

• Many APIs are used to run Python models with BigQuery: Dataproc, Cloud Storage, VPC networks, etc. A basic understanding of all those tools is needed to master Python models.

Because of those limitations, it’s recommended to use Python models to achieve transformations that are harder to achieve in SQL. Let’s take a look at the Python model structure: