Building a Data Pipeline Using the tf.data API

Building data pipelines using tf.data

tf.data provides us with a convenient way to build data pipelines in TensorFlow. Input pipelines are designed for more heavy-duty programs that need to process a lot of data. For example, if we have a small dataset (for example, the MNIST dataset) that fits into the memory, input pipelines would be excessive. However, when working with complex data or problems, where we might need to work with large datasets that don’t fit in memory, we augment the data (for example, for adjusting image contrast/brightness), numerically transform it (for example, standardize), and so on. The tf.data API provides convenient functions that can be used to easily load and transform our data. Furthermore, it streamlines our data ingestion code with the model training.

Additionally, the tf.data API offers various options to enhance the performance of our data pipeline, such as multiprocessing and prefetching data. Prefetching refers to bringing data into the memory before it’s required and keeping it ready.

Creating a data pipeline

When creating an input pipeline, we intend to perform the following:

• Source the data

...