Ingestion Methods—SFTP, API, and Object Storage

To handle various data sources, data engineers should know at least a few ingestion methods to adapt to different scenarios. Let’s look at three common ingestion methods: SFTP, API, and object storage.

SFTP

SSH File Transfer Protocol (SFTP) is a network protocol that provides file access, file transfer, and file management over any reliable data stream. The SSH protocol supports encryption and other security methods to better protect file transfers. Many financial institutions use SFTP to send customer transaction files for security reasons.

Fetch files from a local SFTP server

Follow the steps below to establish a connection and perform basic file operations:

• Connect to the local SFTP server: Initiate a connection to the local SFTP server by executing the following commands: