Understand VPC Flow Logs

After setting up the network infrastructure, the next step is to configure adequate network monitoring capabilities for future debugging and troubleshooting. Appropriate monitoring allows for the early detection of network issues and aids in the timely resolution of network problems.

Amazon VPC provides a network monitoring service called VPC Flow Logs that collects logs of network traffic going to and from VPC network interfaces. Flow logs capture traffic metadata such as the source address, destination address, protocol, port number, etc., in the form of a flow log record. These are stored at the maximum aggregation level of 10 minutes by default but can be changed to 1 minute when creating a flow log. The contents of traffic packets are not recorded in flow logs.

Flow log levels

A flow log can be created on the following levels:

1. VPC level: When created on the VPC level, flow logs are enabled for all the network interfaces within the VPC.

2. Subnet level: When created on the subnet level, flow logs are enabled for all the network interfaces within the specified subnet.

3. Network interface level: On the network interface level, flow logs are captured for the specified network interface only.

Flow log destination

When creating a flow log, it is required to specify the destination for the log delivery. The supported destinations for the flow logs delivery include the following:

• AWS CloudWatch logs: Flow logs can be published to AWS CloudWatch log groups, and a unique log stream is created within the log group for each network interface. Furthermore, CloudWatch makes it simple to search and filter logs, as well as add alerts to track specific events in the logs.

• Amazon S3: Flow logs can also be delivered to S3 buckets, where a consolidated log object in the form of a gzip compressed file is stored. For analysis and querying, the log objects can be integrated with other AWS services, such as Amazon Athena or Elasticsearch.

• Amazon Kinesis Data Firehose: Flow logs can also be delivered to Kinesis Data Firehose. The logs can be published to the specified delivery stream and then integrated with other services for custom processing.

Flow log record structure

A flow log record includes values for several fields that describe the traffic flow. Flow log records are stored in a default format. A custom format can also be specified to capture additional fields in the flow log record.

Example flow log record

The following example illustrates the flow record format and describes its available fields.