AWS VPC Flow Logs: Configuration, Storage, and Analysis

VPC Flow Logs provide network connection metadata for everything traversing your VPC's elastic network interfaces. They're essential for security investigation — when an incident involves network activity, flow logs tell you what connected to what, on what ports, and how much data moved. They're also valuable for network architecture decisions, capacity planning, and compliance evidence.

Configuring Flow Logs involves non-obvious choices about log format, delivery destination, and aggregation interval that meaningfully affect analysis capability and cost. This guide covers those decisions and the analysis patterns that make flow log data operationally useful.

Log Format Selection

VPC Flow Logs support a customizable format that determines which fields are included in each log record. The default format provides basic connectivity information (source IP, destination IP, source port, destination port, protocol, packets, bytes, action). The custom format supports 29+ additional fields including instance ID, VPC ID, subnet ID, traffic type (IPv4 vs IPv6), and TCP flags.

The additional fields are worth including for security analysis. The instance-id field correlates flow log entries directly to EC2 instances without requiring a separate IP-to-instance lookup. The vpc-id and subnet-id fields are essential for multi-VPC environments. TCP flags enable detection of connection scanning (SYN-only packets to multiple destinations).

Recommended custom format string that includes the most useful security fields:

${version} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status} ${vpc-id} ${subnet-id} ${instance-id} ${tcp-flags} ${type} ${region}

The tradeoff is storage cost — more fields mean larger log records. For high-traffic environments, the storage cost of extended format is measurable. Evaluate the cost against the investigation value of the additional fields; for security-focused deployments, the extended fields are generally worth it.

Aggregation Interval

Flow Logs records are aggregated over a time window before being written. The options are 1-minute or 10-minute aggregation intervals. The 1-minute interval gives more granular timing data — useful for understanding whether connections happened before or after a specific event — at approximately 10x the storage cost of 10-minute aggregation.

For security monitoring environments, use 1-minute aggregation for critical VPCs (production environments, VPCs containing sensitive data) and 10-minute aggregation for less critical environments (development, sandbox). The finer granularity is valuable for incident investigation timelines.

Destination: CloudWatch Logs vs. S3

The destination choice affects analysis latency, query capability, and cost:

CloudWatch Logs: Logs become queryable within minutes of generation. Supports near-real-time metric filters and alarms. Log Insights provides a good query interface for investigation. Storage cost is higher than S3 (~$0.03/GB ingestion + $0.03/GB/month storage vs S3 standard $0.023/GB/month). Best for environments where real-time detection and short-retention-window analysis are the primary use case.

S3: Lower storage cost and simpler long-term retention management. Athena provides powerful SQL-based analysis across large time ranges. No real-time alerting capability from S3 alone (though you can configure S3 event notifications to trigger Lambda for near-real-time processing). Best for environments with high traffic volumes where cost matters or where you need historical analysis across months of data.

Both (hybrid): Deliver to CloudWatch Logs for real-time monitoring and to S3 for long-term archival. Higher total cost but provides both real-time and historical analysis. Reserve this for high-security environments where investigation capability across long time ranges is required.

Enabling Flow Logs Across an Organization

For organizations with many VPCs across multiple accounts, enabling Flow Logs in each VPC manually doesn't scale. Use CloudFormation StackSets or Terraform to enable Flow Logs in all VPCs as part of your account baseline. An event-driven Lambda function triggered by CreateVpc CloudTrail events can automatically enable Flow Logs in newly created VPCs.

AWS Config managed rule vpc-flow-logs-enabled detects VPCs without Flow Logs enabled. Add this rule to your Config baseline and configure alerting on non-compliance — it's a simple check that ensures your network monitoring coverage doesn't have gaps.

Analysis Patterns for Security Investigation

Common investigation queries for security incidents involving network activity:

All connections from a specific instance to external IPs:

fields srcaddr, dstaddr, dstport, bytes, action
| filter instance-id = 'i-0123456789abcdef0'
  and not dstaddr like /^10./
  and not dstaddr like /^172.(1[6-9]|2[0-9]|3[0-1])./
  and not dstaddr like /^192.168./
| stats sum(bytes) as total_bytes by dstaddr, dstport
| sort total_bytes desc

Rejected inbound connections (potential port scanning):

fields srcaddr, dstport
| filter action = 'REJECT' and srcaddr not like /^10./
| stats count() as attempts by srcaddr
| sort attempts desc
| limit 20

Related Reading

• VPC security monitoring — flow logs in the context of the full network monitoring stack
• GuardDuty setup — threat intelligence layer that analyzes flow log data
• CloudTrail analysis — complementary audit trail for API-level activity
• AWS security monitoring tools — where flow logs fit in the complete monitoring stack

FAQ

How much storage do VPC Flow Logs consume?

Storage depends heavily on traffic volume. A rough estimate for production environments: 1 GB of flow log data per 1 million flow records. A moderately busy production VPC might generate 5-50 million flow records per day. At 10-minute aggregation with default format, typical storage is 0.5-5 GB/day/VPC. At 1-minute aggregation with extended format, multiply by 10-20x. Estimate your specific volumes with a 24-hour test before committing to a long-term retention period.

Do Flow Logs have any impact on network performance?

No. Flow Logs are generated from copies of network metadata — they don't intercept or inspect actual network traffic. There's no added latency, no bandwidth impact, and no performance degradation on the monitored network interfaces. The processing happens asynchronously and independently of the network path.

Can I monitor traffic between EC2 instances in the same security group?

Yes. Flow Logs capture all traffic through ENIs, including traffic between instances in the same security group or subnet. Traffic between instances in the same security group that isn't explicitly allowed by a security group rule is still logged as rejected. This is valuable for detecting lateral movement within a VPC segment that your security groups are designed to restrict.