Terraform State Locking in AWS: DynamoDB vs S3

Introduction

When managing infrastructure as code with Terraform in collaborative environments, preventing concurrent modifications is critical. This is where state locking becomes essential – it prevents multiple team members from simultaneously changing the same infrastructure, which could lead to state corruption or inconsistent deployments.

In AWS, you have two main options for Terraform state locking:

Using DynamoDB as a dedicated locking mechanism
Leveraging S3’s native locking capabilities (introduced in Terraform v1.10)

This guide explores both approaches, comparing their features, implementation details, and use cases to help you choose the right solution for your infrastructure needs.

Understanding Terraform State and the Need for Locking

Before diving into locking mechanisms, let’s understand why state management is crucial for Terraform deployments.

Terraform state is a JSON file that maps real-world resources to your configuration, tracks metadata, and improves performance for large infrastructures. By default, Terraform stores state locally, but in team environments, remote state storage becomes essential.

When multiple users attempt to modify infrastructure simultaneously, conflicts can arise, potentially corrupting the state file or creating race conditions. This is precisely where state locking provides protection.

For more details on how Terraform state works, refer to the official HashiCorp documentation on S3 backend.

Option 1: S3 Backend with DynamoDB Locking (Traditional Approach)

The traditional and well-established approach for Terraform state management in AWS combines two services:

Amazon S3: Provides high availability storage for the state file
DynamoDB: Implements a distributed locking mechanism

Setting Up DynamoDB for State Locking

First, create a DynamoDB table with the required schema:

aws dynamodb create-table \
     --region us-east-1 \
     --table-name terraform-state-lock \
     --attribute-definitions AttributeName=LockID,AttributeType=S \
     --key-schema AttributeName=LockID,KeyType=HASH \
     --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5

The table requires a partition key named LockID of type String (AttributeType=S).

Configuring the S3 Backend with DynamoDB Locking

Create an S3 bucket for state storage (if you don’t already have one):

aws s3api create-bucket \
     --bucket terraform-state-storage \
     --region us-east-1

aws s3api put-bucket-versioning \
     --bucket terraform-state-storage \
     --versioning-configuration Status=Enabled

Then, in your Terraform configuration, specify the S3 backend with DynamoDB locking:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
  
  backend "s3" {
    bucket         = "terraform-state-storage"
    key            = "environments/production/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-state-lock"
    encrypt        = true
  }
}

provider "aws" {
  region = "us-east-1"
}

Advantages of DynamoDB Locking

Battle-tested solution: Used by the Terraform community for years with proven reliability
Robust locking mechanism: DynamoDB provides a strong consistency model for reliable locking
Fine-grained control: Customizable read/write capacity units and TTL settings
Lock visibility: Lock information can be inspected directly in the DynamoDB table
Conditional writes: Ensures only one change is made to the state at a given time
Compatible with all Terraform versions: Works with both older and newer Terraform versions

Limitations

Additional cost: Maintaining a separate DynamoDB table incurs extra AWS charges
More complex setup: Requires creation and management of an additional AWS resource
Multi-service dependency: Relies on both S3 and DynamoDB functioning correctly

Option 2: Native S3 Locking (Terraform v1.10+)

Starting with Terraform v1.10, HashiCorp introduced native S3 locking capabilities, eliminating the need for a separate DynamoDB table. This feature significantly streamlines Terraform state management in AWS.

Setting Up S3 for Native Locking

Create an S3 bucket with versioning enabled (required for native locking):

aws s3api create-bucket \
     --bucket terraform-state-bucket \
     --region us-east-1

aws s3api put-bucket-versioning \
     --bucket terraform-state-bucket \
     --versioning-configuration Status=Enabled

Configuring Terraform for Native S3 Locking

In your Terraform configuration, enable native S3 locking with the use_lockfile parameter:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
  
  backend "s3" {
    bucket       = "terraform-state-bucket"
    key          = "environments/production/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true
    use_lockfile = true  # Enables native S3 locking
  }
}

provider "aws" {
  region = "us-east-1"
}

The use_lockfile parameter is the key configuration that enables Terraform to use S3’s native locking capabilities instead of requiring a DynamoDB table.

Advantages of Native S3 Locking

Simplified architecture: No need for an additional DynamoDB table
Cost efficiency: Eliminates DynamoDB costs
Reduced management overhead: Fewer AWS resources to monitor and maintain
Streamlined implementation: Single-service approach simplifies setup and troubleshooting
Modern approach: Takes advantage of the latest Terraform features

Limitations

Version requirement: Only available in Terraform v1.10 and later
Less feature-rich: Offers fewer customization options compared to DynamoDB
S3 versioning requirement: Bucket versioning must be enabled
Relatively new feature: Has less community experience and troubleshooting resources

Performance Comparison

Here’s how both approaches compare in terms of performance:

Aspect	DynamoDB	S3 Native Lock
Latency	Potentially higher due to cross-service calls	Potentially lower with single-service design
Throughput	Highly scalable with provisioned capacity	Limited by S3 request rates
Consistency	Strong consistency model	Strong consistency for locking operations
Concurrency	Well-tested with high concurrency	Designed for concurrent operations
Implementation complexity	Higher (multiple services)	Lower (single service)
Resource footprint	Higher (S3 + DynamoDB)	Lower (S3 only)

When to Choose Each Approach

DynamoDB Locking Makes Sense When:

You’re using Terraform versions prior to v1.10
Your team has existing expertise with the DynamoDB locking approach
You need advanced lock management features
You have complex CI/CD pipelines with specific locking requirements
You need detailed lock audit trails for compliance purposes

Native S3 Locking Makes Sense When:

You’re using Terraform v1.10 or later
You want to simplify your AWS resource footprint
You’re looking to reduce AWS costs
You’re setting up new Terraform projects from scratch
You prefer a more streamlined approach to state management

Best Practices for Terraform State Management

Regardless of which locking method you choose, follow these best practices:

Enable encryption: Always protect your state files with encryption at rest backend "s3" { # Other configuration... encrypt = true }
Implement proper IAM policies: Apply least-privilege access controls to your state storage
Enable versioning: Always enable versioning on your S3 bucket to maintain state history aws s3api put-bucket-versioning \ --bucket terraform-state-bucket \ --versioning-configuration Status=Enabled
Use workspaces: Organize infrastructure with Terraform workspaces to minimize contention terraform workspace new production terraform workspace new development
Structure state files logically: Separate state files based on environment or component backend "s3" { # For production networking component key = "environments/production/networking/terraform.tfstate" }
Implement state validation checks: Regularly validate state integrity terraform validate terraform plan -refresh-only
Consider remote state data sources: Use terraform_remote_state data sources to share outputs between states
Document your approach: Maintain clear documentation about your state management strategy
Set up CI/CD pipelines: Automate your Terraform workflows using CI/CD pipelines like GitHub Actions. For a detailed guide on integrating Terraform with GitHub Actions, check out this article on DevToolHub.

Implementation Examples

Example 1: Complete DynamoDB Locking Setup

# backend.tf

terraform {
  required_version = ">= 1.0.0"
  
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
  
  backend "s3" {
    bucket         = "my-terraform-states"
    key            = "projects/microservice-api/terraform.tfstate"
    region         = "us-west-2"
    dynamodb_table = "terraform-state-locks"
    encrypt        = true
  }
}

provider "aws" {
  region = "us-west-2"
}

Example 2: Complete Native S3 Locking Setup

# backend.tf

terraform {
  required_version = ">= 1.10.0"
  
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
  
  backend "s3" {
    bucket       = "my-terraform-states"
    key          = "projects/microservice-api/terraform.tfstate"
    region       = "us-west-2"
    encrypt      = true
    use_lockfile = true
  }
}

provider "aws" {
  region = "us-west-2"
}

Conclusion

The introduction of native S3 locking in Terraform v1.10 represents a significant improvement in simplifying state management for AWS users. While the traditional DynamoDB approach has served the community well for years, the native S3 locking feature offers a more streamlined alternative that reduces complexity and cost.

For new Terraform projects using v1.10 or later, native S3 locking provides an excellent default choice. Existing projects using DynamoDB locking can continue with their current approach or consider migrating to native locking when convenient.

Ultimately, both methods achieve the same critical goal: protecting your infrastructure from concurrent modifications that could lead to state corruption or inconsistent deployments. Whichever approach you choose, implementing proper state locking is an essential best practice for any team using Terraform.

Remember that Terraform continues to evolve, so stay updated with the latest features and best practices from HashiCorp’s official documentation and the broader community.

DevToolHub