Managing Multiple Resources in Terraform: Loops and Key Considerations

TL:DR
Terraform is a tool that made it easy, efficient and fast for us to build out our Infrastructure as a Code (IaaC). However, when it came to Day 2 operations, we were running into some issues where Terraform was destroying our resources, causing us headaches that were slowing us down.  Here is a recap of the problems we are facing and how we plan to avoid these issues in the future by effectively using loops and choosing the right keys.

The Problem: Terraform Destroying Resources

Terraform is an Infrastructure as a Code (IaaC) tool widely used to manage infrastructure. It allows you to define required resources in a declarative manner. All our cloud infrastructure such as VPCs, Kubernetes Clusters, RDS, S3 Buckets, Service Accounts are provisioned and maintained by Terraform.

Our product is growing at a rapid pace and Terraform allows us to build our IaaC in a fast and easy manner. But on Day 2, we faced an issue that Terraform was taking unexpected actions that were destroying our resources. We were using Loops incorrectly, and were not able to change existing resources anymore, just add new ones. We refactored our code and would like to share it to help you avoid a headache.

Using Loops in Terraform

Sometimes we need to create multiple resources at once. So what do we do?

resource "aws_s3_bucket" "bucket0" {
  bucket = "wgk24wt55-demo0"
}
resource "aws_s3_bucket" "bucket1" {
  bucket = "wgk24wt55-demo1"
}
resource "aws_s3_bucket" "bucket2" {
  bucket = "wgk24wt55-demo2"
}

I'm kidding, of course, we would not make it like this as it would break the DRY principle. We use loops instead.

Let’s make a small recap about available loops in Terraform/HCL:

1. Count Loops: The simplest way to make a loop based on predefined or counted numbers to create resources.
2. For_each Loops: This loop allows you to iterate over variables in maps and sets based on keys to create resources.
3. For Loops: These loops don’t allow you to directly create resources, but allow you to filter or transform lists, sets, tuples, or maps.

Count Loop

Let's look for some examples of how we can create multiple S3 buckets with the count loops:

resource "aws_s3_bucket" "bucket" {
  count = 3
  bucket = "wgk24wt55-demo${count.index}"
}

# Or lets add more controll, and iterate over tuple
locals {
  buckets = [
    { name = "demo0" },
    { name = "demo1" },
    { name = "demo2" },
  ]
}

resource "aws_s3_bucket" "bucket" {
  count = length(local.buckets)
  bucket = "wgk24wt55-${local.buckets[count.index].name}"
}

After terraform apply our S3 buckets are created. We can see them listed in our state:

aws_s3_bucket.bucket[0]
aws_s3_bucket.bucket[1]
aws_s3_bucket.bucket[2]

But later, we decided that we don't need the demo1 bucket anymore, let's delete it:

locals {
  buckets = [
    { name = "demo0" },
    { name = "demo2" },
  ]
}

resource "aws_s3_bucket" "bucket" {
  count = length(local.buckets)
  bucket = "wgk24wt55-${local.buckets[count.index].name}"
}

Running terraform apply again, and what do we see?

Plan: 1 to add, 0 to change, 2 to destroy.

But wait, why would Terraform to destroy 2 resources when we only deleted 1?

# aws_s3_bucket.bucket[1] must be replaced

~ bucket = "wgk24wt55-demo1" -> "wgk24wt55-demo2" # forces replacement

So demo1 and demo2 will be deleted, and only after that demo2 will be recreated.

The reasons is key of state objects. When we use count it marks aws_s3_bucket.bucket[X] with numbered index. And object with index 1 had a different name before.

It might be ok for some stateless resources, but not for S3 buckets with data. So how can this be avoided?

For_each Loop

Let’s try switching to another loop: for_each

locals {
  buckets = [
    { name = "demo0" },
    { name = "demo1" },
    { name = "demo2" },
  ]
}
# !Will not work!
resource "aws_s3_bucket" "bucket" {
  for_each = local.buckets
  bucket = "wgk24wt55-${each.value.name}"
}

But this snippet will not work, because for_each can only iterate over a map or a set of strings. Here we should use a for loop, which can help us transform tuple to map where key would be presented as bucket.name.

locals {
  buckets = [
    { name = "demo0" },
    { name = "demo1" },
    { name = "demo2" },
  ]
}
resource "aws_s3_bucket" "bucket" {
  for_each = { for bucket in local.buckets : bucket.name => bucket }
  bucket = "wgk24wt55-${each.value.name}"
}

The above for loop transformed the buckets variable:

# From:
buckets = [
    { name = "demo0" },
    { name = "demo1" },
    { name = "demo2" },
  ]

# Into:
buckets = {
       demo0 = {
           name = "demo0"
        }
       demo1 = {
           name = "demo1"
        }
       demo2 = {
           name = "demo2"
        }
    }

State Migration

However, if we apply these updates, it still will lead to deleting all our buckets. To avoid it, we need to migrate our state first:

terraform state mv "aws_s3_bucket.bucket[0]" "aws_s3_bucket.bucket[\"demo0\"]"
terraform state mv "aws_s3_bucket.bucket[1]" "aws_s3_bucket.bucket[\"demo1\"]"
terraform state mv "aws_s3_bucket.bucket[2]" "aws_s3_bucket.bucket[\"demo2\"]"

Let’s check the state list:

aws_s3_bucket.bucket["demo0"]
aws_s3_bucket.bucket["demo1"]
aws_s3_bucket.bucket["demo2"]

Now if we apply a new code we will get:

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Change as expected

The key right now is a bucket name that we specified via the for loop, so if we delete demo1 bucket now, other buckets will stay untouched as we expected.

locals {
  buckets = [
    { name = "demo0" },
    { name = "demo2" },
  ]
}
resource "aws_s3_bucket" "bucket" {
  for_each = { for bucket in local.buckets : bucket.name => bucket }
  bucket = "wgk24wt55-${each.value.name}"
}

Plan: 0 to add, 0 to change, 1 to destroy.

Conclusion

We can also do more complex keys. For example:

locals {
  buckets = [
    { name = "demo0", namespace = "A" },
    { name = "demo2", namespace = "B" },
  ]
}

resource "aws_s3_bucket" "bucket" {
  for_each = { for bucket in local.buckets : "${bucket.namespace}-${bucket.name}" => bucket }
  bucket = "${each.value.namespace}-wgk24wt55-${each.value.name}"
}

# Which will give us:
aws_s3_bucket.bucket["A-demo0"]
aws_s3_bucket.bucket["B-demo2"]

We are striving to keep keys simple but unique. Mostly we use a combination of name + namespace, but you can also add your domain or environment name. Keep in mind, changing the key requires either deletion or one more state movement, so I don’t recommend including frequently changing parameters such as tags or access policies to your keys.

As you have seen before, choosing the right key can help you avoid state manipulation in the future. Chose them carefully from the beginning.

‍