Avoid secrets in Terraform State

Learn how to identify and prevent sensitive data leaks in Terraform state files. Step-by-step security guide with remediation strategies.

Introduction

As I was working on one of my Terraform labs, I had a gut feeling that something was amiss. I am aware that Terraform state files can inadvertently store sensitive information, which can lead to security vulnerabilities if not managed properly. To ensure that my Terraform configurations are secure, I decided to dig deeper in my Terraform State.

Table of Contents

• Prerequisites
• Set up the lab environment
• Investigate the Terraform State
• 🐞Bug Identified
• 🔧Remediation Steps
• Conclusion

What You'll Learn

In this lab, you will learn how to:

• Keep security in mind when working with Infrastructure as Code (IaC) tools like Terraform.
• Explore Terraform State to identify potential security issues.
• Understand the implications of storing sensitive information in Terraform State.
• Implement best practices to avoid storing sensitive information in Terraform State.

Prerequisites

Before we begin, ensure you have the following knowledge and prerequisites installed on your machine:

• Basic understanding of Terraform and Infrastructure as Code (IaC) concepts.
• Terraform CLI installed (version 1.0 or later).
• Access to a terminal or command prompt.
• Git installed to clone repositories.
• An IDE or text editor for editing Terraform files (e.g., VSCode, Sublime Text).
• Basic knowledge of version control systems, particularly Git.

Set up the lab environment

I assume you have some Terraform experience to follow along. I am skipping detail about how to configure Terraform backends, providers, and other basics to avoid TL;DR. The README.md in the repo has more details if you need them.

Step 1: Clone the Repository

If you want to try this out yourself, clone this repo first. Tag v2.3.1 has the bug we will explore.

git clone --branch v2.3.1 https://github.com/Jihillestad/tflab-linux-public.git
cd ./tflab-linux-public
git switch -c main # make the git repo your own main branch.
git remote remove origin # remove the original remote

Step 2: Review the Terraform Configuration

Open the Terraform configuration files in your preferred text editor or IDE.

Step 3: Set up the Terraform infrastructure in Azure

Initialize Terraform in the cloned repository to set up the backend and download necessary provider plugins. I use the following commands in sequence:

terraform init -backend-config="tflab-linux-public.tfbackend"
terraform validate
terraform plan
terraform apply -auto-approve

Step 4: Validate your infrastructure

Once the apply is complete, you can validate that the infrastructure has been created successfully by logging into the Azure portal or using the Azure CLI to check for the resources defined in your Terraform configuration.

Investigate the Terraform State

Now that we have our infrastructure set up, it's time to investigate the Terraform state file to see if any sensitive information is being stored.

To get an overview of the current state, use the following command:

terraform state list

Line 1, 2, and 4 give me the chills, I suspect that sensitive information is stored under these resources. Let's inspect them further.

terraform state show tls_private_key.main

This is not good at all. My suspicion is confirmed, the private key is stored in plain text within the Terraform state file. This is a significant security risk, as anyone with access to the state file can retrieve this sensitive information.

Let's confirm the same for the other resources built on output from tls_private_key.main.

terraform state show azurerm_key_vault_secret.ssh_private_key

The value is labeled as sensitive, but it's still present in the state file in plain text. To confirm, let's open the state file in Azure Blob Storage.

As we can see, the private key is stored in plain text within the state file. This is a serious security concern, as it exposes sensitive information that should be protected.

🐞Bug Identified

To fix this issue, we need to ensure that sensitive information is not stored in the Terraform state file. First, let's identify the culprit in our Terraform code:

iam.tf

# Description: This file contains the resources for IAM, including Key Vault and SSH key pair generation.
 
# Create an Azure Key Vault
module "kv" {
  source              = "./modules/kv/"
  resource_group_name = azurerm_resource_group.tflab_linux.name
  location            = azurerm_resource_group.tflab_linux.location
  prefix              = var.prefix
  project_name        = var.project_name
  environment         = var.environment
 
  tags = local.common_tags
}
 
 
# Generate an SSH key pair
resource "tls_private_key" "main" {
  algorithm = "RSA"
  rsa_bits  = 2048
}
 
 
# Store the SSH keys in the Key Vault as secrets
resource "azurerm_key_vault_secret" "ssh_public_key" {
  key_vault_id = module.kv.kv_id
  name         = "ssh-public"
  value        = tls_private_key.main.public_key_openssh
 
  tags = local.common_tags
}
 
resource "azurerm_key_vault_secret" "ssh_private_key" {
  key_vault_id = module.kv.kv_id
  name         = "ssh-private"
  value        = tls_private_key.main.private_key_pem
 
  tags = local.common_tags
}

The tls_private_key resource is causing the security issue. We can resolve this by removing the tls_private_key resource and generating the SSH key pair outside of Terraform. We can then store the generated keys in the Key Vault without exposing them in the state file.

🔧Remediation Steps

Manual Remediation - Short-lived Environments

1. Remove the unsafe resources from your Terraform configuration. In this case, delete the iam.tf file which contains multiple unsafe resources.

2. Remove the unsafe resources from the Terraform state:

   terraform state rm tls_private_key.main
   terraform state rm azurerm_key_vault_secret.ssh_public_key
   terraform state rm azurerm_key_vault_secret.ssh_private_key

3. Generate the SSH key pair outside of Terraform using the following command:

   ssh-keygen -t rsa -b 4096 -f ./ssh_key -N "" -C "tflab-linux-key"

4. Store the generated keys in the Azure Key Vault using the Azure CLI:

   az keyvault secret set --vault-name <your-key-vault-name> --name "ssh-public" --file ./ssh_key.pub
   az keyvault secret set --vault-name <your-key-vault-name> --name "ssh-private" --file ./ssh_key
    

5. Move the kv module to a separate repository for better separation of concerns and to avoid managing sensitive resources in the same Terraform configuration as other infrastructure.

6. Destroy the existing Terraform infrastructure:

   terraform destroy -auto-approve

7. Build the Azure Key Vault using the separate repository created in step 5.

8. Refactor main.tf in the compute module to reference the existing Key Vault instead of creating a new one:

    data "azurerm_key_vault" "main" {
      name                = var.key_vault_name
      resource_group_name = var.key_vault_rg
    }
    
    data "azurerm_key_vault_secret" "ssh_public_key" {
      name         = "ssh-public-key"
      key_vault_id = data.azurerm_key_vault.main.id
    }
    
    
    resource "azurerm_linux_virtual_machine" "ubuntu_vm1" {
    
    #... other configurations ...
      admin_ssh_key {
        username   = var.username
        public_key = data.azurerm_key_vault_secret.ssh_public_key.value
      }
    #... other configurations ...
    }

Manual Remediation - Live Environments

1. Remove the unsafe resources from your Terraform configuration. In this case, delete the iam.tf file which contains multiple unsafe resources.

2. Back up the existing SSH keys from the Terraform State:

   terraform output -raw ssh_private_key > backup-private-key.pem
   terraform output -raw ssh_public_key > backup-public-key.pub

3. Move the kv module out of this repository to a separate repository for better separation of concerns and to avoid managing sensitive resources in the same Terraform configuration as other infrastructure.

4. Remove the unsafe resources and the Key Vault from the Terraform state. This makes sure the Key Vault and secrets are not destroyed when you apply the changes:

   terraform state rm azurerm_key_vault_key_vault.main
   terraform state rm tls_private_key.main
   terraform state rm azurerm_key_vault_secret.ssh_public_key
   terraform state rm azurerm_key_vault_secret.ssh_private_key

5. Verify that the secrets have been removed from the state:

   terraform state list

6. Refactor main.tf in the compute module to reference the existing Key Vault which is not managed by Terraform anymore:

    data "azurerm_key_vault" "main" {
      name                = var.key_vault_name
      resource_group_name = var.key_vault_rg
    }
    
    data "azurerm_key_vault_secret" "ssh_public_key" {
      name         = "ssh-public-key"
      key_vault_id = data.azurerm_key_vault.main.id
    }
    
    #... other data sources ...
    
    resource "azurerm_linux_virtual_machine" "ubuntu_vm1" {
    
    #... other configurations ...
      admin_ssh_key {
        username   = var.username
        public_key = data.azurerm_key_vault_secret.ssh_public_key.value
      }
    #... other configurations ...
    }

7. Run terraform plan to make sure no disruptive changes will happen to your infrastructure.

8. Delete the key backup files after verifying that the infrastructure is working as expected.

9. If something went wrong, and you need a new Azure Key Vault secret with the backed-up keys, you can use the following commands:

   az keyvault secret set --vault-name <your-key-vault-name> --name "ssh-public" --file ./backup-public-key.pub
   az keyvault secret set --vault-name <your-key-vault-name> --name "ssh-private" --file ./backup-private-key.pem

Improving after the Remediation

To get the automation value for the Key Vault module back, consider implementing CI/CD pipelines that deploy the Key Vault module separately. This way, you can maintain the benefits of Infrastructure while ensuring that sensitive information is handled securely.

Conclusion

In this lab, we identified a critical security issue where sensitive information was being stored in Terraform State. We explored the implications of this practice and provided remediation steps to secure our Terraform configurations. By following best practices and ensuring that sensitive data is not stored in state files, we can enhance the security of our infrastructure managed by Terraform. Some settings are better handled outside of Terraform, leveraging CI/CD pipelines will help us keep the automation benefits while ensuring security.

Coming Up Next

In the next lab, we will work on improving the security and protecting critical resources against accidental deletion in our Terraform configurations. Stay tuned!