Use Temporary Credentials with Terraform Cloud

1/1/2022

Note: While this post refers only to Terraform Cloud, it applies to Terraform Enterprise as well.

In my post on using Terraform Cloud and AWS Route 53, I created IAM credentials for an IAM user with an IAM policy scoped to the actions Terraform would need to perform in managing DNS records.

While that approach works, it requires the use of long-lived credentials. It is recommended to use short-lived credentials on AWS by assuming roles. But Terraform Cloud doesn't support assuming roles. Fortunately, we're not without options. This post explains how to use short-lived credentials with Terraform Cloud.

Workspace Variables

Terraform Cloud has a concept of workspace variables. You can set workspace environment variables, and they will be used as environment variables in the execution environment that Terraform Cloud uses. This is described in Terraform's documentation:

Terraform Cloud performs Terraform runs on disposable Linux worker VMs using a POSIX-compatible shell. Before running Terraform operations, Terraform Cloud uses the export command to populate the shell with environment variables.

These workspace environment variables can be set from the Terraform Cloud console. How does this help us?

When you assume a role from the AWS CLI, you receive back three needed pieces of information:

Access key ID
Secret access key
Session token

By taking those values and setting them as environment variables in your shell, subsequent AWS CLI commands will be executed using the assumed role's permissions. Likewise, if we take those values and set them as workspace environment variables in Terraform Cloud, then when Terraform makes AWS CLI calls by means of the Terraform AWS provider, it will use that same role's permissions.

Following the same order as above, the environment variables should be named:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN

Why This is More Secure

Before going any further, let's talk about the point of using a role. You could just issue yourself IAM credentials and use them as workspace environment variables and be done with it. We'll talk about two reasons not to do that:

Short-lived credentials are safer than long-lived credentials.
Achieving fine-grained IAM permissions for Terraform is harder that way.

Short-Lived Credentials Over Long-Lived Credentials

For as long as credentials remain valid, they present a risk. If they by any means become known to a threat actor, then that individual will have the same permissions in your system as you do. Short-lived credentials automatically expire after a specified period of time. At that point, if they become known to a third-party, they have no value.

Acheiving Fine-Grained Permissions is Hard

A Terraform config can manage a variety of AWS resources, and thus can call a variety of AWS services. Figuring out precisely what IAM permissions are needed for Terraform to do everything it needs to do can be challenging and time-consuming. You likely want to just get your infrastructure built and see if your idea even works, first, and then later tighten up the permissions. Figuring out what permissions Terraform needs can involve looking at verbose logs or CloudTrail logs to see what it did. But to do that, you most likely are going to start out with an initial set of permissions that are broader than necessary.

So now you have an IAM user with a role that has greater permissions than it really needs. If you're really in a hurry, you might even be tempted to put your IAM user in a group with the AdministratorAccess AWS managed policy. Imagine if those IAM credentials were leaked!

Using short-lived credentials mitigates the risk of starting out with broad permissions and then scoping them down to only what is needed.

Workflow

There are only two ways to set workspace environment variables in Terraform Cloud:

Copy and paste them into the Terraform Cloud console.
Use the Terraform Cloud API to set them programmatically.

If you're using long-lived credentials, then copying and pasting the credentials once is no big deal. But if you're using short-lived credentials that expire every hour (for example, you can change the TTL), then that would obviously get tedious.

No, You Can't Use the Terraform Cloud CLI

The Terraform documentation says that you can set workspace variables a variety of ways. For instance, you can use environment variables in your shell that start with TF_VAR_. You can set them in files named *.auto.tfvars. That's all true. But none of that is talking about workspace environment variables. They can only be set in the workspace.

This is confirmed by the README for the hashicorp/terraform-guides project, which says ("TFE" means "Terraform Enterprise", but this applies to Terraform Cloud, too):

Note that variables added via a *.auto.tfvars file will not show up on the variables tab of the workspace in the TFE UI. Additionally, you cannot add environment variables in a tfvars file.

Use the Terraform Cloud API

Fortunately, you can programmatically get workspace environment variables set in the Terraform Cloud UI using the Terraform Cloud API. The workspace variables API documentation has useful code snippets to help with this, such as this example for setting a variable:

curl \
  --header "Authorization: Bearer $TOKEN" \
  --header "Content-Type: application/vnd.api+json" \
  --request POST \
  --data @payload.json \
  https://app.terraform.io/api/v2/workspaces/ws-4j8p6jX1w33MiDC7/vars

The example source for payload.json is:

{
  "data": {
    "type":"vars",
    "attributes": {
      "key":"some_key",
      "value":"some_value",
      "description":"some description",
      "category":"terraform",
      "hcl":false,
      "sensitive":false
    }
  }
}

Automating Setting Workspace Variables

Here's the general idea:

Set up a role for your Terraform config with a low session duration.
Use the AWS CLI to assume the role.
Parse the output from the assume role operation.
Set the three workspace environment variables using the output.
Perform a run in Terraform cloud using the assumed role credentials.

Prerequisites

To successfully make calls to the Terraform workspace variables API, you need your workspace ID. You also need a Team API token.

You can get your workspace ID from the Terraform Cloud UI. Navigate to your workspace, and then go to Settings > General. The ID field is the workspace ID.

To get a Team token, click on Settings in the top menu bar. This takes you to your organization settings, as opposed to your workspace settings. Click on Teams. Issue yourself a Team API token and save it in your password manager.

Assuming a Role

You'll need to already have some credentials established to be able to assume a role. And you should have already set up your role with a low session duration. Then, you can assume the role:

aws sts assume-role --role-arn "arn:aws:iam::123456789012:role/example-role" --role-session-name AWSCLI-Session

You can likely set the role session name to whatever you want, if you're working with your own AWS account. That value shows up in AWS CloudTrail logs and can also be used as a condition in IAM policy. You can read more about that in this AWS blog on role session names.

The aws sts assume-role command can return a JSON response (depending on how your AWS CLI is configured). You can parse the response using jq. You could also learn JMESPath syntax and not use jq.

If you're using JSON, then the output is going to look something like this:

{
    "Credentials": {
        "AccessKeyId": "ACCESS_KEY_ID",
        "SecretAccessKey": "SECRET_ACCESS_KEY",
        "SessionToken": "SESSION_TOKEN",
        "Expiration": "1970-01-01T00:00:00+00:00"
    },
    "AssumedRoleUser": {
        "AssumedRoleId": "ROLE_ID:ROLE_SESSION_NAME",
        "Arn": "arn:aws:sts::ACCOUNT_ID:assumed-role/ROLE_NAME/ROLE_SESSION_NAME"
    }
}

Parsing the Output of the Assume Role Operation

You could write the output of that command to a file called creds.json, and then pass that as input to jq to set variables. Note that the commands below are intended for use in a shell script. We're not trying to set environment variables in our shell, necessarily. We just need to parse out the values and then use them in future calls to the Terraform Cloud workspace variables API.

AWS_ACCESS_KEY_ID=$(jq -r '.Credentials.AccessKeyId' <creds.json) 
AWS_SECRET_ACCESS_KEY=$(jq -r '.Credentials.SecretAccessKey' <creds.json)
AWS_SESSION_TOKEN=$(jq -r '.Credentials.SessionToken' <creds.json)

Setting Workspace Environment Variables

If you try to create a workspace environment variable that is already defined in your workspace, you'll get an error back from the Terraform Cloud API. If the variable is already defined and you need to update it, then that's a different API call:

curl \
  --header "Authorization: Bearer $TOKEN" \
  --header "Content-Type: application/vnd.api+json" \
  --request PATCH \
  --data @payload.json \
  https://app.terraform.io/api/v2/workspaces/ws-4j8p6jX1w33MiDC7/vars/var-yRmifb4PJj7cLkMG

Note that in this call, the variable ID is at the end of the URL. That's not the only place the variable ID appears: it's also in the JSON payload. This key that isn't present in the payload we'd send for a create operation. Notice the additional "id" key in the example payload for an update, below (copied from Terraform's documentation):

{
  "data": {
    "id":"var-yRmifb4PJj7cLkMG",
    "attributes": {
      "key":"name",
      "value":"mars",
      "description":"some description",
      "category":"terraform",
      "hcl": false,
      "sensitive": false
    },
    "type":"vars"
  }
}

How can we determine if we need to create a new variable or just update an existing one? One way is to use the workspace variables API to list all of the variables in the workspace, and look for our variable by name. That can be accomplished with jq, as well:

curl \
  --header "Authorization: Bearer $TOKEN" \
  --header "Content-Type: application/vnd.api+json" \
  --request POST \
  --data @payload.json \
  https://app.terraform.io/api/v2/workspaces/ws-4j8p6jX1w33MiDC7/vars \
  | jq -r '.data[] | select(.attributes.key == "'$varName'").id'

Set the Workspace Variables

With the variable IDs in hand (if needed), and the three pieces of information obtained from the assume-role AWS CLI command, we're now ready to make the API calls to set the three workspace environment variables.

The terraform-guides project uses a template for the payload in the Terraform Cloud API call. The template has placeholders in it that can be replaced with sed commands. Here is the template:

{
    "data": {
      "type":"vars",
      "attributes": {
        "key":"my-key",
        "value":"my-value",
        "description": "my-descr",
        "category":"env",
        "sensitive":true
      }
    }
  }

Note the use of "sensitive":true in the example above. When a variable is marked as "sensitive", Terraform Cloud encrypts them before storing them, naturally using HashiCorp Vault. The UI will not reveal a "sensitive" variable nor will it let you modify it. Obviously, this is two-way encryption. Otherwise, Terraform Cloud wouldn't be able to use the variables.

For each of the three values we obtained from the AWS CLI assume-role call, we need to make an API call to the Terraform workspace variables API that passes in a value for key, value, and description. Suppose those three values are in varName, varValue, and varDescr, respectively. Borrowing from HashiCorp's helper script, and assuming the above JSON template is in a file named var_template.json and we want to store the actual payload in a file named payload.json, we can do this:

sedDelim=$(printf '\001')
sed -e "s${sedDelim}my-key${sedDelim}$varName${sedDelim}" \
    -e "s${sedDelim}my-value${sedDelim}$varValue${sedDelim}" \
    -e "s${sedDelim}my-descr${sedDelim}$varDescr${sedDelim}" \
    < var_template.json | sed -e 's|\\|\\\\|g' > payload.json

If we are doing an update and need to add in the variable ID, we can do this:

jq --arg newval "$variableId" '.data += { id: $newval }' \
   < payload.json > payload_with_id.json

Perform a Run in Terraform Cloud

If you're using the CLI-driven workflow, then at this point you can run terraform plan or terraform apply and Terraform will have the workspace environment variables it needs to make authorized AWS calls on your behalf.

If you're using the UI/VCS-driven workflow, then you can now either manually trigger a run from the Terraform Cloud UI, or you can trigger a run by pushing code to your VCS.

Conclusion

A relatively simple shell script can automate all of these steps for you. Now, you can start out using a role with too-broad permissions with less risk of those credentials becoming known. And, in general, you can be less concerned about secret credentials being stored with a third-party. There will be times when you'll start a run, and your credentials will have expired. There's nothing wrong with re-issuing your credentials before they expire, however.

It would be great if HashiCorp provided a tool to do this easily, instead of promoting the use of long-lived credentials. Fortunately, they have provided the necessary API endpoints to work more securely with AWS resources using Terraform Cloud.