Stop Terraform drift from reverting your right-sizing

Q: Why did my instance go back to its old size after I resized it?

Because the size lives in your .tf files and you changed it somewhere else. Terraform's job is to make the real world match the configuration, so a plan that refreshes state finds the smaller instance, compares it to the larger size pinned in code, and proposes an update to restore the code's version. The apply that carries out the revert is often an unrelated pull request from a different person.

Q: How do I detect Terraform drift?

Run terraform plan -refresh-only -out=drift.tfplan, convert it with terraform show -json drift.tfplan, and read the resource_drift key. That key holds only the changes Terraform detected between the most recent state and the prior saved state, so it isolates drift from configuration edits you made yourself. For a CI gate, terraform plan -refresh-only -detailed-exitcode returns 2 when drift exists, 0 when state matches reality, and 1 on error.

A right-sized cloud instance being reverted to its larger, more expensive size by a Terraform apply

You resized an oversized instance last quarter and took about $140 a month off the bill. It is back at the old size now. Terraform drift put it there, during an apply for a change that had nothing to do with it, and the plan said so in a note nobody read.

Why Terraform drift undoes a console resize

Terraform holds three pictures of every resource. Your .tf files say what the resource should be. The state file records what Terraform last saw. The provider API holds what is actually running and actually billing.

terraform plan refreshes the middle one. It reads each resource from the provider, writes what it finds into the working state, then compares that against your configuration. Anything in the real world your code does not ask for becomes a change to bring the real world back into line.

A console resize moves only the third picture. Your code still pins the old size, so the next refreshed plan reads the difference as an error to correct.

Terraform is honest about this. The refresh prints a note listing what changed outside Terraform, and it warns that "the following plan may include actions to undo or respond to these changes." That sentence is the whole problem. The plan is doing its job.

Price the correction with the size step you gave up. In us-east-1, an m5.xlarge lists at $0.192 an hour and an m5.2xlarge at $0.384. Over 730 hours that is $140.16 against $280.32. Every step up an instance family doubles the vCPU and the memory, and doubles the hourly rate along with them. The revert costs you the exact saving you booked, per instance, for every month it stands.

The revert does not land when you make the change

Plans are slow, so plenty of pipelines run terraform plan -refresh=false. HashiCorp's plan command reference describes that flag as disabling the synchronization of state with remote objects before checking for configuration changes. It skips the provider read entirely.

So the plan compares your configuration against the stored state, and nothing else. Both of those still say m5.2xlarge. The plan comes back empty, the pipeline goes green, and your resize survives another week. The drift is real. The run that would have caught it declined to look.

The correction lands on the first refreshed apply after that. In most teams that belongs to somebody else's pull request, touching a subnet tag or a security group rule, days or weeks later. Their plan refreshes state, finds your instance at the smaller size, and folds the revert into their apply.

Now read that plan as the reviewer. It proposes a change to an instance the pull request never mentions, sitting underneath the change the author actually made. People check their own diff closely and skim the rest. That is how a resize gets undone by someone who never opened the instance.

The bill climbs back a month later. By then the apply that did it is buried in CI history, and the saving reads like it never worked.

Find your drifted resources by hand

The human-readable plan output mixes drift in with the changes you meant to make. Ask for the machine-readable version instead, where the two are separate fields.

Start with a refresh-only plan. It reconciles state against reality and proposes no infrastructure changes at all, so it is safe to run against production from a laptop.

terraform plan -refresh-only -out=drift.tfplan
terraform show -json drift.tfplan > drift.json

The JSON plan has a key for exactly this. HashiCorp's JSON output format documents resource_drift as a description of the changes Terraform detected when it compared the most recent state to the prior saved state, using the same object structure as resource_changes. Everything that moved outside Terraform is in there, and nothing you wrote is.

jq -r '.resource_drift[]? | "\(.address)\t\(.change.actions | join(","))"' drift.json

Narrow that to the attributes carrying a price. On EC2 the attribute is instance_type. The before value is what your state believed, and the after value is what the provider just reported.

jq -r '.resource_drift[]?
  | select(.type == "aws_instance")
  | select(.change.before.instance_type != .change.after.instance_type)
  | "\(.address)  state=\(.change.before.instance_type)  real=\(.change.after.instance_type)"' drift.json

The same query works elsewhere by swapping the type and the attribute. Compute Engine pins size in machine_type on google_compute_instance. Azure uses size on azurerm_linux_virtual_machine. Managed databases carry the same exposure in instance_class on aws_db_instance, where a size step is worth considerably more than $140.

jq -r '.resource_drift[]?
  | select(.type == "google_compute_instance"
        or .type == "azurerm_linux_virtual_machine"
        or .type == "aws_db_instance")
  | "\(.address)\t\(.type)"' drift.json

To gate this in CI, read the exit code instead of parsing text. terraform plan -detailed-exitcode returns 0 for an empty diff, 1 for an error, and 2 for a non-empty diff. Pair it with refresh-only and a 2 means state and reality disagree.

# -lock=false keeps a nightly read from blocking a real apply.
terraform plan -refresh-only -detailed-exitcode -lock=false
# 0 = state matches reality, 1 = error, 2 = drift found

Run that nightly per workspace. Exit code 2 on a workspace nobody touched that day is your signal, and it arrives while the change is still recent enough that someone remembers making it.

One caution on -detailed-exitcode in an ordinary plan. It returns 2 for any pending change, so it cannot tell drift apart from a configuration edit you made on purpose. The resource_drift key can. Use the exit code to decide whether to look, and the JSON to decide what you are looking at.

Fix it in code, and mind the stop and start

The durable fix is dull. Put the size you actually want into the configuration, so your code and your cloud agree and no future plan has anything to correct.

resource "aws_instance" "api" {
  ami           = data.aws_ami.base.id
  instance_type = "m5.xlarge" # was m5.2xlarge
}

When the instance already runs at the smaller size, that edit costs nothing. The next refresh writes m5.xlarge into state, your configuration asks for m5.xlarge, and the plan is empty for the first time in months.

When you drive the resize through Terraform instead, the AWS provider is explicit about the price: updates to instance_type trigger a stop and start of the EC2 instance. The instance ID and the EBS volumes survive, and the machine is still down for the duration. Run it behind an autoscaling group or a load balancer with more than one healthy target, or take a window. An instance store backed instance loses its ephemeral data on stop, so check the root device type before you queue anything.

Rollback runs the same path backwards. The size lives in version control now, so git revert on the commit plus one apply restores the old size, at the cost of a second stop and start.

The same drift catches storage. Change a disk tier in the console, say Premium SSD v1 to v2 on Azure, and the new SKU sits in the provider API while your code still pins the old one. Azure documents no direct conversion back from Premium SSD v2, so read what your plan intends to do about that difference before you apply it.

Two shortcuts that look like fixes

lifecycle { ignore_changes = [instance_type] } stops the plan from proposing the revert. It also stops your code from describing the machine you run. HashiCorp's lifecycle documentation is precise about the limit: ignored attributes are skipped when planning an update, and they are still considered when planning a create. So the day anything replaces that instance, a new AMI, a changed subnet, user_data_replace_on_change firing, Terraform builds the replacement at the old expensive size, and the plan looks clean while it does it.

terraform apply -refresh-only is the other one. It accepts reality into state without touching infrastructure, which silences the drift note. Your configuration still asks for the larger size, so the disagreement moves out of the note and into the plan body as a pending change. The next ordinary apply reverts the instance exactly as before, with less warning than you had to begin with.

Both commands have honest uses. ignore_changes belongs on attributes another system genuinely owns, like a tag an autoscaler writes. -refresh-only belongs on an out-of-band change you made deliberately and are about to encode. Neither one settles a size you want to keep.

A quick decision tree

graph TD
  A[Instance is bigger than the size you chose] --> B{Does resource_drift list it?}
  B -- No --> C{Did the plan run with -refresh=false?}
  C -- Yes --> D[Re-run the plan with refresh on, then look again]
  C -- No --> E[State matches reality: the size in code is the size you have]
  B -- Yes --> F{Do you want the smaller size?}
  F -- Yes --> G[Edit instance_type in the .tf file, apply, expect a stop and start]
  F -- No --> H[Apply -refresh-only to record it, then set the size in code deliberately]

Common questions

What is Terraform drift? Drift is the gap between what your Terraform state records and what the provider API actually reports. It appears whenever a resource changes outside the Terraform workflow, through a console click, a support ticket, or another automation. terraform plan refreshes state before comparing it to your configuration, so drift shows up as a note listing objects that changed outside Terraform, and usually as a proposed change to put them back.

Why did my instance go back to its old size after I resized it? Because the size lives in your .tf files and you changed it somewhere else. Terraform's job is to make the real world match the configuration, so a plan that refreshes state finds the smaller instance, compares it to the larger size pinned in code, and proposes an update to restore the code's version. The apply that carries out the revert is often an unrelated pull request from a different person.

How do I detect Terraform drift? Run terraform plan -refresh-only -out=drift.tfplan, convert it with terraform show -json drift.tfplan, and read the resource_drift key. That key holds only the changes Terraform detected between the most recent state and the prior saved state, so it isolates drift from configuration edits you made yourself. For a CI gate, terraform plan -refresh-only -detailed-exitcode returns 2 when drift exists, 0 when state matches reality, and 1 on error.

Does terraform plan -refresh=false hide drift? Yes. That flag disables synchronizing state with remote objects before the comparison, so the plan checks your configuration against the stored state alone. If both still hold the old value, the plan is empty and the pipeline passes while the drift sits there undetected. The revert then lands on the next plan that does refresh, which is usually a later apply from a different change.

Does changing instance_type in Terraform cause downtime? It causes a stop and start. The AWS provider documents that updates to instance_type trigger a stop and start of the EC2 instance, which keeps the instance ID and the attached EBS volumes but takes the machine offline for the duration. Schedule it behind an autoscaling group or a load balancer with another healthy target. An instance store backed instance also loses its ephemeral data on stop.

Should I use ignore_changes to stop Terraform reverting a resize? Only when another system genuinely owns that attribute. HashiCorp documents that ignored attributes are skipped when planning an update and still considered when planning a create, so the setting hides the conflict rather than resolving it. When the instance is eventually replaced, by a new AMI or a changed subnet, Terraform rebuilds it at the expensive size still written in your code, and the plan gives no warning.

Does terraform apply -refresh-only fix drift? It records reality into state without changing infrastructure, which removes the drift note. Your configuration is untouched, so it still asks for the old size, and that disagreement simply moves into the plan body as a pending change. The next ordinary apply performs the revert. Use refresh-only to acknowledge a deliberate out-of-band change, then edit the configuration to match.

How much does a reverted right-size actually cost? The full saving, every month it stands. Instance families roughly double in price at each size step, so in us-east-1 an m5.2xlarge at $0.384 an hour against an m5.xlarge at $0.192 is about $280.32 versus $140.16 over 730 hours. A revert on a managed database is worse, since an instance_class step on a Multi-AZ deployment carries the doubled rate twice over.

How OhChimp approaches this

OhChimp reads the Terraform in your repositories and pulls out the cost-relevant resources with the instance type each one declares, so the sizes pinned in code sit next to what your cloud accounts actually bill. Each oversized size becomes a reviewable plan with a confidence score, a risk level, and rollback steps, and the fix ships as a change to the code rather than a click in a console, which is what keeps it from being reverted later. Nothing moves until you press apply, and you are the one who presses it. The saving is then checked against your real bill, so a size that came back shows up as money that never left.

See how the code scanning connects on the GitHub integration page, and pair it with the Kubernetes right-sizing walkthrough if your oversized workloads are running as pods rather than instances.

All OhChimp posts