Stop Terraform drift from reverting your right-sizing
Published · Updated · OhChimp Engineering
You resized an oversized instance last quarter and took about $140 a month off the bill. It is back at the old size now. Terraform drift put it there, during an apply for a change that had nothing to do with it, and the plan said so in a note nobody read.
Why Terraform drift undoes a console resize
Terraform holds three pictures of every resource. Your .tf files say what the resource should be. The state file records what Terraform last saw. The provider API holds what is actually running and actually billing.
terraform plan refreshes the middle one. It reads each resource from the provider, writes what it finds into the working state, then compares that against your configuration. Anything in the real world your code does not ask for becomes a change to bring the real world back into line.
A console resize moves only the third picture. Your code still pins the old size, so the next refreshed plan reads the difference as an error to correct.
The refresh prints a note listing what changed outside Terraform, and it warns that "the following plan may include actions to undo or respond to these changes." The plan is doing its job, and its job is to put the instance back at the old size.
In us-east-1, an m5.xlarge lists at $0.192 an hour and an m5.2xlarge at $0.384. Over 730 hours that is $140.16 against $280.32. Every step up an instance family doubles the vCPU and the memory, and doubles the hourly rate along with them. The revert costs you the exact saving you booked, per instance, for every month it stands.
The revert does not land when you make the change
Plans are slow, so plenty of pipelines run terraform plan -refresh=false. HashiCorp's plan command reference describes that flag as disabling the synchronization of state with remote objects before checking for configuration changes. It skips the provider read entirely.
So the plan compares your configuration against the stored state, and nothing else. Both of those still say m5.2xlarge. The plan comes back empty, the pipeline goes green, and your resize survives another week. The drift is still there, and the run that would have caught it declined to look at the provider.
The correction lands on the first refreshed apply after that. In most teams that belongs to somebody else's pull request, touching a subnet tag or a security group rule, days or weeks later. Their plan refreshes state, finds your instance at the smaller size, and folds the revert into their apply.
The plan the reviewer opens proposes a change to an instance the pull request never mentions, sitting underneath the change the author actually made. People check their own diff closely and skim the rest. That is how a resize gets undone by someone who never opened the instance.
The bill climbs back a month later. By then the apply that did it is buried in CI history, and the saving reads like it never worked.
Find your drifted resources by hand
The human-readable plan output mixes drift in with the changes you meant to make. Ask for the machine-readable version instead, where the two are separate fields.
A refresh-only plan reconciles state against reality and proposes no infrastructure changes at all, so it is safe to run against production from a laptop.
terraform plan -refresh-only -out=drift.tfplan
terraform show -json drift.tfplan > drift.jsonHashiCorp's JSON output format documents resource_drift as a description of the changes Terraform detected when it compared the most recent state to the prior saved state, using the same object structure as resource_changes. Everything that moved outside Terraform is in there, and nothing you wrote is.
jq -r '.resource_drift[]? | "\(.address)\t\(.change.actions | join(","))"' drift.jsonNarrow that to the attributes carrying a price. On EC2 the attribute is instance_type. The before value is what your state believed, and the after value is what the provider just reported.
jq -r '.resource_drift[]?
| select(.type == "aws_instance")
| select(.change.before.instance_type != .change.after.instance_type)
| "\(.address) state=\(.change.before.instance_type) real=\(.change.after.instance_type)"' drift.jsonThe same query works elsewhere by swapping the type and the attribute. Compute Engine pins size in machine_type on google_compute_instance. Azure uses size on azurerm_linux_virtual_machine. Managed databases carry the same exposure in instance_class on aws_db_instance, where a size step is worth considerably more than $140.
jq -r '.resource_drift[]?
| select(.type == "google_compute_instance"
or .type == "azurerm_linux_virtual_machine"
or .type == "aws_db_instance")
| "\(.address)\t\(.type)"' drift.jsonTo gate this in CI, read the exit code instead of parsing text. terraform plan -detailed-exitcode returns 0 for an empty diff, 1 for an error, and 2 for a non-empty diff. Pair it with refresh-only and a 2 means state and reality disagree.
# -lock=false keeps a nightly read from blocking a real apply.
terraform plan -refresh-only -detailed-exitcode -lock=false
# 0 = state matches reality, 1 = error, 2 = drift foundRun that nightly per workspace. Exit code 2 on a workspace nobody touched that day is your signal, and it arrives while the change is still recent enough that someone remembers making it.
In an ordinary plan, -detailed-exitcode returns 2 for any pending change, so it cannot tell drift apart from a configuration edit you made on purpose. The resource_drift key separates them. Use the exit code to decide whether to look, and the JSON to decide what you are looking at.
Fix it in code, and mind the stop and start
The durable fix is dull: put the size you actually want into the configuration, so your code and your cloud agree and no future plan has anything to correct.
resource "aws_instance" "api" {
ami = data.aws_ami.base.id
instance_type = "m5.xlarge" # was m5.2xlarge
}When the instance already runs at the smaller size, that edit costs nothing. The next refresh writes m5.xlarge into state, your configuration asks for m5.xlarge, and the plan is empty for the first time in months.
When you drive the resize through Terraform instead, the AWS provider is explicit about the price: updates to instance_type trigger a stop and start of the EC2 instance. The instance ID and the EBS volumes survive, and the machine is still down for the duration. Run it behind an autoscaling group or a load balancer with more than one healthy target, or take a window.
AWS's stop and start guide warns that "when you stop an instance, the data on any instance store volumes is erased," and checking the root device type alone misses an EBS-backed instance that carries them. An m5d.2xlarge boots from an EBS root with local NVMe instance store attached beside it, so its root device type reads ebs and the apply still wipes those disks. An instance whose root volume is itself instance store can't be stopped at all, so the stop and start the provider needs never happens, and AWS's instance type change guide routes that case to a migration onto a new instance.
Check both before you queue the apply: the first command returns the type and the root device type, and the second reports whether that type supports instance store.
aws ec2 describe-instances --instance-ids i-0123456789abcdef0 \
--query 'Reservations[].Instances[].[InstanceType,RootDeviceType]' \
--output text
aws ec2 describe-instance-types --instance-types m5d.2xlarge \
--query 'InstanceTypes[].[InstanceType,InstanceStorageSupported]' \
--output textA root device type of instance-store means you are migrating, and the in-place instance_type update is off the table. InstanceStorageSupported reporting true means that type can carry local disks, so copy anything you still need off them before the apply.
Rollback runs the same path backwards. The size lives in version control now, so git revert on the commit plus one apply restores the old size, at the cost of a second stop and start.
The same drift catches storage. Change a disk tier in the console, say Premium SSD v1 to v2 on Azure, and the new SKU sits in the provider API while your code still pins the old one. Azure documents no direct conversion back from Premium SSD v2, so read what your plan intends to do about that difference before you apply it.
Two shortcuts that look like fixes
lifecycle { ignore_changes = [instance_type] } stops the plan from proposing the revert, at the cost of a configuration that no longer describes the machine you actually run. HashiCorp's lifecycle documentation names the limit: ignored attributes are skipped when planning an update, and they are still considered when planning a create. So the day anything replaces that instance, a new AMI, a changed subnet, user_data_replace_on_change firing, Terraform builds the replacement at the old expensive size, and the plan looks clean while it does it.
terraform apply -refresh-only is the other one. It accepts reality into state without touching infrastructure, which silences the drift note. Your configuration still asks for the larger size, so the disagreement moves out of the note and into the plan body as a pending change. The next ordinary apply reverts the instance exactly as before, with less warning than you had to begin with.
Both commands have honest uses. ignore_changes belongs on attributes another system genuinely owns, like a tag an autoscaler writes. -refresh-only belongs on an out-of-band change you made deliberately and are about to encode. Neither one settles a size you want to keep.
A quick decision tree
graph TD
A[Instance is bigger than the size you chose] --> B{Does resource_drift list it?}
B -- No --> C{Did the plan run with -refresh=false?}
C -- Yes --> D[Re-run the plan with refresh on, then look again]
C -- No --> E[State matches reality: the size in code is the size you have]
B -- Yes --> F{Do you want the smaller size?}
F -- Yes --> G[Edit instance_type in the .tf file, apply, expect a stop and start]
F -- No --> H[Apply -refresh-only to record it, then set the size in code deliberately]
Common questions
What is Terraform drift?
Drift is the gap between what your Terraform state records and what the provider API actually reports. It appears whenever a resource changes outside the Terraform workflow, through a console click, a support ticket, or another automation. terraform plan refreshes state before comparing it to your configuration, so drift shows up as a note listing objects that changed outside Terraform, and usually as a proposed change to put them back.
Why did my instance go back to its old size after I resized it?
Because the size lives in your .tf files and you changed it somewhere else. Terraform's job is to make the real world match the configuration, so a plan that refreshes state finds the smaller instance, compares it to the larger size pinned in code, and proposes an update to restore the code's version. The apply that carries out the revert is often an unrelated pull request from a different person.
How do I detect Terraform drift?
Run terraform plan -refresh-only -out=drift.tfplan, convert it with terraform show -json drift.tfplan, and read the resource_drift key. That key holds only the changes Terraform detected between the most recent state and the prior saved state, so it isolates drift from configuration edits you made yourself. For a CI gate, terraform plan -refresh-only -detailed-exitcode returns 2 when drift exists, 0 when state matches reality, and 1 on error.
Does terraform plan -refresh=false hide drift?
Yes. That flag disables synchronizing state with remote objects before the comparison, so the plan checks your configuration against the stored state alone. If both still hold the old value, the plan is empty and the pipeline passes while the drift sits there undetected. The revert then lands on the next plan that does refresh, which is usually a later apply from a different change.
Does changing instance_type in Terraform cause downtime?
It causes a stop and start. The AWS provider documents that updates to instance_type trigger a stop and start of the EC2 instance, which keeps the instance ID and the attached EBS volumes but takes the machine offline for the duration. Schedule it behind an autoscaling group or a load balancer with another healthy target. The stop also erases every instance store volume, including the local NVMe disks on an EBS-backed type like m5d, so copy that data off first. An instance with an instance store root volume can't be stopped at all, which makes resizing it a migration to a new instance rather than an in-place update.
Should I use ignore_changes to stop Terraform reverting a resize?
Only when another system genuinely owns that attribute. HashiCorp documents that ignored attributes are skipped when planning an update and still considered when planning a create, so the setting hides the conflict rather than resolving it. When the instance is eventually replaced, by a new AMI or a changed subnet, Terraform rebuilds it at the expensive size still written in your code, and the plan gives no warning.
Does terraform apply -refresh-only fix drift?
It records reality into state without changing infrastructure, which removes the drift note. Your configuration is untouched, so it still asks for the old size, and that disagreement simply moves into the plan body as a pending change. The next ordinary apply performs the revert. Use refresh-only to acknowledge a deliberate out-of-band change, then edit the configuration to match.
How much does a reverted right-size actually cost?
The full saving, every month it stands. Instance families roughly double in price at each size step, so in us-east-1 an m5.2xlarge at $0.384 an hour against an m5.xlarge at $0.192 is about $280.32 versus $140.16 over 730 hours. A revert on a managed database is worse, since an instance_class step on a Multi-AZ deployment carries the doubled rate twice over.
How OhChimp approaches this
OhChimp reads the Terraform in your repositories and pulls out the cost-relevant resources with the instance type each one declares, so the sizes pinned in code sit next to what your cloud accounts actually bill. Each oversized size becomes a plan you read before anything runs. It grades the risk and scores the confidence, and the rollback is written out step by step. The fix ships as a change to the code in your repository rather than a click in a console, which is what stops the next apply from undoing it. Applying is one button, and nothing in your account moves until you press it. OhChimp cannot yet measure what one resized cloud instance saves, so the figure on that plan stays an estimate after the change merges.
The GitHub integration page covers connecting a repository so the scan can read your Terraform. If your oversized workloads are pods rather than instances, the Kubernetes right-sizing walkthrough picks up from there.