4.02 Mutable vs Immutable Infrastructure
Overview
Mutable infrastructure updates servers in place; immutable infrastructure replaces them entirely. Terraform defaults to the immutable approach, which is why some resource updates show as destroy-then-create instead of an in-place change.
Abstract
Mutable infrastructure changes software and configuration on existing servers. Immutable infrastructure provisions new servers with the change already applied, then removes the old ones. Terraform follows the immutable model by default.
Why It Matters in Production
Mutable updates are prone to configuration drift — servers slowly diverging in software version, OS patch level, or config over time, making troubleshooting and future updates harder. Immutable infrastructure avoids this by guaranteeing every server matches its defined configuration at all times, which is central to reliable Infrastructure as Code workflows, versioning, and rollback.
Key Concepts
| Concept | Description |
|---|---|
| Mutable infrastructure | Existing servers are updated in place (in-place update); underlying infrastructure stays the same, only software/config changes |
| Immutable infrastructure | A new server is provisioned with the change, then the old one is deleted; servers are never modified after creation |
| Configuration drift | Gradual divergence between servers in a pool due to inconsistent in-place updates over time |
| Destroy-and-recreate | Terraform's default behavior for changes that can't be applied in place — destroy old resource, then create new one |
Common Use Cases
- Rolling out a new application or OS version across a server pool without risking partial, inconsistent upgrades.
- Using Terraform's default destroy-and-recreate behavior when changing an attribute that cannot be updated in place (e.g. file permissions on a
local_fileresource). - Maintaining a fleet of identical, versioned server images (AMIs, container images) instead of patching long-lived servers.
- Auditing infrastructure state reliably, since immutable servers can't have undocumented manual changes accumulate.
Example Configuration or Commands
In-place update (mutable) — manual upgrade path
A web server pool running NGINX is upgraded version by version on the same servers:
If web server 3 fails to upgrade due to unmet dependencies (network issues, full filesystem, or a different OS version), it stays behind:
Immutable update — Terraform's approach
Instead of patching a server, Terraform provisions a new one and deletes the old one if the new one succeeds:
resource "local_file" "pet" {
filename = "/root/pets.txt"
content = "We love pets!"
file_permission = "0700"
}
# local_file.pet must be replaced
-/+ resource "local_file" "pet" {
content = "We love pets!"
directory_permission = "0777"
~ file_permission = "0777" -> "0700" # forces replacement
filename = "/root/pet.txt"
~ id = "5f8fb950ac60f7f23ef968097cda0a1fd3c11bdf" -> (known after apply)
}
Plan: 1 to add, 0 to change, 1 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
local_file.pet: Destroying...
[id=5f8fb950ac60f7f23ef968097cda0a1fd3c11bdf]
local_file.pet: Destruction complete after 0s
local_file.pet: Creating...
local_file.pet: Creation complete after 0s
[id=5f8fb950ac60f7f23ef968097cda0a1fd3c11bdf]
Apply complete! Resources: 1 added, 0 changed, 1 destroyed.
Changing file_permission from 0777 to 0700 is not an in-place-updatable attribute, so Terraform marks the resource -/+ (destroy and recreate) and reports # forces replacement.
Note
By default, Terraform destroys the existing resource first, then creates the replacement. Lifecycle rules (covered separately) can change this order — e.g. create-before-destroy, or ignore changes entirely.
Best Practices
Best Practices
- Treat servers as disposable; never SSH in to make manual changes that aren't reflected in code.
- Bake new versions into a new image/server and replace old instances rather than patching live ones.
- Use Terraform's plan output to confirm whether a change will update in place or force replacement before applying.
- Pair immutable infrastructure with automated health checks so failed replacements are caught before old resources are removed.
Security Best Practices
Security
- Manual in-place changes (mutable infrastructure) are a common source of undocumented privilege escalation or stale credentials lingering on long-lived servers.
- Immutable infrastructure reduces attack surface over time since servers are short-lived and rebuilt from a known-good, version-controlled definition.
- Always review
# forces replacementin plan output — unexpected replacements can briefly expose a window where the resource doesn't exist (e.g. DNS, certificates) if not handled carefully.
Do and Don't
| ✅ Do | ❌ Don't |
|---|---|
| Replace servers with new versions instead of patching in place | Manually upgrade software on individual servers in a pool |
| Use Terraform plan to check for forced replacements before applying | Apply changes blindly without reviewing the -/+ diff |
| Keep old resources intact until the replacement succeeds | Delete the old resource before confirming the new one works |
| Version infrastructure definitions in code | Allow servers to drift from their defined configuration over time |
Common Mistakes
Common Mistakes
- Assuming all Terraform updates happen in place — many attribute changes force a destroy-and-recreate instead.
- Letting configuration drift build up across a mutable server pool until troubleshooting becomes difficult because each server behaves differently.
- Not checking plan output for
# forces replacement, leading to unexpected resource downtime during apply.
Troubleshooting
# Inspect whether a change will update in place or force replacement
terraform plan
# Confirm current state matches what you expect before applying
terraform show
# After an unexpected replacement, check resource history in state
terraform state list
terraform state show local_file.pet
Real-World Examples
Web Hosting Company — Drift From Inconsistent Patch Windows
Scenario: A pool of NGINX web servers patched manually during separate maintenance windows. Problem: One server failed to upgrade due to a dependency issue and was left on an older version, causing inconsistent behavior under load and confusing on-call debugging. Solution: Migrated to an immutable model — new server images were built per release and rolled into the pool, with failed servers simply discarded instead of left in a half-upgraded state. Outcome: Eliminated version drift across the server pool and simplified incident triage since every server matched a known image version.
Platform Team — Terraform Replacement Catching a Risky Change
Scenario: A team changed a file permission attribute on a managed resource expecting a quiet in-place update.
Problem: The plan showed # forces replacement, which would have silently destroyed and recreated the resource in production without review.
Solution: Caught the forced replacement in terraform plan output during code review before applying, and scheduled the change during a maintenance window.
Outcome: Avoided an unplanned resource recreation during business hours.
Startup — Immutable Images for Zero-Downtime Releases
Scenario: A small engineering team frequently shipped application updates to a fleet of VMs. Problem: In-place deploys occasionally left some VMs on stale code due to failed deploy scripts, requiring manual cleanup. Solution: Switched to building a new machine image per release and replacing instances behind a load balancer instead of patching live ones. Outcome: Failed deploys no longer left partially-updated servers in the fleet; rollbacks became as simple as redeploying the previous image.
Quick Recap
- Mutable infrastructure updates existing servers in place; immutable infrastructure replaces them with new ones.
- In-place updates risk configuration drift when some servers fail to update and others succeed.
- Immutable infrastructure avoids drift since failed updates only affect the new (not-yet-promoted) resource.
- Terraform defaults to immutable behavior: destroy old resource, then create the new one, for changes that can't be applied in place.
- Lifecycle rules can change this default ordering (create-before-destroy) or prevent destruction entirely.
Interview / Revision Notes
- Q: What's the key difference between mutable and immutable infrastructure? Mutable updates existing resources in place; immutable replaces resources entirely rather than modifying them.
- Q: What is configuration drift? Divergence between servers in a pool caused by inconsistent in-place updates over time.
- Q: What does Terraform do by default when an attribute change can't be applied in place? Destroys the existing resource, then creates a new one with the updated configuration.
- Q: How can you tell from
terraform planoutput that a resource will be replaced? The-/+symbol and the# forces replacementcomment next to the changed attribute. - Q: How can the default destroy-then-create order be changed? Using lifecycle rules in the resource block, such as
create_before_destroy.