When PROD Must Die but DEV Must Live: A Tale of Terraform Stacks Day 2 Ops
Day 2 operations are where Terraform Stacks starts to feel different from the “classic” workflow. You’re no longer thinking in terms of a single workspace and a single lifecycle. You’re thinking about a shared codebase that drives multiple environments, each with its own state, approvals, and operational intent. That’s powerful, but it also introduces a new kind of tension: sometimes you need to do “surgery” on one environment while leaving the others completely untouched.
This is exactly the scenario I ran into when I had a healthy and stable DEV environment, but I needed to roll out PROD — and then immediately make a big change to PROD: redeploy it into a different region.
The Problem: PROD Needs a Reset, DEV Must Stay Stable
Let’s say PROD already exists in westus3, and now I need to move it to eastus2. On paper, this sounds like a normal variable change. In practice, Azure resource reality gets in the way.
I have resources like Key Vault and Cosmos DB that embed random strings into names. Those names can’t simply be “morphed” across regions, and when you add Azure soft delete into the mix, you can easily end up with the dreaded “resource name is unavailable” type issues because the old resource is still recoverable behind the scenes. I don’t want to spend my day fighting soft delete glitches.
So my intent is simple:
I want to purge the environment so I can recreate the resources cleanly in the new region.
But there’s a catch: I don’t want to purge everything. I want to preserve the Entra ID-related configuration I’ve already done — things like groups, assignments, identity wiring — so I don’t have to redo that work or risk breaking access patterns.
In my stack, that logic is split across components. I want to remove the component that owns the regional Azure resources (Key Vault, Cosmos, etc.) while leaving the Entra ID/access-control component intact.
The Scary Part: One Branch, Multiple Environments
Here’s where Stacks can feel a little unsettling at first. My deployments are structured such that DEV and PROD are driven from the same codebase and more precisely, the same branch. This is not how I normally manage environments. I typically maintain different environments on their own branch.
With the approach that Terraform Stacks takes, I can’t just “change PROD’s code” and not DEV’s code. I have to commit to main, and now the stack sees a new desired configuration for all environments.
At this moment, Terraform Stacks is ready to do the deed… on both environments.
But wait — I don’t want destructive action in DEV. I only want to remove the “shared” component from PROD. DEV is healthy. DEV should be a spectator.
In my case, I had approvals in place, which means nothing will auto-apply. That saved me from accidentally applying the change to DEV. But it also means the plan is now sitting there for both environments, waiting for me to approve it. And if DEV were set up for auto-apply (which today is only available in the HCP Terraform Premium tier… more on that later), this situation would be even more stressful: DEV would happily run the destructive change the moment the commit landed.
So the challenge becomes: how do I do a destructive operation in PROD without doing it in DEV, when both are tied to the same code?
The Mechanism: Removing a Component (Safely) Using removed
This is where I used a technique that feels a bit like defusing a bomb: comment out the component and introduce a removed block.
I took my shared component and commented it out:
/* //temporarily comment out the component... ick, right?
component "shared" {
source = "./src/terraform/shared"
inputs = {
location = var.primary_location
application_name = var.application_name
environment_name = var.environment_name
tags = var.tags
groups = component.access-control.groups
}
providers = {
azurerm = provider.azurerm.this
random = provider.random.this
}
}*/
Then I added a removed block:
removed {
source = "./src/terraform/shared"
from = component.shared
providers = {
azurerm = provider.azurerm.this
random = provider.random.this
}
}
That removed block is me being explicit: I’m not just “forgetting” the component exists. I’m telling Stacks, “this component used to exist, and it must be removed as part of the lifecycle.”
Operationally, this is exactly what I want for PROD. I want the shared component (the one owning those purge-worthy Azure resources) to be destroyed and removed from that environment’s state, while I keep my Entra ID/access-control component untouched.
The Key Intent: Destructive Action Only in PROD
This still feels scary because it looks like a global change. It’s sitting in the same repo and the same stack definition that governs DEV.
But my intention is not ambiguous: I am only going to approve the destructive plan in PROD.
And that’s what happened.
The destroy operation ran successfully on PROD only, because that’s the environment where I approved it. DEV did not experience destructive action. DEV remained stable.
This is the most important practical takeaway: in a multi-environment stack, the approval gate becomes your scalpel. You can commit something that could apply everywhere, but your operational intent is enforced by which environment you approve.
That’s not the same as a true “environment-scoped change,” but it does give you a workable Day 2 pattern when you need to operate on only one environment.
The Follow-Up: Reverting the Change to Restore the Component
Once PROD was purged and ready for the new region, I didn’t want to leave the code in this “component removed” posture.
So I reverted my changes to bring everything back.
Now something interesting happens that’s reassuring: DEV produces a no changes plan. That’s exactly what I want, because DEV never actually removed anything. From DEV’s perspective, nothing happened — it saw a plan waiting for approval, and it never got approved, so the state and real resources stayed aligned.
Meanwhile, PROD is now ready to be created again. The shared component is back in code, and since I already destroyed it in PROD, Terraform sees this as a fresh create in the new region.
This is the clean outcome I was aiming for:
DEV stays stable and unchanged, and PROD is rebuilt where I need it.
Why This Works: Purge What You Must, Preserve What Matters
I didn’t do this just to be dramatic. I did it because I wanted to preserve part of the environment while resetting the rest.
In my case, I specifically wanted to preserve components — most importantly the access-control component—so I could carry forward Entra ID configuration I’d already invested in. The shared component held the “regional infrastructure” resources that I wanted to replace cleanly, especially given random naming and soft delete behavior.
So this workflow ends up being a practical Day 2 pattern:
You surgically remove a component from one environment, approve destruction only there, revert the code to restore the desired topology, and then rebuild the environment cleanly — without burning down identity and access scaffolding that should remain stable over time.
Conclusion
This is the part that Terraform Stacks makes you confront: in multi-environment deployments, a single commit can represent multiple possible realities. The code is shared, but the operational intent is executed per environment — especially when approvals are in place.
In this scenario, I used the removed block to explicitly delete the shared component, approved the destructive plan only in PROD, kept DEV safe by not approving it, and then reverted the code so DEV converged back to “no changes” while PROD was ready to recreate in the new region.
I did it because I needed to purge Azure resources cleanly and avoid soft-delete issues, but I also needed to preserve the Entra ID and access-control setup I’d already configured.
That’s the real lesson here: Day 2 isn’t just about redeploying. It’s about knowing which parts of an environment are disposable — and which parts you should treat as long-lived foundations.