One Character That Broke My Terraform Migration

TL;DR: If you're getting "Provider produces inconsistent result after apply" while migrating from Terraform 0.12 to 0.13, check if any of your AzureRM resource names start with $. Remove it and move on.

For those of us in development, every version upgrade comes with its share of headaches and the occasional existential crisis. Over the years I've done my share of upgrades, mostly in the .NET world - but nothing prepared me for a Terraform upgrade.

A Brief History of Terraform Versions

Terraform is a tool that allows you to describe infrastructure as code instead of clicking through a Cloud portal. All resources are stored in state, which tracks every change (adding, deleting, or updating attributes), and during terraform apply it only touches what actually changed - otherwise it would delete and recreate everything from scratch.

Terraform has come a long way (state especially - more on that below), but not enough is said about how significant the version jumps can be, specifically between 0.11 and 0.14.

0.12 - HashiCorp introduced a shift from stringly typed to properly typed configurations. Reference syntax changed:

# Before (0.11)
"${var.foo}"

# After (0.12)
var.foo

State gained explicit type metadata:

# Before (0.11)
"count": "3",
"enabled": "true",
"tags": "map[env:prod]"

# After (0.12)
"count": 3,
"enabled": true,
"tags": { "env": "prod" }

and for_each and dynamic blocks were introduced.

0.13 - The biggest change was adding a source address (essentially a namespace) for each provider. Before this, if you had two providers with the same name, there was no clear way to specify which one to use:

# 0.13 — explicit provider namespace
terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 3.0"
    }
  }
}

0.14 - Sensitive attributes are now tracked in state, and a dependency lock file for providers (.terraform.lock.hcl) was introduced (similar to package-lock.json in Node.js world).

Why You Can't Skip Versions

This means that if your repo is running Terraform 0.11 or lower, upgrading to the latest version (1.9.5 in my case) requires four separate deployments - meaning you need to run terraform apply four times:

0.11 -> 0.12 -> 0.13 -> 0.14 -> 1.9.5

The reason is that each version only knows how to migrate state from the previous one - 0.14 simply does not understand a 0.12 state file. This is an intentional decision by HashiCorp, as supporting arbitrary version jumps would be far more complex to maintain and test.

And that was exactly my task - upgrade Terraform from 0.11 to 1.9.5. The first deployment from 0.11 to 0.12 went smoothly, but the upgrade from 0.12 to 0.13 introduced a bug that is the topic of this post.

During the terraform apply step, I got an exception with the following message:

Error: Provider produces inconsistent result after apply

When applying changes to azurerm_servicebus_subscription_rule.example,
provider "registry.terraform.io/hashicorp/azurerm" produced an unexpected new
value: Root resource was present, but not absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

The resource itself didn't show any obvious problems:

resource "azurerm_servicebus_subscription_rule" "example" {
  name            = "$Default"
  subscription_id = azurerm_servicebus_subscription.example.id
  filter_type     = "SqlFilter"
  sql_filter      = "colour = 'red'"
}

I went with a trial and error approach - and honestly, I had the time and freedom to experiment.

Why This Is Happening

The first thing I tried was changing attributes one by one. Renaming from $Default to test-name made terraform apply succeed - so the problem was either the $ or Default itself, which might be a reserved keyword. Next try was deploying with just Default (no $), and bingo.

Why? AzureRM automatically creates a $Default rule for every Service Bus subscription - it's a catch-all that matches all messages when no other rules are defined. The $ prefix is reserved for system-level resources, meaning the Azure will refuse to create any user-defined resource with that prefix - and that conflict is what produces this very unhelpful error message.

Solution

resource "azurerm_servicebus_subscription_rule" "example" {
  name            = "$Default" # <--- Before
  subscription_id = azurerm_servicebus_subscription.example.id
  filter_type     = "SqlFilter"
  sql_filter      = "colour = 'red'"
}

resource "azurerm_servicebus_subscription_rule" "example" {
  name            = "default-rule" # <--- After
  subscription_id = azurerm_servicebus_subscription.example.id
  filter_type     = "SqlFilter"
  sql_filter      = "colour = 'red'"
}

The next two deployments went smoothly.

Solution is simple, just delete $ from resource name.