Terraform Patterns for Observability Backends

Module Shape (all backends)

Every generated module has exactly three files:

• main.tf — all resources + provider block
• variables.tf — all input variables
• outputs.tf — key output values (dashboard URL, monitor IDs, SLO IDs)

Header comment required in every main.tf:

# Generated by otel-as-code v0.1.0 on YYYY-MM-DD.
# Re-run /otel-backend <vendor> to regenerate.
# Drift detection: v2 roadmap.

Run terraform fmt and terraform validate immediately after writing the files.
Emit the exact commands the user should run next (init, plan, apply) as a post-generation note.

Service name → resource identifiers (sanitize, but ONLY in identifier positions)

service.name is frequently an npm-scoped package name like @myorg/web (from package.json#name,
confidence 0.97) — @ and / are invalid in most resource identifiers. terraform validate
does NOT catch this (the invalid string is inside an embedded YAML / a uid the provider accepts
as an opaque string), so it only surfaces at apply against a live account. Derive a sanitized
slug and use it in identifier positions:

locals {
  service_slug = trim(replace(lower(var.service_name), "/[^a-z0-9]+/", "_"), "_")
}

• Sanitize (use local.service_slug) wherever the name becomes an identifier: a Grafana
  dashboard uid / rule-group name, a Prometheus/Dash0 alert: name, a Kubernetes
  metadata.name, any slug. Pick the separator the target format allows: _ for Prometheus alert
  names and Grafana uids (both permit _); - for a DNS-1123 metadata.name (which forbids
  _) — the two constraints have no common separator, so choose per field, don't assume one slug
  fits all.
• Do NOT sanitize the value used in a query/filter ({service_name="..."},
  {job="..."}, WHERE service.name = '...') — it must equal the emitted service.name, so it
  keeps the raw var.service_name; a slug there silently matches nothing.
• Do NOT sanitize a display title (name = "High error rate — ${var.service_name}") —
  free text, the real name reads better.

---

Grafana Cloud (grafana/grafana ~> 4.0)

Authentication variables

variable "grafana_url" {
  description = "Grafana Cloud URL (e.g. https://yourorg.grafana.net)"
  type        = string
}
variable "grafana_service_account_token" {
  description = "Service account token with Editor role"
  type        = string
  sensitive   = true
}

Required resources

1. grafana_folder — create a folder to scope all generated resources
2. grafana_dashboard — uses config_json with a JSON-encoded dashboard model
3. grafana_rule_group — unified alerting (NOT the deprecated grafana_alert_notification)
4. grafana_slo — Grafana Cloud SLOs (requires grafana_slo resource, available on Cloud plans)

Key gotchas

• grafana_folder.uid is auto-generated on create; reference it via grafana_folder.<resource_label>.uid (e.g. grafana_folder.otel_folder.uid)
• Dashboard config_json must be valid Grafana JSON; use jsonencode() to construct it safely
• grafana_rule_group requires a folder_uid (from the folder resource) and interval_seconds
• SLO query block uses PromQL expressions; prefer rate() over irate() for stability
• Pin ~> 4.0 (current major); the 2.x → 3.x → 4.x jumps each changed resource schemas — don't copy older-major patterns
• Filter by job, NOT service_name. On a default OTLP → Prometheus pipeline, service.name is mapped to the job label (and service.instance.id → instance); resource attributes are NOT added as metric labels — they live only on the separate target_info series (otlp.promote_resource_attributes defaults to []). So {service_name="..."} returns NO DATA — a silently-empty panel/alert. Use {job="<name>"}; when service.namespace is set, job becomes <namespace>/<name>, so filter {job="<namespace>/<name>"}. (Only emit {service_name="..."} if the user's Prometheus receiver sets otlp.promote_resource_attributes: [service.name].)

OTel-specific dashboard panels to generate

• Request rate: rate(http_server_request_duration_seconds_count{job="..."}[5m])
• Error rate: rate(http_server_request_duration_seconds_count{job="...",http_response_status_code=~"5.."}[5m])
• P99 latency: histogram_quantile(0.99, rate(http_server_request_duration_seconds_bucket{job="..."}[5m]))

Business-attribute panels (from confirmed businessAttrs)

Grafana panels query Prometheus, so a business attribute is only visible as a metric (or a
metric label). Sanitize the attribute name to a Prometheus metric name (dots/dashes →
underscores) as <M>, and scope every query to the service by job — exactly as the generic
panels do (service.name → job; <namespace>/<name> when a namespace is set). Emit the caveat
as the panel description so an empty panel is self-explanatory.

• kind: counter → sum(rate(<M>_total{job="<name>"}[5m])) (OTLP→Prometheus appends _total to
  counters). Caveat: requires the app to emit a counter named <M>.
• kind: gauge → avg(<M>{job="<name>"}) (or avg_over_time(<M>{job="<name>"}[5m])) — plot the
  value directly. Do NOT wrap a gauge in rate()/sum(rate()): rate() is defined only for
  counters, and summing a ratio like conversion_rate is meaningless.
• kind: dimension → sum by (<label>) (rate(http_server_request_duration_seconds_count{job="<name>"}[5m])).
  Caveat: the breakdown needs <label> to be a metric label. If the attribute is recorded as a
  per-request metric data-point attribute, it already is one; if it is only a resource
  attribute, it is NOT a label on a default OTLP→Prometheus pipeline (see the job-label gotcha
  above) unless the producer promotes it (otlp.promote_resource_attributes).

---

Datadog (DataDog/datadog ~> 4.0)

Authentication variables

variable "datadog_api_key" {
  description = "Datadog API key"
  type        = string
  sensitive   = true
}
variable "datadog_app_key" {
  description = "Datadog application key"
  type        = string
  sensitive   = true
}
variable "datadog_site" {
  description = "Datadog site (e.g. datadoghq.com, datadoghq.eu)"
  type        = string
  default     = "datadoghq.com"
}

Required resources

1. datadog_dashboard — use widget blocks (not JSON string)
2. datadog_monitor — type "metric alert" for threshold-based; "query alert" for formula-based
3. datadog_service_level_objective — type "metric" for metric-based SLOs

Key gotchas

• Both api_key AND app_key are required; setting only one will silently fail
• datadog_monitor.type must be one of: "metric alert", "service check", "event alert", "query alert", "composite", "log alert", "rum alert", "trace-analytics alert"
• SLO timeframe must be exactly "7d", "30d", or "90d" — no other values accepted
• Monitor tags must include the service tag: "service:<service_name>"
• Dashboard layout_type is either "ordered" or "free"

OTel-specific monitor queries

Datadog APM trace metrics are named trace.<operation_name>.{hits,errors,duration}. For an
OTel HTTP server span, Datadog's operation-name logic v2 (default on OTel Collector
>= v0.126.0 / Datadog Agent >= v7.65) assigns the operation name http.server.request —
NOT http.request. Use the trace.http.server.request stem:

• Request rate: sum(last_5m):sum:trace.http.server.request.hits{service:<name>}.as_rate()
• Error rate: sum(last_5m):sum:trace.http.server.request.errors{service:<name>}.as_rate() / sum:trace.http.server.request.hits{service:<name>}.as_rate() > 0.05
• P99 latency: avg(last_5m):p99:trace.http.server.request{service:<name>} > 500000000

Two gotchas baked into those queries:

• Percentiles need the bare distribution metric. trace.<op>.duration is a COUNT (total time) and does NOT support p50/p95/p99. Query the suffix-less distribution metric trace.http.server.request for percentile aggregations.
• Durations are in nanoseconds. 500000000 = 500 ms. Do not write > 500 or > 0.5.
• The operation name is pipeline-dependent. On pre-v2 collectors, or where a transform processor sets operation.name / legacy span_name_as_resource_name is configured, the stem differs. Emit a comment telling the user to confirm theirs in Datadog under APM > Metrics (Metrics Explorer, search trace.) before trusting the monitors — a monitor on a metric that never populates silently never fires.

Business-attribute panels (from confirmed businessAttrs)

• kind: counter → sum:<name>{service:<name>}.as_rate(). Caveat: requires the counter reported
  to Datadog; .as_rate() is documented for StatsD/DogStatsD rate/count metrics — for an
  OTLP-ingested counter, have the user confirm it populates, or drop .as_rate() for a plain
  sum:<name>{service:<name>}.
• kind: gauge → avg:<name>{service:<name>} — the value as-is, no .as_rate().
• kind: dimension → group the request metric by the tag:
  sum:trace.http.server.request.hits{service:<name>} by {<tag>}.as_rate(). Caveat: the tag must
  be present on the spans/metrics (an OTel span attribute promoted to a Datadog tag).

---

New Relic (newrelic/newrelic ~> 3.0)

Authentication variables

variable "newrelic_account_id" {
  description = "New Relic account ID"
  type        = number
}
variable "newrelic_api_key" {
  description = "New Relic User API key (NRAK-...)"
  type        = string
  sensitive   = true
}
variable "newrelic_region" {
  description = "New Relic region: US or EU"
  type        = string
  default     = "US"
}

Required resources

1. newrelic_one_dashboard — pages with widget blocks; use widget_line and widget_table
2. newrelic_nrql_alert_condition — NRQL-based alerts; type = "static" for threshold
3. newrelic_service_level — SLOs; events block uses valid_events / good_events (and optionally bad_events) NRQL query blocks — NOT valid / good

Key gotchas

• account_id has no provider-level default and must be set per resource — but WHERE differs by
  resource. It is a top-level argument on newrelic_one_dashboard, newrelic_alert_policy, and
  newrelic_nrql_alert_condition. On newrelic_service_level there is no top-level
  account_id — it goes inside the events block (a top-level account_id there fails
  terraform validate with "An argument named account_id is not expected here"). Verified against
  newrelic/newrelic v3.x via terraform providers schema -json; re-check if you bump the pin.
• newrelic_nrql_alert_condition requires an alert_policy_id; always create newrelic_alert_policy first
• Service level good query denominator must return a rate between 0 and 1 — divide by total count
• NRQL uses FROM Span for OTel trace data; attribute names follow OTel semconv directly
• Dashboard widgets require account_id inside the nrql_query block, not just on the resource
• Alert-condition NRQL must NOT contain SINCE / UNTIL / TIMESERIES / COMPARE WITH — New Relic rejects them in newrelic_nrql_alert_condition (the condition's own aggregation window drives timing). Those clauses are dashboard-only.

OTel-specific NRQL queries

Dashboard widget NRQL (a time window is expected — SINCE / TIMESERIES are fine here):

• Request rate: SELECT rate(count(*), 1 MINUTE) FROM Span WHERE service.name = '<name>' SINCE 5 MINUTES AGO
• Error rate: SELECT filter(count(*), WHERE otel.status_code = 'ERROR') / count(*) FROM Span WHERE service.name = '<name>' SINCE 5 MINUTES AGO
• P99 latency: SELECT percentile(duration.ms, 99) FROM Span WHERE service.name = '<name>' SINCE 5 MINUTES AGO

Alert-condition NRQL (newrelic_nrql_alert_condition.nrql.query — NO SINCE/TIMESERIES):

• Error rate: SELECT filter(count(*), WHERE otel.status_code = 'ERROR') / count(*) FROM Span WHERE service.name = '<name>'
• P99 latency: SELECT percentile(duration.ms, 99) FROM Span WHERE service.name = '<name>'

Business-attribute widgets (New Relic can query spans/metrics directly — the strongest of the four
for this). Backtick-quote dotted attribute names:

• kind: counter → SELECT sum(`<name>`) FROM Metric WHERE service.name = '<name>' TIMESERIES
  (a reported OTel counter; use rate(sum(`<name>`), 1 minute) for a per-minute rate).
• kind: gauge → SELECT average(`<name>`) FROM Metric WHERE service.name = '<name>' TIMESERIES
  (latest(...) for a level) — never sum() a ratio/level.
• kind: dimension → SELECT count(*) FROM Span WHERE service.name = '<name>' FACET `<name>` SINCE 5 MINUTES AGO — a native breakdown of traffic by the business dimension (no metric-label caveat: the attribute is on the span).

---

Dash0 (dash0hq/dash0)

Authentication variables

variable "dash0_auth_token" {
  description = "Dash0 API auth token (must start with \"auth_\" or \"dash0_at_\"). Requires management/write API access, NOT an ingestion-only scope (this module creates dashboards/check rules) — a 403 on every resource means check the token's permission scope. Create one under Organization Settings > Auth Tokens."
  type        = string
  sensitive   = true
  default     = ""
}
variable "dash0_url" {
  description = "Dash0 API endpoint (region-specific)"
  type        = string
  default     = "https://api.us-west-2.aws.dash0.com"
}
variable "dash0_dataset" {
  description = "Dash0 dataset identifier (not display name). A data-partitioning concept, UNRELATED to the shared `environment` variable — never default this to \"production\" (not a dataset Dash0 provisions; produces a 403 that reads like a permissions failure but isn't). A fresh org's actual default dataset is named \"default\"."
  type        = string
  default     = "default"
}

Provider version

Use the latest available version from the Terraform registry. Check https://registry.terraform.io/providers/dash0hq/dash0/latest for the current version and add a version constraint to required_providers. Do not omit the version constraint.

Key gotchas

• check_rule_yaml must be a full PrometheusRule document — never a flat alert body. It
  needs apiVersion: monitoring.coreos.com/v1, kind: PrometheusRule, metadata.name, and
  spec.groups wrapping the rule; Dash0 currently supports exactly one group containing exactly
  one rule per check_rule_yaml. A flat alert: / expr: / for: / ... document with no groups
  wrapper fails live apply with error converting check rule YAML to Dash0 format: currently only one group is supported (confirmed via a live apply — the golden shipped with exactly this flat,
  broken shape until this was caught). Shape (mirror this, one rule per resource):

  apiVersion: monitoring.coreos.com/v1
  kind: PrometheusRule
  metadata:
    name: <dns-1123-safe-slug>-<alert-purpose>
  spec:
    groups:
      - name: Alerting
        rules:
          - alert: <AlertName>
            expr: <promql>
            for: 5m
            labels: { severity: critical }
            annotations: { summary: "..." }

  metadata.name follows Kubernetes object-naming convention (DNS-1123: lowercase
  alphanumeric + -, no underscores) — a separate constraint from the alert: field (a free-text
  Prometheus label value, where underscores are fine and were NOT the cause of the failure above).
  Sanitize service.name into a hyphen-based slug for metadata.name specifically (the same
  @scope/pkg problem as the "Service name → resource identifiers" rule above, but with - as
  the separator here since DNS-1123 forbids _) — do not reuse an underscore-based slug for it.
• dash0_dataset must never default to "production". See the variable description above —
  confirmed live: a nonexistent dataset produces a 403 that is easily misdiagnosed as a
  token-permission problem (#102 → #103). Default it to "default" (a fresh org's actual
  out-of-the-box dataset) and say so explicitly in the description, so a later regeneration does
  not reintroduce the mistake by pattern-matching onto the shared environment variable's
  "production" default — datasets and deployment environments are unrelated Dash0 concepts.
• A dash0.com/folder-path annotation, if you add one, MUST start with a leading /. Dash0's
  live API rejects an omitted leading slash with dash0 api error: folder path must start with '/' (status: 400) (confirmed via a live apply, #104) — "otel-as-code" fails,
  "/otel-as-code" succeeds. This is a fixed literal in the generated template, not
  account-specific, so it reproduces identically for every user if it regresses.
• The auth token needs management/write scope, not ingestion-only. See the dash0_auth_token
  description above — confirmed live: an insufficiently-scoped token produces a 403 on every
  resource (#102). Dash0's own docs distinguish ingestion-scoped tokens (send telemetry) from
  management-scoped ones (manage dashboards/check rules via the API this module uses); verify the
  exact current UI label for the broader scope against Dash0's own docs/account rather than
  asserting one here, since it was not independently confirmed.

Notes for implementors

Dash0 is an OTel-native backend; their Terraform provider models resources around OTel data
directly. Before writing the terraform-gen subagent's Dash0 template, verify current resource
names at https://registry.terraform.io/providers/dash0hq/dash0/latest/docs.

As of mid-2026, the provider supports dashboards and monitoring rules. Use the registry docs
as the authoritative source for resource names and required fields — do not rely on this skill
alone for Dash0-specific field names.

Key principle: Dash0 uses OTel attribute names natively in query expressions, so no translation
layer is needed between OTel semconv and the monitoring query syntax.

Service filter: because Dash0 is OTel-native, its queries CAN filter by service directly —
service.name in the Query Builder, or service_name (dots→underscores) in PromQL panels. This
is unlike a vanilla OTLP→Prometheus pipeline, where service.name is only the job label and
{service_name="..."} returns no data (see the Grafana gotcha above). So the Dash0 golden keeps
{service_name="..."} intentionally — verify against a live Dash0 instance, as the exact PromQL
label spelling depends on the panel/query surface.

Business-attribute panels (from confirmed businessAttrs)

Dash0 panels are PromQL, but its OTel-native pipeline makes business attributes far more likely to
be queryable than a vanilla Prometheus setup (it filters by service_name directly). Sanitize the
attribute name to a PromQL metric/label (dots→underscores) as <M>.

• kind: counter → sum(rate(<M>_total{service_name="<name>"}[5m])). Caveat: requires the counter
  to be emitted.
• kind: gauge → avg(<M>{service_name="<name>"}) — the value as-is, not rated.
• kind: dimension → sum by (<M>) (rate(http_server_request_duration_seconds_count{service_name="<name>"}[5m]))
  — a breakdown by the business attribute, which Dash0's OTel-native ingestion keeps queryable as a
  label where a vanilla Prometheus pipeline would not. Verify the label spelling against a live
  instance.