Upgrades

Compatibility policy

github-sts follows Semantic Versioning. While the project is pre-1.0:

• Minor (0.X.0): new features and may contain breaking changes; read release notes before upgrading.
• Patch (0.0.X): bug fixes and non-breaking improvements.

Breaking changes are called out at the top of each release’s notes on GitHub.

Release process

1. Release notes are auto-generated by release-please from Conventional Commits.
2. Breaking changes are called out manually in the release description.
3. Docker images are built and signed with cosign keyless signing.
4. The Helm chart is updated in a coordinated release.

See Releases for the full history.

Migration: audience field

The audience field is mandatory in every trust policy. A policy without it is rejected at parse time.

Before:

issuer: https://token.actions.githubusercontent.com
subject: repo:org/repo:ref:refs/heads/main
permissions:
  contents: read

After:

issuer: https://token.actions.githubusercontent.com
subject: repo:org/repo:ref:refs/heads/main
audience: https://sts.example.com
permissions:
  contents: read

Step-by-step migration:

1. Choose an audience value unique to your STS deployment (e.g., https://sts.example.com).
2. Add audience: to every trust policy.
3. Update every workflow’s core.getIDToken() call to use the chosen audience.
4. (Optional) Set GITHUBSTS_OIDC_REQUIRED_AUDIENCE as defense-in-depth.
5. Deploy the updated github-sts version.
6. Verify exchanges succeed; check audience_mismatch errors in metrics and logs.

immutable_subject is not a supported field

The immutable_subject field is not part of the current policy schema. Unknown keys are ignored by the YAML parser, so a policy containing immutable_subject is accepted but the field has no effect. Use subject for an exact match or subject_pattern for a regex.

Before:

issuer: https://token.actions.githubusercontent.com
immutable_subject: repo:myorg/myrepo:ref:refs/heads/main
permissions:
  contents: read

After:

issuer: https://token.actions.githubusercontent.com
subject: repo:myorg/myrepo:ref:refs/heads/main
audience: https://sts.example.com
permissions:
  contents: read

Migration: immutable GitHub identity

GitHub.com trust policies require github.sources[] and github.target (exact immutable owner_id/repository_id pairs) in addition to a workload selector (subject, subject_pattern, or claim_pattern). See Trust Policies for the schema and examples.

For a repository still using GitHub’s legacy subject format, coordinate these steps to avoid locking out its workflow:

1. Record the immutable owner and repository IDs.
2. Opt the repository in to immutable subject claims in GitHub.
3. Confirm a newly minted token contains the expected IDs in sub, repository_owner_id, and repository_id.
4. Update its target trust policies to add github.sources[]/github.target.
5. Deploy the default-on broker configuration (oidc.require_immutable_subject_claims: true).

The checked-in Depthmark/github-sts release policy expects owner ID 268749784 and repository ID 1198676434. Opt that repository in before merging the policy migration. The legacy server opt-out (GITHUBSTS_OIDC_REQUIRE_IMMUTABLE_SUBJECT_CLAIMS=false) does not make an exact immutable trust policy match a legacy token.

Migration: pool metrics instance label

Upgrading to a version with app pools adds an instance label to every GitHub App, rate-limit, and reachability metric that already existed. This applies to every deployment, not only ones that configure instances:: a single-instance app is normalized into a pool of one internally, so it still carries the label (see Metrics):

githubsts_token_exchanges_total
githubsts_token_exchange_duration_seconds
githubsts_github_api_calls_total
githubsts_github_tokens_issued_total
githubsts_github_rate_limit_limit
githubsts_github_rate_limit_remaining
githubsts_github_rate_limit_used
githubsts_github_rate_limit_reset_timestamp
githubsts_github_rate_limit_remaining_percent
githubsts_github_rate_limit_exceeded_total
githubsts_github_secondary_rate_limit_total
githubsts_github_secondary_rate_limit_retry_after_seconds
githubsts_github_reachable
githubsts_github_reachability_check_duration_seconds
githubsts_github_reachability_failures_total

If every app stays single-instance, series counts are unchanged: each of the metrics above still has exactly one series per app (per resource, per endpoint, etc.), just with one more label attached. A query that doesn’t assert an exact label set continues to return the same result. Most dashboards and alerts need no change.

If an app adopts a pool of N instances, that app’s series count for each metric above multiplies by N: one series per instance, not one aggregate. A panel or alert written when the label didn’t exist will now show, or alert on, N separate series instead of one.

Before (single instance, or before upgrading):

githubsts_github_rate_limit_remaining_percent{github_app="checkout"} < 10

After (aggregate across however many instances the app has, so the query keeps its old meaning regardless of pool size):

min(githubsts_github_rate_limit_remaining_percent{github_app="checkout"}) by (github_app) < 10

Or, to alert on a specific credential instead of the whole logical app (new capability, not required):

githubsts_github_rate_limit_remaining_percent{github_app="checkout", github_app_instance="checkout-2"} < 10

Step-by-step migration:

1. Before upgrading, list dashboard panels and alert rules that reference any metric in the list above.
2. For each one, decide whether it should keep reporting one value per app (wrap it in sum/min/max by (...) with instance excluded from the grouping) or start reporting per-instance detail (add instance to the by (...) clause instead).
3. Update those queries before or immediately after the upgrade; the label change takes effect the moment the new version starts serving traffic.
4. Deploy.
5. Confirm panels render the expected series count and alerts still fire on the expected condition.

Migration: permissions in the exchange response

Upgrading to a version with permission narrowing changes what permissions in a successful /sts/exchange response describes.

Before: the trust policy’s permission set, echoed back from the policy file.

After: the grant GitHub actually attached to the token, read back from GitHub’s create-token response.

For a caller that does not narrow, the two agree on every permission the policy names, with one addition: GitHub attaches metadata: read to every installation token, so the response now includes it where it previously did not. Callers that assert exact equality between this field and their trust policy will see a diff:

// policy: {"contents": "write"}
// before
{ "permissions": {"contents": "write"} }
// after
{ "permissions": {"contents": "write", "metadata": "read"} }

Step-by-step migration:

1. Search clients for code that compares the response permissions against an expected set. Assertions on individual keys are unaffected; assertions on the whole map need metadata accounted for.
2. If a client used this field to discover what an identity is allowed to request, point it at the trust policy instead. The response describes one issued token, not the policy ceiling.
3. Deploy.
4. Confirm githubsts_github_token_permission_divergence_total stays at zero. A non-zero direction="above_requested" means GitHub issued a broader token than was asked for and should be investigated before relying on narrowing.

Migration: narrowed label on githubsts_github_tokens_issued_total

The same upgrade adds a narrowed label ("true" / "false") to githubsts_github_tokens_issued_total. The change is additive: a query that does not assert an exact label set returns the same result as before, and the existing permissions label keeps reporting the trust policy’s ceiling rather than the possibly-narrowed effective set, so its cardinality is unchanged.

Only queries that pin an exact label set, or that group by every label, need updating. To keep the previous meaning regardless of narrowing:

sum(rate(githubsts_github_tokens_issued_total[5m])) by (github_app, scope)

The same release also makes the permissions label value deterministic. It was previously built by iterating a Go map, whose order is randomized, so one permission set could render as contents:write,issues:write on one call and issues:write,contents:write on the next – silently splitting a single logical series across up to n! variants. The value is now sorted by permission name.

Expect a one-time series shift on upgrade: the old unsorted variants stop receiving samples and a single sorted series takes over. Counters are cumulative per series, so a rate() or increase() over a window spanning the upgrade may dip. Aggregate away the label if you need continuity across the boundary:

sum(rate(githubsts_github_tokens_issued_total[5m])) by (github_app, scope)

Migration: metric labels renamed to avoid Kubernetes collisions

This release renames three metric labels on every githubsts_ metric. Every dashboard query and alert rule that selects on them must be updated.

A collision here never raises an error. The scrape wins silently, and the application’s own value is either renamed to exported_* or lost outright. The failure only shows up as a panel that returns no rows.

That was already happening. A “GitHub App Instance” dashboard variable built on exported_instance was permanently empty, because that label never existed on any series: the scrape had taken instance and the pool member’s value did not survive. Under the old names, there was no queryable per-instance label on metrics at all. After this rename, github_app_instance exists and works.

Before:

githubsts_github_rate_limit_remaining_percent{app="checkout", instance="checkout-2"} < 10
sum(rate(githubsts_github_tokens_issued_total[5m])) by (app)
githubsts_github_api_calls_total{endpoint="create_token"}

After:

githubsts_github_rate_limit_remaining_percent{github_app="checkout", github_app_instance="checkout-2"} < 10
sum(rate(githubsts_github_tokens_issued_total[5m])) by (github_app)
githubsts_github_api_calls_total{api_endpoint="create_token"}

Step-by-step migration:

1. Grep dashboards and alert rules for githubsts_ and list every query selecting or grouping on app, instance, or endpoint.
2. Rewrite them with the new names. Any query that previously used exported_app or exported_instance should now use the plain github_app / github_app_instance – the exported_ prefix existed only because of the collision and is gone with it.
3. Deploy, then confirm panels return rows. A silently empty panel is the exact failure mode this rename fixes, so verify rather than assume.
4. Series continuity does not carry across a label rename: old series stop and new ones begin. Range queries spanning the upgrade will show a break.

A guard test (internal/metrics/labels_test.go) now fails the build if any metric declares app, instance, job, endpoint, service, namespace, pod, container, or node, so this class of collision cannot come back.

Pre-upgrade checklist

1. Read the release notes for breaking changes.
2. Check the Compatibility matrix for verified component combinations.
3. Test in a staging environment.
4. Update trust policies if schema changes are required.
5. Deploy with a canary or blue-green strategy.
6. Monitor githubsts_oidc_validation_errors_total and githubsts_token_exchanges_total for regressions.