Metrics

All metrics are exposed at GET /metrics in Prometheus text format with the githubsts_ prefix.

Upgrading from a version without app pools? Every GitHub App, rate-limit, and reachability metric below now carries an additional instance label; see Migration: pool metrics instance label for the affected metric list, before/after PromQL, and a step-by-step checklist.

Authenticate scrapes

Bearer shared-secret authentication is the only authentication method specific to GET /metrics. Set GITHUBSTS_METRICS_AUTH_TOKEN to enable it:

export GITHUBSTS_METRICS_AUTH_TOKEN="replace-with-a-random-secret"

This example assumes github-sts serves HTTPS on port 8443 with a certificate trusted by the Prometheus host. See Native TLS and mTLS for server setup.

Configure the Prometheus scrape job with the same token. Prefer a file mounted from your secret manager instead of storing the token directly in prometheus.yml:

scrape_configs:
  - job_name: github-sts
    scheme: https
    static_configs:
      - targets: ["github-sts:8443"]
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/secrets/github-sts-metrics-token
    tls_config:
      ca_file: /etc/prometheus/certs/github-sts-ca.crt

The credentials file must contain only the token value. When the setting is non-empty, verify that a request without the token returns 401 and a request with the token returns 200:

curl --cacert /path/to/github-sts-ca.crt -o /dev/null -s -w '%{http_code}\n' \
  https://github-sts:8443/metrics
# 401

curl --cacert /path/to/github-sts-ca.crt -o /dev/null -s -w '%{http_code}\n' \
  -H "Authorization: Bearer $GITHUBSTS_METRICS_AUTH_TOKEN" \
  https://github-sts:8443/metrics
# 200

When the setting is empty, the endpoint remains unauthenticated. HTTPS protects the Bearer token in transit but does not authenticate the scraper. Native mTLS can authenticate clients for the entire server, including /health and /ready. A reverse proxy or service mesh can provide other authentication methods outside github-sts.

HTTP metrics

Token exchange metrics

JTI replay prevention

Policy metrics

GitHub API metrics

Every pool member (instance label) has its own rate limit series: an app with 3 instances reports 3 independent githubsts_github_rate_limit_remaining series, not one aggregate. A single-instance (non-pooled) app still carries the label, with instance equal to that app’s one normalized instance.

Where the rate limit values come from

The gauges describe each pool member’s GitHub App installation bucket. Two sources write them:

• The rate limit poller. When rate_limit_poll_enabled is on, each replica probes every installation of every pool member once per rate_limit_poll_interval. It sends a conditional GET /emojis request with a token it mints for itself, narrowed to metadata:read. The first request per token returns 200 and spends a request; later probes return 304 Not Modified and spend nothing, until the cached response is older than rate_limit_poll_force_counted_interval, at which point the poller drops the conditional header and pays for another counted 200 rather than keep trusting an untrustworthy 304 (see below). The poller also probes a member again just after a partly used bucket resets, and within seconds of a rate limit response on that member.
• github-sts’s own installation-token calls, such as target repository resolution during an exchange.

Only a response that spent a request is recorded. For installation tokens, GitHub answers requests that spend nothing (GET /rate_limit, and 304 Not Modified) with a separate, unused bucket: used is 0 and the reset is about an hour out. github-sts ignores such a reading whenever it already holds a counted one for a window that has not reset. As a result, between two counted responses a reading can overstate what is left, and githubsts_github_rate_limit_observed_timestamp shows when it was taken; rate_limit_poll_force_counted_interval bounds how long that gap can grow, at the cost of one extra spent request per instance each time it forces a re-read.

Things these gauges do not show:

• Buckets that only client tokens spend. github-sts never keeps or probes a token it minted for a client. If GitHub moves a heavily used client token onto a bucket of its own, that bucket is invisible here.
• The App JWT bucket. Headers from App JWT calls (installation lookup, token minting) are not recorded, because those calls do not use an installation token and so do not describe an installation bucket. Their 403 and 429 responses still count in githubsts_github_rate_limit_exceeded_total and githubsts_github_secondary_rate_limit_total.

When an App is installed on more than one organization, each gauge reports the installation with the least remaining.

githubsts_github_rate_limit_probe_total’s result label is one of not_modified, ok, rate_limited, unauthorized, incomplete_headers, or error. ok is a counted probe and the only probe result whose headers describe the installation bucket. not_modified spends nothing and, for installation tokens, carries no usable reading. unauthorized, incomplete_headers and error mean the poller is not producing readings.

GitHub reachability

App pool metrics

Visibility into instance selection for a pooled app (apps.<name>.instances; see Configuration). A non-pooled app is a pool of one and still emits these, with instances=1 and every selection outcome selected. The instance label throughout this page identifies one physical GitHub App within a pool, not a github-sts server replica.

githubsts_app_pool_selection_total’s outcome label is one of selected, failover, skipped_unreachable, or skipped_rate_limited. skipped_rate_limited only appears with rotation.strategy: rate_limit_aware: it counts a member that request moved to the back of the order for low quota and never tried. See Configuration.

githubsts_app_pool_exhausted_total is the signal worth alerting on: it means every instance in that app’s pool failed for one request. A single instance’s rate-limit gauge dropping doesn’t by itself mean requests are failing: the pool has already failed over around it.

Enterprise Rego bundle metrics

Check /health.security.bundle_enforcement and yaml_only_authorization_possible after every rollout. The latter is always false in required mode. In optional mode, it is true only when at least one configured App lacks both global and App-scoped bundle coverage. Each configured bundles[] status reports mandatory and the admitted policy_revision alongside digest, age, enabled state, and pull errors. Health remains liveness-only even when an exchange would fail closed.

Exchange audit events include bundle_enforcement. Once the bundle path is reached, they add bundle_digest and org_decision.applicable / org_decision.evaluated; completed evaluations add bundle_decisions, whose entries include the exact evaluated digest and policy_revision. Preserve these fields in the SIEM so an evaluated 403 org_policy_denied remains distinguishable from the bundle-related 503 errors.

Audit metrics

Other metrics

Recommended alerts

# Sustained OIDC validation errors: possible misconfiguration or attack
rate(githubsts_oidc_validation_errors_total[5m]) > 0

# JTI replay attempts: investigate immediately
rate(githubsts_jti_replay_attempts_total[5m]) > 0

# GitHub API unreachable: tokens will fail
githubsts_github_reachable == 0

# Audit events dropped: increase buffer or sink throughput
rate(githubsts_audit_events_dropped_total[5m]) > 0

# Exchange latency regression
histogram_quantile(0.99, rate(githubsts_token_exchange_duration_seconds_bucket[5m])) > 1

# Rate limit approaching exhaustion
githubsts_github_rate_limit_remaining_percent < 10

# Rate limit reading is stale: the poller has not refreshed it
time() - githubsts_github_rate_limit_observed_timestamp > 300

# Secondary rate limit active
githubsts_github_secondary_rate_limit_total > 0

# App pool exhausted: every instance failed a request
rate(githubsts_app_pool_exhausted_total[5m]) > 0

# Deployment is running in explicitly optional enterprise-policy posture
githubsts_bundle_enforcement_required == 0

# Bundle pull is failing
rate(githubsts_bundle_pull_total{result="failure"}[5m]) > 0

# Bundle signature verification is failing
rate(githubsts_bundle_verify_total{result="failure"}[5m]) > 0

# Bundle is stale; fail-closed bundles return 503 bundle_stale
githubsts_bundle_age_seconds > max_staleness_seconds

# Active Rego revision changed: review decision impact metrics
increase(githubsts_bundle_policy_revision_changes_total{result="changed"}[10m]) > 0

# Exception expires in less than seven days
githubsts_bundle_policy_exception_seconds_until_expiration < 604800

# Expired exceptions remain in the loaded bundle
githubsts_bundle_policy_exceptions_total{status="expired"} > 0

Permission narrowing

githubsts_github_tokens_issued_total labels permissions with the trust policy’s ceiling, not with the permission set the token was actually minted with. Callers may narrow a request to any subset of that ceiling at any level below it, so labelling by the effective set would create one series per requestable combination, under the control of anyone holding a valid identity. The bounded narrowed label records whether a request asked for less than the policy allows:

# share of exchanges that requested less than their policy grants
sum(rate(githubsts_github_tokens_issued_total{narrowed="true"}[1h]))
  / sum(rate(githubsts_github_tokens_issued_total[1h]))

The exact permission sets are recorded per exchange in the audit log (policy_permissions, requested_permissions, granted_permissions), which has no cardinality budget to protect.

githubsts_github_token_permission_divergence_total fires when the grant GitHub returned in its create-token response differs from what was requested. The two directions are not equally serious:

• direction="above_requested" means the token can do more than the caller asked for. This is a privilege-boundary failure and should be alerted on: it means narrowing is not being honoured upstream, and every narrowed token is broader than it claims.
• direction="below_requested" means the token can do less. It fails safe, but callers will see unexpected 403s from the GitHub API.

Permissions GitHub attaches implicitly (currently metadata) are excluded, so a healthy broker sits at zero on both.

# any over-grant at all is worth paging on
sum(increase(githubsts_github_token_permission_divergence_total{direction="above_requested"}[15m])) > 0