Skip to main content

Security & Implementation Patterns

Use these controls to run Grafana alerting actions safely in production. The primary goals are to keep alerting access scoped, minimize token exposure, and prevent unintended silence mutations.

Guardrails & Restricted Operations​

Restrict each tool to approved actions and alert rule scopes. Avoid exposing unrestricted silence creation or deletion when use cases only need focused alerting visibility.

Dynamic Execution Risk

Allowing unconstrained runtime selection of operations such as Create silence or Delete Silence can lead to unintended alert silences or alert gaps. Enforce allowlists for operations or disable these tools if not required.

Fully Dynamic AI Operations​

Dynamic alert workflows can let the model choose which silences to create, which alerts to filter, or which rules to inspect at runtime. This improves flexibility but increases risk of unintended silences, alert fatigue, or access beyond intended operational boundaries.

Avoid Unconstrained Write Operation in Production

Only enable dynamic silence operations when action allowlists, duration caps, and approval workflows are all enforced.

Token Handling​

  • Treat Grafana Token as a secret. Never share it in tickets, chats, screenshots, or prompts.
  • Store tokens in secure secret storage and rotate them on a schedule.
  • Use dedicated service account tokens for MCP Express only.
  • Use a dedicated service account with alerting-only permissions.
  • Do not use personal user tokens for production automation.
  • Immediately revoke any token you suspect is exposed.

Operational Patterns​

  • Short Silence Durations: Keep silence durations short and always include a clear comment explaining why.
  • Identify Silence Creators: Use created_by values that clearly identify your on-call team or runbook identity.
  • Regular Audits: Review active silences regularly using List silence to ensure no orphaned silences.
  • Alert Rule Review: Periodically review active alert rules using List alert rule to keep your alerting strategy current.

Data Governance & Scopes​

  • Use a service account token with only alerting permissions enabled.
  • Separate read-only alert visibility tools from write-capable silence workflows to keep audits clear.
  • Validate alert rule UIDs and silence IDs before execution.

Common Setup Risks​

  • Token authentication failures: Invalid, expired, revoked, or under-scoped tokens cause authentication failures.
  • Missing or wrong field types: Missing required fields or incorrect field types result in validation errors.
  • Permission gaps: Service account missing alerting permissions will fail on alert operations.

References​