Back to Hub

k8s-incident-triage

Read-only Kubernetes incident triage using kubectl. Accepts natural language or structured input. Produces root-cause hypotheses, evidence, and next-step commands.

Trusted

Install

forge skills add k8s-incident-triage

Requirements

Binary Dependencies

kubectl

Egress Domains

$K8S_API_DOMAIN

Kubernetes Incident Triage

Performs read-only triage of Kubernetes workloads and namespaces using kubectl.

Supports:

  • Natural language input (human mode)
  • Structured JSON input (automation mode)

This skill NEVER mutates cluster state.

Tool Usage

This skill uses cli_execute with kubectl commands exclusively. NEVER use http_request or web_search to interact with Kubernetes. All cluster operations MUST go through kubectl via the cli_execute tool.

Tool: k8s_triage

Diagnose unhealthy Kubernetes workloads, pods, or namespaces.

Output format: Use markdown tables for pod/workload status summaries. Wrap kubectl output and log excerpts in text code blocks. Use bash for recommended next-step commands.

Input Modes

1) Human Mode (Natural Language)

Input is a plain string.

Examples:

  • triage payments-prod
  • triage deployment payments-api in payments-prod
  • why are pods pending in checkout-prod?
  • investigate crashloop in payments-prod
  • triage pod api-7c9f6d7f86-abcde in payments-prod
  • check rollout of deployment payments-api in prod

Behavior:

  • Parse namespace, workload, pod, or selector intent.
  • If namespace omitted, use $DEFAULT_NAMESPACE if set.
  • If ambiguity exists, default to namespace-level triage.
  • Never require the user to remember JSON fields.

2) Automation Mode (Structured JSON)

Input JSON schema:

{ “namespace”: “payments-prod”, “workload_kind”: “deployment”, “workload_name”: “payments-api”, “pod_name”: null, “label_selector”: null, “include_logs”: true, “logs_tail_lines”: 200, “include_previous_logs”: true, “events_limit”: 50, “include_node_diagnostics”: true, “include_metrics”: false, “output_format”: “markdown” }

Rules:

  • namespace is required.
  • If pod_name provided → pod-level triage.
  • If workload fields provided → workload-level triage.
  • Else → namespace scan.

Triage Process

Step 0 — Preconditions

Verify cluster access:

kubectl version —client kubectl cluster-info

If RBAC denies access:

  • Continue with allowed operations.
  • Explicitly report denied commands in output.

Step 1 — Fast Health Snapshot

Namespace scope:

kubectl get pods -n -o wide kubectl get deploy,sts,ds,job,cronjob -n

Workload scope:

kubectl get -n kubectl rollout status / -n —timeout=10s

Select pods in states:

  • CrashLoopBackOff
  • ImagePullBackOff
  • ErrImagePull
  • Pending / Unschedulable
  • Error
  • NotReady
  • OOMKilled
  • High restart count

Limit deep triage to $TRIAGE_MAX_PODS (default 5).

Step 2 — Events Timeline

kubectl get events -n —sort-by=.lastTimestamp | tail -n <events_limit>

Look for:

  • FailedScheduling
  • FailedMount
  • Unhealthy (probe failures)
  • Back-off pulling image
  • Evicted
  • Node pressure signals

Step 3 — Describe Pods & Workloads

For each selected pod:

kubectl describe pod -n

Capture:

  • Container state and reason
  • Restart count
  • Last termination reason
  • Probe failures
  • Volume mount issues
  • Node assignment
  • Taints / tolerations
  • Affinity constraints

If workload-level triage:

kubectl describe -n

Step 4 — Node Diagnostics (Optional)

kubectl get nodes -o wide kubectl describe node

Check for:

  • NotReady
  • MemoryPressure
  • DiskPressure
  • PIDPressure
  • Evictions

Step 5 — Logs (Optional)

kubectl logs -n —all-containers —tail=

If restart loops and include_previous_logs=true:

kubectl logs -n —previous —all-containers —tail=

Rules:

  • Prefer previous logs for CrashLoopBackOff.
  • Redact obvious sensitive patterns.
  • Never print Secret values.

Step 6 — Optional Metrics

If enabled:

kubectl top pods -n kubectl top node

Gracefully skip if metrics-server is unavailable.

Detection Heuristics

Classify detected issues into:

  • CrashLoop / Application Crash
  • OOMKilled / Resource Limits
  • Image Pull Failure
  • Scheduling Constraint
  • Probe Failure
  • PVC / Volume Failure
  • Node Pressure / Eviction
  • Rollout Stuck
  • Unknown / Escalation Needed

For each issue provide:

  • Hypothesis
  • Supporting evidence
  • Confidence score (0.0–1.0)
  • Recommended next commands

Output Structure

1) Incident Summary

  • Namespace
  • Scope (pod/workload/namespace)
  • Affected resources count
  • Timestamp

2) Top Findings

Concise bullet summary.

3) Likely Root Causes (Top 3)

For each:

  • Description
  • Evidence excerpt
  • Confidence
  • Recommended actions

4) Next Commands

Copy-paste kubectl commands.

5) Evidence Appendix

  • Events (recent)
  • Describe excerpts
  • Log excerpts

Safety Constraints

This skill MUST:

  • Perform read-only kubectl operations only.
  • Never execute:
    • apply
    • patch
    • delete
    • exec
    • port-forward
    • scale
    • rollout restart
  • Never print Secret values.
  • Avoid dumping full environment variables.

Autonomous Compatibility

This skill is designed to be invoked by:

  • k8s_alert_handler (alert-triggered triage)
  • k8s_patrol (scheduled patrol scans)
  • Humans via natural language CLI

It must:

  • Be idempotent
  • Produce deterministic fingerprints
  • Avoid excessive cluster load
  • Limit deep triage scope

Security Summary

Trust Level: Trusted

This skill has been reviewed and signed by the Forge team.

Network Access

This skill communicates with: $K8S_API_DOMAIN

Publisher: Forge

Back to Hub