Skip to content
Documentation
Operations
Readme

This directory contains operational runbooks for common Keystone Core procedures. Runbooks are written for operators on production hosts — if you’re looking for a first-time install tutorial, use ../project/GETTING-STARTED.md instead; if you’re hacking on the source, see ../project/DEVELOPMENT.md .

v0.1.x scope note. Every runbook in this directory was rewritten during Phase E of the public-launch checklist to match v0.1 reality — the apt install / dnf install package path, systemd-managed services, and kscorectl for CLI interaction. Each runbook carries an inline **v0.x scope note:** callout wherever v0.1 narrows the full v1.0 surface (e.g. multi-node cluster operations that depend on capabilities not yet wired at boot).

I’m here because…

…I need to install Keystone Core on a new host. → bootstrap-new-cluster.md . The dense reference walkthrough — package install, postinst behavior, filesystem layout, port table, verification matrix.

…I’m upgrading or scheduling maintenance. → upgrade-cluster.md for the single-node upgrade path. scheduled-maintenance.md for planned-window procedures.

…something broke; I need to roll back fast. → emergency-rollback.md . Four v0.1 scenarios (package downgrade, state rollback, config revert, catastrophic restore).

…I need to back up or restore. → backup-restore.md for routine procedures; disaster-recovery.md for the six concrete DR scenarios (SQLite / Postgres / JetStream / Identity / Config / Total-host-loss) + the RTO/RPO drill procedure.

…there’s a security incident. → security-incident.md . Five-phase response (immediate / investigation / containment / verification / post-incident) with real kscore-audit log / kscore-events subscribe / kscore-identity * commands.

…something is slow or timing out. → performance-degradation.md for the diagnostic tree; capacity-scaling.md for “is it time to scale?” decisions.

…I need to rotate TLS certificates. → certificate-rotation.md .

…something’s wrong but I’m not sure what. → troubleshooting.md . The general diagnostic tree — server-won’t-start, agent-won’t-register, journal walkthroughs, diagnostic-bundle script.

Available Runbooks

RunbookDescriptionWhen to Use
bootstrap-new-cluster.mdBootstrap a new Keystone Core clusterInitial deployment
scheduled-maintenance.mdPerform scheduled maintenancePlanned maintenance windows
emergency-rollback.mdEmergency rollback procedureFailed upgrade or critical issues
disaster-recovery.mdDisaster recovery proceduresComplete cluster loss
certificate-rotation.mdRotate TLS certificatesCertificate expiry or compromise
backup-restore.mdBackup and restore proceduresData protection and recovery
upgrade-cluster.mdUpgrade Keystone Core clusterVersion upgrades
troubleshooting.mdCommon troubleshooting proceduresDiagnosing issues
performance-degradation.mdDiagnose and remediate performance issuesAPI latency, timeouts, slowness
security-incident.mdSecurity incident response proceduresBreaches, credential compromise, attacks
capacity-scaling.mdScale infrastructure for increased loadGrowth, resource constraints

Runbook Format

Each runbook follows a consistent format:

  1. Overview - What the runbook covers
  2. Prerequisites - Required access, tools, information
  3. Trigger Conditions - When to execute this runbook
  4. Procedure - Step-by-step instructions
  5. Verification - How to verify success
  6. Rollback - How to undo if needed
  7. Post-Procedure - Cleanup and documentation

Using Runbooks

  1. Read the entire runbook before starting
  2. Gather all prerequisites
  3. Notify stakeholders if required
  4. Execute steps in order
  5. Document any deviations
  6. Complete verification steps
  7. Update incident/change ticket

Contributing

When creating or updating runbooks:

  • Use clear, unambiguous language
  • Include exact commands (copy-pasteable)
  • Add verification steps after each major action
  • Include rollback procedures
  • Test procedures in staging first
  • Keep runbooks up to date with product changes
Security Incident