DocumentationNeuronDB Production
Documentation Branch: You are viewing documentation for the main branch (3.0.0-devel). Select a branch to view its documentation:

Upgrade and Rollback Guide

Overview

Complete procedures for upgrading and rolling back NeuronDB ecosystem components.

⚠️ WARNING: Always back up your database before upgrading. Test upgrades in a development environment first.

Pre-Upgrade Checklist

TaskCommandRequired
Review release notesCheck CHANGELOG.md⚠️ Critical
Backup databaseSee Backup Guide⚠️ Critical
Verify current versionhelm list -n neurondb✅ Yes
Check migration statuskubectl get jobs -n neurondb✅ Yes
Ensure resourcesCheck cluster capacity✅ Yes

Upgrade Procedure

Step 1: Backup

Ensure backup is recent

# Ensure backup is recent
kubectl get cronjob neurondb-backup -n neurondb
kubectl create job --from=cronjob/neurondb-backup manual-backup-$(date +%s) -n neurondb

Step 2: Review Changes

Dry-run upgrade

# Dry-run upgrade
helm upgrade neurondb ./helm/neurondb \
  --version <new-version> \
  --dry-run --debug \
  -f values-production-external-postgres.yaml \
  -n neurondb

Step 3: Run Migrations

Migrations run automatically via pre-upgrade hooks. Monitor:

Watch migration job

# Watch migration job
kubectl get jobs -n neurondb -w | grep migration
kubectl logs -f job/neurondb-migration-<revision> -n neurondb

Step 4: Upgrade Chart

Upgrade Helm chart

helm upgrade neurondb ./helm/neurondb \
  --version <new-version> \
  -f values-production-external-postgres.yaml \
  -n neurondb

Step 5: Monitor Upgrade

Monitor upgrade progress

# Watch pods
kubectl get pods -n neurondb -w

# Check rollout status
kubectl rollout status statefulset/neurondb-neurondb -n neurondb
kubectl rollout status deployment/neurondb-neuronagent -n neurondb

# Check logs
kubectl logs -f deployment/neurondb-neuronagent -n neurondb

Step 6: Verify Upgrade

Verify upgrade success

# Check versions
helm list -n neurondb
kubectl get pods -n neurondb -o jsonpath='{.items[*].spec.containers[*].image}'

# Test functionality
kubectl port-forward svc/neurondb-neuronagent 8080:8080 -n neurondb
curl http://localhost:8080/health

Rollback Procedure

Step 1: Identify Previous Revision

List release history

# List release history
helm history neurondb -n neurondb

# Note the revision number to rollback to

Step 2: Rollback Helm Release

Rollback to previous revision

# Rollback to previous revision
helm rollback neurondb <revision> -n neurondb

# Or rollback to previous version
helm rollback neurondb -n neurondb

Step 3: Database Rollback (if needed)

If database migrations were applied, rollback them:

Rollback database migrations

# Connect to database
kubectl exec -it statefulset/neurondb-neurondb -n neurondb -- \
  psql -U neurondb -d neurondb

# Run rollback script (if provided)
\i /path/to/rollback.sql

Step 4: Verify Rollback

Verify rollback success

# Check pod versions
kubectl get pods -n neurondb -o jsonpath='{.items[*].spec.containers[*].image}'

# Verify functionality
kubectl port-forward svc/neurondb-neuronagent 8080:8080 -n neurondb
curl http://localhost:8080/health

Zero-Downtime Upgrades

StatefulSet Rolling Update

Configure update strategy:

neurondb:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: null  # Update all pods
      maxUnavailable: 1

Canary Deployment

Gradual rollout using partition:

neurondb:
  replicas: 3
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 2  # Only update pod 2 and 3

After verification, reduce partition:

Complete canary deployment

helm upgrade neurondb ./helm/neurondb \
  --set neurondb.updateStrategy.rollingUpdate.partition=1 \
  -n neurondb

Migration Management

Check Migration Status

View migration history

# View migration history
kubectl get jobs -n neurondb | grep migration

# Check database migration version
kubectl exec -it statefulset/neurondb-neurondb -n neurondb -- \
  psql -U neurondb -d neurondb -c \
  "SELECT version, name, applied_at FROM neurondb_agent.schema_migrations ORDER BY version DESC;"

Manual Migration

If automatic migration fails:

Create migration job manually

# Create migration job manually
kubectl create job --from=cronjob/neurondb-migration manual-migration-$(date +%s) -n neurondb