Upgrade and Rollback Guide

[!WARNING] Always back up your database before upgrading. Test upgrades in a development environment first.

Pre-Upgrade Checklist

Upgrade Procedure

Step 1: Backup

Copy
# Ensure backup is recent
kubectl get cronjob neurondb-backup -n neurondb
kubectl create job --from=cronjob/neurondb-backup manual-backup-$(date +%s) -n neurondb

Step 2: Review Changes

Copy
# Dry-run upgrade
helm upgrade neurondb ./helm/neurondb \
  --version <new-version> \
  --dry-run --debug \
  -f values-production-external-postgres.yaml \
  -n neurondb

Step 3: Run Migrations

Migrations run automatically via pre-upgrade hooks. Monitor:

Copy
# Watch migration job
kubectl get jobs -n neurondb -w | grep migration
kubectl logs -f job/neurondb-migration-<revision> -n neurondb

Step 4: Upgrade Chart

Copy
helm upgrade neurondb ./helm/neurondb \
  --version <new-version> \
  -f values-production-external-postgres.yaml \
  -n neurondb

Step 5: Monitor Upgrade

Copy
# Watch pods
kubectl get pods -n neurondb -w

# Check rollout status (PostgreSQL/NeuronDB)
kubectl rollout status statefulset/neurondb-neurondb -n neurondb

# Check logs
kubectl logs -f statefulset/neurondb-neurondb -n neurondb

Step 6: Verify Upgrade

Copy
# Check versions
helm list -n neurondb
kubectl get pods -n neurondb -o jsonpath='{.items[*].spec.containers[*].image}'

# Test database and extension
kubectl port-forward svc/neurondb-neurondb 5432:5432 -n neurondb
# In another terminal:
psql -h localhost -p 5432 -U neurondb -d neurondb -c "SELECT neurondb.version();"

Rollback Procedure

Step 1: Identify Previous Revision

Copy
# List release history
helm history neurondb -n neurondb

# Note the revision number to rollback to

Step 2: Rollback Helm Release

Copy
# Rollback to previous revision
helm rollback neurondb <revision> -n neurondb

# Or rollback to previous version
helm rollback neurondb -n neurondb

Step 3: Database Rollback (if needed)

If database migrations were applied, rollback them:

Copy
# Connect to database
kubectl exec -it statefulset/neurondb-neurondb -n neurondb -- \
  psql -U neurondb -d neurondb

# Run rollback script (if provided)
\i /path/to/rollback.sql

Step 4: Verify Rollback

Copy
# Check pod versions
kubectl get pods -n neurondb -o jsonpath='{.items[*].spec.containers[*].image}'

# Verify database and extension
kubectl port-forward svc/neurondb-neurondb 5432:5432 -n neurondb
# In another terminal:
psql -h localhost -p 5432 -U neurondb -d neurondb -c "SELECT neurondb.version();"

Zero-Downtime Upgrades

StatefulSet Rolling Update

Configure update strategy:

neurondb:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: null  # Update all pods
      maxUnavailable: 1

Canary Deployment

Gradual rollout using partition:

neurondb:
  replicas: 3
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 2  # Only update pod 2 and 3

After verification, reduce partition:

Copy
helm upgrade neurondb ./helm/neurondb \
  --set neurondb.updateStrategy.rollingUpdate.partition=1 \
  -n neurondb

Migration Management

Check Migration Status

Copy
# View migration history
kubectl get jobs -n neurondb | grep migration

# Check database migration version
kubectl exec -it statefulset/neurondb-neurondb -n neurondb -- \
  psql -U neurondb -d neurondb -c \
  "SELECT version, name, applied_at FROM neurondb_agent.schema_migrations ORDER BY version DESC;"

Manual Migration

If automatic migration fails:

Copy
# Create migration job manually
kubectl create job --from=cronjob/neurondb-migration manual-migration -n neurondb

# Or run migration script directly
kubectl exec -it statefulset/neurondb-neurondb -n neurondb -- \
  psql -U neurondb -d neurondb -f /path/to/migration.sql

Troubleshooting

Upgrade Fails

1. Check migration job:

Copy
kubectl describe job neurondb-migration-<revision> -n neurondb
kubectl logs job/neurondb-migration-<revision> -n neurondb

2. Check pod status:

Copy
kubectl describe pod <pod-name> -n neurondb
kubectl logs <pod-name> -n neurondb

3. Rollback if needed:

Copy
helm rollback neurondb -n neurondb

Migration Fails

1. Check migration logs:

Copy
kubectl logs job/neurondb-migration-<revision> -n neurondb

2. Fix migration issues and retry:

Copy
# Delete failed job
kubectl delete job neurondb-migration-<revision> -n neurondb

# Retry upgrade
helm upgrade neurondb ./helm/neurondb --version <version> -n neurondb

Rollback Fails

1. Check release history:

Copy
helm history neurondb -n neurondb

2. Manual rollback:

Copy
# Delete current resources
helm uninstall neurondb -n neurondb

# Reinstall previous version
helm install neurondb ./helm/neurondb \
  --version <previous-version> \
  -f values-production-external-postgres.yaml \
  -n neurondb

Best Practices

[!IMPORTANT] Follow these practices for safe upgrades.

Related Documentation