[SERVER] Resolving RabbitMQ Upgrade Issues in CircleCI Server 4.7.5

Overview

When upgrading CircleCI Server from version 4.7.4 to 4.7.5, users may encounter issues with RabbitMQ due to a required feature flag not being enabled. "Feature flags: `classic_mirrored_queue_version`: required feature flag not enabled! It must be enabled before upgrading RabbitMQ."  The problem arises from the upgrade of RabbitMQ from version 3.11 to 3.12, which necessitates enabling all feature flags from the previous version.

Prerequisites

• Access to the Kubernetes cluster where CircleCI Server is deployed.

Instructions

1. Rollback RabbitMQ StatefulSet Revision:
   • If you have already upgrade to 4.7.5. Revert the RabbitMQ StatefulSet to its previous revision to restore functionality temporarily. If not run the 2. command before upgrading.
2. Enable Required Feature Flags:
   • Execute the following command to enable all feature flags on RabbitMQ: kubectl exec -it rabbitmq-0 -- rabbitmqctl enable_feature_flag all
3. Re-run Helm Upgrade:
   • Perform the Helm upgrade again to apply the changes with the feature flags enabled.

Solution

The above steps resolve the issue by ensuring that all necessary feature flags are enabled before upgrading RabbitMQ. This allows the upgrade to proceed without errors, and all pods should start successfully, ensuring CircleCI Server operates correctly.

RabbitMQ 3.12.0 will require all feature flags from the 3.11.x release series to be enabled before upgrading, similarly to how 3.11.0 required all feature flags introduced before 3.9.0.

If the feature flags are not enabled, RabbitMQ 3.12 and later nodes will refuse to start.
https://github.com/rabbitmq/rabbitmq-server/blob/main/release-notes/3.12.0.md#required-feature-flags

Additional Resources

• RabbitMQ 3.12.0 Release Notes
• RabbitMQ Feature Flags Documentation
• CircleCI Documentation Update