Rotate the payment signing key

Use this runbook to rotate HL_V1_ESHOPV1_PAYMENT_SECRETS, the server-side keyring that signs checkout recovery claims and derives stable PayPal idempotency identifiers.

This secret is generated by the operator. It is not supplied by PayPal, must not reuse the PayPal client secret, and must never be exposed to the browser, logs, tickets, documentation, or chat.

What you'll need

• write access to the E-Shop v1 Bitwarden Secrets Manager project;
• access to the E-Shop v1 GitOps and Argo CD deployment evidence;
• the ability to verify the Sandbox checkout route without exposing secret values;
• confirmation whether the rotation is planned or responds to a suspected compromise.

Understand the keyring

The Bitwarden secret value is an ordered, comma-separated keyring:

PRIMARY_KEY,PREVIOUS_KEY

The first key signs new recovery tokens and derives new provider request identifiers. Retained older keys validate and reproduce identifiers for already-persisted checkout attempts.

danger

Never remove a trusted previous key until its outstanding checkout attempts have been resolved or deliberately retired. Removing it makes those browser recovery tokens unverifiable.

Create a signing key

Generate 32 random bytes and encode them as a single-line Base64 value:

openssl rand -base64 32

Create or update this Bitwarden Secrets Manager entry:

Field	Value
Project	jarvis-at-skic-v1-hermes
Key	HL_V1_ESHOPV1_PAYMENT_SECRETS
Value	Generated key, or the ordered keyring described below

Do not paste the generated value into a shell history, Git repository, CI variable, issue, or chat message.

Perform a planned rotation

Use this path when the current key remains trusted.

1. Generate a new key.
2. Read the existing Bitwarden value through an approved secret-management interface.
3. Replace the Bitwarden value with NEW_KEY,OLD_KEY. Do not add spaces or quotes.
4. Wait for the corp-v1-eshopv1-paypal-sandbox ExternalSecret to refresh, or trigger its approved reconciliation.
5. Roll the storefront Deployment so every replica receives the same ordered keyring.
6. Verify without printing values:
   • the ExternalSecret is Ready;
   • the generated Kubernetes Secret contains the payment-secrets key;
   • every storefront pod is ready and uses the intended image digest;
   • /checkout is rendered dynamically and reports Sandbox payment as configured;
   • a new Sandbox checkout completes;
   • an existing recovery attempt signed with the previous key still targets the same PayPal order.
7. Keep the old key until all checkout attempts that may reference it are resolved or deliberately retired.
8. Remove the old key in a later reviewed change, repeat reconciliation, and repeat the verification steps.

Do not rotate the checkout-attempt identity, create a replacement PayPal order, or delete browser recovery state merely because a prior key remains in use.

Respond to a suspected compromise

A compromised key is no longer trusted and must not remain in the verification keyring.

1. Pause new Sandbox checkout activity.
2. Replace the Bitwarden value with a newly generated key only.
3. Reconcile the ExternalSecret and roll every storefront replica.
4. Treat all recovery tokens signed by the removed key as untrusted. They will fail closed and must not be automatically re-signed.
5. Reconcile outstanding provider orders through an authorized operator path before permitting any replacement checkout.
6. Record the incident timeline, affected deployment revisions, reconciliation evidence, and the decision for each outstanding provider order without recording secret values.
7. Resume checkout only after the new key is active on every replica and payment-safety verification passes.

The browser journal is not an authoritative server-side payment ledger. Re-signing old claims based only on a compromised signature would allow forged claims to become trusted. Seamless compromise recovery therefore requires a future server-side ledger that can independently prove the checkout attempt, provider order, amount, currency, and capture state.

Rotate PayPal credentials separately

Rotating CORP_V1_ESHOPV1_PAYPAL_CLIENT_SECRET does not require rotating the payment signing key. Keeping these secrets independent preserves pending recovery tokens and PayPal idempotency identifiers during ordinary provider-credential rotation.

Roll back safely

If the rollout fails before new operations begin, restore the last trusted ordered keyring and the last known-good storefront revision through the reviewed GitOps path.

If the former key may be compromised, do not restore it merely to recover availability. Keep checkout paused and reconcile outstanding orders through the incident procedure instead.

Evidence to retain

Record only non-secret evidence:

• GitOps commit and Argo CD revision;
• ExternalSecret readiness and refresh time;
• Kubernetes Secret key-name presence, never its value;
• storefront image digest, pod readiness, and restart counts;
• new-checkout and previous-key recovery outcomes;
• provider-order identifiers only where the approved incident record permits them.