← NodeJS Teams

Deployment

Configuration

Services are configured and deployed through the banno-infra and banno-apps repos.

Services are deployed to GCP through Kubernetes (k8s). See the infrastructure docs for setting up a service.

  • A NODE_ENV environment variable should be set in each environment. Use a production value for production environments.
  • If the service needs to read secrets, cryptographic keys, or database credentials (static or dynamic) from Vault, they must be added to a policy configuration. Secrets can be added to Vault manually by the infrastructure team – ask in #org-launch-control.

Deployment Process

Some services, such as node-consumer-login-proxy, have a regular deployment cadence where the firefighter on rotation deploys the latest release. Otherwise, changes should be deployed by the engineer(s) who developed them.

Builds (CI)

The GitHub “build” action will automatically run for PRs and commits to build and test the project.

Projects should define an npm script named ci:build-and-test that builds the project and runs tests.

Deploys (CD)

Note: Avoid deploying to production on Fridays or the day before a holiday.

Steps to Deploy

To kick off a deploy (e.g. once a PR is merged):

  1. Navigate to the “Actions” tab of the project repo, select “release”, and click the “Run workflow” button. Release action in GitHub

    • Releases will be tagged as the next minor version unless commit messages specify other bumps. See the github-tag-action docs.
    • Deployments normally originate from the master or main branch. A release from another branch will be tagged as a prerelease.

  2. The code will automatically be deployed to lower environments (depending on the project’s configuration). Verify that the updated code has been deployed and run your QA tests.

  3. When ready to deploy to production, navigate to the in-progress GitHub action and copy the URL. Request approval in our PR channel and include the GitHub action link. View button on Release action page

    • Approval can be provided by another team member or designated managers. Approval is granted from the GitHub action page.
    • You cannot approve your own releases.

  4. Verify with the rest of the team that the release is safe to deploy.

    • Confirm with other contributors that their changes are tested and ready. (See “Deployments That Include Outside Team Members” below.)

    • As a courtesy, post the deployment in #team-prism-ff with the release changelog or summary. If it is more than low-risk/low-impact, ask for approval with a 👍.

      • :fyinfo: Deploying node-links-aggregator v0.227.0 to prod.
      • :triage_question_or_request: Okay to deploy node-remote-deposit-capture v0.189.0 to prod?

      Alternatively, you can accomplish this in the next step by posting the banno-apps pull request in the channel for approval.

  5. Navigate to https://github.com/Banno/banno-apps/pulls and locate the pull request for the project and environment. (If you initiated the release, you will be assigned to it.)

    1. Obtain at least one approval on the PR. Contributors who are included in the changelog cannot approve the PR.
    2. Merge the pull request. This will kick off the ArgoCD process.

  6. Navigate to ArgoCD to proceed with the Kubernetes deployment. The link should be added to the GitHub PR automatically. ArgoCD links in the PR

    • Click on the Rollout object and go to the Rollout tab in the popover to view the steps.
    • Depending on the application config, there will be preset steps for percentage rollouts (canarying).
    • Use the 3-dot menu to select “Resume” to proceed to the next canary step, or “Promote-Full” for a full deployment. ArgoCD links in the PR
    • To abort a rollout, select the “Abort” option from the 3-dot menu. This will terminate the canaries and revert to the current stable version. Note that this leaves the service in a de-synced state.

Post-Deployment Monitoring

Monitor the service after deployment to check for:

  • Service availability
  • Performance degradation
  • Elevated error rates
  • Increased support issues

The Honeycomb Deployment Monitoring dashboard can assist with monitoring.

Rollbacks

To roll back from a version that has already been deployed:

  1. Navigate to the closed banno-apps PR for the release that was deployed. Scroll to the bottom and click the “Revert” button to open a new PR.
  2. Obtain an approval and merge the PR.
  3. Follow the standard rollout process in ArgoCD.

Restarts

To restart a service’s pods:

  1. Open the application in ArgoCD.
  2. Click on the 3-dot button for the Rollout object (either on the application page or in the rollout details) and choose “Restart Pods”.

“Restart Pods” option in ArgoCD

Deployments That Include Outside Team Members

If a release includes contributors from outside the Prism team, the release should be managed in an open channel. Follow this process:

  1. The firefighter will act as a “release manager” throughout the process.
  2. The release will be canaried, at no more than 5% to start.
    1. This will be initiated via a post on #org-prism, tagging each individual who has a PR that is in the build (including the release manager, if they have a PR). The canary process won’t start until each PR has been accounted for.
      • Example: Is <service name> version <version> ready to be canaried in production? @contributors
      • If an individual with a PR is unavailable, the release manager may at their discretion:
        • Contact their team and request someone to fill in for them.
        • Take responsibility for verifying the changes themselves.
        • Revert the changes and create another build.
    2. The release manager will begin the canary process.
      • If significant time has passed since requesting permission to start the canary, the release manager may ask contributors to reapprove.
        • Example: Is everyone ready for the canary to start? :thumbsup: to approve. @contributors
    3. The release manager will then request a second verification from the same individuals after they have verified their canaried code. Additional checks may be requested at higher canary percentages before moving to production.
      • Example: The canary is running at <canary percentage>%. Indicate you are verifying your changes with :eyes:. Approve for full production with :thumbsup:. @contributors
      • The canary percentage will not exceed 20% due to potential resource constraints.
      • If a change is causing issues, the release manager can terminate the canary. The release manager may then:
        • Revert the changes and create another build.
        • Wait for the contributor to resolve the issue.
  3. Once all involved contributors have verified their code, the release manager will proceed with a full production release.
    • Each contributor must allocate time to monitor their code in production for an appropriate duration to ensure there are no issues.

More Info