Skip to main content

Deployment Strategy

This document outlines the suggested patterns for deploying software and infrastructure.

Description​

Software and infrastructure requires deployment to environments such that it can be used effectively. Having a robust, continuous deployment strategy reduces the burden to push code, verify code is pushed correctly and frees up developers to develop rather than manage releases.

At CRUK we suggest two patterns of deploying infrastructure and code; these are GitHub Actions and CDK Pipelines.

It is also worth noting that what works for one team may not work for another. Consider what services will need to be deployed and how long this would take to chose what CD strategy to take. These are our recommended patterns but if you know better ones that would suite your product team better then, provided it has the rationale, you can chose this.

GitHub Actions​

GitHub actions uses workflows and jobs to run arbitrary commands and actions against any API (providing access available) and source. As it works outside of AWS you can apply this strategy to non-AWS projects and ones that don't specifically use the CDK. GitHub actions are really powerful and can be used for automated checks, integration tests and deployments. Because of this it is the preferred approach to CI/CD at CRUK.

It is recommended to structure your workflow files as follows. More details on each section and best practices follows.

  1. PR Checks - used to run static code checks against a PR (unit tests, cypress, eslint, prettier etc)
  2. PR Deployment - used to deploy an ephemeral environment of your infrastructure; to use to test at the PR level. This can also include integration and e2e tests where applicable.
  3. PR Destroy - used to destroy an ephemeral environment.
  4. Environment Deployment (Integration, Prod etc) - the CI/CD pipeline used to deploy to other environments. This can also include checks between stages and verification/smoke tests where applicable.
note

Some teams have historically used a combination of GitHub actions and CodePipelines with GitHub actions being used to invoke a CodePipeline. We do not recommend this approach as it reduces visibility and makes it difficult to manage two CD processes.

When we refer to deployments in the following documentation we refer to the deployment of the source itself not the deployment of another CD process. In the example of AWS this would be the cdk deploy command.

PR Checks​

This workflow is used to run static code checks against a PR. Each check should have it's own job within the workflow and run in non-blocking parallel. This check should be marked as required and should be run on every PR no matter how small.

Ideally these checks are intended to be quick and provide confidence that the PR meets basic requirements.

Checks can include, but are not limited to:

  • Unit tests
  • Cypress tests (against localhost)
  • ESLint verification
  • Prettier verification
  • Spell checking
  • Commit message checking
  • PR message checking

The idea for these tests are to reduce time manually reviewing trivial things. Only after these checks have passed should a human review the code.

Example
name: Pull Request Checks

on:
  pull_request:
    branches:
      - main

jobs:
  test:
    name: unit tests
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - uses: actions/setup-node@v2
        with:
          node-version: "16"
      - name: Run Tests for Example App
        working-directory: apps/example-app
        run: |
          npm ci
          npm run test
  formatting:
    name: formatting check
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - uses: actions/setup-node@v2
        with:
          node-version: "16"
      - name: Run Prettier Check
        run: |
          npm ci
          npm run prettier-check
  eslint:
    name: eslint check
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - uses: actions/setup-node@v2
        with:
          node-version: "16"
      - name: Run ESLint Check
        run: |
          npm ci
          npm run eslint-check

PR Deployment​

This workflow is used to deploy an ephemeral environment of your infrastructure; to use to test at the PR level. For this AWS connectivity is usually required.

Use the cruk-configure-aws-credentials action to configure credentials to the AWS account in question; ensuring secrets are stored and rotated frequently in GitHub secrets. This action provides the workflow access to run AWS commands such as CDK deploy.

In GitHub it is recommended to use Environments. By doing this you can departmentalize Development, Integration and Production secrets such that you cannot deploy to Production accidentally. In the below example we see the line environment: Development which means environment rules are applied to this GitHub action. The DEV_DEPLOY_ROLE secret should be assigned under this Environment.

This workflow should run on the existence of the Deploy 🚀 label against the PR. The rationale for this is to avoid deploying infrastructure on trivial changes. (Alternatively if you are using monorepo tools and a smart filter you could work this out in the action itself.)

Further checks can be placed in this workflow to verify a successful PR deployment such as integration, e2e and UI checks.

Example
name: Pull Request Deploy

on:
  pull_request:
    branches:
      - main
    types: [labeled, opened, synchronize]

env:
  ENVIRONMENT_NAME: "pr"
  GITHUB_PR_NUMBER: ${{ github.event.number }}

concurrency:
  group: pr-deploy-${{ github.event.number }}

jobs:
  deploy:
     if: |
      (github.event.action == 'labeled' && github.event.label.name == ':rocket: Deploy') ||
      (github.event.action != 'labeled' && contains(github.event.pull_request.labels.*.name, ':rocket: Deploy'))
    runs-on: ubuntu-latest
    environment: Development
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-node@v3
        with:
          node-version: "16"
      - name: Configure AWS Credentials
        uses: CRUKorg/cruk-configure-aws-credentials@v1
        with:
          aws-region: eu-west-2
          aws-role-arn: ${{ secrets.DEV_DEPLOY_ROLE }}
      - name: Deploy Example App
        working-directory: apps/example-app
        run: |
          npm ci
          npm run cdk deploy -- --all --require-approval never --outputs-file ./cdk-outputs.json
      - name: Post Deployment Checks
        run: true # Put any post checks/tests here

PR Destroy​

This workflow is used to destroy an ephemeral environment. It's access is the same as PR Deploy but is used to clean up resources when a PR is merged, closed or Deploy 🚀 label removed.

Example
name: Pull Request Destroy

on:
  pull_request:
    branches:
      - main
    types: [unlabeled, closed]

env:
  ENVIRONMENT_NAME: "pr"
  GITHUB_PR_NUMBER: ${{ github.event.number }}

concurrency:
  group: pr-deploy-${{ github.event.number }}

jobs:
  deploy:
    if: |
      (github.event.action == 'unlabeled' && github.event.label.name == ':rocket: Deploy') ||
      (github.event.action == 'closed' && contains(github.event.pull_request.labels.*.name, ':rocket: Deploy'))
    runs-on: ubuntu-latest
    environment: Development
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-node@v3
        with:
          node-version: "16"
      - name: Configure AWS Credentials
        uses: CRUKorg/cruk-configure-aws-credentials@v1
        with:
          aws-region: eu-west-2
          aws-role-arn: ${{ secrets.DEV_DEPLOY_ROLE }}
      - name: Deploy Example App
        working-directory: apps/example-app
        run: |
          npm ci
          npm run destroy -- --all --force

Environment Deployment (Integration, Prod etc)​

This is the core workflow, the CI/CD pipeline used to deploy to other environments. This can also include checks between stages and verification/smoke tests where applicable.

It should be triggered on a merge to the main branch to automatically execute. If this is not possible automatically an alternative is to trigger on GitHub releases; creating a release in GitHub is the manual step where the user must define the version, tag and changelog.

The access for this will be the different per-environment. Use the same structures and GitHub Actions best practices to define secrets in the appropriate location.

Automated pre-prod and post-prod checks should also be run as part of the CI/CD workflow. Checks such as smoke tests and monitor counts can be run post-deployment to verify that the system is still working normally. If it is not trigger alerts to Slack on failures.

This step can be hooked into service-now to automatically create a standard change request.

Example
name: "My Cool App Deployment"

on:
  push:
    branches:
      - main

env:
  CI: true

jobs:
  deploy-dev-static:
    name: Deploy Development Static Resources
    runs-on: ubuntu-latest
    environment: Development
    concurrency: my-cool-app-dev-static
    permissions:
      id-token: write
      contents: read
      actions: read
    steps:
      - uses: actions/checkout@v3
      - name: Configure AWS Credentials
        uses: CRUKorg/cruk-configure-aws-credentials@v1
        with:
          aws-region: eu-west-2
          aws-role-arn: ${{ secrets.DEV_DEPLOY_ROLE }}
      - name: Deploy Shared Development Infrastructure
        run: true # Any shared development infrastructure (large databases, user pools, etc)
  deploy-int:
    name: Deploy Integration
    runs-on: ubuntu-latest
    environment: Integration
    concurrency: my-cool-app-int
    permissions:
      id-token: write
      contents: read
      actions: read
    steps:
      - uses: actions/checkout@v3
      - name: Configure AWS Credentials
        uses: CRUKorg/cruk-configure-aws-credentials@v1
        with:
          aws-region: eu-west-2
          aws-role-arn: ${{ secrets.INT_DEPLOY_ROLE }}
      - name: Deploy Infrastructure
        run: npm run deploy
  deploy-prod:
    needs: [deploy-int]
    name: Deploy Prod
    runs-on: ubuntu-latest
    environment: Production
    concurrency: my-cool-app-prod
    permissions:
      id-token: write
      contents: read
      actions: read
    steps:
      - uses: actions/checkout@v3
      - name: Configure AWS Credentials
        uses: CRUKorg/cruk-configure-aws-credentials@v1
        with:
          aws-region: eu-west-2
          aws-role-arn: ${{ secrets.PROD_DEPLOY_ROLE }}
      - name: Deploy Infrastructure
        run: npm run deploy

GitHub Actions Best Practices​

Caching Technique with Nx​

Using caching techniques in GitHub Actions with Nx helps reduce build times by reusing previous build outputs instead of recomputing them. This is particularly useful in CI/CD pipelines, where repeatable builds and optimized speeds are crucial.

Prerequisites​

  • Nx monorepo with packages managed by either pnpm or yarn or npm.
  • GitHub Actions configured as your CI/CD solution.
  • Familiarity with Nx caching concepts and GitHub Action YAML syntax.

Installation​

To get started with Nx and explore more about its capabilities, you can refer to the official Nx documentation here.

This comprehensive resource provides detailed information on installing, configuring, and using Nx, along with examples and advanced guides. It’s an excellent reference to deepen your understanding of Nx, including caching strategies, CI/CD integration, and optimization techniques.

Step 1: Configure Nx for Caching​

Ensure Nx caching is set up properly in your workspace. In nx.json, check that tasksRunnerOptions includes a caching option:

{
  "tasksRunnerOptions": {
    "default": {
      "runner": "nx/tasks-runners/default",
      "options": {
        "cacheableOperations": ["build", "test", "lint"]
      }
    }
  }
}

Step 2: Set Up GitHub Actions Workflow with Nx Caching​

In your .github/workflows/ directory, create or edit a YAML file for your deployment pipeline, such as deploy.yml.

Basic Structure of deploy.yml Below is a basic setup for Nx caching using pnpm in GitHub Actions.

Example
name: Deploy with Nx Caching

on:
  push:
    branches:
      - main # Or your target branch

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v3

      - name: Set up Node
        uses: actions/setup-node@v3
        with:
          node-version: "18" # Use the appropriate Node version
          cache: "pnpm"

      - name: Set up pnpm
        uses: pnpm/action-setup@v4 # Use the appropriate pnpm version
        with:
          version: 9

      - name: Install dependencies
        run: pnpm install # Or 'yarn install' for Yarn

      - name: Restore Nx cache
        id: nx-cache
        uses: actions/cache@v4
        with:
          path: |
            .nx/cache
          key: ${{ runner.os }}-nx-cache-${{ hashFiles('**/project.json', 'nx.json', 'package.json', 'pnpm-lock.yaml') }}
          restore-keys: |
            ${{ runner.os }}-nx-cache-

      - name: Build and Test with Nx
        run: |
          pnpm nx run-many --target=build --all
          pnpm nx run-many --target=test --all

      - name: Deploy Application
        run: |
          # Add deployment script here

Step 3: Customizing Cache Key Strategy (Optional)​

If only a subset of packages needs to be rebuilt frequently, adjust the cache keys to reflect these dependencies. For example, use project-specific cache keys if you want each package to have its own cache.

key: ${{ runner.os }}-nx-cache-${{ hashFiles('apps/my-app/project.json', 'libs/my-lib/project.json', 'nx.json', 'package.json', 'pnpm-lock.yaml') }}

This approach enables caching per project, so changes in one project don't invalidate the cache of other unrelated projects.

Step 4: Forcing Rebuilds for Specific Packages (Optional)​

If you want a package to be rebuilt every time, despite caching, you can skip its cache. Here’s an example of how to do this for a package named A:

- name: Build Package A
  run: pnpm nx run A:build --skip-nx-cache

CDK Pipelines​

caution

The integration with GitHub using this method is very limited and is one way; GitHub to AWS. If you require extra functionality, such as running integration tests only on certain requests or orchestrating other actions, see the GitHub Actions deployment method.

CDK Pipelines is a high-level construct that is part of the CDK library. It utilizes AWS CodePipeline to deploy CDK stacks to multiple different AWS regions and accounts. The pipeline itself is also mastered in the CDK allowing the pipeline to "self-mutate" meaning no manual deployments are required (other than the first). The CDK pipeline can also hook into GitHub using AWS CodeStar for secure access to invoke the pipeline on every push to the main branch.

This article details how to setup a CDK pipeline in great detail. Examples of CDK pipelines can also be found below.

Further documentation can be found on the CDK documentation site here: @aws-cdk/pipelines

For PR environments CodeBuild deploy and destroy projects can be created (mastered in the CDK pipeline) that are invoked on new pull requests and destroyed on merge. These can synth and deploy the CDK for a stage of the pipeline with PR specific parameters. This is then hooked into the Pull Request as a check.

Rationale​

There are many advantages of having a continuous deployment strategy such as:

  • Removes the need for manual deployments - these are time consuming for developers and are prone to error. Having a fully automated deployment strategy helps keep code being continuously deployed and aligns with agile methodology.
  • Aligns with a fast release cycle with smaller incremental improvements to an application.
  • Automation checks as part of the deployment - these can catch issues in a particular change and can be setup to automatically rollback upon a caught error.
  • Infrastructure as code - keeping deployment as infrastructure as code means that it is reproducible and can be peer reviewed without the need for manual changes.

From a software engineering perspective generally GitHub actions is preferred for the following reasons:

  • More visibility - we can link commits and release directly with deployments and see the entire end to end process presented in a visually helpful way.
  • Already used for other test/check processes
  • Better developer experience - we work in GitHub; having our CD here as well keeps it in the same place and avoids confusion.
  • Can make use of the marketplace - allowing us to quickly use standard actions created by others.
  • Filtering on CodePipelines is limited - it is trivial on GitHub to not invoke workflows on certain criteria such as .md files. Doing this in CodePipelines requires custom work. We can link to GitHub releases for example trivially.
  • Retrying CodePipelines actions can be more difficult.

See further reading for more articles as to the benefits of continuous deployment.

Examples​

References & Further Reading​