Release Process
This document outlines the CRUK release process for products.
For additional context on this best practice see Environments.
Description
Software that is developed within CRUK must be appropriately tested and reviewed before being pushed to production.
Flow
To help describe the release process the following flow chart was created which provides a simplified view of an engineer making a change and seeing it through to production. Specific details on each area of the flow chart can be found below.
Process
1,2 and 3. Engineer makes changes locally and creates a pull request
These steps follow a common approach where engineers make changes on their local machines on a git branch. This is then pushed to the git repository and a PR (pull request) created.
4. PR Approval
The PR requires approval by the team before the change can be released. This is done with three methods that can run in parallel.
4a - An PR ephemeral environment is created using the software in the PR. This is a short lived environment that is spun up per PR and asserts that the software can be build and deployed. Tests can also be run against this stack.
4b - Automated checks via GitHub actions are performed. This can include unit tests, UI tests, static code checks, security checks, integration checks and more. These are performed for all PR and are created by the team. When following this process it is essential to have sufficient automated approval checks to reduce manual effort in asserting a change.
4c - Another engineer reviews the code itself and approves if they are happy with the content.
Once the PR has been approved via these methods it is ready for release. First the type of change must be determined to understand it's release process.
5. Change Types
The change type will be determined at the task level prior to this stage in the flow. It is at this point in the flow however where the change type dictates the next steps.
The change request itself should be created by the PR automatically by integrating with the Service Now API.
| Change Type | Team Approval | External Approval | Examples |
|---|---|---|---|
| Standard | ✔️ | ❌ | Cosmetic CSS change, updating dependencies, form validation changes |
| Normal | ✔️ | ✔️ | Fixing a security breach, new service integrations, change to downstream contracts |
For simplicity the 'Standard' change template covers all aspects of self-served product releases with the exception of the areas listed below.
6. Semantic versioning
Github releases can be further automated with the help of conventional commits and semantic versioning.
A release configuration file release.config.js at the root repository level defines the plugins used for the release.
The commit-analyser plugin in the release config defines the type of commits that will cause a major or minor version change, the automatic generation of release notes and upload of releases to github among other features.
{
"plugins": [
[
"@semantic-release/commit-analyzer",
{
"preset": "angular",
"releaseRules": [
{ "type": "docs", "scope": "README", "release": "patch" },
{ "type": "refactor", "scope": "core-*", "release": "minor" },
{ "type": "refactor", "release": "patch" },
{ "scope": "no-release", "release": false }
]
}
],
"@semantic-release/release-notes-generator"
]
}
Examples for semantic release configuration
Automating releases with semantic versioning is optional and only useful where the development and QA teams agree to use conventional commits in pull requests.
Semantic versioning in monorepos
If you need to configure the release of individual packages within a monorepo, Nx offers a nx release command to help with running the semantic release before/during the merge to the root branch.
Example nx.json release config:
"release": {
"changelog": {
"automaticFromRef": true,
"projectChangelogs": {
"createRelease": "github"
}
},
"projects": ["apps/*", "packages/*"],
"projectsRelationship": "independent",
"version": {
"conventionalCommits": true,
"generatorOptions": {
"fallbackCurrentVersionResolver": "disk"
}
}
}
In the example above, nx is configured to run the github release of individual packages within apps and packages and use conventional commits. You can then configure github actions to run the release.
Examples:
semantic-release section in Online Payments Services
Exceptions
This list will evolve over time and should be used as a guideline only. It is not authoritative; documentation will be provided where this will be sourced and this document will be updated to reference that when available.
- Anything that involves working with resource outside the product team.
- Anything that impacts any other digital product
- Anything that impacts any production service other than the product being released.
- Anything affecting or changing the underpinning architecture.
- Anything affecting or changing the underpinning infrastructure.
- Anything with security implications or fixes resulting from a security breach.
- Changes where there are other dependent systems.
There may be other exceptions on a case-by-case basis; if you are unsure speak with the product manager.
6a. Standard
A pre-approved change type that has been determined to have no impact outside of the application or data solution being changed and therefore only needs to be reviewed by the product team (the Product Manager, Lead Software/Data/CRM Engineer, QA Engineer).
These changes are approved within the team and require no external approval. They can be deployed whenever the team chooses to do so, however we caution against deploying changes at times when there is insufficient support in place to deal with any issues that arise from the change (e.g. deploying on a Friday afternoon if there is no support in place on the Saturday).
The change must be tracked within ServiceNow with a change request and a deployment notification must be posted in the #releases channel in Slack. The change request creation and Slack post can be done automatically from Github Actions.
6b. Normal
A change that has impact outside of the application or data solution being changed and therefore requires:
- review of people from other teams
- scheduling to avoid conflict with other planned changes and
- additional comms to ensure all stakeholders are aware
These are changes that impact upstream or downstream systems or other teams' activity and therefore need coordination between teams or stakeholders in order to safely release. These changes are approved by the product team as with a standard change but also require approvals from impacted services, stakeholders and service management. The change must be tracked and approved within ServiceNow with a change request and communicated ahead of time on the #releases channel in Slack. This change type requires CAB (Change Advisory Board) and external stakeholder approval.
7. Merge PR into to main branch
Once approvals have been granted and the change request created. The PR can be merged, using the UI of GitHub, into the main branch. This should be done by the engineer who made the change itself.
8. Integration Deployment
Now that the change has been merged into the main branch the PR will be closed resulting in the PR environment being destroyed (8a). Automated processes should be setup that deploy to integration on each push to the main branch (8b).
9. Versioned Release and Production Deploy
If the integration deploy was successful and a production deploy is required, first create a release with a tag version number that coincides with the release version mentioned in the change request. Github releases is a good way to manage these. This could trigger though Github actions a production deploy or a separate process could handle the production environment deploy, deploying the code in the tag number mentioned in the release. production
Advice
Different products have different levels of dependence on each other, also that different products may choose to release at different times to suite the product requirements or restrictions, and different products teams have different capabilities capacity to respond to issues that may arise from adjacent deploys.
- Avoid deploying on a Friday or a day before a bank holiday unless you have to, for example there is P1 bug that is causing a loss of income.
- Avoid deploying too late in the working day unless your are prepared to work late to iron out any issues.
- It is preferable to deploy when traffic to your products is at its lowest (Speak to the Google Analytics team about tracking this).
- The smaller the change and more frequent the push value out to our users the better. Faster iterations and the ability to fix fast is preferable than aiming for perfection and loosing the ability to fix fast.
10. Verification
Post deployment checks are performed. Automated checks should ensure that the deployment to production was successful. Metrics should be monitored to ensure that the traffic remains as expected and no errors have resulted. Should checks fail the deployment should automatically rollback to the last successful change. In events where this has not happened the PR commit can be reverted and pushed following the same approval process; for normal changes this can be done under the same change request.
Once the change is successfully in production and verified this completes the release process. The change request should be closed and any stakeholders notified that the change is now complete. This process should be integrated with the change management Service Now API to automatically update the status of the deployment against the change request itself.
Rationale
We wish to have a robust release process that captures issues early in the process and prevents issues in production whilst also ensuring that the process is not a burden for engineers.
A lot of effort has been put in to minimize the process around releases and as a result the standard change was created. By reducing the process and approval for trivial changes software can be deployed and developed faster and saves the organisation time.
Why "Standard" and "Normal"?
These terms are defined by ITIL (Information Technology Infrastructure Library) which is a set of practices for IT activities and describes the processes for an organisation on software delivery. The Service Now application uses these terms and so these terms were chosen to align with this.
Examples
This release process was defined whilst building the online payments product. It can be used as an example for a product that encapsulates the release process above.
The PWS change management criteria can be viewed here: PWS Change Management