Skip to main content

smoke_testing

Smoke Testing​

Overview​

Smoke testing is a means of quickly testing the key features or capabilities of a system, giving rapid feedback on the stability of an environment, feature or release. Its main purpose is to try to detect major defects, thereby validating the application's stability ahead of potentially further test phases.

When to use smoke testing​

Smoke testing can be used in a variety of circumstances. Some of these might include:

  • after a new build/version or system update as a basic regression test
  • before spending time on extensive testing, to ensure the system is stable before completing more lengthy and potentially costly or time consuming forms of testing (like regression or user acceptance testing)
  • where testing time might be tight, to ensure that testing is focused on the key features of an application (e.g. the core user journeys)

Manual or Automatic​

Smoke testing can be done via manual or automated tests. The 2 approaches can also be combined if needed, to allow human testers to validate more complex scenarios and automated tests to give fast feedback within CI/CD pipelines.

In CRUK, where we always want to be mindful of costs being a charity, we might consider running a subset of smoke tests on a PR commit for example to ensure that key functionality is not broken (and to ensure we don't waste time and resources running all our tests on a broken build). We might then run the full regression pack manually once the development work is complete.

What to include in a smoke test​

When identifying what to include for a smoke test we should consider:

  • testing the key functionality of a system
  • aim for broad and shallow tests
  • the tests should be quick to run (we want fast feedback)

The benefits of smoke testing​

Smoke testing has a number of benefits including:

  • saves time and resources by rejecting/identifying unstable builds
  • identifies critical defects early
  • rapid feedback
  • easy to perform
  • improves the quality of the system

What environments to run a smoke test on​

Smoke testing can be done on any environment, some examples of where below:

  • Pull requests (PR): as a sanity test, to check build stability and key functionality
  • Staging/UAT: as a post deployment validation
  • Production: Post-deployment checks, often using synthetic testing for early issue detection

Tools for smoke testing​

Smoke testing can be done manually, automated, or a combination of both. When using tools ensure they are approved for use by CRUK. Some examples below:

  • Automation: Playwright (frontend/API)
  • Manual: Postman for API testing
  • Synthetic Monitoring: Datadog Synthetic Monitoring

Synthetic tests involves simulating user interactions or system behaviors using scripted or automated tools. An example of a synthetic test would be to test a simple login journey, or validate a page is up. To be useful they should be configured to run frequently, how frequently will depend on how critical the functionality is. For crucial, high risk journeys or pages this could be every 5 or 10 minutes, for less critical functionality it might be once an hour or once a day.

Alerting and notifications​

By having proactive monitoring with synthetic tests it allows us to be notified as soon as there is an issue with an environment. This is particularly important on Production. Datadog Synthetic Monitoring integrates with slack so you can be notified as soon as an synthetic test fails. It's good practise to have dedicated slack channels for production and test environments alerts so that production alerts are not accidentally missed.

Test tagging​

Smoke tests are often a subset of tests that might sit within a larger, more extensive regression pack. To easily identify which tests to run and to allow the tests to still be structured logically, we can use test tagging to identify which tests to run as a smoke test e.g. @smoke. Test tagging is common in frameworks like Cucumber, for Playwright you can use custom tags to create the same effect.

e.g.

test.describe("Login Feature", () => {
  test("should log in successfully @smoke @login", async ({ page }) => {
    // test steps
  });
});

Then to only run tests with the smoke tag you can run:

npx playwright test --grep "@smoke"