Spot the False Positive Test

Make sure your tests aren’t false positives

Unit tests are a great way to isolate code and test small pieces to ensure the intended outcome is met. Many modern (and many not so modern but stand up to the test of time) testing frameworks have helped to ease the pain of unit testing.

Outside of a few frameworks, most don’t offer a way to spot false positives. False positive unit tests are tests that continue to pass even when a value is changed in the code that is being tested.

Let’s look at a at an example to show how we can identify a false positive unit test.

Example False Positive

// someModule.js
export const someConstant = 42;
export const multiplyToConstant = (multiplier) => {
  return someConstant * multiplier
}
// someModule.test.js
import {someConstant, multiplyToConstant} from 'someModule.js';
import test from 'node:test';
import { strict as assert } from 'node:assert';
// ignore the bad test case description
test('returns expected result' () => {
  const multiplier = 24;
  const expectedAnswer = multiplier * someConstant;
  assert.strictEqual(multiplyConstant(multiplier), expectedAnswer)
})

If you take a close look at the example above, you will notice that we are using the constant to test that the result is correct.

It may not be entirely clear why this is an issue, but what if the example was this:

//refreshTimer.js
import refreshToken from 'refreshToken.js'
import { interval } from 'rxjs'
export const timeInMsToRefresh = 900000 // 15 mins
export const refreshTimer = interval(timeInMsToRefresh)
export const startRefreshTimer = () => 
  refreshTimer.subscribe(refreshToken)

The above code will refresh the token every 15 mins. What if a developer set the refresh token to 1 minute to test out the functionality. If the unit test used the constant to mock the time before the interval ticks, the unit test would still pass, but be invalid.

We would not be able to catch the unintentional code commit.

A simple solution is to have a test for the constant. It is a very good idea to test your constants/configurations. More often than not, they should not change, and when changed we want to make sure that it is intended. Therefore a developer updates the test or realizes that they made a mistake. Going back to the original example we simply just need to add a test to assert the value of someConstant:

// someModule.test.js
import {someConstant, multiplyToConstant} from 'someModule.js';
import test from 'node:test';
import { strict as assert } from 'node:assert';
// it really is this simple
test('someConstant should equal 42', () => {
  assert.strictEqual(someConstant, 42)
})
// ignore the bad test case description
test('returns expected result' () => {
  const multiplier = 24;
  const expectedAnswer = multiplier * someConstant;
  assert.strictEqual(multiplyConstant(multiplier), expectedAnswer)
})

Adding constant validation test cases, in this case, prevents a false positive test.

There are many more ways that a false positive test can present itself. However, in almost all false positive cases we can identify them by asking the following question:

If I change a variable, will the tests still pass? If this is true, you may have a set of false positive test cases on your hands.