Global setup file for native node test runner

[jamm3e3333]

What is the problem this feature will solve?

I'm sorry if there's already such a feature and I haven't found a way how to solve this. Basically what I'm looking for is to run some function that would run beofre all the tests and setup the environment. So far the only way how to setup some environment is doing it with the run function but that is only limited for each test suite from what I've tried. I would love to have the have some setup file in the way how does it work for example in jest or mocha or other testing framework.

What is the feature you are proposing to solve the problem?

Have a setup.js/setup.ts file that will run before all the tests, you would somehow specify in command that starts the tests this setup file.

What alternatives have you considered?

node version: v20.6.1
npm version: 9.8.1

I've tried this setup file

setup.ts

import { performance } from 'perf_hooks'
import { run } from 'node:test'

run({
    files: ['src/node-test-runner/suite.test.ts'],
    setup: () => {
        let memStats: Record<string | number, number> = {};
        const perf = performance;

        const start = perf.now();
        const cpuUsage = process.cpuUsage()

        process.nextTick(() => {
            const current = process.memoryUsage();
            for (const key in current) {
              memStats[key] = Math.max(memStats[key] ?? 0, current[key as keyof ReturnType<typeof process.memoryUsage>]);
            }
          });

          process.on("exit", () => {
            logger.info({
              memStats,
              timeMilliseconds: perf.now() - start,
              cpuUsage: process.cpuUsage(cpuUsage),
              pid: process.pid,
            });
          });
    },
})

suite.test.ts

import { describe, it } from 'node:test'
import {equal} from 'node:assert'
import { testBody } from '../index'
import { logger } from '../logger'

const TEST_ITERATIONS = Number(process.env.TEST_ITERATIONS);

describe(__filename, () => {
    it(`${__filename}`, async () => {
            equal(true, true)
    })
})

and I'm running this command to execute tests: node --require ts-node/register --test

but neither the setup.ts file is executed nor the suite.test.ts file is executed

[koshic]

You should run it via 'node --require ts-node/register setup.ts'.

I use similar solution - kind of wrapper around 'run' function to setup different reporters (CI / local run), collect coverage with c8, globs for .ts files, additional loaders, etc. Because this wrapper is a package with 'bin' script named 'test', I can run tests via 'yarn test'.

Also, adding your own config (as an example, to exclude some files from coverage) is a few lines:

// wrapper
let config;
try {
 config = await import('test.config.js');
} catch {
 config = { /* default */ };
}

run(/* your test config */)

// config
export default {
 // anything you may need
}

[benjamingr]

Yeah this is just a syntax thing, node --require ts-node/register setup.ts as it invokes the runner programatically.

If you build useful stuff on top of the native test runner or notice missing features please do tell us and put it on npm if you're comfortable with it being open source :)

[koshic]

If you build useful stuff on top of the native test runner or notice missing features please do tell us and put it on npm if you're comfortable with it being open source :)

In my case it's an internal and very specific package created to keep some (a lot of...) defaults inside. I believe when we get native coverage reporter (with source maps & lcov output) and globs, such wrappers will be redundant for most of us. Only one thing may be really required - some way to organize reporters (but we already have .env support, right?).

[benjamingr]

Right, and since the programatic invocation is just a Node.js file you can use whatever you want for config there :)

[jamm3e3333]

Yeah this is just a syntax thing, node --require ts-node/register setup.ts as it invokes the runner programatically.

If you build useful stuff on top of the native test runner or notice missing features please do tell us and put it on npm if you're comfortable with it being open source :)

Sure I'll try to come up with something

[rluvaton]

Reopening as you also can't do global teardown

[rluvaton]

actually you can do:

const {tap} = require('node:test/reporters');
const process = require('node:process');
const path = require("node:path");
const {run} = require("node:test");
const {finished} = require('node:stream');

const stream = run({
    files: ['./src/a.test.js', './src/b.test.js'],
    concurrency: false,

    setup: () => {
        console.log('setup');
    }
})
    .compose(tap);

finished(stream, () => {
    console.log('teardown');
});

stream.pipe(process.stdout);

But it's only for when using run I think we should document it though

[cjihrig]

There has been some discussion about Node.js as a whole supporting a configuration file. I'm not sure if that will lead to anything, but having a separate file for just the test runner would not be great.

[ernestdolog]

But it's only for when using run I think we should document it though

Would be awesome to have it documented. What would be even better is to have it working with some sort of file detection, like: files: [Application.baseDir + '/__tests__/**/*.test.ts'],

My global test setup is the following:

/**
 * The Global Test Runner
 * =========
 * Provides global test setup.
 *
 * Because of node test runners concurrency this is elemental.
 */
import './framework.setup';
import {run} from 'node:test';
import process from 'node:process';
import {tap} from 'node:test/reporters';
import {Application} from '#app/application';

const stream = run({
    files: [
        Application.baseDir + '/__tests__/e2e/auction-api.resolver.test.ts',
        Application.baseDir + '/__tests__/e2e/user-api.resolver.test.ts',
        Application.baseDir + '/__tests__/unit/logger.unit.test.ts',
    ],
    /**
     * As after integration, and e2e tests we truncate databases
     * This should not happen paralelly, unless we figure a way to only delete the data added at that test with a one-liner
     * Switch on concurrency locally for unit tests and such
     */
    concurrency: false,
}).compose(tap);

stream.pipe(process.stdout);

I have a couple of test suites coming. Would be nice, if all tests unit-integration-e2e could be imported with a one-liner.

[Twipped]

I think the issue I'm having is related to this.

I'm writing an internal utility for our tests that is built around the native test-runner. I need to make sure a module executes inside each test process before the test file is actually run. The --import flag works for this if invoking node --test directly, but I haven't found a way to do this when using the run function. Simply importing the module before invoking run doesn't do the job, because the tests all execute in separate child processes. Is there some way to programmatically inject an --import flag?

edit:
I solved it by pushing to execArgv, but it'd be nice if there was an official API for this.

process.execArgv.push('--import', path.resolve(__dirname, 'index.js'));

[koshic]

I solved it by pushing to execArgv, but it'd be nice if there was an official API for this.

It's always been here - https://nodejs.org/dist/latest-v21.x/docs/api/cli.html#node_optionsoptions. Just set that variable in process.env before run call.

[Twipped]

@koshic I'd still consider that a hack compared to explicitly having a means of specifying flags and env on the run options.

[fvictorio]

Thanks for sharing the workaround @Twipped!

I agree it feels like a hack, especially because both using process.execArgv.push and process.env.NODE_OPTIONS are passed down to the tests. This means that if the tests also start child processes for some reason, then those will also load the setup module (probably not important in practice, but it feels wrong).

I guess a way to fix that is to start your setup module with process.execArgv.splice(-2). But having some sort of builtin solution would be great.

[mdrobny]

My use cases for global setup file is to e.g. start a database container or create local cloud resources before running integration tests.
All integration tests run in parallel and use the same database or cloud resource so it needs to be executed only once.
I prefer not to use run function because e.g. it doesn't cooperate well with running or debugging selected tests directly from editor

This functionality is supported by popular test runners:

• Jest: globalSetup and globalTeardown files defined in config
• Mocha: global setup and teardown fixtures file defined via CLI flag (--require) or in config
• Vitest: globalSetup file defined in config
• Playwright: globalSetup and globalTeardown files defined in config
• Bun: preload file with hooks defined via CLI flag or in config

Using --require or --import CLI flag is similar but it will run the "setup file" for every test file, instead of doing it just once. I could manage to use it if there would be a straightforward way of sharing info between different test child processes to know if setup was already started or finished.
But beside that, it doesn't provide a way to do a teardown as it was mentioned in the comments.

[bukowa]

Landed here looking for a global setup. Ideally in such a way that each of my tests receives the context I created in the global setup function.

globalSetup(context) {
    context.driver = something
}

test("Can ....", context){
    context.driver.do()
}

[mdrobny]

I have some ideas how to define globalSetup and globalTeardown functions but I don't know if they are possible to implement.
I am trying to avoid adding new command-line options

1. Use --import or --require option and new hooks

User defines a file, for example setup.js

import { globalSetup, globalTeardown } from 'node:test'

globalSetup(async () => {
  await prepareSomething()
})

globalTeardown(async () => {
  await cleanupSomething()
})

Then user runs tests like this

node --import setup.js --test

Challenges:

• setup.js will be executed for each test file but globalSetup and globalTeardown functions must be executed only once.
  So first globalSetup to execute, must register an async function to be executed before running test files
  First globalTeardown to execute, must register an async function to be executed after all tests completed (or failed).
• I am not sure if such communication between test file process and main process is possible

2. Use --import or --require option and export hooks

User defines a file, for example setup.js

export const globalSetup = async () => {
  await prepareSomething()
}

export const globalTeardown = async () => {
  await cleanupSomething()
}

Then user runs tests like this

node --import setup.js --test

Challenges:

• how to import from setup.js and execute globalSetup and globalTeardown functions in the main test runner process only

Pros:

• setup and teardown functions are not executed in every test file

3. Add new command-line options --test-setup and --test-teardown

User defines a file, for example setup.js

import { globalSetup } from 'node:test'

globalSetup(async () => {
  await prepareSomething()
})

and teardown.js

import { globalTeardown } from 'node:test'

globalTeardown(async () => {
  await cleanupSomething()
})

Then user runs tests like this

node --test-setup setup.js --test-teardown teardown.js --test

Challenges:

• how hard it is to import/require file provided via command-line option

Pros:

• run function in test runner already has setup option. teardown would need to be added

[cjihrig]

I think option 1 or 2 sound the best (they honestly sound like the same thing just a difference of where the hook function lives). A few things to note are:

• I believe --import and --require will run for each test file as well as the main runner process.
• node:test knows if the current process is the main runner or a test file. I think that would simplify some things such as needing to communicate between processes. It would also allow the test runner to "no-op" hooks in the appropriate places.
• Since the test files are all separate processes, users will need to know that a one time global setup will not be reflected in each child process. That's fine if you are doing something like starting a database, but will not work if you are doing something like defining a global variable.
• In the future, I'd really like to make it possible to run the test files in the same process. Any solution we come up with should keep that future use case in mind. For example, the limitation I listed above would no longer apply if everything were in the same process. Maybe that is fine to just document.
• We also need to keep the use case of running a single file without --test in mind. I think the solutions proposed here would work in that case, but wanted to point it out.

run function in test runner already has setup option. teardown would need to be added

We should probably add this anyway.

EDIT: After thinking a bit, option 1 is probably better. With option 2 the test runner will have to search for the exported functions in all cases, which slows things down.