-
Notifications
You must be signed in to change notification settings - Fork 28.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
New CLI flag to exit with an error status if test coverage is incomplete #48739
Comments
I've been wondering when someone would open this issue 😄 . Option 1 (changing the existing flag) would be preferable to limit the number of flags. It's fine to change its behavior because it's still experimental. We would need to support:
If coverage does not meet the threshold, we would need to set the process exit code, and ideally include some indication in the coverage output. We should also include the threshold information in the reporter coverage event. We also need to define how the threshold is calculated. The simplest way would be to look only at the total lines vs. lines covered. However, we could also include branch taken / not taken information in the calculation. |
I am not sure I would use this before there is sourcemaps support, but wont block it |
Is there an issue open for this? |
nope |
I like to configure my branch coverage differently from my line coverage - more specifically, I use a strict setting for the line-coverage and a looser setting for the branch coverage (I just don't find it's worth it to test every single optional parameter and what-not in the project(s) I'm working in). It would be nice if whatever config option is chosen is capable of supporting the configuration of both values independently. I personally don't use warning thresholds, but I would imagine a good number of people would care about being able to configure those as well And I know some tools also support configurable thresholds for the percentage of functions you have under test, though I don't know how much people care about that in practice. All together, if this were to be a single CLI argument, I guess we could invent some sort of short-hand syntax for it. Something like:
If warning thresholds aren't supplied, we could default to some fixed percentage above the error or something. Pro: concise and easier for humans to read and write. I was also seeing somewhere in another ticket (can't remember where) some discussion about maybe adding a config file so these sorts of customizations don't get too crazy. That could be another option. |
There has been no activity on this feature request for 5 months. To help maintain relevant open issues, please add the
never-stale
|
I think this is now relevant again |
What is the problem this feature will solve?
When using the Node.js test runner in combination with the
--experimental-test-coverage
flag, it would be very useful to be able to also flag Node.js to exit the process with an error status (i.e.1
) if the coverage was not 100% complete, to be able to enforce complete code coverage in CI.Until this feature exists, I won't be able to migrate projects from using
coverage-node
:https://github.com/jaydenseric/coverage-node/tree/v8.0.0#command-coverage-node
Personally I only have interest in enforcing 100% code coverage or not enforcing it at all, but I can see how some teams might want to enforce a minimum percent of code coverage so they can iteratively work towards 100%, raising the minimum over time, and notice in CI if any contributions go against that goal.
What is the feature you are proposing to solve the problem?
We currently run tests like this:
So I propose another flag
--minimum-coverage
(bikeshed the name) that can accept a percent number for the minimum coverage percent required to avoid the process exiting with an error status1
. E.g:What alternatives have you considered?
Changing the
--experimental-test-coverage
from a boolean to accepting a percent number for the minimum coverage percent required to avoid the process exiting with an error status1
.I actually quite like this because it reduces the number of flags, but I'm not sure it's ok to change an existing flag already in use? Perhaps that's not a concern since it's experimental and also if the value is optional and defaults to the current behaviour it won't break projects already using
--experimental-test-coverage
.Enforce 100% code coverage by default, and use an environment variable to opt out of the enforcement. This is how
coverage-node
currently works:https://github.com/jaydenseric/coverage-node/tree/v8.0.0#command-coverage-node
The text was updated successfully, but these errors were encountered: