-
Notifications
You must be signed in to change notification settings - Fork 9.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
tests(unit): retry failures and upload failure artifacts #15378
Conversation
4604a20
to
a98ba71
Compare
@@ -56,7 +56,7 @@ | |||
"unit-viewer": "yarn mocha --testMatch viewer/**/*-test.js", | |||
"unit-flow": "bash flow-report/test/run-flow-report-tests.sh", | |||
"unit": "yarn unit-flow && yarn mocha", | |||
"unit:ci": "NODE_OPTIONS=--max-old-space-size=8192 npm run unit --forbid-only", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oh. we need to support this in the script. can you not remove ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The --forbid-only
flag doesn't work when we use it like this. This PR adds it to run-mocha-tests.js
and gives it a default value of true
in CI instead.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What do you mean it doesn't work when used like this?
I know it doesn't work at all w/o this PR or #15383 , but I don't understand your meaning here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Two reasons:
- Before this PR we didn't pipe the
--forbid-only
option into mocha - We would need to do
npm run unit -- --forbid-ony
to pass the flag all the way down
Instead of messing with the flags in the package scripts, I would prefer to just look at process.env.CI
. We already do this for --parallel
in run-mocha-tests.js
:
lighthouse/core/test/scripts/run-mocha-tests.js
Lines 127 to 132 in 06ebc36
// Although much faster, mocha's parallel test runner defers printing errors until | |
// all tests have finished. This may be undesired for local development, so enable | |
// parallel mode by default only in CI. | |
// Also, good to default to false locally because that avoids missing cross-file | |
// test contamination by chance of mocha splitting up the work in a way that hides it. | |
default: Boolean(process.env.CI), |
Unit failures are only happening in CI, failure artifacts might give us more insight. Retry unit tests 5 times to unblock CI.
CI failures added to backlog: #15382