Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: worker being killed after being spawned and other worker bugs #13107

Merged
merged 53 commits into from Aug 11, 2022
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
71d0080
test: ad integration test case for worker not executing test
phawxby Aug 8, 2022
8c6ab73
test: add test case for worker being killed just once
phawxby Aug 9, 2022
86acdf1
fix: worker being killed after spawning
phawxby Aug 9, 2022
718425b
Merge branch 'master' into worker-restart-bug
phawxby Aug 9, 2022
de0c0e8
fix: missing license
phawxby Aug 9, 2022
d83aa84
fix: flaky tests
phawxby Aug 9, 2022
9435d0b
chore: add some diagnostic logging
phawxby Aug 9, 2022
29e1193
chore: break apart tests to see where the problem is occuring
phawxby Aug 9, 2022
c23f581
fix: handling of work ready status
phawxby Aug 9, 2022
e8a8058
chore: add fallback timeout
phawxby Aug 9, 2022
4e0b399
refactor: add abstract class with common functions and vars
phawxby Aug 9, 2022
790f772
fix: restarting on init
phawxby Aug 9, 2022
b4151ea
refactor: event emitting is a much cleaner way of tracking what's goi…
phawxby Aug 9, 2022
2753069
chore: better handle shutdown events
phawxby Aug 9, 2022
617e102
fix: only set out of memory when necessary
phawxby Aug 9, 2022
28230da
chore: windows debugging
phawxby Aug 9, 2022
60b5b1b
chore: try and debug single test
phawxby Aug 9, 2022
a9a3779
chore: remove debugging
phawxby Aug 9, 2022
cdf9627
chore: remove logging
phawxby Aug 9, 2022
7a79c92
chore: simplify tests and move more to abstract worker
phawxby Aug 9, 2022
11fd6f8
chore: only run flaky test
phawxby Aug 10, 2022
55f4097
chore: enable tests again
phawxby Aug 10, 2022
723951b
fix: missing snapshot
phawxby Aug 10, 2022
a062850
chore: try to resolve the flake
phawxby Aug 10, 2022
66c83bd
chore: logging
phawxby Aug 10, 2022
c47404d
chore: see if this changes debugging output
phawxby Aug 10, 2022
9331bd2
chore: make noisy
phawxby Aug 10, 2022
5058e14
chore: debugging
phawxby Aug 10, 2022
ddbfb5e
chore: more
phawxby Aug 10, 2022
811e228
chore: even more robust logging
phawxby Aug 10, 2022
197c0da
chore: refactor error streaming monitoring
phawxby Aug 10, 2022
0d0d03e
chore: error handling
phawxby Aug 10, 2022
11e8b25
chore: log this
phawxby Aug 10, 2022
8f427a4
fix: tests and try to detect process crashes differently on windows
phawxby Aug 10, 2022
57d3b20
chore: remove debug logging
phawxby Aug 10, 2022
d3abe14
chore: remove only
phawxby Aug 10, 2022
f704620
chore: add win32 specific logic to try and work around the issue
phawxby Aug 10, 2022
5a840bb
chore: back to adding debug logging
phawxby Aug 10, 2022
d6b2db0
chore: try adding more possible detections
phawxby Aug 10, 2022
202323b
chore: try a different event
phawxby Aug 10, 2022
9e7d081
chore: disconnect handling
phawxby Aug 10, 2022
75037d2
fix: test bug
phawxby Aug 10, 2022
37c5a43
chore: log child and buffer
phawxby Aug 10, 2022
3e8b3ae
chore: forgot to make public
phawxby Aug 10, 2022
fab410a
chore: log
phawxby Aug 10, 2022
8e4a31e
chore: use up memory quicker and more logging
phawxby Aug 10, 2022
b974eea
chore: try removing force exit to see if that helps with the confusin…
phawxby Aug 10, 2022
1e1023d
chore: different strat
phawxby Aug 10, 2022
ab2931d
chore: remove debugging
phawxby Aug 10, 2022
c610bdb
chore: revert to silent worker
phawxby Aug 11, 2022
ee20013
Update CHANGELOG.md
SimenB Aug 11, 2022
bd786b1
chore: pr feedback
phawxby Aug 11, 2022
613fcf2
chore: pr changes
phawxby Aug 11, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
6 changes: 3 additions & 3 deletions packages/jest-worker/src/types.ts
Expand Up @@ -190,12 +190,12 @@ export type WorkerOptions = {
*/
on?: {
[WorkerEvents.STATE_CHANGE]:
| onStateChangeHandler
| Array<onStateChangeHandler>;
| OnStateChangeHandler
| ReadonlyArray<OnStateChangeHandler>;
};
};

export type onStateChangeHandler = (
export type OnStateChangeHandler = (
state: WorkerStates,
oldState: WorkerStates,
) => void;
Expand Down
5 changes: 5 additions & 0 deletions packages/jest-worker/src/workers/ChildProcessWorker.ts
Expand Up @@ -117,6 +117,11 @@ export default class ChildProcessWorker
const silent = this._options.silent ?? true;

if (!silent) {
// NOTE: Detecting an out of memory crash is independent of idle memory usage monitoring. We want to
// monitor for a crash occurring so that it can be handled as required and so we can tell the difference
// between an OOM crash and another kind of crash. We need to do this because if a worker crashes due to
// an OOM event sometimes it isn't seen by the worker pool and it just sits there waiting for the worker
// to respond and it never will.
console.warn('Unable to detect out of memory event if silent === false');
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should only log this if _childIdleMemoryUsageLimit is passed I think? Then we can probably also make it an error instead of warning

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comment applies to thread worker - shouldn't we only monitor for OOM if the option is passed, otherwise keep old behavior?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should only log this if _childIdleMemoryUsageLimit is passed I think? Then we can probably also make it an error instead of warning

Yes to the first bit, i'm inclined to say no to the second. It can be useful for debugging to see the child output when a memory limit is in place, for example to see the exact thrown error message.

Copy link
Contributor Author

@phawxby phawxby Aug 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comment applies to thread worker - shouldn't we only monitor for OOM if the option is passed, otherwise keep old behavior?

Actually, ignore my previous comment....

Detecting an out of memory event and the worker crashing is independent of monitoring of the idle memory usage and rebooting as necessary. The _childIdleMemoryUsageLimit applies to the latter case, the former is building and improving on #13054

Do I need better docs to explain that?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added some comments to explain

}

Expand Down
22 changes: 14 additions & 8 deletions packages/jest-worker/src/workers/WorkerAbstract.ts
Expand Up @@ -17,7 +17,11 @@ export default abstract class WorkerAbstract
extends EventEmitter
implements Pick<WorkerInterface, 'waitForWorkerReady' | 'state'>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Create a new interface within types.ts instead of Picking here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's an extremely limited use case just to prevent the class throwing typing errors, it didn't really seem necessary to create a new class just for that as it would have no utility elsewhere. Someone could create a different worker type in future and create a different abstract with totally different base props.

{
private __state = WorkerStates.STARTING;
/**
* DO NOT WRITE TO THIS DIRECTLY.
* Use this.state getter/setters so events are emitted correctly.
*/
#state = WorkerStates.STARTING;

protected _fakeStream: PassThrough | null = null;

Expand All @@ -28,12 +32,12 @@ export default abstract class WorkerAbstract
protected _resolveWorkerReady: (() => void) | undefined;

public get state(): WorkerStates {
return this.__state;
return this.#state;
}
protected set state(value: WorkerStates) {
if (this.__state !== value) {
const oldState = this.__state;
this.__state = value;
if (this.#state !== value) {
const oldState = this.#state;
this.#state = value;

this.emit(WorkerEvents.STATE_CHANGE, value, oldState);
}
Expand All @@ -44,12 +48,14 @@ export default abstract class WorkerAbstract

if (typeof options.on === 'object') {
for (const [event, handlers] of Object.entries(options.on)) {
if (Array.isArray(handlers)) {
// Can't do Array.isArray on a ReadonlyArray<T>.
// https://github.com/microsoft/TypeScript/issues/17002
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh my 😅

if (typeof handlers === 'function') {
super.on(event, handlers);
} else {
for (const handler of handlers) {
super.on(event, handler);
}
} else {
super.on(event, handlers);
}
}
}
Expand Down
74 changes: 35 additions & 39 deletions packages/jest-worker/src/workers/__tests__/WorkerEdgeCases.test.js
Expand Up @@ -19,7 +19,6 @@ import {
import ChildProcessWorker, {SIGKILL_DELAY} from '../ChildProcessWorker';
import ThreadsWorker from '../NodeThreadsWorker';

jest.retryTimes(0);
jest.setTimeout(10000);

const root = join('../../');
Expand Down Expand Up @@ -58,6 +57,15 @@ test.each(filesToBuild)('%s.js should exist', async file => {
await expect(async () => await access(path)).not.toThrowError();
});

async function closeWorkerAfter(worker, testBody) {
try {
await testBody(worker);
} finally {
worker.forceExit();
await worker.waitForExit();
}
}

describe.each([
{
name: 'ProcessWorker',
Expand Down Expand Up @@ -101,62 +109,50 @@ describe.each([
}

test('should get memory usage', async () => {
let worker;

try {
worker = new workerClass({
await closeWorkerAfter(
new workerClass({
childWorkerPath: workerPath,
maxRetries: 0,
workerPath: join(__dirname, '__fixtures__', 'EdgeCasesWorker'),
});
}),
async worker => {
const memoryUsagePromise = worker.getMemoryUsage();
expect(memoryUsagePromise).toBeInstanceOf(Promise);

const memoryUsagePromise = worker.getMemoryUsage();
expect(memoryUsagePromise).toBeInstanceOf(Promise);

expect(await memoryUsagePromise).toBeGreaterThan(0);
} finally {
if (worker) {
worker.forceExit();
await worker.waitForExit();
}
}
expect(await memoryUsagePromise).toBeGreaterThan(0);
},
);
});

test('should recycle on idle limit breach', async () => {
let worker;

try {
worker = new workerClass({
await closeWorkerAfter(
new workerClass({
childWorkerPath: workerPath,
// There is no way this is fitting into 1000 bytes, so it should restart
// after requesting a memory usage update
idleMemoryLimit: 1000,
maxRetries: 0,
workerPath: join(__dirname, '__fixtures__', 'EdgeCasesWorker'),
});

const startSystemId = worker.getWorkerSystemId();
expect(startSystemId).toBeGreaterThanOrEqual(0);
}),
async worker => {
const startSystemId = worker.getWorkerSystemId();
expect(startSystemId).toBeGreaterThanOrEqual(0);

worker.checkMemoryUsage();
worker.checkMemoryUsage();

await waitForChange(() => worker.getWorkerSystemId());
await waitForChange(() => worker.getWorkerSystemId());

const systemId = worker.getWorkerSystemId();
expect(systemId).toBeGreaterThanOrEqual(0);
expect(systemId).not.toEqual(startSystemId);
const systemId = worker.getWorkerSystemId();
expect(systemId).toBeGreaterThanOrEqual(0);
expect(systemId).not.toEqual(startSystemId);

await new Promise(resolve => {
setTimeout(resolve, SIGKILL_DELAY + 100);
});
await new Promise(resolve => {
setTimeout(resolve, SIGKILL_DELAY + 100);
});

expect(worker.isWorkerRunning()).toBeTruthy();
} finally {
if (worker) {
worker.forceExit();
await worker.waitForExit();
}
}
expect(worker.isWorkerRunning()).toBeTruthy();
},
);
});

describe('should automatically recycle on idle limit breach', () => {
Expand Down