Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: do not run benchmark tests on 'make test' #34434

Merged
merged 8 commits into from Jul 22, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
3 changes: 2 additions & 1 deletion CHANGELOG.md
Expand Up @@ -39,7 +39,8 @@ release.
<a href="doc/changelogs/CHANGELOG_V14.md#14.0.0">14.0.0</a><br/>
</td>
<td valign="top">
<b><a href="doc/changelogs/CHANGELOG_V12.md#12.18.2">12.18.2</a></b><br/>
<b><a href="doc/changelogs/CHANGELOG_V12.md#12.18.3">12.18.3</a></b><br/>
<a href="doc/changelogs/CHANGELOG_V12.md#12.18.2">12.18.2</a><br/>
<a href="doc/changelogs/CHANGELOG_V12.md#12.18.1">12.18.1</a><br/>
<a href="doc/changelogs/CHANGELOG_V12.md#12.18.0">12.18.0</a><br/>
<a href="doc/changelogs/CHANGELOG_V12.md#12.17.0">12.17.0</a><br/>
Expand Down
14 changes: 8 additions & 6 deletions Makefile
Expand Up @@ -298,8 +298,8 @@ v8:
jstest: build-addons build-js-native-api-tests build-node-api-tests ## Runs addon tests and JS tests
$(PYTHON) tools/test.py $(PARALLEL_ARGS) --mode=$(BUILDTYPE_LOWER) \
--skip-tests=$(CI_SKIP_TESTS) \
$(CI_JS_SUITES) \
$(CI_NATIVE_SUITES)
$(JS_SUITES) \
$(NATIVE_SUITES)
Comment on lines +301 to +302
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW the reason we split tests into CI_JS_SUITES and CI_NATIVE_SUITES was that to accommodate the arm-fanned setup we split the tests into test-ci-native (requires a build toolchain set up to build the native addons) and test-ci-js (which "should not use a native compiler at all" (see comment in Makefile)).

For testing outside of the CI we don't have the same split in Makefile test targets, so you could optionally have a single make variable instead of two.


.PHONY: tooltest
tooltest:
Expand Down Expand Up @@ -492,9 +492,11 @@ test-all-valgrind: test-build
test-all-suites: | clear-stalled test-build bench-addons-build doc-only ## Run all test suites.
$(PYTHON) tools/test.py $(PARALLEL_ARGS) --mode=$(BUILDTYPE_LOWER) test/*

JS_SUITES ?= default
NATIVE_SUITES ?= addons js-native-api node-api
# CI_* variables should be kept synchronized with the ones in vcbuild.bat
CI_NATIVE_SUITES ?= addons js-native-api node-api
CI_JS_SUITES ?= default benchmark
CI_NATIVE_SUITES ?= $(NATIVE_SUITES)
CI_JS_SUITES ?= $(JS_SUITES) benchmark
ifeq ($(node_use_openssl), false)
CI_DOC := doctool
else
Expand Down Expand Up @@ -654,8 +656,8 @@ test-with-async-hooks:
$(MAKE) build-node-api-tests
$(MAKE) cctest
NODE_TEST_WITH_ASYNC_HOOKS=1 $(PYTHON) tools/test.py $(PARALLEL_ARGS) --mode=$(BUILDTYPE_LOWER) \
$(CI_JS_SUITES) \
$(CI_NATIVE_SUITES)
$(JS_SUITES) \
$(NATIVE_SUITES)


.PHONY: test-v8
Expand Down
2 changes: 2 additions & 0 deletions README.md
Expand Up @@ -311,6 +311,8 @@ For information about the governance of the Node.js project, see
**Gireesh Punathil** &lt;gpunathi@in.ibm.com&gt; (he/him)
* [guybedford](https://github.com/guybedford) -
**Guy Bedford** &lt;guybedford@gmail.com&gt; (he/him)
* [HarshithaKP](https://github.com/HarshithaKP) -
**Harshitha K P** &lt;harshitha014@gmail.com&gt; (she/her)
* [hashseed](https://github.com/hashseed) -
**Yang Guo** &lt;yangguo@chromium.org&gt; (he/him)
* [himself65](https://github.com/himself65) -
Expand Down
2 changes: 1 addition & 1 deletion doc/api/dns.md
Expand Up @@ -96,7 +96,7 @@ The following methods from the `dns` module are available:
<!-- YAML
added: v8.3.0
changes:
- version: v14.5.0
- version: v12.18.3
pr-url: https://github.com/nodejs/node/pull/33472
description: The constructor now accepts an `options` object.
The single supported option is `timeout`.
Expand Down
5 changes: 4 additions & 1 deletion doc/api/worker_threads.md
Expand Up @@ -621,6 +621,9 @@ if (isMainThread) {
<!-- YAML
added: v10.5.0
changes:
- version: REPLACEME
pr-url: https://github.com/nodejs/node/pull/34394
description: The `trackUnmanagedFds` option was set to `true` by default.
- version:
- v14.6.0
pr-url: https://github.com/nodejs/node/pull/34303
Expand Down Expand Up @@ -689,7 +692,7 @@ changes:
[`fs.close()`][], and close them when the Worker exits, similar to other
resources like network sockets or file descriptors managed through
the [`FileHandle`][] API. This option is automatically inherited by all
nested `Worker`s. **Default**: `false`.
nested `Worker`s. **Default**: `true`.
* `transferList` {Object[]} If one or more `MessagePort`-like objects
are passed in `workerData`, a `transferList` is required for those
items or [`ERR_MISSING_MESSAGE_PORT_IN_TRANSFER_LIST`][] will be thrown.
Expand Down
192 changes: 192 additions & 0 deletions doc/changelogs/CHANGELOG_V12.md

Large diffs are not rendered by default.

14 changes: 13 additions & 1 deletion doc/guides/onboarding-extras.md
Expand Up @@ -24,7 +24,19 @@ request.
* `tsc-agenda` - Open issues and pull requests with this label will be added to
the Technical Steering Committee meeting agenda

--
---

* `author-ready` - A pull request is _author ready_ when:
* There is a CI run in progress or completed.
* There is at least one Collaborator approval (or two TSC approvals
for semver-major PRs).
* There are no outstanding review comments.

Please always add the `author ready` label to the pull request in that case.
Please always remove it again as soon as the conditions are not met anymore,
such as if CI run fails or a new outstanding review comment is posted.

---

* `semver-{minor,major}`
* be conservative – that is, if a change has the remote *chance* of breaking
Expand Down
2 changes: 1 addition & 1 deletion lib/internal/worker.js
Expand Up @@ -152,7 +152,7 @@ class Worker extends EventEmitter {
env === process.env ? null : env,
options.execArgv,
parseResourceLimits(options.resourceLimits),
!!options.trackUnmanagedFds);
!!(options.trackUnmanagedFds ?? true));
if (this[kHandle].invalidExecArgv) {
throw new ERR_WORKER_INVALID_EXEC_ARGV(this[kHandle].invalidExecArgv);
}
Expand Down
22 changes: 10 additions & 12 deletions lib/zlib.js
Expand Up @@ -664,18 +664,13 @@ function Zlib(opts, mode) {
// to come up with a good solution that doesn't break our internal API,
// and with it all supported npm versions at the time of writing.
this._writeState = new Uint32Array(2);
if (!handle.init(windowBits,
level,
memLevel,
strategy,
this._writeState,
processCallback,
dictionary)) {
// TODO(addaleax): Sometimes we generate better error codes in C++ land,
// e.g. ERR_BROTLI_PARAM_SET_FAILED -- it's hard to access them with
// the current bindings setup, though.
throw new ERR_ZLIB_INITIALIZATION_FAILED();
}
handle.init(windowBits,
level,
memLevel,
strategy,
this._writeState,
processCallback,
dictionary);

ZlibBase.call(this, opts, mode, handle, zlibDefaultOpts);

Expand Down Expand Up @@ -821,6 +816,9 @@ function Brotli(opts, mode) {
new binding.BrotliDecoder(mode) : new binding.BrotliEncoder(mode);

this._writeState = new Uint32Array(2);
// TODO(addaleax): Sometimes we generate better error codes in C++ land,
// e.g. ERR_BROTLI_PARAM_SET_FAILED -- it's hard to access them with
// the current bindings setup, though.
if (!handle.init(brotliInitParamsArray,
this._writeState,
processCallback)) {
Expand Down
10 changes: 10 additions & 0 deletions node.gyp
Expand Up @@ -1376,6 +1376,11 @@
'HAVE_OPENSSL=1',
],
}],
[ 'node_use_openssl=="true" and experimental_quic==1', {
'defines': [
'NODE_EXPERIMENTAL_QUIC=1',
],
}],
['v8_enable_inspector==1', {
'defines': [
'HAVE_INSPECTOR=1',
Expand Down Expand Up @@ -1430,6 +1435,11 @@
'HAVE_OPENSSL=1',
],
}],
[ 'node_use_openssl=="true" and experimental_quic==1', {
'defines': [
'NODE_EXPERIMENTAL_QUIC=1',
],
}],
['v8_enable_inspector==1', {
'defines': [
'HAVE_INSPECTOR=1',
Expand Down
59 changes: 45 additions & 14 deletions src/node_zlib.cc
Expand Up @@ -139,8 +139,8 @@ class ZlibContext : public MemoryRetainer {
CompressionError ResetStream();

// Zlib-specific:
CompressionError Init(int level, int window_bits, int mem_level, int strategy,
std::vector<unsigned char>&& dictionary);
void Init(int level, int window_bits, int mem_level, int strategy,
std::vector<unsigned char>&& dictionary);
void SetAllocationFunctions(alloc_func alloc, free_func free, void* opaque);
CompressionError SetParams(int level, int strategy);

Expand All @@ -157,7 +157,10 @@ class ZlibContext : public MemoryRetainer {
private:
CompressionError ErrorForMessage(const char* message) const;
CompressionError SetDictionary();
bool InitZlib();

Mutex mutex_; // Protects zlib_init_done_.
bool zlib_init_done_ = false;
int err_ = 0;
int flush_ = 0;
int level_ = 0;
Expand Down Expand Up @@ -615,13 +618,8 @@ class ZlibStream : public CompressionStream<ZlibContext> {
AllocScope alloc_scope(wrap);
wrap->context()->SetAllocationFunctions(
AllocForZlib, FreeForZlib, static_cast<CompressionStream*>(wrap));
const CompressionError err =
wrap->context()->Init(level, window_bits, mem_level, strategy,
std::move(dictionary));
if (err.IsError())
wrap->EmitError(err);

return args.GetReturnValue().Set(!err.IsError());
wrap->context()->Init(level, window_bits, mem_level, strategy,
std::move(dictionary));
}

static void Params(const FunctionCallbackInfo<Value>& args) {
Expand Down Expand Up @@ -724,6 +722,15 @@ using BrotliEncoderStream = BrotliCompressionStream<BrotliEncoderContext>;
using BrotliDecoderStream = BrotliCompressionStream<BrotliDecoderContext>;

void ZlibContext::Close() {
{
Mutex::ScopedLock lock(mutex_);
if (!zlib_init_done_) {
dictionary_.clear();
mode_ = NONE;
return;
}
}

CHECK_LE(mode_, UNZIP);

int status = Z_OK;
Expand All @@ -742,6 +749,11 @@ void ZlibContext::Close() {


void ZlibContext::DoThreadPoolWork() {
bool first_init_call = InitZlib();
if (first_init_call && err_ != Z_OK) {
return;
}

const Bytef* next_expected_header_byte = nullptr;

// If the avail_out is left at 0, then it means that it ran out
Expand Down Expand Up @@ -897,6 +909,11 @@ CompressionError ZlibContext::GetErrorInfo() const {


CompressionError ZlibContext::ResetStream() {
bool first_init_call = InitZlib();
if (first_init_call && err_ != Z_OK) {
return ErrorForMessage("Failed to init stream before reset");
}

err_ = Z_OK;

switch (mode_) {
Expand Down Expand Up @@ -930,7 +947,7 @@ void ZlibContext::SetAllocationFunctions(alloc_func alloc,
}


CompressionError ZlibContext::Init(
void ZlibContext::Init(
int level, int window_bits, int mem_level, int strategy,
std::vector<unsigned char>&& dictionary) {
if (!((window_bits == 0) &&
Expand Down Expand Up @@ -974,6 +991,15 @@ CompressionError ZlibContext::Init(
window_bits_ *= -1;
}

dictionary_ = std::move(dictionary);
}

bool ZlibContext::InitZlib() {
Mutex::ScopedLock lock(mutex_);
if (zlib_init_done_) {
return false;
}

switch (mode_) {
case DEFLATE:
case GZIP:
Expand All @@ -995,15 +1021,15 @@ CompressionError ZlibContext::Init(
UNREACHABLE();
}

dictionary_ = std::move(dictionary);

if (err_ != Z_OK) {
dictionary_.clear();
mode_ = NONE;
return ErrorForMessage("zlib error");
return true;
}

return SetDictionary();
SetDictionary();
zlib_init_done_ = true;
return true;
}


Expand Down Expand Up @@ -1040,6 +1066,11 @@ CompressionError ZlibContext::SetDictionary() {


CompressionError ZlibContext::SetParams(int level, int strategy) {
bool first_init_call = InitZlib();
if (first_init_call && err_ != Z_OK) {
return ErrorForMessage("Failed to init stream before set parameters");
}

err_ = Z_OK;

switch (mode_) {
Expand Down
15 changes: 14 additions & 1 deletion test/parallel/test-worker-track-unmanaged-fds.js
@@ -1,10 +1,13 @@
'use strict';
const common = require('../common');
const assert = require('assert');
const { Worker } = require('worker_threads');
const { Worker, isMainThread } = require('worker_threads');
const { once } = require('events');
const fs = require('fs');

if (!isMainThread)
common.skip('test needs to be able to freely set `trackUnmanagedFds`');

// All the tests here are run sequentially, to avoid accidentally opening an fd
// which another part of the test expects to be closed.

Expand Down Expand Up @@ -37,6 +40,16 @@ process.on('warning', (warning) => parentPort.postMessage({ warning }));
assert.throws(() => fs.fstatSync(fd), { code: 'EBADF' });
}

// The same, but trackUnmanagedFds is used only as the implied default.
{
const w = new Worker(`${preamble}
parentPort.postMessage(fs.openSync(__filename));
`, { eval: true });
const [ [ fd ] ] = await Promise.all([once(w, 'message'), once(w, 'exit')]);
assert(fd > 2);
assert.throws(() => fs.fstatSync(fd), { code: 'EBADF' });
}

// There is a warning when an fd is unexpectedly opened twice.
{
const w = new Worker(`${preamble}
Expand Down
37 changes: 37 additions & 0 deletions test/parallel/test-zlib-reset-before-write.js
@@ -0,0 +1,37 @@
'use strict';
const common = require('../common');
const assert = require('assert');
const zlib = require('zlib');

// Tests that zlib streams support .reset() and .params()
// before the first write. That is important to ensure that
// lazy init of zlib native library handles these cases.

for (const fn of [
(z, cb) => {
z.reset();
cb();
},
(z, cb) => z.params(0, zlib.constants.Z_DEFAULT_STRATEGY, cb)
]) {
const deflate = zlib.createDeflate();
const inflate = zlib.createInflate();

deflate.pipe(inflate);

const output = [];
inflate
.on('error', (err) => {
assert.ifError(err);
})
.on('data', (chunk) => output.push(chunk))
.on('end', common.mustCall(
() => assert.deepStrictEqual(Buffer.concat(output).toString(), 'abc')));

fn(deflate, () => {
fn(inflate, () => {
deflate.write('abc');
deflate.end();
});
});
}