zlib: expose zlib.crc32() #52692

joyeecheung · 2024-04-25T17:53:00Z

This patch exposes the crc32() function from zlib to user-land.

It computes a 32-bit Cyclic Redundancy Check checksum of data. If
value is specified, it is used as the starting value of the checksum,
otherwise, 0 is used as the starting value.

The CRC algorithm is designed to compute checksums and to detect error
in data transmission. It's not suitable for cryptographic authentication.

To be consistent with other APIs, if the data is a string, it will
be encoded with UTF-8 before being used for computation. If users only
use Node.js to compute and match the checksums, this works well with
other APIs that uses the UTF-8 encoding by default.

Some third-party JavaScript libraries compute the checksum on a
string based on str.charCodeAt() so that it can be run in browsers.
If users want to match the checksum computed with this kind of library
in the browser, it's better to use the same library in Node.js
if it also runs in Node.js. If users have to use zlib.crc32() to
match the checksum produced by such a third-party library:

If the library accepts Uint8Array as input, use TextEncoder
in the browser to encode the string into a Uint8Array with UTF-8
encoding, and compute the checksum based on the UTF-8 encoded string
in the browser.
If the library only takes a string and compute the data based on
str.charCodeAt(), on the Node.js side, convert the string into
a buffer using Buffer.from(str, 'utf16le').

const zlib = require('node:zlib');
const { Buffer } = require('node:buffer');

let crc = zlib.crc32('hello');  // 907060870
crc = zlib.crc32('world', crc);  // 4192936109

crc = zlib.crc32(Buffer.from('hello', 'utf16le'));  // 1427272415
crc = zlib.crc32(Buffer.from('world', 'utf16le'), crc);  // 4150509955

NOTE: it's exposed in zlib because that's also what Python and Ruby do.

src/node_zlib.cc

This patch exposes the crc32() function from zlib to user-land. It computes a 32-bit Cyclic Redundancy Check checksum of `data`. If `value` is specified, it is used as the starting value of the checksum, otherwise, 0 is used as the starting value. ```js const zlib = require('node:zlib'); const { Buffer } = require('node:buffer'); let crc = zlib.crc32('hello'); // 907060870 crc = zlib.crc32('world', crc); // 4192936109 crc = zlib.crc32(Buffer.from('hello')); // 907060870 crc = zlib.crc32(Buffer.from('world'), crc); // 4192936109 ```

doc/api/zlib.md

lpinca · 2024-04-25T19:33:08Z

Wouldn't it be better to expose it under crypto?

Edit: On second thought, maybe not.

joyeecheung · 2024-04-25T19:54:39Z

Wouldn't it be better to expose it under crypto?

It's not a cryptographic hash, but a checksum. I was thinking about copying the python documentation (python also puts it on zlib):

The algorithm is not cryptographically strong, and should not be used for authentication or digital signatures. Since the algorithm is designed for use as a checksum algorithm, it is not suitable for use as a general hash algorithm.

(Not sure if it's okay to copy other documentation, but if we have to write one from scratch it wouldn't be too different anyway?)

panva · 2024-04-26T09:49:30Z

Wouldn't it be better to expose it under crypto?

Edit: On second thought, maybe not.

I had the exact same thought process.

The algorithm is not cryptographically strong, and should not be used for authentication or digital signatures. Since the algorithm is designed for use as a checksum algorithm, it is not suitable for use as a general hash algorithm.

I agree this documentation bit or something to its effect is necessary to circumvent the same brain exercises myself and @lpinca went through :)

joyeecheung · 2024-04-26T09:51:26Z

The doc currently puts it as

The CRC algorithm is designed to compute checksums and to detect error
in data transmission. It's not suitable for cryptographic authentication.

Also, note that both Python and Ruby puts crc32 under a zlib module

github-actions · 2024-04-26T12:20:55Z

Failed to start CI

   ⚠  Something was pushed to the Pull Request branch since the last approving review.
   ✘  Refusing to run CI on potentially unsafe PR

https://github.com/nodejs/node/actions/runs/8848132665

joyeecheung · 2024-04-26T12:24:33Z

huh, this is...amusing. I'll start from Jenkins.

nodejs-github-bot · 2024-04-26T12:24:35Z

CI: https://ci.nodejs.org/job/node-test-pull-request/58716/

src/node_zlib.cc

Co-authored-by: Anna Henningsen <github@addaleax.net>

nodejs-github-bot · 2024-04-26T17:32:22Z

CI: https://ci.nodejs.org/job/node-test-pull-request/58721/

nodejs-github-bot · 2024-04-27T08:13:52Z

CI: https://ci.nodejs.org/job/node-test-pull-request/58741/

doc/api/zlib.md

lib/zlib.js

nodejs-github-bot · 2024-04-28T05:40:03Z

CI: https://ci.nodejs.org/job/node-test-pull-request/58768/

nodejs-github-bot · 2024-04-29T18:16:00Z

CI: https://ci.nodejs.org/job/node-test-pull-request/58803/

nodejs-github-bot · 2024-04-30T15:32:11Z

CI: https://ci.nodejs.org/job/node-test-pull-request/58821/

joyeecheung · 2024-05-01T16:56:50Z

Updated to run clang-format on the code. Can I get a re-approval to land it? @benjamingr @jasnell @addaleax @lpinca @anonrig @VoltrexKeyva

nodejs-github-bot · 2024-05-02T12:54:47Z

Landed in 2cfabb1

This patch exposes the crc32() function from zlib to user-land. It computes a 32-bit Cyclic Redundancy Check checksum of `data`. If `value` is specified, it is used as the starting value of the checksum, otherwise, 0 is used as the starting value. ```js const zlib = require('node:zlib'); const { Buffer } = require('node:buffer'); let crc = zlib.crc32('hello'); // 907060870 crc = zlib.crc32('world', crc); // 4192936109 crc = zlib.crc32(Buffer.from('hello')); // 907060870 crc = zlib.crc32(Buffer.from('world'), crc); // 4192936109 ``` PR-URL: nodejs#52692 Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

This patch exposes the crc32() function from zlib to user-land. It computes a 32-bit Cyclic Redundancy Check checksum of `data`. If `value` is specified, it is used as the starting value of the checksum, otherwise, 0 is used as the starting value. ```js const zlib = require('node:zlib'); const { Buffer } = require('node:buffer'); let crc = zlib.crc32('hello'); // 907060870 crc = zlib.crc32('world', crc); // 4192936109 crc = zlib.crc32(Buffer.from('hello')); // 907060870 crc = zlib.crc32(Buffer.from('world'), crc); // 4192936109 ``` PR-URL: #52692 Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run. zlib Issues and PRs related to the zlib subsystem. labels Apr 25, 2024

anonrig reviewed Apr 25, 2024

View reviewed changes

src/node_zlib.cc Show resolved Hide resolved

anonrig approved these changes Apr 25, 2024

View reviewed changes

joyeecheung force-pushed the crc32 branch 2 times, most recently from b2b9ece to 23223c6 Compare April 25, 2024 19:05

joyeecheung force-pushed the crc32 branch from 23223c6 to 53fb0e4 Compare April 25, 2024 19:16

joyeecheung commented Apr 25, 2024

View reviewed changes

doc/api/zlib.md Show resolved Hide resolved

joyeecheung added 2 commits April 25, 2024 22:01

fixup! zlib: expose zlib.crc32()

d9057fc

fixup! fixup! zlib: expose zlib.crc32()

d4fc5ca

joyeecheung force-pushed the crc32 branch from 278dfef to d4fc5ca Compare April 25, 2024 20:04

fixup! fixup! zlib: expose zlib.crc32()

b26c0c6

joyeecheung added the request-ci Add this label to start a Jenkins CI on a PR. label Apr 26, 2024

github-actions bot added request-ci-failed An error occurred while starting CI via request-ci label, and manual interventon is needed. and removed request-ci Add this label to start a Jenkins CI on a PR. labels Apr 26, 2024

joyeecheung mentioned this pull request Apr 26, 2024

feat(ncu-ci)!: require --certify-safe flag nodejs/node-core-utils#798

Merged

lpinca approved these changes Apr 26, 2024

View reviewed changes

lpinca added the semver-minor PRs that contain new features and should be released in the next minor version. label Apr 26, 2024

addaleax approved these changes Apr 26, 2024

View reviewed changes

src/node_zlib.cc Outdated Show resolved Hide resolved

fixup! zlib: expose zlib.crc32()

905c481

Co-authored-by: Anna Henningsen <github@addaleax.net>

VoltrexKeyva approved these changes Apr 26, 2024

View reviewed changes

benjamingr reviewed Apr 27, 2024

View reviewed changes

doc/api/zlib.md Show resolved Hide resolved

benjamingr approved these changes Apr 27, 2024

View reviewed changes

jasnell reviewed Apr 27, 2024

View reviewed changes

lib/zlib.js Show resolved Hide resolved

jasnell approved these changes Apr 27, 2024

View reviewed changes

fixup! fixup! zlib: expose zlib.crc32()

58d9da4

lpinca approved these changes May 1, 2024

View reviewed changes

nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label May 2, 2024

nodejs-github-bot merged commit 2cfabb1 into nodejs:main May 2, 2024
57 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

zlib: expose zlib.crc32() #52692

zlib: expose zlib.crc32() #52692

joyeecheung commented Apr 25, 2024 •

edited

lpinca commented Apr 25, 2024 •

edited

joyeecheung commented Apr 25, 2024 •

edited

panva commented Apr 26, 2024

joyeecheung commented Apr 26, 2024 •

edited

github-actions bot commented Apr 26, 2024

joyeecheung commented Apr 26, 2024

nodejs-github-bot commented Apr 26, 2024

nodejs-github-bot commented Apr 26, 2024

nodejs-github-bot commented Apr 27, 2024

nodejs-github-bot commented Apr 28, 2024

nodejs-github-bot commented Apr 29, 2024

nodejs-github-bot commented Apr 30, 2024

joyeecheung commented May 1, 2024

nodejs-github-bot commented May 2, 2024

zlib: expose zlib.crc32() #52692

zlib: expose zlib.crc32() #52692

Conversation

joyeecheung commented Apr 25, 2024 • edited

lpinca commented Apr 25, 2024 • edited

joyeecheung commented Apr 25, 2024 • edited

panva commented Apr 26, 2024

joyeecheung commented Apr 26, 2024 • edited

github-actions bot commented Apr 26, 2024

joyeecheung commented Apr 26, 2024

nodejs-github-bot commented Apr 26, 2024

nodejs-github-bot commented Apr 26, 2024

nodejs-github-bot commented Apr 27, 2024

nodejs-github-bot commented Apr 28, 2024

nodejs-github-bot commented Apr 29, 2024

nodejs-github-bot commented Apr 30, 2024

joyeecheung commented May 1, 2024

nodejs-github-bot commented May 2, 2024

joyeecheung commented Apr 25, 2024 •

edited

lpinca commented Apr 25, 2024 •

edited

joyeecheung commented Apr 25, 2024 •

edited

joyeecheung commented Apr 26, 2024 •

edited