feat(stream-transform-from): create a new package (#68)

* chore(stream-transform-from): create directory for `@sounisi5011/stream-transform-from` package * test(stream-transform-from): add tests * test(stream-transform-from): remove the `createPipeline()` function from test code * feat(stream-transform-from): add a type definition for the public API * feat(stream-transform-from): add `transformFrom()` function and `TransformFromAsyncIterable` class * fix(stream-transform-from): change the complex `createSource()` method to a simple `createSource()` function * test(stream-transform-from): fix the tests + Re-create the `Transform` object every time testing. + Simplify tests that differ only in options with `describe.each()`. * refactor(stream-transform-from): use the `catch()` method instead of the `try ... catch` statement * fix(stream-transform-from): don't call the callback multiple times & call the callback passed to `_flush()` method after finish * test(stream-transform-from): use the `break` keyword while transforming a stream * fix(stream-transform-from): use the `break` keyword while transforming a stream * refactor(stream-transform-from): refactoring the `TransformFromAsyncIterable` class + Rename the `done` property to `transformCallback + Rename the `callDoneFn` method to `callTransformCallback` + Remove the `pushError` method and add `finish` method in its place * test(stream-transform-from): add tests + Added a test that returns multiple chunks + Added a test for the type of chunks contained in the source * fix(stream-transform-from): fix the type definition of return values If there is a possibility that either the `objectMode` or the `readableObjectMode` option is not `true`, the return value should not be `unknown`. * test(stream-transform-from): add a test to merge multiple chunks * fix(stream-transform-from): `transformCallback` should be called when the next chunk is needed, not after transforming a chunk * test(stream-transform-from): add type definition tests * fix(stream-transform-from): fix `InputChunkType` Index signatures and `boolean` types are now supported. * test(stream-transform-from): add type definition tests for output values * docs(stream-transform-from): add example code * test(stream-transform-from): add a test for the timing of data flowing in the stream * test(stream-transform-from): fix the timing test for data flowing in the stream I noticed that the timing of the flow changes when using the asynchronous API. * test(stream-transform-from): eliminate the PromiseRejectionHandledWarning Promise objects should only be created in tests. Creating it during test case generation could cause Promise errors to affect unrelated code. * test(stream-transform-from): fix the timing test for data flowing in the stream We have not found a way to control when data is read from the Readable stream. * fix(stream-transform-from): create a source iterator in the constructor * refactor(stream-transform-from): remove `src/utils.ts` * refactor(stream-transform-from): reduce Cognitive Complexity in the `TransformFromAsyncIterable#finish()` method Code Climate reported: + Function `finish` has a Cognitive Complexity of 6 (exceeds 5 allowed). Consider refactoring. * test(stream-transform-from): add tests that inherit from the `TransformFromAsyncIterable` class * revert: test(stream-transform-from): add tests that inherit from the `TransformFromAsyncIterable` class This reverts commit 28e9901. * test(stream-transform-from): add tests to convert strings of different encodings * feat(stream-transform-from): can use the encoding passed with the chunk * feat(stream-transform-from): export some types * test(stream-transform-from): can't get the error after processing all chunks see nodejs/node#34274 * test(stream-transform-from): the bug with not being able to get errors after processing all chunks has been fixed in Node v15 * test(stream-transform-from): fix the bug that no error occurs after getting all the chunks This bug has been fixed in Node.js v15. + https://github.com/nodejs/node/blob/master/doc/changelogs/CHANGELOG_V15.md#15.0.0 + nodejs/node#34314 However, we have found a workaround that works for anything less than Node.js v15, so we will attempt to fix this bug in this package. * fix(stream-transform-from): fix the bug that no error occurs after getting all the chunks * test(stream-transform-from): make the conditional branches of the test code readable and organized * test(stream-transform-from): refactoring test code * build(stream-transform-from): disable the inlineSources option in TypeScript The size of the package is now smaller if `src` directory contents are included. * docs(stream-transform-from): add `README.md` * ci(stream-transform-from): add custom publish scripts * docs(stream-transform-from): update `README.md` * test(stream-transform-from): add tests for options that should be ignored * fix(stream-transform-from): ignore some options * fix(stream-transform-from): add "construct" in the options to ignore * docs(stream-transform-from): update `README.md`: add fields that are disallowed in options * refactor(stream-transform-from): don't use the global object `process` * docs(stream-transform-from): fix the code for the example in `README.md` * refactor(stream-transform-from): move utility type functions to the top of file * style(stream-transform-from): update `src/index.ts`
sounisi5011 · May 22, 2021 · 2d7d879 · 2d7d879
1 parent 4782812
commit 2d7d879
Show file tree

Hide file tree

Showing 16 changed files with 2,016 additions and 1 deletion.
diff --git a/.eslintrc.yaml b/.eslintrc.yaml
@@ -56,7 +56,7 @@ overrides:
     parserOptions:
       sourceType: module
       project:
-        - ./{packages{,/ts-type-utils},actions}/*{,/tests{,/helpers}}/tsconfig.json
+        - ./{packages{,/ts-type-utils},actions}/*{,/tests{,/helpers},/test-d}/tsconfig.json
     settings:
       node:
         # see https://github.com/mysticatea/eslint-plugin-node/blob/v11.1.0/docs/rules/shebang.md

diff --git a/packages/stream-transform-from/.github/workflows/publish.sh b/packages/stream-transform-from/.github/workflows/publish.sh
@@ -0,0 +1,10 @@
+#!/bin/bash
+
+# https://qiita.com/yudoufu/items/48cb6fb71e5b498b2532#comment-87e291b98f4cabf77138
+readonly DIR_PATH="$(cd "$(dirname "${BASH_SOURCE:-${(%):-%N}}")"; pwd)"
+
+outputs_tag_name="${outputs_tag_name}" \
+  node "${DIR_PATH}/../../scripts/publish-convert-readme.js"
+
+# see https://stackoverflow.com/a/62675843/4907315
+pnpm publish --access=public --no-git-checks
diff --git a/packages/stream-transform-from/README.md b/packages/stream-transform-from/README.md
@@ -0,0 +1,166 @@
+# @sounisi5011/stream-transform-from
+
+[![Go to the latest release page on npm](https://img.shields.io/npm/v/@sounisi5011/stream-transform-from.svg)](https://www.npmjs.com/package/@sounisi5011/stream-transform-from)
+![Supported Node.js version: ^12.17.x || 14.x || 15.x || 16.x](https://img.shields.io/node/v/@sounisi5011/stream-transform-from)
+[![Tested with Jest](https://img.shields.io/badge/tested_with-jest-99424f.svg)](https://github.com/facebook/jest)
+[![Commitizen friendly](https://img.shields.io/badge/commitizen-friendly-brightgreen.svg)](http://commitizen.github.io/cz-cli/)
+<!-- [![Minified Bundle Size Details](https://img.shields.io/bundlephobia/min/@sounisi5011/stream-transform-from)](https://bundlephobia.com/result?p=@sounisi5011/stream-transform-from) -->
+<!-- [![Install Size Details](https://packagephobia.com/badge?p=@sounisi5011/stream-transform-from)](https://packagephobia.com/result?p=@sounisi5011/stream-transform-from) -->
+[![Dependencies Status](https://status.david-dm.org/gh/sounisi5011/npm-packages.svg?path=packages%2Fstream-transform-from)](https://david-dm.org/sounisi5011/npm-packages?path=packages/stream-transform-from)
+[![Build Status](https://github.com/sounisi5011/npm-packages/actions/workflows/ci.yaml/badge.svg)](https://github.com/sounisi5011/npm-packages/actions/workflows/ci.yaml)
+[![Maintainability Status](https://api.codeclimate.com/v1/badges/26495b68302f7ff963c3/maintainability)](https://codeclimate.com/github/sounisi5011/npm-packages/maintainability)
+
+[`stream.Transform` class]: https://nodejs.org/docs/latest/api/stream.html#stream_class_stream_transform
+[`Buffer` object]: https://nodejs.org/api/buffer.html
+
+Create a [transform stream][`stream.Transform` class] from an async iterator.
+This is [the last piece](https://github.com/nodejs/node/issues/27140#issuecomment-533266638) needed to convert between streams and async iterators/generators.
+
+## Features
+
+* No dependencies
+
+    This package uses only the Node.js built-in [`stream.Transform` class].
+
+* Strict type definition
+
+    The exact type definitions for arguments and return values will be generated based on the `objectMode` option.
+
+* Encoding arguments can be used
+
+    You can use `encoding`, which is passed as the second argument of the [`transform._transform()` method](https://nodejs.org/docs/latest/api/stream.html#stream_transform_transform_chunk_encoding_callback).
+    This allows you to safely convert a string to [`Buffer` object].
+
+## Installation
+
+```sh
+npm install @sounisi5011/stream-transform-from
+```
+
+```sh
+yarn add @sounisi5011/stream-transform-from
+```
+
+```sh
+pnpm add @sounisi5011/stream-transform-from
+```
+
+## Usage
+
+### Convert [`Buffer` objects][`Buffer` object]
+
+```js
+const fs = require('fs');
+const stream = require('stream');
+
+const { transformFrom } = require('@sounisi5011/stream-transform-from');
+
+stream.pipeline(
+  fs.createReadStream('input.txt', 'utf8'),
+  transformFrom(async function*(source) {
+    for await (const { chunk } of source) {
+      yield chunk.toString('utf8').toUpperCase();
+    }
+  }),
+  fs.createWriteStream('output.txt'),
+  error => {
+    if (error) {
+      console.error(error);
+    } else {
+      console.log('done!');
+    }
+  }
+);
+```
+
+### Convert any type value
+
+```js
+const stream = require('stream');
+
+const { transformFrom } = require('@sounisi5011/stream-transform-from');
+
+stream.pipeline(
+  stream.Readable.from([1, 2, 3]),
+  transformFrom(
+    async function*(source) {
+      for await (const { chunk } of source) {
+        yield chunk + 2;
+      }
+    },
+    { objectMode: true }
+  ),
+  // ...
+  error => {
+    if (error) {
+      console.error(error);
+    } else {
+      console.log('done!');
+    }
+  }
+);
+```
+
+### Convert string to [`Buffer`][`Buffer` object] using encoding
+
+```js
+const stream = require('stream');
+
+const { transformFrom } = require('@sounisi5011/stream-transform-from');
+
+stream.pipeline(
+  // ...
+  transformFrom(
+    async function*(source) {
+      for await (const { chunk, encoding } of source) {
+        if (typeof chunk === 'string') {
+          yield Buffer.from(chunk, encoding);
+        }
+      }
+    },
+    { writableObjectMode: true }
+  ),
+  // ...
+  error => {
+    if (error) {
+      console.error(error);
+    } else {
+      console.log('done!');
+    }
+  }
+);
+```
+
+## API
+
+```js
+const { transformFrom } = require('@sounisi5011/stream-transform-from');
+
+// The return value is a Transform stream.
+const transformStream = transformFrom(
+  async function*(source) {
+    // `source` is `AsyncIterableIterator<{ chunk: Buffer, encoding: BufferEncoding }>`
+    //          or `AsyncIterableIterator<{ chunk: unknown, encoding: BufferEncoding }>` type
+
+    // The value returned by `yield` keyword will be passed as the first argument of `transform.push()` method.
+  },
+
+  // The second argument is an options for the Transform stream.
+  // The options are passed to the constructor function of the Transform class.
+  // However, the following fields are not allowed:
+  // + `construct`
+  // + `read`
+  // + `write`
+  // + `writev`
+  // + `final`
+  // + `destroy`
+  // + `transform`
+  // + `flush`
+  // The fields listed above will be ignored if specified.
+  {}
+);
+```
+
+## Related
+
+* [generator-transform-stream](https://github.com/bealearts/generator-transform-stream)
diff --git a/packages/stream-transform-from/examples/index.js b/packages/stream-transform-from/examples/index.js
@@ -0,0 +1,44 @@
+const stream = require('stream');
+
+const { transformFrom } = require('@sounisi5011/stream-transform-from');
+
+stream.pipeline(
+  stream.Readable.from([1, 2, 3, 4, 5]),
+  // Convert a number to a Buffer object.
+  transformFrom(
+    async function*(source) {
+      for await (const { chunk: inputChunk } of source) {
+        console.log({ inputChunk });
+
+        if (typeof inputChunk === 'number') {
+          const code = inputChunk;
+          yield Buffer.from([code]);
+        }
+      }
+    },
+    { writableObjectMode: true },
+  ),
+  // Transform a Buffer object.
+  transformFrom(async function*(source) {
+    for await (const { chunk } of source) {
+      yield Buffer.concat([
+        Buffer.from([0xF0]),
+        chunk,
+        Buffer.from([0xFF]),
+      ]);
+    }
+  }),
+  new stream.Writable({
+    write(outputChunk, _, done) {
+      console.log({ outputChunk });
+      done();
+    },
+  }),
+  error => {
+    if (error) {
+      console.error(error);
+    } else {
+      console.log('done!');
+    }
+  },
+);
diff --git a/packages/stream-transform-from/examples/package.json b/packages/stream-transform-from/examples/package.json
@@ -0,0 +1,9 @@
+{
+  "private": true,
+  "dependencies": {
+    "@sounisi5011/stream-transform-from": "link:.."
+  },
+  "engines": {
+    "node": "^12.3.x"
+  }
+}
diff --git a/packages/stream-transform-from/jest.config.js b/packages/stream-transform-from/jest.config.js
@@ -0,0 +1,11 @@
+module.exports = {
+  preset: 'ts-jest',
+  coverageDirectory: 'coverage',
+  globals: {
+    'ts-jest': {
+      tsconfig: '<rootDir>/tests/tsconfig.json',
+    },
+  },
+  testEnvironment: 'node',
+  testMatch: ['<rootDir>/tests/**/*.ts'],
+};
diff --git a/packages/stream-transform-from/package.json b/packages/stream-transform-from/package.json
@@ -0,0 +1,86 @@
+{
+  "name": "@sounisi5011/stream-transform-from",
+  "version": "0.0.0",
+  "description": "Create a transform stream from an async iterator",
+  "keywords": [
+    "async",
+    "asyncgenerator",
+    "asyncgeneratorfunction",
+    "asyncgeneratorfunctions",
+    "asyncgenerators",
+    "asynciterable",
+    "asynciterables",
+    "asynciterator",
+    "asynciterators",
+    "from",
+    "generator",
+    "generatorfunction",
+    "generatorfunctions",
+    "generators",
+    "iterable",
+    "iterables",
+    "iterator",
+    "iterators",
+    "stream",
+    "transform",
+    "util",
+    "utility"
+  ],
+  "homepage": "https://github.com/sounisi5011/npm-packages/tree/main/packages/stream-transform-from#readme",
+  "bugs": {
+    "url": "https://github.com/sounisi5011/npm-packages/issues"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/sounisi5011/npm-packages.git",
+    "directory": "packages/stream-transform-from"
+  },
+  "license": "MIT",
+  "author": "sounisi5011",
+  "type": "commonjs",
+  "exports": {
+    ".": "./dist/index.js",
+    "./package.json": "./package.json"
+  },
+  "main": "./dist/index.js",
+  "types": "./dist/index.d.ts",
+  "directories": {
+    "lib": "./src/",
+    "example": "./examples/",
+    "test": "./tests/"
+  },
+  "files": [
+    "dist/",
+    "src/"
+  ],
+  "scripts": {
+    "build": "tsc",
+    "lint:tsc": "run-p lint:tsc:*",
+    "lint:tsc:src": "tsc --noEmit",
+    "lint:tsc:test": "tsc -p ./tests/ --noEmit",
+    "lint:tsc:test-d": "tsc -p ./test-d/ --noEmit",
+    "test": "run-if-supported-node run-p test:*",
+    "test:examples": "run-s build test:examples:*",
+    "test:examples:index": "node ./examples/index.js",
+    "test:jest": "jest",
+    "test:tsd": "run-s build test:tsd:*",
+    "test:tsd:exec": "tsd"
+  },
+  "devDependencies": {
+    "@sounisi5011/scripts--run-if-supported-node": "workspace:^0.0.0",
+    "@types/node": "15.x",
+    "tsd": "0.15.1"
+  },
+  "engines": {
+    "node": "^12.17.x || 14.x || 15.x || 16.x"
+  },
+  "runkitExampleFilename": "./examples/index.js",
+  "tsd": {
+    "compilerOptions": {
+      "lib": [
+        "es2019"
+      ],
+      "rootDir": "./"
+    }
+  }
+}