How to read remote text file and write it in reverse line order?

[mariusa]

Hi,

This works fine for reading a .jsonl text file from URL and writing the same contents to a local file:

 await pipeline(
        got.stream(url),
        fs.createWriteStream(filename)
    )

How could one write contents in reverse order line by line? Or, have an event to process remote file in reverse order line by line?

Example remote file:

line 1
long line 2
line 3
long line 4
another line 5

Looking to get a local file or readline event, but in reverse line order (not character order)

another line 5
long line 4
line 3
long line 2
line 1

Found this https://github.com/dominictarr/fs-reverse , but it's no longer maintained and works only for local files.
Note: file size could be GBs, can't read it all and reverse it in memory.

Thank you!

[szmarczak]

This is a problem totally unrelated to Got, best if asked on StackOverflow :)

Writing in reverse order is highly inefficient. Instead, read it in reverse.

You can write offests of the line breaks (you can use a TransformStream). For example the offsets for

line 1
long line 2
line 3
long line 4
another line 5

are [6, 18, 25, 37]. Once the file is saved on disk, you can just pass start and end arguments in https://nodejs.org/api/fs.html#fscreatereadstreampath-options

start: 37 + 1 (line break) = 38, end: Infinity
start: 25 + 1 (line break) = 26, end: 37
start: 18 + 1 (line break) = 19, end: 25
start: 6 + 1 (line break) = 7, end: 18
start: 0, end: 6

[mariusa]

Thank you, Szymon! Very useful.

The TransformStream should be added when reading the stream with got, right? eg

 await pipeline(
        got.stream(url),
        myTransform(), // is this correct? Or it needs to be passed some parameters?
        fs.createWriteStream(filename)
    )


const myTransform = new Transform({
   transform(chunk, encoding, cb) {
      // find offsets of the line breaks
      cb();
  }
});

(found this Transform example: #1184 )