Detect streaming process

[ZababurinSergei]

Hi!

For streaming downloading, the FIFO class is used.

Here is the cycle where reading occurs during cloning.

class FIFO {
  constructor() {
    this._queue = [];
  }

  write(chunk) {
    if (this._ended) {
      throw Error('You cannot write to a FIFO that has already been ended!')
    }
    console.log('######## Cycle ########')
    if (this._waiting) {
      const resolve = this._waiting;
      this._waiting = null;
      resolve({ value: chunk });
    } else {
      this._queue.push(chunk);
    }
  }

Process:

After compression, data downloading begins.
And this process is not in the callbacks

// Convert a web ReadableStream (not Node stream!) to an Async Iterator
// adapted from https://jakearchibald.com/2017/async-iterators-and-generators/
function fromStream(stream) {
  // Use native async iteration if it's available.
  if (stream[Symbol.asyncIterator]) return stream
  const reader = stream.getReader();
  return {
    next() {
      console.log('download....')
      return reader.read()
    },
    return() {
      reader.releaseLock();
      return {}
    },
    [Symbol.asyncIterator]() {
      return this
    },
  }
}

After downloading the following process occurs.

Нow can I display data about the loading progress for the progress bar in percent ?

[jcubic]

Are you asking how to calculate lodaded / total * 100?

[ZababurinSergei]

No.

The onProgress callback does not display the moment the repository is downloaded using the fetch method.

• downloading the repository takes 5 minutes. In the picture this is the output to the download console. I put console.log inside your code
• Between the Compressing object status and the onMessage Total call, the status is not included in the onProgress callback.

It seems that the clone method is stuck.

I would like to send fetch progress to the onProgress method instead of console.log('download...') output
https://javascript.info/fetch-progress

[jcubic]

Do you know to fix this? Do you want to contribute and add this feature to the library?

[ZababurinSergei]

The onProgress method can be passed.

Conditionally something like this.

function fromStream(stream, onProgress) { 
.......

 return {
    next() {
      onProgress({
        phase: 'download'
      })
      return reader.read()
    },
    return() {
      reader.releaseLock();
      return {}
    },
    [Symbol.asyncIterator]() {
      return this
    },
  }
  .......

or this

  async _next() {
    this.started = true
    let { done, value } = await this.stream.next()
    if (done) {
      this._ended = true
      if (!value) return Buffer.alloc(0)
    }
    if (value) {
      value = Buffer.from(value)
    }
    return value
  }

The length of the downloaded file can be transferred.

But it uses header Transfer-Encoding: chunked

And in the case-proxy header Content-length is removed

 for (let h of exposeHeaders) {
                if (h === 'content-length') {continue;}
                if (f.headers.has(h)) {
                    res.setHeader(h, f.headers.get(h));
                }
            }

I can try to do....
If you check my code.

I can write very crookedly...

And I don't know the project.
Maybe it’s better to do this in some other better way.

That's why I wrote to you. You must know everything about the project.

There is also a method onMessage. It might be better to do it through him

[jcubic]

Sorry, I know very little about the project, I mostly do maintenance stuff. I rarely write code myself. The original author who was writing all the code left the project, I decided to maintain it, so the project wouldn't die. I don't know the internals. So if you want this feature you will need to implement it yourself. It would be a nice feature when progress executes more frequently.

[ZababurinSergei]

I did what I wanted.
Displayed all the statuses that come to the onMessage method and everything that comes to onProgress

It seems to me that it would be good to display all messages in the onMessage method and set the download status.

But we need to better understand the project.

If the project is large, then these statuses are needed to understand when the code can actually be used.

In the video I showed all the statuses that occur when cloning a repository.

If I have time, I'll try to do this.

Screencast.from.2024-05-20.18-02-50.webm