Replaced glob with readdir-glob to be memory efficient

[Yqnn]

As described in the following issue: #422, node-glob is not designed to handle a huge amount of files: it requires a quantify of memory that is proportional to the number of matched files.

Why ? Because, it lists only the folders that appears in the pattern, and it has to memorise all the found files to ensure a same file is not emitted twice.
It makes sense to proceed like this when the pattern matches only a little proportion of the filesystem.
But as it's not a common use case when creating an archive, it would be more efficient to list all the files, and then check if they match the given pattern.

Advantage:

• memory consumptions is fixed, no matter the number of matched files
• it's faster when the proportion of matching files is high

Drawback:

• absolute glob patterns are not supported: archiver.glob('*.txt',{cwd: '/tmp'}) has to be used instead of archiver.glob('/tmp/*.txt')
• it's slower when few files matches in a big filesystem

The current PR implements this approach by replacing glob with readdir-glob that is memory-efficient.
It also pauses the glob stream when archiving is on-going, to keep the memory usage stable.

Maybe it would be better to offer this as a new option, or only to replace the directory() function.
Feel free to give feedback :)

[melitus]

@Yqnn Is memory consumptions also fixed for archiver.append(file, {name: filename})

[Yqnn]

@Yqnn Is memory consumptions also fixed for archiver.append(file, {name: filename})

Nop, if you use append() the library has to remember all the appended files, so it's not possible to make it memory efficient in that case.
You have to put in place a throttling mechanism on your side. I guess you could check if archiver._queue.length is small enough before appending new files.

[melitus]

@Yqnn Thanks. But is there any alternative to append(). I am passing buffer and filename to append()