Skip to content

simplifylabs/uniquer

Repository files navigation

Uniquer

Save equal files only once, avoid duplicates.


Install

NPM

npm install uniquer

Yarn

yarn add uniquer

Usage

Synchronously

import Uniquer from 'uniquer';

const fileName = Uniquer.writeSync('./output/', '.txt', 'Hello, world!');

// 315f5bdb...5894edd3.txt
console.log(fileName);

Asynchronously

import Uniquer from 'uniquer';

const fileName = await Uniquer.write('./output/', '.txt', 'Hello, world!');

// 315f5bdb...5894edd3.txt
console.log(fileName);

This may be all you need to use Uniquer. For advanced usage, see API


Advantages

📦 Save storage

No duplicates, more storage.

💻 Save performance

Only write a file if it doesn't exist.

📁 Get unique file names

No need to think of a creative file name.

🧹 Keep it clean

No duplicates, less files, cleaner.


How it works

The concept is that simple:

  1. Create the SHA256 hash of the passed data.
  2. Get the file name by appending the file extension.
  3. Check if a file with this name already exists. If...
    • ...not, create it.
    • ...yes, do nothing.

This means: Same data, same file name. So, if you try to write the same data multiple times, it will:

  • write the data only once.
  • only create a single file.

There is no need to worry that there could be a duplicate file name, as there wasn't any collision with SHA256 ever. More here.


API

Uniquer.writeSync(directory, extension, data[, options])

Write data to a unique file synchronously.
Calling multiple times with the exact same data, will result in the same file name.

directory String Path to the output directory
extension String File extension
data String | ArrayBufferView The file's data
[options] String | WriteFileOptions Encoding or WriteFileOptions
returns String File name of created file

Example

import Uniquer from 'uniquer';
// 315f5bdb...5894edd3.txt
const fileName = Uniquer.writeSync('./output/', '.txt', 'Hello, world!');

Uniquer.write(directory, extension, data[, options])

Write data to a unique file asynchronously.
Calling multiple times with the exact same data, will result in the same file name.

directory String Path to the output directory
extension String File extension
data String | ArrayBufferView The file's data
[options] String | WriteFileOptions Encoding or WriteFileOptions
returns Promise<String> File name of created file

Example

import Uniquer from 'uniquer';
// 315f5bdb...5894edd3.txt
const fileName = await Uniquer.write('./output/', '.txt', 'Hello, world!');

Uniquer.getFileName(extension, data)

Resolve the unique file path, based on the file hash + extension.

extension String File extension
data String | ArrayBufferView The file's data
returns String Resolved file name

Example

import Uniquer from 'uniquer';
// 315f5bdb...5894edd3.txt
const fileName = Uniquer.getFileName('.txt', 'Hello, world!');

Uniquer.getFilePath(directory, extension, data)

Resolve the unique file path, based on the file hash + extension.

directory String Path to the output directory
extension String File extension
data String | ArrayBufferView The file's data
returns String Resolved file path

Example

import Uniquer from 'uniquer';

// /home/user/output/315f5bdb...5894edd3.txt
const filePath = Uniquer.getFilePath('/home/user/output/', '.txt', 'Hello, world!');

Uniquer.getFileHash(data)

Get the SHA256 hash of the data, converted to hex format.

data String | ArrayBufferView The file's data
returns String SHA256 file hash

Example

import Uniquer from 'uniquer';

const hash = Uniquer.getFileHash('Hello, world!');

// 315f5bdb...5894edd3
console.log(hash);