Ideas for enhancement

[itinerant-fox]

1. Support for taking input from multiple input files.
   Wordlist can be spread across multiple files. Currently I am merging it and then passing to duplicut.
   Need to work something like : duplicut -p 1.txt 2.txt 3.txt 4.txt -o output.txt

2. Progress bar. Need not be accurate. Can be a guesstimate.

Thanks for the software 👍

[nil0x42]

Hi ! Thank you for your suggestions :)

---

1. Support for taking input from multiple input files.

I did not implement multiple input files because it will not be faster than doing:
cat 1.txt 2.txt 3.txt > all.txt && duplicut -i all.txt -o output.txt
I prefer focusing on features that would do things better than existing tools. But of course, if someone is willing to make a PR implementing it, i would be happy to merge it !

---

2. Progress bar.

All the needed source for progress bar is already implemented (status.c & uinput.c).
Current progress-tracking implementation is good enough, but the UX might be a little old-fashioned: Each time you press any key during execution, a line reports current progress info with ETA, to feel like john-the-ripper's progress tracking.

So implementing a progress-bar would be actually very easy, as it's only a matter of display, i'll add- it to my TODO for sure 👍

[sectroyer]

I have few dictionaries and I would like to remove duplicates across them. Therefore it would be good to have an option to remove words from a dict that are in some other dict. I want to keep those dicts separate but make sure that I am not testing same pws when using them in sequence( NOT always the case) :)

[yanncam]

Good morning all,

First of all, thank you for duplicut, the tool is particularly powerful!

However, I encountered the same problem than @sectroyer and @itinerant-fox: I needed to use duplicate in a unitary way on each of my wordlists, then to deduplicate the wordlists between them.

Duplicut is only designed to process a single file, so I designed a wrapper (in bash) that automates the process for N wordlists while relying on duplicut.

This wrapper generates a single temporary file concatenating all the wordlists with delimiters (need disk space), then after deduplication, recuts this single file to regenerate the initial deduplicated unitary wordlists, accompanied by some optimization statistics.

You will find the wrapper here : https://github.com/yanncam/multiduplicut

Hope it can help others!

Thanks again for this great tool :)!

[nil0x42]

the delimiters idea is intreresting, actually it's probably the easiest way to implement it inside of duplicut without needing to rewrite a large part of the codebase.
I'll consider implementing multi file when i have time, so no ETA from now (always busy)
Anyway, your script is very nice, and i think it will help many people