"Bullet-proof" tool to upload documents

[felixebert]

As an aleph user which would like to upload large amounts of files to aleph, I struggle to get them into aleph - e.g. in case of connection errors, after the machine got suspended, after I accidentally closed the browser and so on - and quickly give up.

It would be cool to get the upload process less error-prone - e.g. by providing an electron app to upload documents, a service worker job running in the background, a browser extension, ...

[pudo]

An electron app seems like a massive maintenance burden to incur for this, I'd rather look a bit at multi-part file upload managers for React. To be honest we do all of our bulk uploads using alephclient, so it's not really an issue, the user-facing upload is only ever used for a dozen documents or so.

[pudo]

Should we declare this done after your recent improvements to the upload dialog?

[PlainSite]

Even though there are obvious security implications involved, it would be super helpful to be able to see a tree of the local filesystem where large numbers of documents are stored as well as some kind of icon-based representation of which files have already been ingested and which are outstanding. For us this changes on a nightly basis as more files come in from the court system. Along those lines, it's unclear if "Re-Process" will try to re-ingest millions of files that are already processed and indexed, or if it will just scan for files that have not yet been ingested in some folder.

[PlainSite]

One additional thing that would be really nice is a pause/play button given how resource-intensive ingestion is. On some production servers, this may cause problems at certain times of the day. It's of course possible to use kill -STOP and kill -CONT to effect the same outcome, but not quite as elegant.