Documents with German Umlaute (ä,ö,ü) do not show previews and are not downloadable

[rick0ff]

Hi community,

I've been running paperless-ngx for a couple of months and I love it so far. But I've come across a weird issue that is likely related to German Umlaute (ä, ö, ü), since any documents

• that have filenames or correspondents with them
• do not show any previews anymore (just shows grey instead of the preview) and
• are not downloadable ->

GET /api/documents/1824/download/?original=true
HTTP 404 Not Found
Allow: GET, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

{
"detail": "Nicht gefunden."
}

Logs of the webserver show:

[2024-04-26 17:30:19,328] [WARNING] [django.request] Not Found: /api/documents/1824/preview/
[2024-04-26 17:30:22,550] [WARNING] [django.request] Not Found: /api/documents/1824/download/

The sanity checker picks up on these files as well.

Samples of the names/correspondents, i have about 250 of them:

• ./2014/0_Vermieter Wohnen/Mietzinsreduktion wegen Lärm.pdf
• ./2014/Allianz/Allianz Police 2 gekündigt 2019.pdf
• ./2015/0_Behörden Sonstige/TARI 2015.pdf
• ./2018/0_Ärzte/18-12-Rechnungen AUS Spital.pdf

Any ideas on how to solve this exactly?
Tried to export the files without thumbnails and reimport it, but all these files seem to stop the exporter with every file with Umlaute anyways.

Thank you for any help! 😃

I am running paperless-ngx on docker, v2.7.2 according to "system status", but 2.6.2 at the bottom of the navigation bar 😕 . My docker-compose.yml:

version: "3.4"
services:
broker:
image: docker.io/library/redis:7
restart: unless-stopped
volumes:
- redisdata:/data

db:
image: docker.io/library/postgres:15
restart: unless-stopped
volumes:
- ./data/db:/var/lib/postgresql/data
environment:
POSTGRES_DB: xyz
POSTGRES_USER: xyz
POSTGRES_PASSWORD: xyz

webserver:
image: ghcr.io/paperless-ngx/paperless-ngx:latest
restart: unless-stopped
depends_on:
- db
- broker
- gotenberg
- tika
ports:
- "6000:8000"
healthcheck:
test: ["CMD", "curl", "-fs", "-S", "--max-time", "2", "http://localhost:8000"]
interval: 30s
timeout: 10s
retries: 5
volumes:
- ./data/appdata:/usr/src/paperless/data
- ./data/media:/usr/src/paperless/media
- ./export:/usr/src/paperless/export
- ./consume:/usr/src/paperless/consume
env_file: docker-compose.env
environment:
PAPERLESS_REDIS: redis://broker:6379
PAPERLESS_DBHOST: db
PAPERLESS_TIKA_ENABLED: 1
PAPERLESS_TIKA_GOTENBERG_ENDPOINT: http://gotenberg:3000
PAPERLESS_TIKA_ENDPOINT: http://tika:9998

gotenberg:
image: docker.io/gotenberg/gotenberg:7.8
restart: unless-stopped

# The gotenberg chromium route is used to convert .eml files. We do not
# want to allow external content like tracking pixels or even javascript.
command:
  - "gotenberg"
  - "--chromium-disable-javascript=true"
  - "--chromium-allow-list=file:///tmp/.*"

tika:
image: ghcr.io/paperless-ngx/tika:latest
restart: unless-stopped

volumes:
redisdata:

docker-compose.env:

PAPERLESS_URL=https://paperless.example.com
PAPERLESS_TIME_ZONE=Europe/Berlin
PAPERLESS_OCR_LANGUAGE=deu+eng
PAPERLESS_SECRET_KEY=xyz
PAPERLESS_FILENAME_FORMAT={created_year}/{correspondent}/{title}

[shamoon]

Other issues have been reported that amount to differences in how the filesystem encodes the diacritics, basically paperless uses UTF-8 and it seems some filesystems dont.

[rick0ff]

Thank you for your reply. So, how do i resolve this?

[alexmaret]

So, how do i resolve this?

Either your database or your filesystem uses a non-UTF-8 character set. You have to understand, which one it is.

Most of the time it is the filesystem. You can check this using iconv, which is a tool to convert character sets. It would be best to execute it from within the Docker container, but you can also start with the host.

find /usr/src/paperless/media | iconv -c -f UTF-8

The command above will list every file in your paperless media folder and remove any character that is not UTF-8 from the output. If you are using a non-UTF-8 character set for your files, then all your umlauts should be missing from the output. If all the umlauts are displayed correctly, then your files should be good.

Next step would be to check the database.

[rick0ff]

Hope you had a nice trip then. ;)

This time you might need to go a bit slower for me :D Sorry!

• Where exactly would i run the script?
• In which docker container?
• NFC? NFD?

Yes, my main computer is a Macbook, so also where all the uploading via web UI takes place. Do you have a Mac at hand for reproducing?

[alexmaret]

NFC and NFD describes Unicode normalization. NFC is the composed normalization where an 'Ö' is a single character, wheras NFD is the decomposed form where 'Ö' is an 'O' plus a diacritic (the two dots). This is the core of your problem. Your files are in NFD form whereas the database is in NFC form. https://en.wikipedia.org/wiki/Unicode_equivalence#Normalization

The script above (updated!), will rename a file to NFC form because that is what your database has stored. So you can just copy it into a file tonfc.sh and make it executable with chmod 755 tonfc.sh. The script requires the Unicode conversion tool that you have to install with apt install icu-devtools first!

The file you want to rename should be in the current directory or otherwise the script might fail if a directory also contains non-nfc characters. So you can do something like this:

cd media/documents/originals/path/to/broken/file/
tonfc.sh bröken-file.pdf
cd media/documents/archive/path/to/broken/file/
tonfc.sh bröken-file.pdf

This should allow Paperless to find the file again, as it is looking for the name in the database, which is in NFC form. If the directory also contains non-nfc characters, you might also have to rename the directories.

Now let's talk about why this happens. @shamoon is not quite right, that this is a filesystem problem, at least not in your case. You are using a Unix filesystem and these don't care about character sets at all. For a Unix filesystems, a filename is simply a byte array, that can contain anything, except for some reserved characters like '/'.

I have a theory for your case, which you can maybe verify somehow: You are using MacOS, which is known to use NFD for files. So your files were most likely in NFD already when you uploaded them. That should not be a problem for Paperless and by default it does not seem to do a NFC conversion. So where does the NFC value come from in your database? I think this is a problem with Safari, which seems to always convert values to NFC when submitting a form: https://stackoverflow.com/questions/63791942/prevent-safari-from-normalizing-unicode-when-sending-post-data

So this is what might have happened:

1. You uploaded a file in NFD
2. You edited document attributes in Paperless with Safari.
3. When submitting the form, Safari converted NFD to NFC which was then stored in the database. For some reason, the operation you did, did not trigger a file rename.

Does this ring a bell somehow? Do you even use Safari?

[shamoon]

Pardon the interruption, but just wanted to thank you for taking all the time to keep helping out with this, Im just sorta following along here 👍

[rick0ff]

I can only second shamoon: Thank you so much for your patience and help!

This time I am not sure if I was able to follow exactly. So here's what I did:

1. installed icu-devtools
2. created tonfc.sh (with the updated code in your previous post) in media/documents/archive/2013/0_ÖV Öffentlicher Verkehr and media/documents/originals/2013/0_ÖV Öffentlicher Verkehr
3. each modified the permissions with 755
4. ran it as "./tonfc.sh ÖV\ Abo.pdf ", script did not output any errors or success notifications.

--> In paperless i still do not get any previews, even after closing the doc and reopening it.
Did I do everything correctly?

No, i never use Safari. Just Chrome. So this cannot be the source for the problem. Any other ideas?

[alexmaret]

Well, the script just renames a file to NFC. If Paperless still does not show anything, then there still has to be a difference between the database filename and the filename of the filesystem. So we need to compare the bytes again and find where they differ. If you hexdump the filename of the filesystem, they should be equal to the database now, if the script worked correctly.

You can also do it the other way round and just update the database filename with a copy & pasted path of the filesystem. In many cases this might be the better option, especially if you have to rename whole directories. If you rename a directory to NFC then all files which are in that directory can become unavailable in Paperless if they still have the NFD path in the database. If you want to revert this, you can replace any-nfc with any-nfd in the script.

When it comes to the source of the problem, then this is a tricky thing. I tested this on my system and uploaded a file in NFD form. Neither Paperless, nor the filesystem had an issue with it. I can see the file just fine and I can also edit its attributes. It stayed in NFD form the whole time.

If we look at this analytically, then we have 4 possibilities only:

1. Something external modifies your database entries.
   In your case this should be unlikely, as you are using a vanilla setup. Unless you have something that connects to the database and reads/writes data outside of Paperless.

2. Something external modifies your filesystem.
   This would usually be the most likely scenario, especially when people export the media directory as a network share. As you are not doing this, it would need to be some Linux service. But which service should that be? A service that somehow renames files to NFD? Why NFD? That makes no sense, outside of Mac pretty much everyone uses NFC. So why would that service rename to NFD?

3. Paperless does the NFC normalization.
   As I have tested, Paperless does not seem to do this by default. I also grepped the source code and found only one instance of Unicode normalization, which should be completely unrelated. So if it really does this, then this will be hard to find.

4. A client normalizes the names.
   That could be anything from a browser to an app which uses the Paperless API. In your case I would assume this is the most likely scenario as you are using a vanilla setup. The problem with this theory however is, that a change in document attributes should usually trigger a rename in Paperless. So while this seems to be the most likely scenario, it also requires a bug in Paperless, where somehow the database is updated, but the file is not renamed accordingly. Maybe the file rename is triggered but simply fails for some reason.

I think we looked at all angles and at this point you are the only person who can move this forward, by closely observing when documents suddenly become unavailable. If no one touches these documents, they should keep their names forever. So something must have modified them.