[BUG] App Crashes When Root Directories With Too Many Sub-Levels Are Loaded

[jobindjohn]

I am trying to open directory, but it shows "No directory selected" on the side tab.

OS: Window 10
Zettlr version: 1.4.1

[nathanlesage]

Did you try to open the "Documents" directory? If so, no wonder, because Zettlr won't allow to open either your complete Harddrive (directory: C:\) or your complete Documents folder (directory: C:\Users\<name>\Documents) due to them being too big to handle, as Zettlr reads in the full directory tree.

[jobindjohn]

Thanks for the reply. No, I am trying to open a folder on Dropbox. Some of them have sub-directories. But its not too many. I got this bug after moving to version 1.4.1

[nathanlesage]

Mh, that's not good. Are any errors thrown on the console?

[dambaras]

I'm having a related problem (OS X): In my Mac Finder, I've moved a small folder of notes from one larger folder to another folder. Zettlr does not recognize the shift, so I can't tell it to open the folder in its new location.

[dambaras]

Actually, that was an error on my end. never mind. :(

[jobindjohn]

I don't see anything. The directory doesn't show up in the directories list when I open and the files list shows 'No directory selected'. Is there any other way I can check for errors?

[johannhuang]

The same situation, version 1.4.1, on macOS 10.14, chosen to open a folder, but nothing happens!

[jobindjohn]

Reinstalled Zettlr. Issue still persists. Tried installing on a different Windows systems, and it seems to be working there.

Where can I find installation file for a previous version of Zettlr? Maybe I can use that till this is fixed. Thanks!

[jobindjohn]

The directories panel shows this when I moved the directories out of dropbox. Maybe it was open in Zettlr all the while but did not display?

[carlocastoldi]

This bug, along with the one that doesn't let you save the files, is the most annoying now.
I've found that it sometimes, indeterministically, open the folders and came upon this very same bug either on my laptop (Arch Linux), on my Desktop (Manjaro Linux) and on my friend's MacOS

[nathanlesage]

Hey,

sorry to hear that you experience something like that. Concerning the broken-link-directories: It's been implemented so that when you move root-directories, you can relocate them (especially useful for directories on removable media). IF you move these root directories @jobindj, then Zettlr obviously won't find them anymore.

I am still not quite sure what the underlying problem is. I don't experience this problem (just set up a new Windows 7 workstation and everything worked like a charm …) so it's really hard for me to understand where to look for the error.

[nathanlesage]

Just found this issue: #345

Does simply waiting for the directories to appear help …? If so, it's not a bug but a UX-related problem!

[nathanlesage]

I am trying to open directory, but it shows "No directory selected" on the side tab.

I just stumbled into that error myself :D I thought "Oh well, the VueComponent must have unloaded the whole directory tree, why did it do that?!" and spent 30 minutes investigating before I realised that this is the message that appears only in thin mode when there is no directory currently selected. I fixed that in the latest commit. Zettlr will now try to always both inform you when it is trying to open a new root directory and when it is done loading it as well as always trying to select a directory.

[carlocastoldi]

Zettlr will now try to always both inform you when it is trying to open a new root directory and when it is done loading it as well as always trying to select a directory.

I'm sorry to report that having slow directory load times isn't the case of the bug i've been experiencing
Video: https://gfycat.com/gratefulagreeableasiandamselfly

To what i can gather it actually opens the directory, but doesn't actually render it on the left side menu (either it's Thin, Expanded or Compacted mode).

When this bug occurs it even lose the ability to save an opened file and just create new files (named %Y%M%D%h%m%s.md) each saving the state of the file in the moment when it was saved.

[nathanlesage]

Ouh, I see what you mean. Are there any error messages either on the command line or in the Inspection Console of the Window? Sounds pretty much like a bug in the GUI (which would also explain why other functionalities are affected).

[carlocastoldi]

Are there any error messages either on the command line or in the Inspection Console of the Window?

Mmh nope, no luck :/
The only error i get in the terminal is [27331:1006/134112.098389:ERROR:buffer_manager.cc(488)] [.DisplayCompositor]GL ERROR :GL_INVALID_OPERATION : glBufferData: <- error from previous GL command

and in the Developers Tools Console i only get font errors:


Failed to load resource: net::ERR_FILE_NOT_FOUND Lato-Light.woff2:1
Failed to load resource: net::ERR_FILE_NOT_FOUND webhostinghub-glyphs.ttf:1
Failed to load resource: net::ERR_FILE_NOT_FOUND Lato-Heavy.woff2:1

[nathanlesage]

Damnit … nevertheless, after opening the directory, could you please check whether or not these two things happened:

1. Typing in the inspection console renderer._store._store.state gives you an object where the items-property contains the wanted directory
2. After quitting Zettlr, does the configuration file contain the requested directory in its "openPaths" (or similar) property?

[carlocastoldi]

Typing in the inspection console renderer._store._store.state gives you an object where the items-property contains the wanted directory

The item-property has length: 0. Apart from the length property, item has two more values: __ob__ and __proto__.
The returned object, though, has selectedDirectory and selectedFile properties with 2 IDs i cannot confirm to be related to the directory & file opened. I although noticed that changing file or directory always changes these IDs with the exception of when Zettlr saves the changes of a new file (it keeps the previous selectedFile ID of the presumably "parent" file).

After quitting Zettlr, does the configuration file contain the requested directory in its "openPaths" (or similar) property?

If you refer to selectedDirectory property, after restarting Zettlr it actually does keep the same ID!

[nathanlesage]

Okay this sounds super weird, as everything you described is just as the app should behave, but there is no reason for the store not to keep the opened directories and pretend there's nothing loaded, because the directory is certainly loaded. o.O

[carlocastoldi]

Oh yeah! The fact that after the update i get the notification " has been loaded." already made me think that it actually loads it without properly rendering it :/

[carlocastoldi]

Even weirder: it sometimes (after i recorded this video i can no longer see the directories i could open during the recording) shows some directories, but not others. I cannot understand based on which criteria though

https://gfycat.com/phonyafraidalligator
(i just noticed gfycat cut the extra seconds of the recording. I just showed that after restarting Zettlr it actually closed the opened directories)

EDIT: i actually discovered that once i delete ~/.config/zettlr it works flawlessly and can open the simpler directories (with few subdirectories) up until i try opening a more "complex" one. Once I try that it just can't no longer open neither the complex folders nor the simpler ones: I have to delete ~/.config/zettlr again!

EDIT2: I've tried to search for the actual limit of directory complexity Zettlr can handle. After a ton of time testing, I found that Zettlr cannot open folders deeper then 8! If I try to open directory with max depth >= 9 it will break and will require to delete the config folder (I imagine because every time you start up Zettlr it tries to open up the 8+ deep directory and fail, so you actually have to force Zettlr to close the "opened" root folder)

Opening folder8 causes Zettlr to break; folder7, on the other hand, is fine

[nathanlesage]

You'd love to see the face I made while watching your video. I never felt so clueless!

Nevertheless, the depth of directories is an interesting finding. But when you say you remove the full ./config/zettlr-directory, this also removes some other data. Could you please try to only remove the directories from the config.json file by setting the property openPaths to [] (so that it looks like: "openPaths": [])? If this also works then we definitely know that it's only pertaining to the config itself.

Then the next step would be to investigate if some other process, i.e. one of the libraries might produce an error. We had a lot of problems in the past with chokidar, so one possibility would be that Chokidar literally chokes when the directories are nested that deep on your operating system (some Linux systems have different approaches to how they actually handle these, so maybe there's some incompatibility with yours? Btw which OS do you use? Never saw that GUI before).

I definitely need to bump up logging in the application, because as it is now it more resembles a game of Pinata than bugfixing :D

[carlocastoldi]

Could you please try to only remove the directories from the config.json file by setting the property openPaths to []

Yup! That was definitely it!

Btw which OS do you use? Never saw that GUI before

I use Manjaro Linux with KDE Plasma (it's very configurable, so you might not recognise it :P)
But i can say that i have the same problem with Arch Linux (\w KDE aswell) on my laptop.
EDIT: oh right: my FS is standard ext4

[..] Chokidar [..]

About that i don't know... I'm not a NodeJS developer so all these libraries are a bit extraneous to me :/

I definitely need to bump up logging in the application, because as it is now it more resembles a game of Pinata than bugfixing :D

Hahahahaha on the plus side I'm liking it! It sure gives me an excuse to procrastinate my studying and feel helpful :P

[carlocastoldi]

Now, after realising the openPaths thingy, i tried again to look at the developer's tool console. This are the outputs i get:

[carlocastoldi]

Oh! I forgot to mention an important aspect: I tried Zettlr 1.3.0 and it worked fine both on my desktop and my laptop!

EDIT: this is what i get when i open the usual directory (it's on Dropbox) on Zettlr 1.3.0 on my laptop

Still on ext4 fs

[nathanlesage]

Mh, I just tried it and I actually could reproduce the issue with 8 nested folders on Windows. So something IS happening. I'll be trying this with the full fanfare of debug messages shortly ...

[nathanlesage]

Alright, can confirm that folder nesting of more than 8 breaks the visuals on every operating system. Nevertheless, I also can confirm that these folders are correctly loaded into the app, hence I suspect there's something going on during the IPC. The simplest explanation would simply be that JSON-encoding just gives up with that deepness (albeit 8 levels seem a little bit of a low ceiling)

[nathanlesage]

Alright, the "official" ceiling of JSON.stringify is not eight levels, the function just ran perfectly. Must be something in the IPC, definitely. Investigating …

[nathanlesage]

Alright, for now I'm out of ideas now. The complete paths WITH the respective folder have been passed to the internal IPC function of electron, but the folder is removed once it's in the renderer, so something weird is happening during the IPC call. It might have to do with the encoding, but I'm not sure.

[nathanlesage]

Alright, after crashing my config once again, I think I found the error:

According to some issues and pull requests on the Electron repository, this is actually a known problem with the way Electron handles messages.

• Here's how complex objects (such as nested folders within Zettlr) can crash the renderer process: Renderer subprocess crash on method invocation. electron/electron#20334
• Here's the bad pull request from April that may have introduced that bug: refactor: use mojo for electron internal IPC electron/electron#17406
• Here's a (hopeful) candidate (potentially …?) for alleviating this error: refactor: use v8 serialization for ipc electron/electron#20214

Nevertheless, until this is done, we may have to refrain from complex nested directory structures :(

[carlocastoldi]

Well at least we managed to find out what the problem is (i guess that also explains why Zettlr 1.3.0 doesn't have this bug: because it doesn't use electron6). Now let's wait for a fix upstream ^^

Do you think you could somehow inform the users not to open a folder nested more than 8 levels? I really think this is a problem many people are experiencing but aren't vocal about it. I suggested to install Zettlr to 3 friends of mine before the start of the semester and all of them experienced this bug. Only one didn't give up and kept using this awesome program :/
I'm truly afraid this bug could push away a considerable amount of users since it could give a bad first impression even before diving into the real potential of the software

That said: many many thanks for the time spent investigating this awkward bug, Nathan! I really love the dedication and ideas YOU put into this project

[nathanlesage]

Well at least we managed to find out what the problem is (i guess that also explains why Zettlr 1.3.0 doesn't have this bug: because it doesn't use electron6). Now let's wait for a fix upstream ^^

Precisely! Let's also hope that Electron 6 doesn't break on macOS Catalina, I got the first messages from concerned users, and I have my doubts, given my experience with Electron …

Do you think you could somehow inform the users not to open a folder nested more than 8 levels?

I did so already! Not all of them will read it, though, but also I don't think that this many people will experience the bug. After all, the documentation and everything tries to nudge you into creating a dedicated folder, so it's not that probable that a lot of people have such deeply nested folders — I myself have an incredible amount of folders loaded into Zettlr, but none of them has eight sub-levels of folders :D

Therefore, I don't think it'll push a lot of users from the app. I think more will stay as soon as this bug is fixed, but if you see any surplus value in using the app I think you'll already be reading GitHub and Twitter so you'll be up to date!

That said: many many thanks for the time spent investigating this awkward bug, Nathan! I really love the dedication and ideas YOU put into this project

Nah, just I like doing it :) Besides, investigating bugs gives you unprecedented insight into any program or programming language! It's really educative

[nathanlesage]

The FSAL refactor was certainly a good idea.