No recorder/browser window drawn on MacOS BigSur 11.1 Intel

[Moominologist]

Versions affected

Webrecorder 2.0.3.131
Webrecorder Player 1.8.0

On BigSur 11.1 (Intel) both Webrecorder and Webrecorder Player (1.8.0) will open but not display a capture/playback window. Also, when opening an existing warc no window is created. The app itself doesn't freeze, menues are accessible. Deleting the stored configuration in //Application Support/Webrecorder doesn't lead to the window being redrawn.
Unfortunately I couldn't find info on how to pass a debug argument to the app - if anyone would kindly direct me in the right direction I'll gladly post the output. (I have tried > open -F -a Webrecorder , but I'm not observing any different behaviour).

[Moominologist]

Let med add some further detail. When I execute the Webrecorder binary directly I get

 /Applications/Webrecorder.app/Contents/MacOS/Webrecorder ; exit;
Dat Share api server listening on
http://localhost:56631
└── /
    ├── s
    │   ├── wagger (GET)
    │   │   └── / (GET)
    │   │       ├── json (GET)
    │   │       ├── yaml (GET)
    │   │       ├── static/
    │   │       │   └── * (GET)
    │   │       └── * (GET)
    │   ├── hare (POST)
    │   └── ync (POST)
    ├── init (POST)
    ├── unshare (POST)
    └── num
        ├── Sharing (GET)
        └── Dats (GET)

--no-browser --loglevel info -d /Users/<my_username>/Documents/Webrecorder-Data -u <my_username> --port 0 --behaviors-tarfile /Applications/Webrecorder.app/Contents/Resources/python-binaries/behaviors.tar.gz --dat-share-port 56631
Waiting...
Waiting...
Waiting...
Waiting...
Waiting...
Waiting...
Waiting...
Waiting...
Waiting...
Waiting...
Set pid: 68593
User <my_username> exists
DEFAULT_USER=<my_username>

WARCSERVER_HOST=http://localhost:56658

RECORD_HOST=http://localhost:56659

[68590] Failed to execute script webrecorder_full

/var/folders/v4/np4zkwmj74q_n5ynhbdnhpg80000gn/T/_MEIB8cEq9/webrecorder/load/wamloader.py:41: YAMLLoadWarning: calling yaml.load_all() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
/var/folders/v4/np4zkwmj74q_n5ynhbdnhpg80000gn/T/_MEIB8cEq9/webrecorder/load/wamloader.py:41: YAMLLoadWarning: calling yaml.load_all() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
Traceback (most recent call last):
  File "webrecorder/standalone/webrecorder_full.py", line 132, in <module>
  File "webrecorder/standalone/standalone.py", line 119, in main
  File "webrecorder/standalone/webrecorder_full.py", line 36, in __init__
  File "webrecorder/standalone/standalone.py", line 60, in __init__
  File "webrecorder/fullstackrunner.py", line 43, in __init__
  File "webrecorder/fullstackrunner.py", line 78, in init_server
  File "webrecorder/fullstackrunner.py", line 37, in app
  File "/Users/travis/.pyenv/versions/3.6.10/lib/python3.6/site-packages/PyInstaller/loader/pyimod03_importers.py", line 631, in exec_module
  File "webrecorder/maincontroller.py", line 22, in <module>
  File "/Users/travis/.pyenv/versions/3.6.10/lib/python3.6/site-packages/PyInstaller/loader/pyimod03_importers.py", line 631, in exec_module
  File "webrecorder/contentcontroller.py", line 14, in <module>
  File "/Users/travis/.pyenv/versions/3.6.10/lib/python3.6/site-packages/PyInstaller/loader/pyimod03_importers.py", line 631, in exec_module
  File "pywb/apps/rewriterapp.py", line 4, in <module>
  File "/Users/travis/.pyenv/versions/3.6.10/lib/python3.6/site-packages/PyInstaller/loader/pyimod03_importers.py", line 631, in exec_module
  File "fakeredis.py", line 105, in <module>
ImportError: fakeredis: unable to find libc or equivalent

Total Conn: 2
Total Unique Procs: 1
Last WR Process, Shutting Down Redis
Terminating Process

Seems, as if fakeredis can't find the path to libc.

[laurelin88]

Just wanted to bump this - same issue and same error on BigSur for me as well.

[mateuszr]

I also confirm, on Mac OS, Big Sur, 11.1 the app launches but doesn't show the main window.

[thewildmage]

I'm also finding that the app launches, and I'm able to access window menus, but no actual window is shown. This is on MacOS Big Sur 11.1, but on an Apple Silicon/M1 MacBook. So maybe this has nothing to do with Intel or ARM, and is instead caused by Big Sur?

I haven't yet been able to build locally to see if I get the same error message as mentioned above. I can say that version 2.0.3 was working just fine for me on my old MacBook though, which was still on Catalina.

[mateuszr]

After googling around, it looks that BigSur changed the behaviour of dynamic linking.

New in macOS Big Sur 11 beta, the system ships with a built-in dynamic linker cache of all system-provided libraries. As part of this change, copies of dynamic libraries are no longer present on the filesystem. Code that attempts to check for dynamic library presence by looking for a file at a path or enumerating a directory will fail. Instead, check for library presence by attempting to dlopen() the path, which will correctly check for the library in the cache. (62986286)

Some similar issues in other projects:
urllib3/urllib3#1905
vispy/vispy#1885
https://stackoverflow.com/questions/63475461/unable-to-import-opengl-gl-in-python-on-macos

[JCBerger]

After googling around, it looks that BigSur changed the behaviour of dynamic linking.

New in macOS Big Sur 11 beta, the system ships with a built-in dynamic linker cache of all system-provided libraries. As part of this change, copies of dynamic libraries are no longer present on the filesystem. Code that attempts to check for dynamic library presence by looking for a file at a path or enumerating a directory will fail. Instead, check for library presence by attempting to dlopen() the path, which will correctly check for the library in the cache. (62986286)

Some similar issues in other projects:
urllib3/urllib3#1905
vispy/vispy#1885
https://stackoverflow.com/questions/63475461/unable-to-import-opengl-gl-in-python-on-macos

So are you able to tell if any of the most recent pull requests address this issue?

[ikreymer]

Hi, sorry for not updating this earlier.

Webrecorder has released a Chrome browser extension available at: ArchiveWeb.page which includes much (but not yet all) of the functionality of the desktop app. A new desktop app based on the ArchiveWeb.page architecture is also in development. (The repo is available here: https://github.com/webrecorder/archiveweb.page)

The new app will be a full replacement for Webrecorder Desktop, and plan is to support importing any existing Webrecorder Desktop collections as well into the ArchiveWeb.page App.

Both the extension (available now) and the new upcoming app should work on all platforms, including BigSur, as they rely on a recent version of Chrome, and no other external dependencies. (This app requires running a python binary and Redis which is likely a source of these issues).

[thewildmage]

That's really good to know. Thanks for the info!

[mateuszr]

The Chrome extension and the electron app from the repo https://github.com/webrecorder/archiveweb.page works well!
If the "webrecorder-desktop" project is not going to be further developed, I suggest updating the Readme.md and point to the other project.

[ikreymer]

@mateuszr Yes, that has been the plan but was waiting to have an initial build of the new app ready! Sorry for the delay!

There is now finally an initial release of ArchiveWeb.page beta available at: https://github.com/webrecorder/archiveweb.page/releases

I've added a comment to the README and will mark this repo as archived.