Remix Support? #1093

jaidetree · 2023-03-03T06:55:42Z

jaidetree
Mar 3, 2023

Currently integrating shadow-cljs builds into an existing TS Remix app at work.

My strat is to target esm, then import it into TS route files as the single component rendered on the page and\or import the loader function.

Just got it building and rendering a reactified reagent view so happy about that. However, one tragic casualty was the repl support. From what I understand only the browser runtime is supported, but unfortunately that caused the remix bundling to break server side due to the missing WebSocket APIs. Remix bundles each route and it's dep twice: Once for server and once for client.

Trying to think through if there's a way to make it work.

One solution off the top of my head would be to create separate bundles via shadow-cljs edn builds directly into the remix build directory. Might mess with the static analysis tooling but should still run.

Any better ideas out there?

thheller · 2023-03-03T07:12:52Z

thheller
Mar 3, 2023
Maintainer

Sorry, I know nothing about remix, and don't have time to investigate.

:esm builds have a :runtime :custom option which also takes a :devtools {:client-ns your.custom.client-ns} to control which "REPL client" ns is loaded. For :runtime :browser this would be this one.

It is just a regular namespace that gets "injected" into the build and establishes the websocket connect when loaded

shadow-cljs/src/main/shadow/cljs/devtools/client/browser.cljs

Line 337 in ac8b33e

(cljs-shared/init-runtime! client-info ws/start ws/send ws/stop))

So, in theory you could try to strip this namespace down to the bare essentials in a way that works in both remix client and server? The node version is substantially smaller since it doesn't do all the HUD stuff? None of this is documented, but maybe you can figure it out. It basically just wants to connect to shadow.remote server and then do stuff based on messages it gets. Maybe there is even a npm package that you can use instead of straight up WebSocket? The websocket stuff is already fairly minimal and should be changable easily. You could even take only that file and put it into your local project with the adjustments you need.

A demo repo would help, even if it doesn't work. I can at least provide tips from the build side.

1 reply

jaidetree Mar 3, 2023
Author

Thanks! I'll setup a demo repo and dig into that.

jaidetree · 2023-03-04T06:01:16Z

jaidetree
Mar 4, 2023
Author

Got the remix minimal setup here:

https://github.com/eccentric-j/cljs-shadow-remix-demo

npm install
npm run watch:cljs in one terminal
npm run dev in another terminal

Bad news, I'm getting a whole new error.

On the project at work, the page renders the reagent component that prints "It works!" just fine. Except when I save changes and an error is thrown from goog about a module already being provided.
In this demo project, I just get an error:

ReferenceError: SHADOW_ENV is not defined
    at Object.<anonymous> (/Users/j/projects/cljs-remix/app/cljs/demo.js:4:1)
    at Module._compile (node:internal/modules/cjs/loader:1275:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1329:10)
    at Module.load (node:internal/modules/cjs/loader:1133:32)
    at Function.Module._load (node:internal/modules/cjs/loader:972:12)
    at Module.require (node:internal/modules/cjs/loader:1157:19)
    at require (node:internal/modules/helpers:119:18)
    at /Users/j/projects/cljs-remix/server.js:35:18
    at Layer.handle [as handle_request] (/Users/j/projects/cljs-remix/node_modules/express/lib/router/layer.js:95:5)
    at next (/Users/j/projects/cljs-remix/node_modules/express/lib/router/route.js:144:13)

Not sure why. I'm going to keep poking at it this weekend but if you by chance have any insight here, it would be vastly appreciated 😅

8 replies

jaidetree Mar 4, 2023
Author

Confirmed my hypothesis: Unfortunately switching to browser target does not work atm either due to same error:

Here's the js remix bundles for the server:

// app/cljs/demo.ts
SHADOW_ENV.setLoaded("shadow.module.demo.prepend.js");
SHADOW_ENV.setLoaded("goog.debug.error.js");
SHADOW_ENV.setLoaded("goog.dom.nodetype.js");
SHADOW_ENV.setLoaded("goog.asserts.asserts.js");
SHADOW_ENV.setLoaded("goog.reflect.reflect.js");
SHADOW_ENV.setLoaded("goog.math.long.js");
SHADOW_ENV.setLoaded("goog.math.integer.js");
SHADOW_ENV.setLoaded("goog.dom.asserts.js");
SHADOW_ENV.setLoaded("goog.functions.functions.js");
SHADOW_ENV.setLoaded("goog.string.typedstring.js");
SHADOW_ENV.setLoaded("goog.string.const.js");
SHADOW_ENV.setLoaded("goog.i18n.bidi.js");
SHADOW_ENV.setLoaded("goog.html.trustedtypes.js");
SHADOW_ENV.setLoaded("goog.html.safescript.js");
SHADOW_ENV.setLoaded("goog.fs.url.js");
SHADOW_ENV.setLoaded("goog.fs.blob.js");
SHADOW_ENV.setLoaded("goog.html.trustedresourceurl.js");
SHADOW_ENV.setLoaded("goog.string.internal.js");
SHADOW_ENV.setLoaded("goog.html.safeurl.js");
SHADOW_ENV.setLoaded("goog.html.safestyle.js");
SHADOW_ENV.setLoaded("goog.object.object.js");
SHADOW_ENV.setLoaded("goog.html.safestylesheet.js");
SHADOW_ENV.setLoaded("goog.dom.htmlelement.js");
SHADOW_ENV.setLoaded("goog.dom.tagname.js");
SHADOW_ENV.setLoaded("goog.array.array.js");
SHADOW_ENV.setLoaded("goog.labs.useragent.useragent.js");
SHADOW_ENV.setLoaded("goog.labs.useragent.util.js");
SHADOW_ENV.setLoaded("goog.labs.useragent.browser.js");
SHADOW_ENV.setLoaded("goog.dom.tags.js");
SHADOW_ENV.setLoaded("goog.html.safehtml.js");
SHADOW_ENV.setLoaded("goog.html.uncheckedconversions.js");
SHADOW_ENV.setLoaded("goog.dom.safe.js");
SHADOW_ENV.setLoaded("goog.string.string.js");
SHADOW_ENV.setLoaded("goog.collections.maps.js");
SHADOW_ENV.setLoaded("goog.structs.structs.js");
SHADOW_ENV.setLoaded("goog.uri.utils.js");
SHADOW_ENV.setLoaded("goog.uri.uri.js");
SHADOW_ENV.setLoaded("goog.string.stringbuffer.js");
SHADOW_ENV.setLoaded("cljs.core.js");
SHADOW_ENV.setLoaded("clojure.string.js");
SHADOW_ENV.setLoaded("clojure.set.js");
SHADOW_ENV.setLoaded("shadow.cljs.devtools.client.env.js");
SHADOW_ENV.setLoaded("shadow.esm.esm_import$react.js");
SHADOW_ENV.setLoaded("clojure.walk.js");
SHADOW_ENV.setLoaded("reagent.debug.js");
SHADOW_ENV.setLoaded("reagent.impl.util.js");
SHADOW_ENV.setLoaded("reagent.impl.batching.js");
SHADOW_ENV.setLoaded("reagent.impl.protocols.js");
SHADOW_ENV.setLoaded("reagent.ratom.js");
SHADOW_ENV.setLoaded("reagent.impl.component.js");
SHADOW_ENV.setLoaded("reagent.impl.input.js");
SHADOW_ENV.setLoaded("reagent.impl.template.js");
SHADOW_ENV.setLoaded("reagent.core.js");
SHADOW_ENV.setLoaded("remix.demo.page.js");
SHADOW_ENV.setLoaded("shadow.module.demo.append.js");
var DemoPage = remix.demo.page.demo_page;
shadow.cljs.devtools.client.env.module_loaded("demo");

All the actual imports are missing that should come before their respective SHADOW_ENV.setLoaded line.

thheller Mar 4, 2023
Maintainer

I looked into this for a bit, and I don't see a quick way of getting this to work. There seem to be many assumptions baked in on the remix side that I can't easily decipher. I don't understand how its live-reload works and how to get rid of it. Just removing the import doesn't seem to be enough. I also can't find a way to tell remix to re-render the client side, so even with working hot-reload in the client side from shadow-cljs it won't do anything. Is there a "render" function that would be callable from a ^:dev/after-load hook?

thheller Mar 4, 2023
Maintainer

SHADOW_ENV supposed to be defined in <output-dir>/cljs_env.js, and in the case of :esm it is assigned to globalThis. So, it should be available as a global from everywhere. Remix however does not seem to honor this. At least I can't find where it puts that one.

jaidetree Mar 4, 2023
Author

Remix produces a server build in build/index.js and browser build in public/build/ with a dynamic name.

Even with a minimal bit of code like the following:

// app/lib/sideEffect.js
globalThis.SIDE_EFFECT = "I RAN"
console.log("HEY THERE");

In app/routes/index.ts:

import '~/lib/sideEffect.js'

export default function DemoPage () {
  return (<div>{globalThis.SIDE_EFFECT}</div>)
}

The sideEffect import is omitted from the bundle. Again, this is a very unwelcome surprise as in my repo at work, the cljs-runtime/cljs_env.js and other imports are included in the bundle correctly.

jaidetree Mar 6, 2023
Author

Good news! After 3 days of debugging, searching, and asking around I found the issue:

The create-remix CLI sets sideEffects: false on the root package.json which causes esbuild to omit the cljs source from the bundle.

In my example repo, I set sideEffects: true.

Now I get a cljs specific error on reloads (after making changes) about modules already being provided.

I'll poke around on my own too but if you try out that demo you can reproduce by changing the cljs view.

jaidetree · 2023-03-06T23:06:00Z

jaidetree
Mar 6, 2023
Author

Traced down the original error more:

Error: Namespace "cljs.core" already declared.
    at Object.goog.provide (/Users/j/projects/cljs-shadow-remix-demo/app/cljs/cljs-runtime/cljs_env.js:62:13)
    at app/cljs/cljs-runtime/cljs.core.js (/Users/j/projects/cljs-shadow-remix-demo/build/index.js:922:10)
    at /Users/j/projects/cljs-shadow-remix-demo/build/index.js:7:50
    at Object.<anonymous> (/Users/j/projects/cljs-shadow-remix-demo/app/cljs/demo.js:79:8)
    at Module._compile (node:internal/modules/cjs/loader:1120:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1174:10)
    at Module.load (node:internal/modules/cjs/loader:998:32)
    at Function.Module._load (node:internal/modules/cjs/loader:839:12)
    at Module.require (node:internal/modules/cjs/loader:1022:19)
    at require (node:internal/modules/cjs/helpers:102:18)

This is due to goog checking to see if a module was already provided, which is established in the compiled cljs_env.js

goog.provide = function(name) {
  if (goog.isInModuleLoader_()) {
    throw new Error("goog.provide cannot be used within a module.");
  }
  if (!COMPILED) {
    if (goog.isProvided_(name)) {
      throw new Error('Namespace "' + name + '" already declared.');
    }
  }
  goog.constructNamespace_(name);
};

It's caused by remix's auto-reloaded solution for integrating with express apps:

app.all(
	"*",
	process.env.NODE_ENV === "development"
		? (req, res, next) => {
				purgeRequireCache()

				return createRequestHandler({
					build: require(BUILD_DIR),
					mode: process.env.NODE_ENV,
				})(req, res, next)
		  }
		: createRequestHandler({
				build: require(BUILD_DIR),
				mode: process.env.NODE_ENV,
		  }),
)

function purgeRequireCache() {
	// purge require cache on requests for "server side HMR" this won't let
	// you have in-memory objects between requests in development,
	// alternatively you can set up nodemon/pm2-dev to restart the server on
	// file changes, but then you'll have to reconnect to databases/etc on each
	// change. We prefer the DX of this, so we've included it for you by default
	for (const key in require.cache) {
		if (key.startsWith(BUILD_DIR)) {
			delete require.cache[key]
		}
	}
}

I've tried adding the following to purgeRequireCache:

function purgeRequireCache() {
	if (globalThis?.goog) {
		// console.log(globalThis.goog.loadedModules_)
		// console.log(globalThis.goog.implicitNamespaces_)
		// console.log(goog.getObjectByName("cljs"))
		// globalThis.goog.loadedModules_ = {}
		// globalThis.goog.implicitNamespaces_ = {}
		console.log(
			globalThis.goog.isProvided_,
			globalThis.goog.isProvided_("cljs.core"),
		)
		globalThis.goog.isProvided_ = () => false
		console.log(
			globalThis.goog.isProvided_,
			globalThis.goog.isProvided_("cljs.core"),
		)
	}
	// purge require cache on requests for "server side HMR" this won't let
	// you have in-memory objects between requests in development,
	// alternatively you can set up nodemon/pm2-dev to restart the server on
	// file changes, but then you'll have to reconnect to databases/etc on each
	// change. We prefer the DX of this, so we've included it for you by default
	for (const key in require.cache) {
		if (key.startsWith(BUILD_DIR)) {
			delete require.cache[key]
		}
	}
}

Unfortunately though, that solution does not work. Going to keep at it, but if any solutions come to mind I'm very interested!

8 replies

jaidetree Mar 7, 2023
Author

Given the friction thus far, I’m trying to take this in baby-steps. Reloading in this case is server-side during development. Meaning I can change a remix or cljs component, refresh the browser, and see updates from the express server response. Not quite as nice as the repl, but sufficent for developing server-routes.

That said, replacing goog APIs does sound like a better solution! I explored that, but found a problem I didn’t have an answer for yet:

The order goes like the following:

Request hits express
Express purges the require module cache
It loads the /build artifact (bundled server-side code)
Continues the request through the remix express request handler

I had tried replacing good.provide and good.isProvided_ between steps 2 and 3. That ended up not working because the goog library uncoditionally sets those functions against the goog object, so on refresh it would reset my overrides to their original implementation and throw that “Namespace x already declared”.

Is there an entry point that would let me modify the goog object after it loads, but before providing any other cljs module or resource via goog.provide?

thheller Mar 7, 2023
Maintainer

Best place seems somewhere arround here, if I understand the purpose of that file currently (i.e. entry point for the server code).

https://github.com/eccentric-j/cljs-shadow-remix-demo/blob/cce80ca37c6097777004abb511bde7cd553b44ce/app/entry.server.tsx#L7

It could just import "~/cljs/cljs-runtime/cljs_env.js"; and after doing so the goog global should exist, so goog.provide = goog.constructNamespace_; should work?

jaidetree Mar 7, 2023
Author

I think that could work, or something similar in server.js as part of the purgeRequireCache function. Thanks!

For reference, the server reloading is handled at:
https://github.com/eccentric-j/cljs-shadow-remix-demo/blob/cce80ca37c6097777004abb511bde7cd553b44ce/server.js#L50

and the server bundle is loaded\used at:
https://github.com/eccentric-j/cljs-shadow-remix-demo/blob/cce80ca37c6097777004abb511bde7cd553b44ce/server.js#L34

thheller Mar 7, 2023
Maintainer

I think the default purgeRequireCache function should be enough. As long as the globals :npm-module creates are preseved everything should be fine. Can't do the hacking in server.js directly AFAICT, needs to be part of something in the build itself.

jaidetree Mar 9, 2023
Author

That's true, trying to override goog from server.js did not work, but was able to do it in entry.server.ts and entry.client.ts using the approach you outlined.

// src/app/lib/setupCljs.ts
import '~/cljs/cljs-runtime/cljs_env'

// Borrowed from https://github.com/thheller/shadow-cljs/blob/af1a77099e69b09c3799a8b64e169cd24d379a72/src/main/shadow/cljs/devtools/client/env.cljs#L152-L161

// Patch away the already declared exception and checks
// otherwise hot-reload may fail
goog.provide = goog.constructNamespace_
goog.require = goog.module.get

Then in entry.server.ts and entry.client.ts:

import '~/lib/setupCljs'

I'll update the demo repo to use that too if anyone ever wants to give it a shot. Going to make some progress as-is towards product goals then can circle back to this discussion for improving the live-repl support.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remix Support? #1093

{{title}}

Replies: 3 comments 17 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Remix Support? #1093

jaidetree Mar 3, 2023

Replies: 3 comments · 17 replies

thheller Mar 3, 2023 Maintainer

jaidetree Mar 3, 2023 Author

jaidetree Mar 4, 2023 Author

jaidetree Mar 4, 2023 Author

thheller Mar 4, 2023 Maintainer

thheller Mar 4, 2023 Maintainer

jaidetree Mar 4, 2023 Author

jaidetree Mar 6, 2023 Author

jaidetree Mar 6, 2023 Author

jaidetree Mar 7, 2023 Author

thheller Mar 7, 2023 Maintainer

jaidetree Mar 7, 2023 Author

thheller Mar 7, 2023 Maintainer

jaidetree Mar 9, 2023 Author

jaidetree
Mar 3, 2023

Replies: 3 comments 17 replies

thheller
Mar 3, 2023
Maintainer

jaidetree Mar 3, 2023
Author

jaidetree
Mar 4, 2023
Author

jaidetree Mar 4, 2023
Author

thheller Mar 4, 2023
Maintainer

thheller Mar 4, 2023
Maintainer

jaidetree Mar 4, 2023
Author

jaidetree Mar 6, 2023
Author

jaidetree
Mar 6, 2023
Author

jaidetree Mar 7, 2023
Author

thheller Mar 7, 2023
Maintainer

jaidetree Mar 7, 2023
Author

thheller Mar 7, 2023
Maintainer

jaidetree Mar 9, 2023
Author