Allow to render component dependencies without middleware

[JuroOravec]

Continuation from this comment #277 (comment)

To refine the previous idea a bit more, I suggest following API for users to render dependencies:

1. Preferably, use {% component_dependencies %} (or the JS/CSS variants) in the top-level component BEFORE any other tags that could render components.

2. Alternatively, instead of {% component_dependencies %} (or the JS/CSS variants) users may also use track_dependencies for the same effect:

   track_dependencies(
   	context,
   	lambda ctx: Template("""
   		{% component 'hello' %}{% endcomponent %}
   	""").render(ctx),
   ) -> str:

3. If users need to place {% component_dependencies %} (or the JS/CSS variants) somewhere else than at the beginning of the top-level component, then we will need to the replacement strategy. I suggest to still use the track_dependencies. Basically, if the template given to track_dependencies contains {% component_dependencies %} tag, then we do the replacement strategy. If {% component_dependencies %} is NOT present, then we first render the text and collect all dependencies, and prepend them to the rendered content.

   For users, the API would still be the same as in 2.:

   track_dependencies(
   	context,
   	lambda ctx: Template("""
   		{% component 'hello' %}{% endcomponent %}
   		{% component_dependencies %} <-- comp_deps tag found, so replacement strategy used
   	""").render(ctx),
   ) -> str:

4. And if users do not want to call track_dependencies for each template render, they can use the Middleware (as it is currently), with the caveat that it works only for non-streaming responses and of content type text/html.

[EmilStenstrom]

To complicate things, I think many will put component_dependencies_js in the footer of the page (where it doesn’t block rendering), and the css-version in the head.

I’m wondering if we’re mixing two concepts here? “Where do I want my link tag” vs “I want to dynamically load my dependencies”.

Is adding a new tag another option?

about the track_dependencies-function, where would the user put it?

[JuroOravec]

To complicate things, I think many will put component_dependencies_js in the footer of the page (where it doesn’t block rendering), and the css-version in the head.

Hm, ok, so in that case the point 1. won't work very well from UX perspective.

I’m wondering if we’re mixing two concepts here? “Where do I want my link tag” vs “I want to dynamically load my dependencies”.

Yeah. Btw, by "dynamic loading", we mean the feature that only JS/CSS dependencies of used components are used, right?

Is adding a new tag another option?

I'm thinking of going in opposite direction, unifying all the dependency-rendering logic under a single tag, e.g.:

• Using type kwarg to specify whether to render JS/CSS/both:

  • Render both JS and CSS:
    {% component_dependencies %}
  • Render only JS:
    {% component_dependencies type="js" %}
  • Render only CSS:
    {% component_dependencies type="css" %}

• Using dynamic=True/False to decide whether to render ALL registered components (False) or only those used (True)

  • Render ALL components:
    {% component_dependencies %}
  • Render ALL components:
    {% component_dependencies dynamic=False %}
  • Render only used components:
    {% component_dependencies dynamic=True %}

about the track_dependencies-function, where would the user put it?

At the place where they initialize the template rendering. For example if I have a view that returns a rendered template, I would use it like so:

from django.shortcuts import render

def my_view(request):
	data = do_something()
	...
	return track_dependencies(
		context,
		lambda ctx: render(request, "my_template.html", ...)
	)

Altho, in this example, there's a question of what is the context - a dict or a Context?

What could work better would be if we defined our own render function. Which could work similarly to Django's render, and would hide the logic of track_dependencies. So we would have

from django_components import render

def my_view(request):
	data = do_something()
	...
	return render(request, "my_template.html", {"some": "data"})

This, supplying our own render (and render_to_string) functions, is also what I do in django-components fork.

[JuroOravec]

Two more things:

1. So in the discussion above I assume that {% component_dependencies %} renders JS/CSS of ALL components. Because of tag name, it could also suggest that it would only render dependencies of a SINGLE (current) component (AKA "render dependencies of a component" instead of "render component dependencies"). Are you aware of people wanting to render JS/CSS like this, meaning handling JS/CSS per component, instead of grouping them all in a single place? From web development perspective, it makes more sense to group it all.

2. One extra idea when it comes to rendering dependencies - mostly just throwing it out there so I won't forget, as it probably deserves it's own discussion. (But it's useful to have it mentioned for when thinking about how to design the interface):
   Currently the idea is to render a single JS/CSS file per component class. We could add a field on the Component class to configure whether to render the dependency file once per class, or once for each render instance. But to allow the latter, we would need to allow users to differenciate between the rendered instances, so we would need to render the JS/CSS deps at the same time (meaning with the same Context) as we use to render the component's HTML. In other words we would allow to use Django templating syntax inside JS and CSS files.

[EmilStenstrom]

I’m wondering if we’re mixing two concepts here? “Where do I want my link tag” vs “I want to dynamically load my dependencies”.

Yeah. Btw, by "dynamic loading", we mean the feature that only JS/CSS dependencies of used components are used, right?

Yes, "only load the dependencies of the components that are used on this page".

Is adding a new tag another option?

I'm thinking of going in opposite direction, unifying all the dependency-rendering logic under a single tag, e.g.:

* Using `type` kwarg to specify whether to render JS/CSS/both:
  
  * Render both JS and CSS:
    `{% component_dependencies %}`
  * Render only JS:
    `{% component_dependencies type="js" %}`
  * Render only CSS:
    `{% component_dependencies type="css" %}`

I like this, but not sure it's worth it because we would be breaking some existing code. Maybe keeping the old tags as reference to the new ones would work though.

* Using `dynamic=True/False` to decide whether to render ALL registered components (`False`) or only those used (`True`)
  
  * Render ALL components:
    `{% component_dependencies %}`
  * Render ALL components:
    `{% component_dependencies dynamic=False %}`
  * Render only used components:
    `{% component_dependencies dynamic=True %}`

The only reason you should render all components is if you somehow are expecting them to be in cache for the next page load. You're essentially trading first load time to speed up all successive page loads. This is likely not the right tradeoff for most people.

I like moving towards using this tag to only load used tags. Feels simple and clean.

about the track_dependencies-function, where would the user put it?

At the place where they initialize the template rendering. For example if I have a view that returns a rendered template, I would use it like so:
(snip)
What could work better would be if we defined our own render function. Which could work similarly to Django's render, and would hide the logic of track_dependencies. So we would have

from django_components import render

def my_view(request):
	data = do_something()
	...
	return render(request, "my_template.html", {"some": "data"})

Thanks for the explanation, makes sense!

The render method has a very wide surface area, that we then need to keep in sync with for new releases. Not sure I like that responsibilities. Also, overriding it won't work for generic views and other use-cases where you don't use the render function. I think I'm in favor of using component_dependencies then.

[dylanjcastillo]

Just to add something I've noticed with component_dependencies + the dynamic loading middleware. In cases where you dynamically replace content in a page (e.g., using hx-get or hx-post in HTMX), then the required JS and CSS aren't included.

So that means you have to render all the dependencies together. Or, at least, that's what I did when I had this issue. Not sure if we should consider this an issue because I guess it'd be hard to solve from django-components side alone.

[EmilStenstrom]

@dylanjcastillo So maybe there should be a view you can call to update the deps as well? Or to we include the new deps in the response from the server too?

[JuroOravec]

Adding new deps sounds like a complex problem:

• How do we establish which dependencies are already present on the webpage? The same call from hx-get/post could be made also from terminal with curl.
• How would we insert the JS / CSS into the HTML?

My first thoughts:

• Either the follow-up request adds some extra metadata, maybe a request header or query (?abc) to specify what it needs.

  • But then we'd need to manage Django endpoints, which to me feels outside of the scope of this project (as it stands currently)

• Or, probably better approach, we could define a global JS variable that keeps track of all the loaded dependencies. And a function or two to CRUD and load the dependencies. If you ever hacked around Facebook or YouTube frontends in devtools, they do something similar to load only that code into the website that is requested.

  So then, the user could access the data on which dependencies are loaded, and it would be up to them on how they want to send the data, whether through HTMX or something else. And they would also need to implement a Django endpoint that responds with the extra dependencies

  • Actually, maybe a better approach, so user doesn't have to do the heavy lifting -> We wrap all the JS / CSS dependencies in a JS script that calls the client-side JS library that manages the dependencies. If a new dependency was sent, it would be loaded onto the page. If it was loaded before, it will be ignored.

    • This seems to me like a good approach. It would mean, however, that we would not try to optimize the payload size at this point (AKA that on a dynamic replacement, the http response could possibly include all the old + new JS/CSS instead of only the new ones), but it would work. And IMO only once we have this way of loading CSS/JS dynamically in the client, then a separate issue could be of optimizing the payloads.

[JuroOravec]

More thoughts - based on the discussion in #399, I'm thinking we could have 2 kinds of JS - class-wide and instance-specific. Class-wide is ran once when the component of certain class is first loaded. Instance-specific is ran each time the component is rendered. Class-wide would be like it is currently, meaning it does not have access to context and nor data from get_context_data. On the other hand, the instance-specific JS would be rendered with the same Context as the HTML is.

[dylanjcastillo]

Actually, maybe a better approach, so user doesn't have to do the heavy lifting -> We wrap all the JS / CSS dependencies in a JS script that calls the client-side JS library that manages the dependencies. If a new dependency was sent, it would be loaded onto the page. If it was loaded before, it will be ignored.

This sounds very interesting, but not sure what you mean by the "client-side JS library that manages the dependencies". Is that something we'd create?

In regards to the 2 kinds of JS, I agree. I often ended up doing some weird hacks with event listeners to simulate the Instance-specific one.

[JuroOravec]

@dylanjcastillo Sorry for convoluted language, basically meant some JS script that would manage the dependencies. So yeah, we'd create/own that.

[EmilStenstrom]

I think it makes sense to have a very lightweight JS script that tracks and updates dependencies makes sense.

I wonder if we could make the existing component views just pass their dependencies as HTTP headers? Then the JS could intercept those and update as needed.

[EmilStenstrom]

I have thought about this more and I think we should implement this like this:

• Every component.as_view-view keeps track of which components it recursively has to render
• In the HTTP request that is returned, we list all dependencies that is needed to render that view
• We provide a vanilla JS script that parses the returned dependencies, looks at the existing deps, and patches any of the missing ones.

[JuroOravec]

@EmilStenstrom Sounds good!

My naive guess is that we'd do something similar also in the middleware, so the JS script would be aware of the CSS/JS files that were included in the initial response (so it wouldn't load them twice).

---

When it comes to sending data via HTTP headers, I think we'll learn more about the best approach when we try some proof of concept. While I feel like HTTP headers may be the best place, I'm also not sure if the requests can be reliably intercepted. So far I've done it only in browser automation like Playwright. In the browser, there could also be an issue that there's multiple ways how data can be fetched (using fetch or "old school" XMLHttpRequest). But I think there should be 100% some JS library that already handles request interception).

Also, with HTTP headers, the header size needs to be considered too. From this thread, we can say that a header may be limited to 8kb. If we'd need to pass on only the component names, NOT the JS/CSS file paths, then that could be maybe 30 characters per component (including formatting), which means a single component could have up to roughly 265 dependencies/subcomponents.

But if we'd need to pass also the URL path to the JS/CSS files, then, assuming up to 100 chars per JS/CSS file, and on average only 2 files per component/dependency (1 JS and 1 CSS file), then we'd need maybe around 230 chars per dependency/subcomponent, which would leave us with up to 32-35 dependencies/subcomponents.

While 32 subcomponents seems like plenty, I can imagine that if someone was heavily using a
component library to define tables, buttons, icons, columns, etc, then one could get over the limit, as the component library components could each hide inside of themselves additional 2-5 components.

So I'd like to also explore the option of passing the data as e.g. <script> tags inlined into an HTML response, so the number of subcomponents isn't a limiting factor. But this could also have it's caveats like - Can we simply "append" the <script> tag, or do we need to parse the response and check if it has <html> or <body> tags, and move the <script> inside of those? And if we need to parse the response HTML, what impact will it have on the response time?

And what if someone returned a response that's not a valid HTML? Or not an HTML at all? After all, django-components can still be in theory used to generate also non-HTML content. *

Possibly we could do a "progressive" approach, where if all the dependencies info can fit into an HTTP header, then we use the header, and if NOT, then we insert the <script> tag into the response body, IF the response has an HTML content-type/MIME. And if there's too many dependencies for a header, but response content type is NOT HTML, then we'd raise an error. *

* I realized that considering non-HTML cases doesn't make sense, because we're talking about importing JS/CSS, and those only makes sense within the context of a browser.

[EmilStenstrom]

Good point with the 8kb limit.

I think we can probably build a simple algo that compresses a list of dependency strings in an efficient way:
https://chatgpt.com/share/df340430-0ba0-424c-a16a-69ebd9dc14e7

But maybe the easiest way would be to use multiple headers, and not try to push everything into one? That should make this practically limitless.

[JuroOravec]

Oh, very intesting ideas with both multiple headers and compression, I like that!

Didn't know that multiple HTTP headers with the same name are possible. For reference, here's an example from StackOverflow:

Cache-control is documented here: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9 like this:

Cache-Control = "Cache-Control" ":" 1#cache-directive
The #1cache-directive syntax defines a list of at least one cache-directive elements (see here for the formal definition of > #values: Notational Conventions and Generic Grammar)

So, yes,

Cache-Control: no-cache, no-store
is equivalent to (order is important)

Cache-Control: no-cache
Cache-Control: no-store

So with 3-4 such headers, we could already fit over 100 subcomponents, which feels like it should be sufficient also for the "heavy user of component library" scenario, which means we could avoid having to touch the response HTML 🎉

Going further with the quick mathz, if we assume the pattern that component's JS and CSS files have the same path and the same name, and differ only in the suffix:

/path/to/component/component.js
/path/to/component/component.css

And we do some sort of compression algo like @EmilStenstrom suggested, in which:

• We would need to define the common path for the JS/CSS files only once, which would save us 100 chars
• There would be some overhead for the algo, e.g. 20 chars

Then we'd bring it down from 230 to around 150 chars per subcomponent, which would allow for up ~50 subcomponents per header, in which case we could do with just 2 headers to fit 100 dependencies/subcomponents.

[EmilStenstrom]

Just for fun I played with GPT-4o to implement this, seems we get a compression ratio of 0.55, that will only grow as we add more urls.

urls = [
    "/path/to/calendar/index.js",
    "/path/to/calendar/style.css",
    "/path/to/dropdown/index.js",
    "/path/to/dropdown/style.css",
    "/path/to/dropdown2/dropdown.css",
    "/path/to/dropdown2/dropdown.js",
    "/path/to/dropdown3/dropdown.css",
    "/path/to/dropdown3/dropdown.js",
    "/separate/path/to/calendar/index.js",
    "/separate/path/to/calendar/style.css",
    "/separate/path/to/dropdown/index.js",
    "/separate/path/to/dropdown/style.css",
]

from collections import Counter

# Extract words
words = []
for url in urls:
    parts = url.split('/')
    for part in parts:
        sub_parts = part.split('.')
        for sub_part in sub_parts:
            if sub_part:  # Only add non-empty strings
                words.append(sub_part)

# Count frequency of each word
word_counts = Counter(words)

# Sort words by frequency in descending order, most common words first
unique_words = sorted(word_counts.keys(), key=lambda x: -word_counts[x])

# Create a dictionary mapping
word_to_index = {word: index for index, word in enumerate(unique_words)}

# Replace words in URLs with their identifiers
compressed_urls = []
for url in urls:
    compressed_url = []
    parts = url.split('/')
    for part in parts:
        sub_parts = part.split('.')
        for i, sub_part in enumerate(sub_parts):
            if sub_part:  # Only add non-empty strings
                compressed_url.append(str(word_to_index[sub_part]))
                if i != len(sub_parts) - 1:  # Don't add '.' after the last sub_part
                    compressed_url.append('.')
        compressed_url.append('/')
    compressed_urls.append(''.join(compressed_url[:-1]))

print(urls)

print(compressed_urls)
print(unique_words)

input_length = sum(len(url) for url in urls)
output_length = sum(len(url) for url in compressed_urls) + sum(len(word) + 1 for word in unique_words)

print(f"Input length: {input_length}")
print(f'Output length: {output_length}')
print(f'Compression ratio: {round(output_length / input_length, 2)}')

# Create a reverse mapping from indices to words
index_to_word = {index: word for word, index in word_to_index.items()}

# Reconstruct the original URLs
reconstructed_urls = []
for compressed_url in compressed_urls:
    reconstructed_url = []
    parts = compressed_url.split('/')
    for part in parts:
        sub_parts = part.split('.')
        for i, sub_part in enumerate(sub_parts):
            if sub_part:  # Only process non-empty strings
                reconstructed_url.append(index_to_word[int(sub_part)])
                if i != len(sub_parts) - 1:  # Don't add '.' after the last sub_part
                    reconstructed_url.append('.')
        reconstructed_url.append('/')
    reconstructed_urls.append(''.join(reconstructed_url[:-1]))

print(reconstructed_urls)

$ python url_compressor.py
['/path/to/calendar/index.js', '/path/to/calendar/style.css', '/path/to/dropdown/index.js', '/path/to/dropdown/style.css', '/path/to/dropdown2/dropdown.css', '/path/to/dropdown2/dropdown.js', '/path/to/dropdown3/dropdown.css', '/path/to/dropdown3/dropdown.js', '/separate/path/to/calendar/index.js', '/separate/path/to/calendar/style.css', '/separate/path/to/dropdown/index.js', '/separate/path/to/dropdown/style.css']
['/0/1/5/6.3', '/0/1/5/7.4', '/0/1/2/6.3', '/0/1/2/7.4', '/0/1/9/2.4', '/0/1/9/2.3', '/0/1/10/2.4', '/0/1/10/2.3', '/8/0/1/5/6.3', '/8/0/1/5/7.4', '/8/0/1/2/6.3', '/8/0/1/2/7.4']
['path', 'to', 'dropdown', 'js', 'css', 'calendar', 'index', 'style', 'separate', 'dropdown2', 'dropdown3']
Input length: 370
Output length: 204
Compression ratio: 0.55
['/path/to/calendar/index.js', '/path/to/calendar/style.css', '/path/to/dropdown/index.js', '/path/to/dropdown/style.css', '/path/to/dropdown2/dropdown.css', '/path/to/dropdown2/dropdown.js', '/path/to/dropdown3/dropdown.css', '/path/to/dropdown3/dropdown.js', '/separate/path/to/calendar/index.js', '/separate/path/to/calendar/style.css', '/separate/path/to/dropdown/index.js', '/separate/path/to/dropdown/style.css']

[JuroOravec]

@EmilStenstrom Haha, that's amazing! I just played with it too. First I tried a "second pass", but that didn't work well.

Then I tried a couple of built-in compression libraries that python has that ChatGPT suggested. Tried with 57 urls (~29 components):

• zlib - compression ratio 0.27 to 0.22
• bz2 - compression ratio 0.36 to 0.25
• lzma - compression ratio 0.43 to 0.27

Also timed them, and got these results

• our algo - 0.1734 seconds
• zlib - 0.0118 seconds
• bz2 - 0.0177 seconds
• lzma - 0.0222 seconds

And lol, zlib is the clear winner here.

But those compression libraries produce bytes, so to pass that as an HTTP header, GPT says we need to encode it to base64 after the compression. That makes the stats a bit worse, but zlib is still the best out of all 4, with ~0.295 compression ratio and 0.029 seconds per 57 urls.

With a bit of extrapolation, a single HTTP header (8kb) could fit ~460 components. That's definitely enough! But I like that it still leaves the doors open for using multiple headers if that's ever needed.

[EmilStenstrom]

I guess you could produce a dictonary for all components on the first load, and then just send the encoded URL:s on all subsequent dynamic loads? I think that makes our algo work best?

[JuroOravec]

Could you add an example of what you have in mind?

[EmilStenstrom]

The code I pasted above does two things:

1. Split on / and put all the parts into a dictionary
2. Replace all the urls with references into that dictionary, leaving just 4/6/8/9/9

when I calculated the compression ratio, and always included the full dictionary. But if we can write that to the page in a script tag, the fronted could skip sending the dict again, and we could just pass the list of numbers.

[JuroOravec]

Hm, not sure about it at the first read. Because it sounds like then this data transmission will be "state-ful" - we would need to be able to tell to the server which words we already have on the frontend. So then we'd need to intercept both requests (frontend --> server) as well as responses (frontend <-- server).

But anyway I can look into that once working on this. I think we already have a good idea of how this should work, and I reckon I'll get to the implementation of this in about 1-2 weeks.

---

To sum up what we have so far:

See the diagram in Mermaid editor

---

One extra thing I've noticed from the diagram is that when we will be parsing the HTTP header on the frontend, we'll also need to distinguish between:

• First request:
  • Fetches a full HTML
  • CSS and JS should be inlined into and tags by the user
  • Frontend script does NOT load the JS/CSS manually
• Fragment request:
  • Fetches an HTML fragment
  • CSS and JS cannot be inlined into and because these tags are NOT present
  • Frontend script loads the JS/CSS manually

From frontend's perspective, this could be a simple boolean value (0 / 1) within the same HTTP header, or a separate header altogether.

The question remains how will the server decide whether to flag the reponse as "page" or "fragment"?

1. We could do a simple check inside the middleware to see if the rendered HTML string contains e.g. <html> and <body> tags, and if not then treat the response as a fragment.
   • We could make it configurable, so that users could potentially override this with their own checker.

      # settings:
      COMPONENTS = {
      	# Import path
      	fragment_checker: "django_components.middleware.fragment_checker",  # Default
      }

2. Or maybe there could be a decorator that, when applied to a view function, would set a private property on the HttpResponse instance, and that's how our middleware would know:

   @html_fragment
   def my_view(request):
   	content = render_content(...)
   	return HttpReponse(content)

Personally I like the first approach more (using config).

[EmilStenstrom]

I'm not sure I follow how this would be stateful? Sorry for being very brief, I'll try to explain my thinking in more detail:

• When calling component_js/css_dependencies, we would render a COMPONENT_URL_DICTIONARY = [...] which contains a dictionary for ALL components.
• When any component view is called, it would include the relevant dependencies encoded with the global dictionary in a header
• When a view was called from the frontend, we would intercept the response, and use the dictionary from the latest full pageload to decode the URL:s into full URL:s and then diff them against the currently included ones.

Since we're always using a global dictionary over all components, I think we can fully avoid any state management.

Since all component view always return just fragments, and we're only intercepting those calls, I don't see why we would need to "mark" different calls as different types either. If we do, let's use the dependencies header, and not send that on the first page load.

Please point out if I'm missing something!

[JuroOravec]

Not sure if I got it 100% right, but my assumption is that the HTML fragments fetched after the page load MAY use different components to render than the components used for the initial page HTML. E.g. if someone triggered HTMX to load a different tab or even a different page altogether.

In such case, the "different" components would not be included in the frontend's dictionary, because it would not be detected in the 2nd step ("When any component view is called...").

[EmilStenstrom]

I'm saying that we should base the dictionary on all components in the registry, not just the ones that are rendered on the initial page HTML. That should include all components I think? We could even build the dict and store it in the registry, so as to offset the construction cost on page load?