How to structure generic "field" components that link to the parent state (django-components/HTMX/Alpine.js/Crispy/Bootstrap)?

[tanssinet]

I built a mid-sized Django app (150+ views) alone in few months, and you just cannot beat the productivity. In this case the user experience with traditional web app was good enough, and it was more important to be fast than to have a "modern" feel.

But some parts got unnecessarily hairy and complex. One page includes (among other things) two-level "nested" inline model formsets, where the lowest leel has multiple instances of the same autocomplete fields. Basically it is list of lists of values selected by autocomplete. I only send the parent entity (with all the children) to backend when the root entity create or update is complete.

And now I need to create an even more complex version of this same page.

I have used several frontend frameworks, but try to use them as little as possible to keep my productivity up. I have built a component library for a customer using Angular 1.x so I have some understanding how to approach component libraries.

I have tried to use component approach in my project as much as possible, but Django wthout django-componts does not make it easy. Currently, I have abstract and generic "autocomplete component" (based on Bloodhound.js/Typeahead.js) and have then "subclassed" ths component for various types of entities I need to search for, with a corresponding REST API for each autocomplete. Autocomplete components (both the generic and model-specific ones) are currently crispy form Fields, but then I build a JSON object inside the level-1 formset of the parent.

I was inspired by the video below, so my idea would be to build the new page with a combination of django-components/Alpine.js/HTMX, possibly using crispy forms there as well.

https://www.youtube.com/watch?v=3GObi93tjZI

I really like the idea that django-components/Alpine.js/HTMX are small libraries that work well together, but I am struggling currently to understand how they should be working together, and how I should structure my application.

So let's start with the autocomplete selection component. In my case, I am searching for a performer, i.e. a singer or orchesra.

1. The user should get the autocomplete experience, but having a REST API makes things unnecessarily complex when we have HTMX. There is django-htmx-autocomplete library, but that appears to have it's own approach to components that does not work that well with django-components nor with standard django forms or crispy forms. So potentially I would need to rewrite a component similar to it, but using djanco-components model.

2. As I need to add multiple performers (and later multiple groups of performers, and even later multiple events containing multiple groups of performers), I must have multiple copies of the component on the page -- although potentially only when editing -- so they need to have a unique identifier. One approach is to pass a unique identifier from the parent, this has the benefit that the parent then knows the identity and can use it for data. Apparently Alpine.js has $id magic property that could be used to generate the identifier. But if I do that, how do I reference the selected vaue from the parent component?

3. The biggest issue when building components is communication between components. Maybe the term "component" is too abstract and generic. Some components are more like "fields" (such as autocomplete), whereas others are more like "forms" (collections of related fields). For field-like components such as autocomplete, it makes sense that the frontend keeps the state until the parent component wants to save the data.

4. Another aspect related to HTMX is data permanence. I think nobody wants to give up the idea that servers are stateless. But sometimes the data is transient and I don't want to keep it i the database. For example, I have currently two classes of users. Perhaps 1000 users enter data. I can store their data in the database even if it is temporary. But then I have those who want to view the data. And I have 50k page loads, which is not a huge amount, but I would prefer the components to store and process the transient data for these users in the frontend.

So to build a component hierarchy, and to start with a autocomplete, my approach could be something like:

1. Implement abstract generic autocomplete field with data fetching using HTMX
2. Use the abstract generic autocomplete field to implement the concrete autocomplete fields
3. Frontend would maintain the state of the autocomplete field, perhaps using Alpine.js. Should the autocomplete be a django-components component? It has HTML, javascript and python, but it needs to integrate with/be a (crispy) form field. Is it a good idea/bad idea to make a component that is a field?
4. Parent component corresponds to a model and a form, and the autocomplete field would be one field within the model, The form would be similar to the current individual level-2 inline formset form, but implemented with HTMX.
5. If I want to use Alpine.js to connect the state of the autocomplete field to the entity attribute represented by the parent component, what should I be linking with what?
6. Parent component that uses the autocomplete field is a django-component. Parent component is updated using HTMX.

Any thoughts?

[JuroOravec]

Hey @tanssinet, there's a lot of info in there, and I'd probably need to re-read it a few times to fully understand all the points you're considering, but here are some ways how I structure Django/HTMX/AlpineJS app:

1. Separation of concerns

   IMO Django makes things a bit TOO convenient, which then makes projects harder to scale. E.g. you can make database requests from within templates just by writing {{ mymodel.related_field }}, but then you lose the awareness of where interactions with DB (or other upstream services) happen. Or the fact that Django's Forms can handle both form rendering AND interaction with the database, and it also serves as entry point for modifying form styling or frontend behavior. IMO that's too much, and it makes the flow hard to follow.

   When using a full-stack framework like Django, then, if you are building anything beyond a small projects, IMO it is critical to maintain clear boundaries between different parts of the system. You should treat the project the same way like if your frontend was a separate project.

   1. I ditched Django's Form.

      • Instead, for UI rendering, I write forms in plain HTML (and build my own Django components). That way, form rendering is just the same as rendering anything else -> you provide inputs to the template, and it renders it accordingly. No extra complexity. Each form is it's own Django component. If I need to modify the form's styling, I add a CSS file. If I need to modify UI behavior, I add JS file (more on that later).
      • For Form submission, ON CLIENT SIDE, I use HTMX/AlpineJS. If it's a simple "submit and redirect/refresh", then HTMX does that well. If I need to pre-process the data, I use AlpineJS for that.
        • NOTE: To use HTMX/AlpineJS to submit forms, I had to add some setup code to pass along CSRF token in requests sent via HTMX/AlpineJS.
      • For Form submission, ON SERVER SIDE, I use regular ol' views/routes as endpoints. I also use Django Ninja for endpoint input parsing and validation. So I effectively have 2 kinds of views
        • ENDPOINTS - They act like REST API - they accept inputs and send response as JSON, GET/POST depending on if it's query or mutation
        • VIEWS - GET only. They respond with HTML
      • So, to sum up, my form submission flow looks like this:
        1. User requests VIEW, gets back HTML that's rendered
        2. Some form is rendered as part of the HTML response, all done within Django templates
        3. User fills in data, submits the form.
        4. Form submission request from client side is done by HTMX/AlpineJS. The form is submitted to an ENDPOINT.
        5. Server-side, there's an endpoint (Django view) that parses the inputs, does some action, and responds with something.
        6. Depending on business logic, this is where possible redirect, or some UI action is triggered to signal that the action was complete.

   2. Forbid making databases requests from within templates.

      E.g. all my VIEWs have a structure like this:

      def my_view(request):
      	# check if requested model exists
      	my_model = get_object_or_404(MyModel, id=xyz)
      
      	# Prepare ALL data required to render the templates
      	children = my_model.children.all()
           child_count = len(children)
      
      	# Render template, given the data
      	render("template.html", {
      		"my_model": my_model,
      		"children": children,
      		"children_count": children_count,
      	})

      So I make all my database queries BEFORE rendering the template. This means that:

      • I have control over and can optimize the database interactions, because I know EXACTLY what kind of data I am fetching, and I can make bulk/batch queries.
      • Templates become simple data transformation functions that I could easily test to ensure that some important aspects of the rendered templates remain still.
      • I'd call it "reduced mental overload", you could call it better productivity - I need to deal with smaller scope when making changes, as all frontend/template changes can now be split into 3 subtasks:
        1. Prepare data that the template will need
        2. Within the template, pass the data to the right component
        3. Inside the right component, with the rigth data, render it somehow

   3. Strict data and event flow

      From my experience, moving data up (events / callbacks) in Django templates is HARD. But it kinda makes sense:

      • At render time, you have ALL the information. So there should be NO reason to need to bubble up.
      • On the other hand, it DOES make sense to bubble up when the HTML is already rendered on the client-side.

      Also, the only way to send data (PUSH) from client to server is via forms/requests.
      The only way to fetch data (PULL) from server to client is via forms/request OR initial data rendered in Django templates.

      Data/events can flow within the UI via AlpineJS events.

      Here's a summary of the constraints:

      Data flow:

      From \ To	Python	JS	HTML
      Python	-	dataset attribute, AlpineJS binding	Django templates
      JS	Submit form	-	AlpineJS ($refs)
      HTML	Submit form	AlpineJS (x-data, emit, ...)	-

      Event flow:

      From \ To	Python	JS	HTML
      Python	-	N/A	N/A
      JS	Submit form	AlpineJS (emit, ...)	N/A
      HTML	Submit form	AlpineJS (x-data, emit, ...)	-

   4. Co-locate AlpineJS and django-components components

      With the Django/DjangoComponents/AlpineJS stack, we have two frameworks that have their own concepts of "components". So it can be a bit confusing. But it's important to note that "Django component" (DC) components operate PURELY on server-side, whereas AlpineJS components are PURELY client-side.

      However, for maintainability, I found that the easiest thing for me is to "co-locate" the two. In other words, if I have a separate DC component, and I need to do some JS event handling, I also create an AlpineJS component inside that DC template.

---

With these in mind, here is how I would approach some of the issues you're mentioning:

1. Autocomplete components

   • I'd look at it from the perspective of data. How many options are there for the autocomplete? Tens, maybe lower hundreds?
     • If so, I think you can embed all the autocomplete options into the HTML response, and then use some UI magic (some 3rd party component or plain JS) to take those options, and make them into an autocomplete.
     • If we're talking about hundreds, thousands, and more, then rendering all options inside the HTML could be heavy. So instead you should delegate the autocomplete search to the server. So you'd have an API endpoint that takes the searched text, and responds with autocomplete suggestions.
   • Whichever of the two approaches above, I'd encapsulate it in an autocomplete django component. And inside that component, all the JS, AlpineJS/HTMX, etc logic would reside and would be hidden.

2. how do I reference the selected vaue from the parent component?
   What does "selected" mean in this case? It sounds like you want to respond to user action. So that means this happens in the UI.

   If the parent component is also in the UI, then you can use Alpine's event emitting to send the child's ID to the parent, and then the parent decides how to handle things further.

   As for whether to use AlpineJS' $id or not - Do the child component have meaning? AKA if the child components are rendered based on some entries from the database, then they probably already have an ID, so I'd use that. That way it's easier to debug and reason about. BUT, you can have situations where the children do not have special meaning. Like if I have a Sortable list, then those individual entries may be meaningless. It's just a piece of HTML I need to identify uniquely. Then I use AlpineJS $id.

3. Same applies here as I wrote for point 1.

   I'd reiterate one thing here - the way I use Django templates/components, NO components have server-side state. They can define client-side state, which would be ultimately owned by AlpineJS, but no state is persisted when a server-side component is rendered.

4. For this one, I'd suggest to again look at the flow from the perspective of data:

   Where does user authentication / authorization happen? It's probably done for you already by Django. So users are already persisten in the database. If you want to store extra data for users you have 2 options:

   1. Store the extra data in the database
   2. Store the extra data on the client-side

   IMO this should be decided based on your app/user/business needs. If you store data in the database, then user can access the website from different devices (e.g. desktop vs mobile) and still access the same data. If you store data on the client-side (e.g. in localStorage), then user would lost access to this data if they used a different device.

   Ofc, there's more options - if the user data we're talking about is like session/cookie, you could use a different kind of database (like Redis) for that. But that feels like already out of scope of this discussion.

---

Hope this helps!

[tanssinet]

Wow, what a thoughtful and long reply. Lots of things to digest. No need for you to go back to my original question, here are some more specific questions for the things that I still did not quite understand.

(1) You mention that you have created your own components. So do you have a component corresponding to "text input field", i.e. generic components corresponding to form fields? Or do you have only higher level components that not only has a type but also business meaning, for example one component for "person name field" (that is not build from other components)?

(2) You appear to agree with me that creating a "autocomplete django component" make sense. I would typically approach this in component-based architectures by making a generic "autocomplete django component" and then thin "venue selector component" on top of that generic component, which would include for instance setting the REST URL for getting the venue search results. I would never use the "autocomplete" directly, but only through a component which has meaning like the "venue selector component".

(3) In my case the "venue selection component" has over 5000 possible results, so there is a REST API and search behind it. (but there is another autocomplete that has 300 results, which might still be doable inlining the data, although it currently uses the same generic autocomplete component/field). If I understood you correctly, you also have some JSON endpoints, so not everything is HTML. So in this case probably the most sensible option is to just wrap the existing autocomplete/venue selection functionality in a Django template.

If I would like to be more HTMX purist, I could have the "venue selection component" render the results (including the text input) on the server side rather than return JSON. In my 5000 result case it would just limit the results to what autocomplete shows, for example 5 results.

(4) The "venue selection component" would communicate with the parent by emitting a "venue selected" event with the id of the venue, which will then be listened to by the parent component. Makes sense.

(5) Ditching the forms is an interesting idea. I still need to convince myself that it is worth it. But I can see that getting Alpine/HTMX tags into the HTML make the forms messy, so I am open to the idea. I guess my open question is that if you take a more HTMX purist appcoch (more on that below), does it look different? Ninja seems to assume you are working with JSON data.

(6) How do you validate the data in practice? All in Django Ninja? If I ditch the forms like you are suggesting, then I need to build all that infastructure by myself that is not provided by Django Ninja. I am not familiar with that library yet. Are you using HTML validations at all?

(7) In my case, I have three main user groups. Anonymous end users interacting with the data (50K page views/day), logged in end users interacting with the data (currently practically none of these, as everything can be used anonymously) and approximately 500 users entering data who are logged in. I want to build new complex logic for the end users. I would prefer to store data in the client side, so that anonymous users can use the new complex feature, and there would be no need to log in, but that is not a hard requirement. I could also require them to register so I could store the data on the server if that is easier for me to implement. If the trade-off is between installing/maintaining Redis vs. requiring registration for the complex features, I would go for the latter. So that is why my question is so vague, because I have very little hard requirements on how to structure code and even where to keep data.

(8) Have you tried being more of a HTMX purist and learned something from this experience? If you are using JSON results, you are necessarily building some logic in the frontend. If I have understood the idea of HTMX correctly, the idea is to bring as much as possible of this back to the backend. So if I think of a continuum inside frontend of more logic (typical SPA with a full-blown frontend framework) and no logic (HTMX purist way), your approach is somewhere in the middle, although closer to the HTMX purist way than the SPA way. It seems to me that in the video I linked they are appear to be full-blown HTMX purists, which means there is almost no need for Alpine.js manipulation.

Let's make the input validation talk a little bit more concrete with an example. Let's say we have a integer input field price, which needs to be an integer and has a minimum and maximum value. I chose the price deliberately, because it is an example where the simple HTML5 validation fails. Here in Europe, we use euros, shortened either "5€" or "5e". Although I want price to be an integer field, "5e" is accepted by most of the browser HTML validators, as "5e" is a valid number, because browser thinks that the e is the exponent (like in 1e5), although 100% of the users actually meant "euros" with e, not the exponent.

I have collected my understanding of different approaches below. I will make the idea more concrete by concentrating on where the responsibilty for setting the "error" CSS class for the input element resides.

Traditional Django - Form validates the input on form POST. Django form validation puts the error CSS class when reloading the page. There could be HTML5 validation, but it confuses the users when there are two types of error messages. In this case I would disable it, and just use the text input field and validate the price in the clean_price() form method.
Traditional Django + splash of Javascript/jQuery - Like choice 1, but with some hand-coded JS/jquery logic so that error class is set when the focus is changed from the input field, in addition to the above. This gives more modern feeling to the UX. We can fix the previous handling and add error class on input with custom logic that checks the"5e"
Framework SPA - Framework component validates that the number is int, and possibly the min and max values. The data is send via REST requests on form submit, which validate the data in the backend and possibly render the backend validation result. The framework component adds the error CSS class if the server responds with the error code. The framework probably does not use the HTML5 validation, but has built-in validation logic for ints. If you want, you could build custom framework component for price on top of the framework component, hand-coding the special validation for price.
Using pure HTML + Frontend UI kit - The validation logic is in the components of the UI kit (such as Tailwind Elements). They probably have a better built-in support for custom validations, but there is still two places where you implement the validation.
Traditional Django Forms + HTMX form checking - When the user is ready with the full form, the form is sent (via HTMX request) to the backend, and Django Forms sets the error CSS class when the page is rendered. The only differene between this and the traditional Django is that the is no full page reload, only the form is reloaded.
<-- Your way? --> Alpine.js component way - When the user is ready with the form, the form is sent (via HTMX request) to the backend, and Django Ninja validation sets the error CSS class.
HTMX purist way - There is only text input field in the browser and no validation code, no Javascript or even Alpine.js code for adding the error class. When focus changes to the next field, data is sent to the backend via HTMX request, which returns the text input field + the HTML for the error message, i.e. the backend component is responsible for rendering the input field with the error CSS class added + the error message-

I guess we have the following trade-offs:

1. Does the user get the validation error message immediately when the focus changes after entering the wrong value, or only when posting the full form
2. Implement/configure the field validation only once (no need to implement/configure separately for the frontend, and obviously backend validation is needed anyway)
3. Minimize the backend load / number of backend requests
4. Can we prevent a full page reload
5. Performance

With traditional Django you get 2 & 3. Performance is not so good, as we reload the full page.

With traditional Django + HTMX you get 2 & 3 & 4 & 5. Performance is better when using HTMX, as we prevent the full page load, but as the user only gets the error message after HTTP request, the performance is not as good as when we are validating in the frontend.

Using SPA, trad Django + custom Javascript/jquery, UI Kit, or Alpine.js component approach we get 1 & 3 & 4 & 5. The performance is the best, as the validation is done in the frontend.

HTMX purist gets 1 & 2 & 4. The 5 is a question mark, but the video I linked seems to suggest that it is not really a problem but actually on complex UIs this solution wins React etc. on performance. Also, it might be possible that the user can continue editing other fields (although submit button might be disabled) while the backend request is active so the user does not suffer from the performance.

So I am interested in hearing if you have tried the HTMX purist way (and learned something along the way) or decided on the approach based on your intuition, or if you have some other rationale for preferring your approach.

[EmilStenstrom]

Another take at this (and there are as many takes on this as there are developers!) from me.

I am not a big fan of splitting my business rules between the backend and frontend. I think it quickly leads to messy and hard to maintain codebases, where every features needs to be implemented twice; once in the backend, and once in the frontend.

A way around this is to say that, yes, we will need a backend no matter what. So let's put our business logic, state handling, and effort there, and try to keep the frontend as minimal as possible. Django-components allows you to build sites like that if you want.

Without going into details about your specific case, and given the above "philosophy" around where to focus ones effort:

1. Build everything as django-components on the backend first. No JS.
2. Then sprinkle HTMX on top, using component.as_view() to fetch component state dynamically and update the front-end.
3. Persist all state on the backend, so full page reloads restore the state to before you reloaded.

Advantages:

• This will minimize the time you spend in "JS-land", and put all your business logic in one place.
• It also makes testing easier, because you can focus your testing on the backend.
• You're not dependent on the accelerated speed of breaking features in JS-land either, your site is likely to be easy to maintain
• Accessibility is baked on, because JS is optional

As I said, there are as many opinions on this as there are developers, but I wanted to add my take.

[tanssinet]

@EmilStenstrom thank you for your perspective. I have a similar aversion towards having the business logic split into the frontend and the backend, and I guess this is why I like Django.

Are you using forms to implement your components?

What are the lowest-level components in your hierarchy? I.e. do you have lower-level "text input" components, business components for fields such as "company name" or are your smallest components even bigger, perhaps closer to Django models?

If you have field level components, how do you store the state in the backend? I.e. if there is a person entity, and the user enters the name, do you already send the name field data using HTMX into the model when the name field loses focus, or do you send it only when the user pushes the save button for the whole person enity?

I guess the former option would mean that validation needs to be postponed until the server has all the data, so I am suspecting you are mostly sending data when you have the whole entity at hand, which you can then validate.

[EmilStenstrom]

@tanssinet I currently do not use django-components at all I'm afraid. Children and work as a non-developer has taken most of my time. So I can't answer concrete questions of how I use the library...

If I did use it, I think I would wrap Django's forms, and build components based on that. I have this idea that django-components one day will be part of Django, and that it would benefit to be. Integration with forms would be one nice way to be able to use what's already in the box.

One way of handling state is to have a notion of "PersonDraft", where you store an unfinished Person object until all the fields are correctly filled in. This could give you a nice way to resume work after reloads, but also after browser crashes, or even resuming work from a different device. I like the resilience of that.

[tanssinet]

@EmilStenstrom Ok, thanks. I think it would be a good fit for being part of Django. Yes, the draft method is how I intend to make this work on the component I am trying this out with. When the users are anonymous, then this is not a viable option, though.

[dylanjcastillo]

Hey @tanssinet,

In case it's useful:

1. Here's my somewhat unorthodox way of managing forms with django-components and HTMX. I like doing hacky stuff so YMMV.
2. I tried building a component that had a somewhat complex autocomplete feature with just HTMX a while ago and eventually gave up because it was simply too painful. I ended up creating a "frontend-heavy" component that used fetch and an small endpoint for retrieving the options when there were too many. I try to do as much as possible with HTMX, but also keep a couple of simple JSON APIs I use for cases such as this one. I feel that way you get the best of both worlds.

[tanssinet]

Thanks, looks interesting. Can you elaborate what was painful about making autocomplete with HTMX?