How to modify database from external script?

[CtrlC-Root]

I'd like to modify the database content from an external script. I've looked through the documentation, open issues, closed issues, and started reading the code to figure out how to do this. I'm posting here asking for help in case there is some really obvious way to do this I've missed in my search so far. Example:

#!/usr/bin/env python

from lektor.project import Project
from lektor.db import Tree
from lektor.metaformat import serialize


def main():
    project = Project.discover()
    env = project.make_env()
    pad = env.new_pad()

    # arbitrary example modification: sort flow blocks in top-level page model by multiple fields
    # so they will show up in a nice order in the admin UI when editing the page
    pad.root['exflow'].blocks = sorted(
        pad.root['exflow'].blocks,
        key=lambda block: (block['category'], block['title']))

    # TODO: how to persist the changes here?
    tree = Tree(pad)
    ts = tree.edit('/')
    with ts:
        ts.data.update(exflow='????')  # XXX: some way to use serialize() maybe?


if __name__ == '__main__':
    main()

I made up an arbitrary example of a DB modification above because the more in-depth changes I plan to make are not relevant here as far as I can tell. I'll keep reading the source code to see if I can figure it out but in the meantime if someone knows how to do this or can point me in the right direction I would really appreciate it.

P.S. I really enjoy using Lektor so thank you to everyone who has contributed to make this awesome tool!

[dairiki]

@CtrlC-Root From your code sample, it looks like you're looking to update flowblock data.

Lektors flowblock-editing code is kinda hokey right now in that half of the serialization/deserialization of .lr data is done Python-side, and half is done in JavaScript in the Admin GUI. (Disclaimer: This is from memory, so may not be 100% correct.)

The EditorSession, from Tree.edit() that you're looking to use in your sample should work for top-level page fields, but it (currently) doesn't know how to handle the packed data that gets stored fields of type flow.

In the normal Admin GUI editing flow, parsing and serialization of flow data happens in the Admin frontend (JavaScript). (See FlowWidget, etc.)

On the Python side, FlowType and friends know how to parse and render the flow data, but do not know how to serialize flowblock data.

I don't think there is currently any Python code in Lektor to help with flow data serialization. For a one-off, it probably wouldn't be too hard to roll-your-own. ([Here's)(https://github.com/lektor/lektor/blob/05ba62fff422dc7ec75fb958085eeb2e09d7b69f/frontend/js/widgets/flow/metaformat.tsx#L74) the javascript code that does what's needed.)

---

In the long term, I think it would be good to move the de/serialzation of flow data to the Python side. Doing it cleanly will take some thought and planning, however.
(A related enhancement would be supporting alternative (or more standard) serialization formats for page data. E.g. #751.)

[CtrlC-Root]

@CtrlC-Root From your code sample, it looks like you're looking to update flowblock data.

Indeed, many of the automated changes I want to make involve flow blocks.

Lektors flowblock-editing code is kinda hokey right now in that half of the serialization/deserialization of .lr data is done Python-side, and half is done in JavaScript in the Admin GUI. (Disclaimer: This is from memory, so may not be 100% correct.)

I did inspect the HTTP requests sent from the frontend to the backend while modifing a flow field and saw it was sending the entire serialized content of the field to the /rawrecord endpoint. So that lines up with what I've observed so far. I just wanted to confirm I hadn't somehow missed some part of the backend that supported this and was used elsewhere.

The EditorSession, from Tree.edit() that you're looking to use in your sample should work for top-level page fields, but it (currently) doesn't know how to handle the packed data that gets stored fields of type flow.

In the normal Admin GUI editing flow, parsing and serialization of flow data happens in the Admin frontend (JavaScript). (See FlowWidget, etc.)

On the Python side, FlowType and friends know how to parse and render the flow data, but do not know how to serialize flowblock data.

I don't think there is currently any Python code in Lektor to help with flow data serialization. For a one-off, it probably wouldn't be too hard to roll-your-own. ([Here's)(https://github.com/lektor/lektor/blob/05ba62fff422dc7ec75fb958085eeb2e09d7b69f/frontend/js/widgets/flow/metaformat.tsx#L74) the javascript code that does what's needed.)

Thank you for the links! Yeah, I'll end up implementing something in my scripts to do this. If this functionality were to exist in Lektor do you know where it would be or how it would look? I ask because if I can write mine that way up front then it should be easier to make a PR for it later.

In the long term, I think it would be good to move the de/serialzation of flow data to the Python side. Doing it cleanly will take some thought and planning, however. (A related enhancement would be supporting alternative (or more standard) serialization formats for page data. E.g. #751.)

I agree with regards to refactoring the existing code but I think having CRUD access to the database through Lektor APIs, even if it's not used by the frontend, would be useful as an interim step. Obviously if there was support for other formats, which already have tooling available, that would sidestep the entire issue.

[dairiki]

If this functionality were to exist in Lektor do you know where it would be or how it would look?

I'm not sure. I think it's complicated and messy.

My first thought (which could easily be wrong) is that code to de/serialize the flowblock data for editing belongs in lektor.editor.EditorSession (and its friends EditorData, etc.) (This is what is returned from Tree.edit.) It currently knows how to parse and update the top-level fields of a page.

EditorSession already has access to the DataModel for the page, so it can access the type of each of the fields. It could treat flowblocks specially. Better still, it could call out to the Type object for each field to further deserialize/serialize the value of each field. (Type does not currently support this. It currently has no support for serialization. Also, note that the deserialization we want here is a slightly different operation than that currently performed by Type.value_from_raw, since value_from_raw sometimes returns a descriptor rather than an actual value. We'd want our deserialize_for_editing method to always return a value with a concrete type — probably a string in the case of those fields for which value_from_raw returns a descriptor.)

If EditorSession is modified to further de/serialize field values according to field type, for the sake of backward-compatibility, this behavior should probably be opt-in. This could be done via an optional argument to the constructor.

I'm envisioning being able to do something like:

>>> tree = Tree(pad)
>>> editor_session = tree.edit("/", deserialize_values=True)
>>> editor_session.data["exflow"]
[ # a list of blocks
    {
        "name": "mytextblock",  # this should be called 'block_type' or similar, but it's called 'name' in current code
        "data": {
            "title": "A Title",
            "body": "Here is some text.",
        },
    },
   # ... more blocks if they exist ...
]

# Update the title of the first block
>>> with tree.edit("/", deserialize_values=True) as editor:
...     editor.data["exflow"][0]["data"]["title"] = "My New Title"

My guess is that it will take a fair amount of work to implement this cleanly.