It should be possible to use simple UDFs in the LiberTEM GUI. Use-case:

• a specific analysis has to be repeatedly run on different data: it's important to have a seamless workflow
• a developer wants to make his UDF usable by his colleagues that don't speak Python
• the UDF is not part of LiberTEM, but possibly packaged as a PyPI Python package
• opening a notebook every time would be a hassle, and also has a higher barrier of entry: this should be usable without knowing Python or being able to use Jupyter notebooks

Implementation

Python/server side

On the Python side, clean up the Analysis interface and transform it into a plugin API. Discover Analysis instances either via specifying the Python package on the libertem-server command line:

$ libertem-server --plugins-from=module_a,module_b

Or by performing auto-discovery and using entry_points:

setup(
    entry_points={'libertem.plugins': [
        'my_analysis_name = package.module_a:Class',
        'other_analysis = package.module_b:Class'
    ]},
    ...
)

The list of plugins will be made available in the Web API, and the job running part will also query the plugin registry to find its analyses. The built-in analyses could possibly behave exactly like external plugins, except that some of them have some extra support in the GUI.

GUI

The MVP would support UDFs without any parameters, which have one or more result channels in signal- or navigation axes. Signal UDFs would be integrated in the Mode drop-down of all analyses, and "navigation UDFs" would be added to the "add analysis" drop-down.

As the second iteration, support for parameters could be added, ranging from simple int/float parameters, to JSON-Schema-like data, to generic mask definitions or other widgets.

Use-cases for the second iteration:

• SSB from https://github.com/ptychography-4-0/ptychography/
• RDF Mapping?
• parameters for pre-processing built into the UDF (thresholds etc.)

Discussion / Questions

• Should we support UDFs that return both signal and navigation-axis results?
  • In the MVP, maybe only either-or?
• Do we need any additional features to make this work well?
  • For example, a UDF written by an inexperienced developer could be slower than the ones built-in, there it would help to have better user feedback (smaller partitions? Adaptive and dynamic partition sizes #382)
  • Better error reporting, in case there is a bug in the UDF? (See also: Introduce error boundaries in the GUI #673)
  • Advanced: support for pick-like analyses? Could, for example, allow to show pre-processing results on a single frame, which might be nice for tuning parameters (example: thresholds)

Developing a new plugin could be done from an interactive jupyter session, using Context.run. We could add some debugging/validation there: it could be possible to check if the structure of the analysis results is suitable for integration into the GUI.

Thoughts?

/cc @jhgee