Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UDF computations on overlapping ROIs with a common merge step #651

Closed
uellue opened this issue Feb 26, 2020 · 3 comments · May be fixed by #764
Closed

UDF computations on overlapping ROIs with a common merge step #651

uellue opened this issue Feb 26, 2020 · 3 comments · May be fixed by #764
Assignees
Labels
design enhancement New feature or request WIP
Milestone

Comments

@uellue
Copy link
Member

uellue commented Feb 26, 2020

Discussions with https://github.com/Ptychography-4-0/ptychography/ team:

Ptychography reconstruction requires partial reconstruction and stitching of regions that are close and compact in sample coordinates and not necessarily in the "natural order" and current partitioning of the dataset, which could be a random scan with temporal ordering.

To enable distributed ptychography, it should be possible to run an UDF in such a way that data that belongs to a global ROI is passed as one data block to a reconstruction algorithm, i.e. "process_partition()" The local result is then merged into a global result buffer with a non-trivial merge function.

The actual reconstruction and merge can be swapped out as needed and the size and shape of regions can be adapted to the "sweet spot" of various algorithms and hardware as needed.

@uellue uellue added enhancement New feature or request design labels Feb 26, 2020
@uellue
Copy link
Member Author

uellue commented Feb 26, 2020

Actually, it should be possible to pass a list of ROIs to define "virtual partitions" that can be processed in parallel / arbitrary order.

@uellue
Copy link
Member Author

uellue commented Feb 27, 2020

Refs Ptychography-4-0/ptychography#10
Refs Ptychography-4-0/ptychography#8 for merge function that is stable for arbitrary overlapping ROIs, i.e. repeated processing
Refs Ptychography-4-0/ptychography#12 as a simple test algorithm to validate the system

@uellue uellue changed the title UDF computations on overlapping ROIs that with a common merge step UDF computations on overlapping ROIs with a common merge step Feb 27, 2020
@sk1p sk1p self-assigned this Mar 5, 2020
@sk1p sk1p added the WIP label Mar 5, 2020
@sk1p sk1p added this to the 0.5 milestone Mar 11, 2020
@uellue uellue modified the milestones: 0.5, 0.6 Apr 7, 2020
@sk1p sk1p modified the milestones: 0.6, 0.7 Jun 10, 2020
@uellue uellue modified the milestones: 0.7, 0.8 May 27, 2021
@sk1p sk1p modified the milestones: 0.8, backlog Aug 24, 2021
@uellue
Copy link
Member Author

uellue commented Jul 28, 2022

Just like #981, this can be solved in an "outer loop" outside of LiberTEM for now . Closing until specific needs arise!

@uellue uellue closed this as completed Jul 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
design enhancement New feature or request WIP
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants