running modules on subregions of one image

[tischi]

I think there are several reasons why one wants to run one module on a sub-region (not only spatial, but all dimensions) of an image, and I therefore feel we should have some rather generic support for this.

I can see the following use cases:

• You have a multi-channel image and you want to segment nuclei, then you want a module that can segment nuclei to only run on the nucleus channel.
• You have a very large image and you want to check whether certain segmentation algorithms perform well on that image. For that it is useful to be able to run these algorithms only a few small parts (e.g. x,y,z subregions) of that image such that you can quickly visually inspect the results before deciding to segment the whole image with a specific algorithm.
• If modules can (generically) process subregions of an image, this could be used to implement tiled analysis of a large image. In this case the analysis module itself would not have to be touched at all. There would be a module before that generates the tiles (TileGenerator) and then we call the analysis module to work in parallel on those tiles. I think this is what @BioinfoTongLI is doing for some modules, right?
• Another use case could be image registration. Let's say we have a module that's called Find_2D_Affine_Transformation, which finds an affine transformation to register two images. Normally the input to such a module would be two paths to two different 2D images. Using the suggested pattern, it would be now be two paths to images with corresponding ROIs, i.e. four inputs: path1 & roi1, path2 & roi2. Like this, such a module could be readily used to register, for example, two z slices of the same image or two time points of the same image, in which case the paths would be pointing to the same images and only the ROIs would be different.

There may be more use cases, let me know, I would then add them to the above bullet points.

If we agree that we want this, the question is how to best implement this. My naive proposal would be that the input to an analysis module is always the path to an image and the path to a small text file that contains the region of interest that it should work on. I don't know if that's possible in NextFlow, but I would suggest that if this small text file is missing, the default would be to take the whole image as an input. If it is difficult to deal with missing files in Nextflow, we could also always have the ROI file and say if it's empty, then the whole image should be processed by default.

Fractal seems to already implement something along those lines:

https://fractal-analytics-platform.github.io/fractal-tasks-core/tables/#roi-tables

@BioinfoTongLI @krokicki what do you think?

[jluethi]

Given that I saw an issue mentioning Fractal in the BioImageTools repo, I felt summoned ;)

Just to mention that we have a much improved (better documented, but also technically more flexible) implementation that we use in Fractal by now, learn more about it here: https://biovisioncenter.github.io/ngio/stable/table_specs/overview/
We really need to update the old tasks-core docs page to point there.

These new ROIs can be saved as CSV files, Parqeut, AnnData or just plain Json. They contain rectangular bounding boxes in up to 4D (e.g. TZYX). Channel selection usually happens separately (as one almost always needs e.g. named or index-based access for it).

We use those heavily in Fractal tasks and ngio has lots of tooling to efficiently load specific regions or build iterators that go over regions. See for example the Fractal Cellpose SAM task for an up-to-date reference of how these ROIs can be used in processing: https://github.com/fractal-analytics-platform/fractal-cellpose-sam-task

[BioinfoTongLI]

I've got no objections in supporting general slicing (in any dimension) for a module.
My question would be: do we want to enforce this to all future modules? Asking because it comes with a cost.
Off the top of my mind, you will involuntary enforcing:

1. all images should have valid metadata.
2. all images should have correct metadata (e.g channel is C but not Z).
3. for the methods which usually doesn't need slicing, i.e. takes an image of shape (X, Y) and do stuff. Wouldn't it be too much work (support slicing for all dimentions etc), for just a very simple task (e.g. thresholding)?

Great that @jluethi is joining the chat, what's your strategy here? Is slicing by default there for all modules?

[jluethi]

Great that @jluethi is joining the chat, what's your strategy here? Is slicing by default there for all modules?

I think it's a great question where which level of slicing makes sense. The more manual such slicing needs to be handled, the higher the burden on every implementation. But it's very hard to predict which thing that e.g. runs fine in memory in 2D really needs to be sliced when processed in 3D or across time points.

That's why we've added Iterators to ngio: https://biovisioncenter.github.io/ngio/stable/getting_started/6_iterators/
This allows us to have some standard processing patterns where all the complexity of the slicing is handled on the library level and we can ask for option ROI table input. If a user specifies a ROI table, we loop over it, otherwise we process the whole image.
The iterators also allow specifying that e.g. a processing function requires 2D or 3D images => if given a 2D + t or 3D + t OME-Zarr, we'll iterate over all timepoints without the need for specific ROI tables.
Or if a function always requires 2D strictly (e.g. a flatfield correction), but is given a 3D image => we'd automatically iterate over Z planes.

The iterator pattern has proven quite useful to slim down our tasks while actually making them more flexible about what OME-Zarrs can be processed.