Support 4D colored, static images

[rly]

NWB does not support 4D colored, static volumes, but volumetric imaging is becoming more popular in neurophysiology experiments.

Context:

Currently, the format of our data is GCAMP time series images (4D: X,Y,Z, time) and static NeuroPAL colored images (4D: X,Y,Z,RGB) as well as ROIs (neuron centers) and associated metadata. It seems like NWB currently at least has support for the GCAMP time series and the ROIs, but not for static 4D colored volumes. The closest I could find in the pynwb documentation was the pynwb.image.RGBImage class, but that seems to only allow for 3D colored images (X,Y,RGB). I was wondering first if there is another class that I'm not aware of that would allow us to include these volumetric images in an NWB file, and if not, if that was a functionality that you were planning on implementing in NWB in the near future.

Our field is increasingly using volumetric imaging to study whole brain dynamics, and with the advent of NeuroPAL, color has also become an important part of our analysis pipelines. I think it would be hugely beneficial to incorporate flexibility for multi-channel volumetric images into the NWB format. Optimally, there would be flexibility for the 4th color channel dimension to include more than just RGB as well, since we are often imaging a range of different channels that don't always clearly map to just RGB values.

Proposal:

Create a new Volume type and make new GrayscaleVolume , RGBVolume, MultiChannelVolume types as subtypes. We could add Volumes and VolumeReferences as analogs to Images and ImageReferences. Maybe that is too much redundancy, but a volume is not an image and I don’t think inheritance is appropriate there.

I have reached out to the author about what metadata are required for the color channels and will report back here.

[CodyCBakerPhD]

To be clear, that extra color channel is not differentiating a separate 'optic channel' as we normally refer to it?

We've seen ophys data before encoded as an extra appended dimension or even RGB when in actuality it represented grayscale of a single optic stream

[CodyCBakerPhD]

Especially w.r.t.

Optimally, there would be flexibility for the 4th color channel dimension to include more than just RGB as well, since we are often imaging a range of different channels that don't always clearly map to just RGB values.

is that not what we have right now, simply with a separate 3D imaging plane (which could be called a 'volume' in that case) for each separate color channel (which they even say doesn't have to be RGB and thus would require the wavelength info anyway)

[rly]

I asked for clarification, but I think that yes, each color channel is a separate "optical channel".

Yes, we can have separate 3D imaging planes for each separate color channel. But if the x, y, and z dimensions are the same across imaging planes, as it seems to be here, it may be more explicit and efficient to store these data as a 4D array rather than slices of the 4D array. But that's minor and probably it's better to minimize changes.

It seems like we still need a data type to represent single (static / non-time-varying) images associated with multiple imaging planes. We could have a grayscale volumetric TwoPhotonSeries with one time point, but we would need N TwoPhotonSeries, one for each of the N imaging planes (N channels). That seems hacky.

[rly]

From the requester:

Yes, I think 4D: XYZC would be the ideal datatype with flexibility to allow any number of channels (the number of channels will likely be between 3-10 for almost all purposes). In terms of metadata, for each channel we would have the floating point emission value in nanometers, the filter used for that channel which would be formatted as center and width: ie if we're doing tagRFP with emission at 561nm and a bandpass filter at 700nm with width 70nm we would need these three pieces of information in the metadata. We also would want to include what each channel is used for- ie panneuronal marker, pseudocolor green, etc - but that can probably just be included in the freeform text description.

These image volumes are currently stored as image stacks in .tiff files and are in the range of ~50-150 MB.

They would also like to see support for time series of these multichannel volumes (5D: TXYZC). These files would also be stored as stacks of .tiff files but could be on the order of 30-50 GB.

[CodyCBakerPhD]

It seems like we still need a data type to represent single (static / non-time-varying) images associated with multiple imaging planes.

I too am seeing requests for this at the NeuroDataShare, but for grayscale. Particular to being ophys-associated (light sheet microscopy) so having additional ophys metadata is nice, hence my draw towards representing this using photon series + optical channels

So really we just need something like a PhotonImage that has ophys metadata and represents the equivalent of a PhotonSeries with a single frame, which is what I currently recommend for maximum metadata

They would also like to see support for time series of these multichannel volumes (5D: TXYZC)

So, again, isn't this currently possible by splitting the PhotonSeries over each color channel, which is done mainly because each imaging plane of each photon series has a different optical channel, and it's the optical channel that actually encodes that essential metadata about the emission (granted we could improve by including optional fields for the filter, which seems pretty common)

The problem of allowing 5D max dimensions in a photon series is the identifiability of the columns in a 4D series: TXYC vs. TXYZ, which is why we specifically assign optical channels as the 'color' dimension indirectly by encoding as different series and capping the dimensionality at 4D to mean TXYZ

These image volumes are currently stored as image stacks in .tiff files and are in the range of ~50-150 MB.

NeuroConv and ROIExtractors would really appreciate examples of these files, are they willing to share a small file?

[rly]

Agreed on all points.

To be clear, the proposed solution for time series of multichannel volumes, is to have N TwoPhotonSeries objects (TXYZ), one for each of the N imaging planes (N channels). The timestamps and ophys metadata would be the same across the objects, except for the OpticalChannel and maybe a description.