Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

olci_l1b-reader and xarray=2023.12.0 are seemingly incompatible #2705

Open
fwfichtner opened this issue Dec 21, 2023 · 16 comments
Open

olci_l1b-reader and xarray=2023.12.0 are seemingly incompatible #2705

fwfichtner opened this issue Dec 21, 2023 · 16 comments
Assignees
@mraspaud
Labels
bug component:readers

Comments

@fwfichtner
Copy link
Contributor

Describe the bug
See below, this is resolved by pinning xarray to an earlier version

To Reproduce

import satpy
import xarray as xr

from satpy import find_files_and_readers

print("xarray version: ", xr.__version__)
print("satpy version: ", satpy.__version__)

if __name__ == '__main__':
    files = find_files_and_readers(
        sensor="olci",
        base_dir="/Users/username/Downloads/SENTINEL3-OLCI-4ikNLD92G6",
        reader="olci_l1b",
    )

    scene = satpy.Scene(filenames=files, reader="olci_l1b", reader_kwargs={'engine': 'netcdf4'})
    print(scene.available_dataset_names())

Expected behavior

Results in:

xarray version:  2023.11.0
satpy version:  0.46.0
/opt/homebrew/Caskroom/miniforge/base/envs/stac/lib/python3.11/site-packages/xarray/core/dataset.py:278: UserWarning: The specified chunks separate the stored chunks along dimension "columns" starting at index 4096. This could degrade performance. Instead, consider rechunking after loading.
  warnings.warn(
['Oa01', 'Oa02', 'Oa03', 'Oa04', 'Oa05', 'Oa06', 'Oa07', 'Oa08', 'Oa09', 'Oa10', 'Oa11', 'Oa12', 'Oa13', 'Oa14', 'Oa15', 'Oa16', 'Oa17', 'Oa18', 'Oa19', 'Oa20', 'Oa21', 'altitude', 'humidity', 'latitude', 'longitude', 'satellite_azimuth_angle', 'satellite_zenith_angle', 'sea_level_pressure', 'solar_azimuth_angle', 'solar_zenith_angle', 'total_columnar_water_vapour', 'total_ozone']

Actual results
Text output of actual results or error messages including full tracebacks if applicable.

xarray version:  2023.12.0
satpy version:  0.46.0
[DEBUG: 2023-12-21 12:42:51 : satpy.readers.yaml_reader] Reading ('/opt/homebrew/Caskroom/miniforge/base/envs/stac/lib/python3.11/site-packages/satpy/etc/readers/olci_l1b.yaml',)
[DEBUG: 2023-12-21 12:42:51 : satpy.readers.yaml_reader] Reading ('/opt/homebrew/Caskroom/miniforge/base/envs/stac/lib/python3.11/site-packages/satpy/etc/readers/olci_l1b.yaml',)
[DEBUG: 2023-12-21 12:42:51 : satpy.readers.yaml_reader] Assigning to olci_l1b: ['/Users/username/Downloads/SENTINEL3-OLCI-4ikNLD92G6/S3A_OL_1_EFR____20191203T230931_20191203T231231_20191204T004026_0179_052_158_3420_LN1_O_NR_002.SEN3/Oa13_radiance.nc', '/Users/username/Downloads/SENTINEL3-OLCI-4ikNLD92G6/S3A_OL_1_EFR____20191203T230931_20191203T231231_20191204T004026_0179_052_158_3420_LN1_O_NR_002.SEN3/Oa08_radiance.nc', '/Users/username/Downloads/SENTINEL3-OLCI-4ikNLD92G6/S3A_OL_1_EFR____20191203T230931_20191203T231231_20191204T004026_0179_052_158_3420_LN1_O_NR_002.SEN3/Oa19_radiance.nc', '/Users/username/Downloads/SENTINEL3-OLCI-4ikNLD92G6/S3A_OL_1_EFR____20191203T230931_20191203T231231_20191204T004026_0179_052_158_3420_LN1_O_NR_002.SEN3/Oa05_radiance.nc', '/Users/username/Downloads/SENTINEL3-OLCI-4ikNLD92G6/S3A_OL_1_EFR____20191203T230931_20191203T231231_20191204T004026_0179_052_158_3420_LN1_O_NR_002.SEN3/Oa03_radiance.nc', '/Users/username/Downloads/SENTINEL3-OLCI-4ikNLD92G6/S3A_OL_1_EFR____20191203T230931_20191203T231231_20191204T004026_0179_052_158_3420_LN1_O_NR_002.SEN3/Oa02_radiance.nc', '/Users/username/Downloads/SENTINEL3-OLCI-4ikNLD92G6/S3A_OL_1_EFR____20191203T230931_20191203T231231_20191204T004026_0179_052_158_3420_LN1_O_NR_002.SEN3/Oa18_radiance.nc', '/Users/username/Downloads/SENTINEL3-OLCI-4ikNLD92G6/S3A_OL_1_EFR____20191203T230931_20191203T231231_20191204T004026_0179_052_158_3420_LN1_O_NR_002.SEN3/Oa17_radiance.nc', '/Users/username/Downloads/SENTINEL3-OLCI-4ikNLD92G6/S3A_OL_1_EFR____20191203T230931_20191203T231231_20191204T004026_0179_052_158_3420_LN1_O_NR_002.SEN3/Oa07_radiance.nc', '/Users/username/Downloads/SENTINEL3-OLCI-4ikNLD92G6/S3A_OL_1_EFR____20191203T230931_20191203T231231_20191204T004026_0179_052_158_3420_LN1_O_NR_002.SEN3/Oa15_radiance.nc', '/Users/username/Downloads/SENTINEL3-OLCI-4ikNLD92G6/S3A_OL_1_EFR____20191203T230931_20191203T231231_20191204T004026_0179_052_158_3420_LN1_O_NR_002.SEN3/Oa09_radiance.nc', '/Users/username/Downloads/SENTINEL3-OLCI-4ikNLD92G6/S3A_OL_1_EFR____20191203T230931_20191203T231231_20191204T004026_0179_052_158_3420_LN1_O_NR_002.SEN3/Oa04_radiance.nc', '/Users/username/Downloads/SENTINEL3-OLCI-4ikNLD92G6/S3A_OL_1_EFR____20191203T230931_20191203T231231_20191204T004026_0179_052_158_3420_LN1_O_NR_002.SEN3/Oa16_radiance.nc', '/Users/username/Downloads/SENTINEL3-OLCI-4ikNLD92G6/S3A_OL_1_EFR____20191203T230931_20191203T231231_20191204T004026_0179_052_158_3420_LN1_O_NR_002.SEN3/Oa14_radiance.nc', '/Users/username/Downloads/SENTINEL3-OLCI-4ikNLD92G6/S3A_OL_1_EFR____20191203T230931_20191203T231231_20191204T004026_0179_052_158_3420_LN1_O_NR_002.SEN3/Oa11_radiance.nc', '/Users/username/Downloads/SENTINEL3-OLCI-4ikNLD92G6/S3A_OL_1_EFR____20191203T230931_20191203T231231_20191204T004026_0179_052_158_3420_LN1_O_NR_002.SEN3/Oa06_radiance.nc', '/Users/username/Downloads/SENTINEL3-OLCI-4ikNLD92G6/S3A_OL_1_EFR____20191203T230931_20191203T231231_20191204T004026_0179_052_158_3420_LN1_O_NR_002.SEN3/Oa21_radiance.nc', '/Users/username/Downloads/SENTINEL3-OLCI-4ikNLD92G6/S3A_OL_1_EFR____20191203T230931_20191203T231231_20191204T004026_0179_052_158_3420_LN1_O_NR_002.SEN3/Oa01_radiance.nc', '/Users/username/Downloads/SENTINEL3-OLCI-4ikNLD92G6/S3A_OL_1_EFR____20191203T230931_20191203T231231_20191204T004026_0179_052_158_3420_LN1_O_NR_002.SEN3/Oa12_radiance.nc', '/Users/username/Downloads/SENTINEL3-OLCI-4ikNLD92G6/S3A_OL_1_EFR____20191203T230931_20191203T231231_20191204T004026_0179_052_158_3420_LN1_O_NR_002.SEN3/Oa20_radiance.nc', '/Users/username/Downloads/SENTINEL3-OLCI-4ikNLD92G6/S3A_OL_1_EFR____20191203T230931_20191203T231231_20191204T004026_0179_052_158_3420_LN1_O_NR_002.SEN3/Oa10_radiance.nc', '/Users/username/Downloads/SENTINEL3-OLCI-4ikNLD92G6/S3A_OL_1_EFR____20191203T230931_20191203T231231_20191204T004026_0179_052_158_3420_LN1_O_NR_002.SEN3/tie_geometries.nc', '/Users/username/Downloads/SENTINEL3-OLCI-4ikNLD92G6/S3A_OL_1_EFR____20191203T230931_20191203T231231_20191204T004026_0179_052_158_3420_LN1_O_NR_002.SEN3/geo_coordinates.nc', '/Users/username/Downloads/SENTINEL3-OLCI-4ikNLD92G6/S3A_OL_1_EFR____20191203T230931_20191203T231231_20191204T004026_0179_052_158_3420_LN1_O_NR_002.SEN3/instrument_data.nc', '/Users/username/Downloads/SENTINEL3-OLCI-4ikNLD92G6/S3A_OL_1_EFR____20191203T230931_20191203T231231_20191204T004026_0179_052_158_3420_LN1_O_NR_002.SEN3/tie_meteo.nc']
/opt/homebrew/Caskroom/miniforge/base/envs/stac/lib/python3.11/site-packages/xarray/namedarray/core.py:487: UserWarning: Duplicate dimension names present: dimensions {'bands'} appear more than once in dims=('bands', 'bands'). We do not yet support duplicate dimension names, but we do allow initial construction of the object. We recommend you rename the dims immediately to become distinct, as most xarray functionality is likely to fail silently if you do not. To rename the dimensions you will need to set the ``.dims`` attribute of each variable, ``e.g. var.dims=('x0', 'x1')``.
  warnings.warn(
/opt/homebrew/Caskroom/miniforge/base/envs/stac/lib/python3.11/site-packages/xarray/core/dataset.py:282: UserWarning: The specified chunks separate the stored chunks along dimension "columns" starting at index 4096. This could degrade performance. Instead, consider rechunking after loading.
  warnings.warn(
Traceback (most recent call last):
  File "/Users/username/PycharmProjects/pythonProject/satpy_stuff.py", line 18, in <module>
    scene = satpy.Scene(filenames=files, reader="olci_l1b", reader_kwargs={'engine': 'netcdf4'})
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/stac/lib/python3.11/site-packages/satpy/scene.py", line 152, in __init__
    self._readers = self._create_reader_instances(filenames=filenames,
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/stac/lib/python3.11/site-packages/satpy/scene.py", line 173, in _create_reader_instances
    return load_readers(filenames=filenames,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/stac/lib/python3.11/site-packages/satpy/readers/__init__.py", line 575, in load_readers
    reader_instance.create_filehandlers(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/stac/lib/python3.11/site-packages/satpy/readers/yaml_reader.py", line 616, in create_filehandlers
    filehandlers = self._new_filehandlers_for_filetype(filetype_info,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/stac/lib/python3.11/site-packages/satpy/readers/yaml_reader.py", line 604, in _new_filehandlers_for_filetype
    return list(filtered_iter)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/stac/lib/python3.11/site-packages/satpy/readers/yaml_reader.py", line 572, in filter_fh_by_metadata
    for filehandler in filehandlers:
  File "/opt/homebrew/Caskroom/miniforge/base/envs/stac/lib/python3.11/site-packages/satpy/readers/yaml_reader.py", line 513, in _new_filehandler_instances
    yield filetype_cls(filename, filename_info, filetype_info, *req_fh, **fh_kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/stac/lib/python3.11/site-packages/satpy/readers/olci_nc.py", line 172, in __init__
    self.cal = cal.nc
               ^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/stac/lib/python3.11/functools.py", line 1001, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/stac/lib/python3.11/site-packages/satpy/readers/olci_nc.py", line 121, in nc
    dataset = xr.open_dataset(f_obj,
              ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/stac/lib/python3.11/site-packages/xarray/backends/api.py", line 578, in open_dataset
    ds = _dataset_from_backend_dataset(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/stac/lib/python3.11/site-packages/xarray/backends/api.py", line 371, in _dataset_from_backend_dataset
    ds = _chunk_ds(
         ^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/stac/lib/python3.11/site-packages/xarray/backends/api.py", line 336, in _chunk_ds
    variables[name] = _maybe_chunk(
                      ^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/stac/lib/python3.11/site-packages/xarray/core/dataset.py", line 327, in _maybe_chunk
    var = var.chunk(
          ^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/stac/lib/python3.11/site-packages/xarray/core/variable.py", line 1042, in chunk
    chunks = {self.get_axis_num(dim): chunk for dim, chunk in chunks.items()}
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/stac/lib/python3.11/site-packages/xarray/core/variable.py", line 1042, in <dictcomp>
    chunks = {self.get_axis_num(dim): chunk for dim, chunk in chunks.items()}
              ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/stac/lib/python3.11/site-packages/xarray/namedarray/core.py", line 661, in get_axis_num
    return self._get_axis_num(dim)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/stac/lib/python3.11/site-packages/xarray/namedarray/core.py", line 664, in _get_axis_num
    _raise_if_any_duplicate_dimensions(self.dims)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/stac/lib/python3.11/site-packages/xarray/namedarray/core.py", line 867, in _raise_if_any_duplicate_dimensions
    raise ValueError(
ValueError: This function cannot handle duplicate dimensions, but dimensions {'bands'} appear more than once on this object's dims: ('bands', 'bands')

Environment Info:

  • OS: OSX, Linux
  • Satpy Version: 0.46.0
@djhoese
Copy link
Member

djhoese commented Dec 21, 2023

Could you try loading one of your files directly without satpy by doing:

import xarray as xr

dataset = xr.open_dataset("/path/to/your/file.nc", decode_cf=True, mask_and_scale=True, chunks="auto")

And see if that fails? As far as I can tell nothing in Satpy is referencing a "bands" dimension in this reader or before so this seems like something that changed in xarray that isn't compatible with the files. That would mean a fix in xarray is required.

Can you also confirm what the latest (most recent) version of xarray is where this works?

@fwfichtner
Copy link
Contributor Author

You are right, it also fails. It does work with 2023.11.0.

It's instrument_data.nc that fails to open, the other files work.

I should have checked this before, feel free to close this here.

@djhoese
Copy link
Member

djhoese commented Dec 21, 2023

Could you file a bug with xarray and link to it here? Could you also for the record paste a ncdump -h of the failing file here? I don't understand what's broken just from the error message.

@fwfichtner
Copy link
Contributor Author

Yes, but I will have to do so tomorrow. That's the dump:

netcdf instrument_data {
dimensions:
	bands = 21 ;
	columns = 4865 ;
	detectors = 3700 ;
	rows = 1953 ;
variables:
	float FWHM(bands, detectors) ;
		FWHM:_FillValue = -1.f ;
		FWHM:ancillary_variables = "detector_index lambda0" ;
		FWHM:long_name = "OLCI bandwidth (Full Widths at Half Maximum)" ;
		FWHM:units = "nm" ;
		FWHM:valid_max = 650.f ;
		FWHM:valid_min = 0.f ;
	short detector_index(rows, columns) ;
		detector_index:_FillValue = -1s ;
		detector_index:coordinates = "time_stamp altitude latitude longitude" ;
		detector_index:long_name = "Detector index" ;
		detector_index:valid_max = 3699s ;
		detector_index:valid_min = 0s ;
	byte frame_offset(rows, columns) ;
		frame_offset:_FillValue = -128b ;
		frame_offset:long_name = "Re-sampling along-track frame offset" ;
		frame_offset:valid_max = 15b ;
		frame_offset:valid_min = -15b ;
	float lambda0(bands, detectors) ;
		lambda0:_FillValue = -1.f ;
		lambda0:ancillary_variables = "detector_index FWHM" ;
		lambda0:long_name = "OLCI characterised central wavelength" ;
		lambda0:units = "nm" ;
		lambda0:valid_max = 1040.f ;
		lambda0:valid_min = 390.f ;
	float relative_spectral_covariance(bands, bands) ;
		relative_spectral_covariance:_FillValue = NaNf ;
		relative_spectral_covariance:ancillary_variables = "lambda0" ;
		relative_spectral_covariance:long_name = "Relative spectral covariance matrix" ;
	float solar_flux(bands, detectors) ;
		solar_flux:_FillValue = -1.f ;
		solar_flux:ancillary_variables = "detector_index lambda0" ;
		solar_flux:long_name = "In-band solar irradiance, seasonally corrected" ;
		solar_flux:units = "mW.m-2.nm-1" ;
		solar_flux:valid_max = 2500.f ;
		solar_flux:valid_min = 500.f ;

// global attributes:
		:absolute_orbit_number = 29437U ;
		:ac_subsampling_factor = 64US ;
		:al_subsampling_factor = 1US ;
		:comment = " " ;
		:contact = "eosupport@copernicus.esa.int" ;
		:creation_time = "2023-12-20T07:20:24Z" ;
		:history = "  2023-12-20T07:20:24Z: PUGCoreProcessor JobOrder.3302865.xml" ;
		:institution = "PS2" ;
		:netCDF_version = "4.2 of Jan 13 2023 10:05:23 $" ;
		:processing_baseline = "OL__L1_.003.03.01" ;
		:product_name = "S3B_OL_1_EFR____20231220T045944_20231220T050110_20231220T072024_0085_087_290_1980_PS2_O_NR_003.SEN3" ;
		:references = "S3IPF PDS 004.1 - i2r6 - Product Data Format Specification - OLCI Level 1, S3IPF PDS 002 - i1r8 - Product Data Format Specification - Product Structures, S3IPF DPM 002 - i2r9 - Detailed Processing Model - OLCI Level 1" ;
		:resolution = "[ 270 294 ]" ;
		:source = "IPF-OL-1-EO 06.17" ;
		:start_time = "2023-12-20T04:59:43.719978Z" ;
		:stop_time = "2023-12-20T05:01:09.611725Z" ;
		:title = "OLCI Level 1b Product, Instrument Data Set" ;
}
`
``

@djhoese
Copy link
Member

djhoese commented Dec 21, 2023 

	float relative_spectral_covariance(bands, bands) ;
		relative_spectral_covariance:_FillValue = NaNf ;
		relative_spectral_covariance:ancillary_variables = "lambda0" ;
		relative_spectral_covariance:long_name = "Relative spectral covariance matrix" ;

Oh yeah so xarray doesn't like this. Maybe this is a bad assumption on xarray's part that a variable can't have the same dimension twice. I could see how that breaks their model. I assume this is an accurate description from the OLCI file's point of view.

@mraspaud this was your reader, right?

@djhoese
Copy link
Member

djhoese commented Dec 25, 2023

@fwfichtner Were you able to file a bug with xarray?

@fwfichtner
Copy link
Contributor Author

@djhoese nope, I noticed that they did quite some changes to handling dimensions in the recent update and that it takes me a bit longer to understand what their roadmap was that I was willing to spend during the holidays. Reference: pydata/xarray#8491

@fwfichtner
Copy link
Contributor Author

fwfichtner commented Jan 2, 2024 •
edited

A happy new year @djhoese! I opened an issue upstream pydata/xarray#8579 and I am not sure how to continue from there, but I guess this has to be changed here accordingly.

@mraspaud
Copy link
Member

mraspaud commented Jan 9, 2024

@fwfichtner Thanks for filing the issue. Nice that the xarray team acknowledges this as a problem.

I see you have a workaround for the problem in that issue. Is this something we need to implement in the OLCI reader or is it ok for you to pin xarray and wait for the fix on their side?

@fwfichtner
Copy link
Contributor Author

Yes, great @mraspaud, I was a little fast accepting our fate here. I currently pinned xarray and I am happy to wait until this is fixed assuming that this will happen within some foreseeable future.

@simonrp84
Copy link
Member

Any updates on this? Still seems to be broken today...

@fwfichtner
Copy link
Contributor Author

Not that I know, we have to wait if anything happens in pydata/xarray#8579. We currently pinned xarray to 2023.11.0

@mraspaud
Copy link
Member

mraspaud commented Apr 9, 2024

Looks like it's not really prioritized on the xarray side, so if we want things to move, we probably have get our hands dirty and give a go at a pull request there...

@fwfichtner
Copy link
Contributor Author

I don't have a lot of time to pick that up myself in the coming weeks, but would you be able to pinpoint what exactly needs to be done in xarray? I guess it is not just removing the raise.

@varisht-tathya
Copy link

varisht-tathya commented May 30, 2024 •
edited

Any updates on this issue other than rolling back xarray? I am facing the same issue with Sentinel 3 OLCI

@mraspaud
Copy link
Member

@varisht-tathya we are still waiting on pydata/xarray#8579 to be solved... which won't happen unless someone takes the time to work on it...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mraspaud mraspaud

Labels
bug component:readers
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@mraspaud @djhoese @fwfichtner @simonrp84 @varisht-tathya