ZeroDivisionError in L70 of api.py

[norlandrhagen]

Hi there,

I'm having an issue when trying to rechunk a 4d zarr store. In the traceback at L70 of algorithm.py (70 headroom = max_mem // chunk_mem), I'm getting the error: ZeroDivisionError: integer division or modulo by zero.

The snippet below should be a MRE.

import xarray as xr
import fsspec
import rechunker 
import zarr 

store_url = "gs://cmip6/CMIP6/CMIP/CCCma/CanESM5/historical/r1i1p1f1/Omon/thetao/gn/v20190429"
ds = xr.open_dataset(store_url, engine='zarr', chunks={}, decode_cf=False)
group = zarr.open_consolidated(store_url)

chunks_dict = {
'i': {'i': 128},
 'j': {'j': 128},
 'latitude': {'i': 128, 'j': 128},
 'lev': {'lev': -1},
 'lev_bnds': {'lev': -1, 'bnds': -1},
 'longitude': {'i': 128, 'j': 128},
 'thetao': {'i': 128, 'j': 128, 'time': 2, 'lev': -1},
 'time': {'time': 2},
 'time_bnds': {'time': 2, 'bnds': -1},
 'vertices_latitude': {'i': 128, 'j': 128, 'vertices': -1},
 'vertices_longitude': {'i': 128, 'j': 128, 'vertices': -1}
}

tmp_mapper = fsspec.get_mapper('temp_store')
tgt_mapper = fsspec.get_mapper('staging_store')

array_plan = rechunker.rechunk(group, chunks_dict, "1000MB", tgt_mapper, temp_store=tmp_mapper)

A few notes:

• It seems like the ZeroDivisionError happens on the thetao var.
• In L70 of algorithm.py, in the headroom = max_mem // chunk_mem calculation, somehow chunk_mem is being assigned to 0 in the above for loop.

    chunk_mem = itemsize * prod(chunks)
    if chunk_mem > max_mem:
        raise ValueError(f"chunk_mem {chunk_mem} > max_mem {max_mem}")
    headroom = max_mem // chunk_mem

    new_chunks = list(chunks)
    # only consolidate over these axes
    axes = sorted(chunk_limit_per_axis.keys())[::-1]
    for n_axis in axes:
        c_new = min(
            chunks[n_axis] * headroom, shape[n_axis], chunk_limit_per_axis[n_axis]
        )
        # print(f'  axis {n_axis}, {chunks[n_axis]} -> {c_new}')
        new_chunks[n_axis] = c_new
        chunk_mem = itemsize * prod(new_chunks)
        headroom = max_mem // chunk_mem

Traceback:

---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
Cell In[11], line 2
      1 # %pdb on 
----> 2 array_plan = rechunker.rechunk(group, chunks_dict, "1000MB", tgt_mapper, temp_store=tmp_mapper)

File ~/opt/anaconda3/envs/install/envs/ncviewjs/lib/python3.10/site-packages/rechunker/api.py:302, in rechunk(source, target_chunks, max_mem, target_store, target_options, temp_store, temp_options, executor)
    299 if isinstance(executor, str):
    300     executor = _get_executor(executor)
--> 302 copy_spec, intermediate, target = _setup_rechunk(
    303     source=source,
    304     target_chunks=target_chunks,
    305     max_mem=max_mem,
    306     target_store=target_store,
    307     target_options=target_options,
    308     temp_store=temp_store,
    309     temp_options=temp_options,
    310 )
    311 plan = executor.prepare_plan(copy_spec)
    312 return Rechunked(executor, plan, source, intermediate, target)

File ~/opt/anaconda3/envs/install/envs/ncviewjs/lib/python3.10/site-packages/rechunker/api.py:457, in _setup_rechunk(source, target_chunks, max_mem, target_store, target_options, temp_store, temp_options)
    454 copy_specs = []
    456 for array_name, array_target_chunks in target_chunks.items():
--> 457     copy_spec = _setup_array_rechunk(
...
---> 70 headroom = max_mem // chunk_mem
     72 if headroom == 1:
     73     break

ZeroDivisionError: integer division or modulo by zero

Results of conda list:

# Name                    Version                   Build  Channel
adal                      1.2.7              pyhd8ed1ab_0    conda-forge
adlfs                     2022.11.2          pyhd8ed1ab_0    conda-forge
aiohttp                   3.8.3           py310h90acd4f_1    conda-forge
aiosignal                 1.3.1              pyhd8ed1ab_0    conda-forge
aiosqlite                 0.18.0                   pypi_0    pypi
alembic                   1.9.1              pyhd8ed1ab_0    conda-forge
anyio                     3.6.2              pyhd8ed1ab_0    conda-forge
appdirs                   1.4.4                    pypi_0    pypi
appnope                   0.1.3              pyhd8ed1ab_0    conda-forge
apprise                   1.2.1                    pypi_0    pypi
argon2-cffi               21.3.0             pyhd8ed1ab_0    conda-forge
argon2-cffi-bindings      21.2.0          py310h90acd4f_3    conda-forge
asciitree                 0.3.3                      py_2    conda-forge
asgi-lifespan             2.0.0                    pypi_0    pypi
asttokens                 2.2.1              pyhd8ed1ab_0    conda-forge
async-timeout             4.0.2              pyhd8ed1ab_0    conda-forge
asyncpg                   0.27.0          py310h90acd4f_1    conda-forge
attrs                     22.2.0             pyh71513ae_0    conda-forge
awscli                    1.27.43         py310h2ec42d9_0    conda-forge
azure-core                1.26.1             pyhd8ed1ab_0    conda-forge
azure-datalake-store      0.0.51             pyh9f0ad1d_0    conda-forge
azure-identity            1.12.0             pyhd8ed1ab_0    conda-forge
azure-storage-blob        12.14.1            pyhd8ed1ab_0    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                pyhd8ed1ab_3    conda-forge
backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
bcrypt                    4.0.1                    pypi_0    pypi
beautifulsoup4            4.11.1             pyha770c72_0    conda-forge
bleach                    5.0.1              pyhd8ed1ab_0    conda-forge
blinker                   1.5                pyhd8ed1ab_0    conda-forge
bokeh                     2.4.3              pyhd8ed1ab_3    conda-forge
boto3                     1.26.43            pyhd8ed1ab_0    conda-forge
botocore                  1.29.43            pyhd8ed1ab_0    conda-forge
brotlipy                  0.7.0           py310h90acd4f_1005    conda-forge
bzip2                     1.0.8                h0d85af4_4    conda-forge
c-ares                    1.18.1               h0d85af4_0    conda-forge
ca-certificates           2022.12.7            h033912b_0    conda-forge
cachetools                5.2.0              pyhd8ed1ab_0    conda-forge
certifi                   2022.12.7          pyhd8ed1ab_0    conda-forge
cf_xarray                 0.7.6              pyhd8ed1ab_0    conda-forge
cffi                      1.15.1          py310ha78151a_3    conda-forge
cfgv                      3.3.1                    pypi_0    pypi
cftime                    1.6.2           py310h936d966_1    conda-forge
charset-normalizer        2.1.1              pyhd8ed1ab_0    conda-forge
click                     8.1.3           unix_pyhd8ed1ab_2    conda-forge
cloudpickle               2.2.0              pyhd8ed1ab_0    conda-forge
colorama                  0.4.4              pyh9f0ad1d_0    conda-forge
comm                      0.1.2              pyhd8ed1ab_0    conda-forge
commonmark                0.9.1                    pypi_0    pypi
contourpy                 1.0.6                    pypi_0    pypi
coolname                  2.2.0                    pypi_0    pypi
croniter                  1.3.8                    pypi_0    pypi
cryptography              39.0.0          py310hdd0c95c_0    conda-forge
cycler                    0.11.0                   pypi_0    pypi
cytoolz                   0.12.0          py310h90acd4f_1    conda-forge
dask                      2022.12.1          pyhd8ed1ab_0    conda-forge
dask-core                 2022.12.1          pyhd8ed1ab_0    conda-forge
dateparser                1.1.5                    pypi_0    pypi
debugpy                   1.6.5           py310h7a76584_0    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
dictdiffer                0.9.0              pyhd8ed1ab_0    conda-forge
distlib                   0.3.6                    pypi_0    pypi
distributed               2022.12.1          pyhd8ed1ab_0    conda-forge
docker                    6.0.1                    pypi_0    pypi
docutils                  0.15.2          py310h2ec42d9_6    conda-forge
entrypoints               0.4                pyhd8ed1ab_0    conda-forge
executing                 1.2.0              pyhd8ed1ab_0    conda-forge
fastapi                   0.88.0             pyhd8ed1ab_0    conda-forge
fasteners                 0.17.3             pyhd8ed1ab_0    conda-forge
fastprogress              1.0.3                    pypi_0    pypi
filelock                  3.9.0                    pypi_0    pypi
flit-core                 3.8.0              pyhd8ed1ab_0    conda-forge
fonttools                 4.38.0                   pypi_0    pypi
freetype                  2.12.1               h3f81eb7_1    conda-forge
frozenlist                1.3.3           py310h90acd4f_0    conda-forge
fsspec                    2022.11.0          pyhd8ed1ab_0    conda-forge
gcsfs                     2022.11.0          pyhd8ed1ab_0    conda-forge
gettext                   0.21.1               h8a4c099_0    conda-forge
glib                      2.74.1               hbc0c0cd_1    conda-forge
glib-tools                2.74.1               hbc0c0cd_1    conda-forge
google-api-core           2.11.0             pyhd8ed1ab_0    conda-forge
google-auth               2.15.0             pyh1a96a4e_0    conda-forge
google-auth-oauthlib      0.8.0              pyhd8ed1ab_0    conda-forge
google-cloud-core         2.3.2              pyhd8ed1ab_0    conda-forge
google-cloud-storage      2.7.0              pyh1a96a4e_0    conda-forge
google-crc32c             1.1.2           py310h6f7428f_4    conda-forge
google-resumable-media    2.4.0              pyhd8ed1ab_0    conda-forge
googleapis-common-protos  1.57.1             pyhd8ed1ab_0    conda-forge
greenlet                  2.0.1           py310h7a76584_0    conda-forge
griffe                    0.25.3                   pypi_0    pypi
grpcio                    1.51.1          py310h0f91fd3_0    conda-forge
gst-plugins-base          1.21.3               h37e1711_1    conda-forge
gstreamer                 1.21.3               h1d18e73_1    conda-forge
gunicorn                  20.1.0          py310h2ec42d9_3    conda-forge
h11                       0.14.0             pyhd8ed1ab_0    conda-forge
h2                        4.1.0                    pypi_0    pypi
heapdict                  1.0.1                      py_0    conda-forge
hpack                     4.0.0                    pypi_0    pypi
httpcore                  0.16.3                   pypi_0    pypi
httpx                     0.23.3                   pypi_0    pypi
hyperframe                6.0.1                    pypi_0    pypi
icu                       70.1                 h96cf925_0    conda-forge
identify                  2.5.12                   pypi_0    pypi
idna                      3.4                pyhd8ed1ab_0    conda-forge
importlib-metadata        6.0.0              pyha770c72_0    conda-forge
importlib_resources       5.10.2             pyhd8ed1ab_0    conda-forge
intake                    0.6.6                    pypi_0    pypi
intake-esm                2022.9.18                pypi_0    pypi
ipykernel                 6.19.4             pyh736e0ef_0    conda-forge
ipython                   8.8.0              pyhd1c38e8_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
ipywidgets                8.0.4              pyhd8ed1ab_0    conda-forge
isodate                   0.6.1              pyhd8ed1ab_0    conda-forge
jedi                      0.18.2             pyhd8ed1ab_0    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
jmespath                  1.0.1              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   hac89ed1_2    conda-forge
jsonpatch                 1.32                     pypi_0    pypi
jsonpointer               2.3                      pypi_0    pypi
jsonschema                4.17.3             pyhd8ed1ab_0    conda-forge
jupyter                   1.0.0           py310h2ec42d9_8    conda-forge
jupyter_client            7.4.8              pyhd8ed1ab_0    conda-forge
jupyter_console           6.4.4              pyhd8ed1ab_0    conda-forge
jupyter_core              5.1.2           py310h2ec42d9_0    conda-forge
jupyter_events            0.4.0              pyhd8ed1ab_0    conda-forge
jupyter_server            2.0.6              pyhd8ed1ab_0    conda-forge
jupyter_server_terminals  0.4.3              pyhd8ed1ab_0    conda-forge
jupyterlab_pygments       0.2.2              pyhd8ed1ab_0    conda-forge
jupyterlab_widgets        3.0.5              pyhd8ed1ab_0    conda-forge
kiwisolver                1.4.4                    pypi_0    pypi
krb5                      1.20.1               h049b76e_0    conda-forge
kubernetes                25.3.0                   pypi_0    pypi
lcms2                     2.14                 h29502cd_1    conda-forge
lerc                      4.0.0                hb486fe8_0    conda-forge
libabseil                 20220623.0      cxx17_h844d122_6    conda-forge
libblas                   3.9.0           16_osx64_openblas    conda-forge
libcblas                  3.9.0           16_osx64_openblas    conda-forge
libclang                  13.0.1          default_he082bbe_0    conda-forge
libcrc32c                 1.1.2                he49afe7_0    conda-forge
libcxx                    14.0.6               hccf4f1f_0    conda-forge
libdeflate                1.14                 hb7f2c08_0    conda-forge
libedit                   3.1.20191231         h0678c8f_2    conda-forge
libffi                    3.4.2                h0d85af4_5    conda-forge
libgfortran               5.0.0           11_3_0_h97931a8_27    conda-forge
libgfortran5              11.3.0              h082f757_27    conda-forge
libglib                   2.74.1               h4c723e1_1    conda-forge
libgrpc                   1.51.1               h966d1d5_0    conda-forge
libiconv                  1.17                 hac89ed1_0    conda-forge
libjpeg-turbo             2.1.4                hb7f2c08_0    conda-forge
liblapack                 3.9.0           16_osx64_openblas    conda-forge
libllvm13                 13.0.1               h64f94b2_2    conda-forge
libogg                    1.3.4                h35c211d_1    conda-forge
libopenblas               0.3.21          openmp_h429af6e_3    conda-forge
libopus                   1.3.1                hc929b4f_1    conda-forge
libpng                    1.6.39               ha978bb4_0    conda-forge
libpq                     15.1                 h3640bf0_2    conda-forge
libprotobuf               3.21.12              hbc0c0cd_0    conda-forge
libsodium                 1.0.18               hbcb3906_1    conda-forge
libsqlite                 3.40.0               ha978bb4_0    conda-forge
libtiff                   4.5.0                h6268bbc_0    conda-forge
libvorbis                 1.3.7                h046ec9c_0    conda-forge
libwebp-base              1.2.4                h775f41a_0    conda-forge
libxcb                    1.13              h0d85af4_1004    conda-forge
libzlib                   1.2.13               hfd90126_4    conda-forge
llvm-openmp               15.0.6               h61d9ccf_0    conda-forge
locket                    1.0.0              pyhd8ed1ab_0    conda-forge
lz4                       4.2.0           py310h8d4e1d9_0    conda-forge
lz4-c                     1.9.3                he49afe7_1    conda-forge
mako                      1.2.4              pyhd8ed1ab_0    conda-forge
markdown                  3.4.1                    pypi_0    pypi
markupsafe                2.1.1           py310h90acd4f_2    conda-forge
matplotlib                3.6.2                    pypi_0    pypi
matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
mistune                   2.0.4              pyhd8ed1ab_0    conda-forge
msal                      1.20.0             pyhd8ed1ab_0    conda-forge
msal_extensions           1.0.0              pyhd8ed1ab_0    conda-forge
msgpack-python            1.0.4           py310ha23aa8a_1    conda-forge
msrest                    0.7.1              pyhd8ed1ab_0    conda-forge
multidict                 6.0.4           py310h90acd4f_0    conda-forge
mypy_extensions           0.4.3           py310h2ec42d9_6    conda-forge
mysql-common              8.0.31               hc4b2c72_0    conda-forge
mysql-libs                8.0.31               h8658499_0    conda-forge
nbclassic                 0.4.8              pyhd8ed1ab_0    conda-forge
nbclient                  0.7.2              pyhd8ed1ab_0    conda-forge
nbconvert                 7.2.7              pyhd8ed1ab_0    conda-forge
nbconvert-core            7.2.7              pyhd8ed1ab_0    conda-forge
nbconvert-pandoc          7.2.7              pyhd8ed1ab_0    conda-forge
nbformat                  5.7.1              pyhd8ed1ab_0    conda-forge
ncurses                   6.3                  h96cf925_1    conda-forge
ncviewjs-backend          0.0.post28               pypi_0    pypi
nest-asyncio              1.5.6              pyhd8ed1ab_0    conda-forge
netcdf4                   1.6.2                    pypi_0    pypi
nodeenv                   1.7.0                    pypi_0    pypi
notebook                  6.5.2              pyha770c72_1    conda-forge
notebook-shim             0.2.2              pyhd8ed1ab_0    conda-forge
nspr                      4.35                 hea0b92c_0    conda-forge
nss                       3.78                 ha8197d3_0    conda-forge
numcodecs                 0.11.0          py310h7a76584_1    conda-forge
numpy                     1.24.1          py310h1b7c290_0    conda-forge
oauthlib                  3.2.2              pyhd8ed1ab_0    conda-forge
openjpeg                  2.5.0                h13ac156_2    conda-forge
openssl                   3.0.7                hfd90126_1    conda-forge
orjson                    3.8.5                    pypi_0    pypi
packaging                 22.0               pyhd8ed1ab_0    conda-forge
pandas                    1.5.2           py310hecf8f37_0    conda-forge
pandoc                    2.19.2               h694c41f_1    conda-forge
pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
paramiko                  2.12.0                   pypi_0    pypi
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
partd                     1.3.0              pyhd8ed1ab_0    conda-forge
pathspec                  0.10.3                   pypi_0    pypi
pcre2                     10.40                h1c4e4bc_0    conda-forge
pendulum                  2.1.2                    pypi_0    pypi
pexpect                   4.8.0              pyh1a96a4e_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    9.4.0           py310hab5364c_0    conda-forge
pip                       22.3.1             pyhd8ed1ab_0    conda-forge
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_0    conda-forge
platformdirs              2.6.2              pyhd8ed1ab_0    conda-forge
ply                       3.11                       py_1    conda-forge
portalocker               2.6.0           py310h2ec42d9_1    conda-forge
pre-commit                2.21.0                   pypi_0    pypi
prefect                   2.7.7                    pypi_0    pypi
prometheus_client         0.15.0             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.36             pyha770c72_0    conda-forge
prompt_toolkit            3.0.36               hd8ed1ab_0    conda-forge
protobuf                  4.21.12         py310h7a76584_0    conda-forge
psutil                    5.9.4           py310h90acd4f_0    conda-forge
psycopg2                  2.9.3           py310h31f19fa_2    conda-forge
pthread-stubs             0.4               hc929b4f_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
pyasn1                    0.4.8                      py_0    conda-forge
pyasn1-modules            0.2.7                      py_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pydantic                  1.10.4          py310h90acd4f_0    conda-forge
pygments                  2.14.0             pyhd8ed1ab_0    conda-forge
pyjwt                     2.6.0              pyhd8ed1ab_0    conda-forge
pynacl                    1.5.0                    pypi_0    pypi
pyopenssl                 23.0.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9                    pypi_0    pypi
pyqt                      5.15.7          py310hdd03f62_2    conda-forge
pyqt5-sip                 12.11.0         py310h415000c_2    conda-forge
pyrsistent                0.19.3          py310h90acd4f_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.10.8          he7542f4_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-fastjsonschema     2.16.2             pyhd8ed1ab_0    conda-forge
python-json-logger        2.0.4              pyhd8ed1ab_0    conda-forge
python-slugify            7.0.0                    pypi_0    pypi
python_abi                3.10                    3_cp310    conda-forge
pytz                      2022.7             pyhd8ed1ab_0    conda-forge
pytz-deprecation-shim     0.1.0.post0              pypi_0    pypi
pytzdata                  2020.1                   pypi_0    pypi
pyu2f                     0.1.5              pyhd8ed1ab_0    conda-forge
pyyaml                    5.4.1           py310h90acd4f_4    conda-forge
pyzmq                     24.0.1          py310hf615a82_1    conda-forge
qt-main                   5.15.6               haeff654_5    conda-forge
qtconsole                 5.4.0              pyhd8ed1ab_0    conda-forge
qtconsole-base            5.4.0              pyha770c72_0    conda-forge
qtpy                      2.3.0              pyhd8ed1ab_0    conda-forge
re2                       2022.06.01           hb486fe8_1    conda-forge
readchar                  4.0.3                    pypi_0    pypi
readline                  8.1.2                h3899abd_0    conda-forge
rechunker                 0.5.0              pyhd8ed1ab_0    conda-forge
regex                     2022.10.31               pypi_0    pypi
requests                  2.28.1             pyhd8ed1ab_1    conda-forge
requests-oauthlib         1.3.1              pyhd8ed1ab_0    conda-forge
rfc3986                   1.5.0                    pypi_0    pypi
rich                      13.0.0                   pypi_0    pypi
rsa                       4.7.2              pyh44b312d_0    conda-forge
ruamel.yaml               0.17.21         py310h90acd4f_2    conda-forge
ruamel.yaml.clib          0.2.7           py310h90acd4f_1    conda-forge
s3fs                      0.4.2                      py_0    conda-forge
s3transfer                0.6.0              pyhd8ed1ab_0    conda-forge
send2trash                1.8.0              pyhd8ed1ab_0    conda-forge
setuptools                65.6.3             pyhd8ed1ab_0    conda-forge
sip                       6.7.5           py310h7a76584_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
skyplane                  0.2.1                    pypi_0    pypi
sniffio                   1.3.0              pyhd8ed1ab_0    conda-forge
sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
soupsieve                 2.3.2.post1        pyhd8ed1ab_0    conda-forge
sqlalchemy                1.4.41          py310h90acd4f_0    conda-forge
sqlalchemy2-stubs         0.0.2a31           pyhd8ed1ab_0    conda-forge
sqlite                    3.40.0               h9ae0607_0    conda-forge
sqlmodel                  0.0.8              pyhd8ed1ab_0    conda-forge
sshtunnel                 0.4.0                    pypi_0    pypi
stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
starlette                 0.22.0             pyhd8ed1ab_0    conda-forge
tblib                     1.7.0              pyhd8ed1ab_0    conda-forge
terminado                 0.17.1             pyhd1c38e8_0    conda-forge
text-unidecode            1.3                      pypi_0    pypi
tinycss2                  1.2.1              pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h5dbffcc_0    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
tornado                   6.2             py310h90acd4f_1    conda-forge
traitlets                 5.8.0              pyhd8ed1ab_0    conda-forge
typer                     0.7.0                    pypi_0    pypi
typing-extensions         4.4.0                hd8ed1ab_0    conda-forge
typing_extensions         4.4.0              pyha770c72_0    conda-forge
tzdata                    2022.7                   pypi_0    pypi
tzlocal                   4.2                      pypi_0    pypi
universal_pathlib         0.0.21             pyh191b570_0    conda-forge
urllib3                   1.26.13            pyhd8ed1ab_0    conda-forge
uvicorn                   0.20.0          py310h2ec42d9_1    conda-forge
virtualenv                20.17.1                  pypi_0    pypi
wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
webencodings              0.5.1                      py_1    conda-forge
websocket-client          1.4.2              pyhd8ed1ab_0    conda-forge
wheel                     0.38.4             pyhd8ed1ab_0    conda-forge
widgetsnbextension        4.0.5              pyhd8ed1ab_0    conda-forge
xarray                    2022.12.0          pyhd8ed1ab_0    conda-forge
xorg-libxau               1.0.9                h35c211d_0    conda-forge
xorg-libxdmcp             1.1.3                h35c211d_0    conda-forge
xz                        5.2.6                h775f41a_0    conda-forge
yaml                      0.2.5                h0d85af4_2    conda-forge
yarl                      1.8.2           py310h90acd4f_0    conda-forge
zarr                      2.13.3             pyhd8ed1ab_0    conda-forge
zeromq                    4.3.4                he49afe7_1    conda-forge
zict                      2.2.0              pyhd8ed1ab_0    conda-forge
zipp                      3.11.0             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               hfd90126_4    conda-forge
zstd                      1.5.2                hfa58983_4    conda-forge

Wondering if anyone has any thoughts on why this might be happening.

Thanks!

[rabernat]

Thanks for sharing. Can reproduce it even more simply with

target_chunks = (2, -1, 128, 128)
rechunker.rechunk(group['thetao'], target_chunks, "1000MB", tgt_mapper, temp_store=tmp_mapper)

I'll investigate.

[rabernat]

So it looks like we can fix it with

target_chunks = (2, 45, 128, 128)

My impression is that we don't support the syntax -1 to mean "the full size of the dimension." This could easily be fixed. Or we could raise an error if you pass a chunk shape as a negative number.

[norlandrhagen]

Thanks for the wildly quick response @rabernat!

Didn't realize the -1 means the full size on the dim wasn't supported. Either of those options sounds like an improvement to me!

[rabernat]

PR welcome! 😉

The check would go here:

rechunker/rechunker/algorithm.py

Line 68 in 46db807

new_chunks = list(chunks)

Just replace the special value of -1 with the corresponding shape (i.e. swap -1 for 45 like I did above), something like

new_chunks = [s if c == -1 else c for c, s in zip(chunks, shape)]

[norlandrhagen]

Sounds great! I'll open up a PR.