BioArgo #211

iuryt · 2022-04-22T21:07:33Z

iuryt
Apr 22, 2022

Hi,

Is BioArgo available for ArgoPy?
I would like to use it to help me to define initial conditions for a biogeochemical model.
Cool stuff!

gmaze · 2022-04-25T07:28:41Z

gmaze
Apr 25, 2022
Maintainer

Hi @iuryt
BGC-Argo data are not yet supported by argopy, but are accessible in "expert" mode using the "gdac" data source.
In other words, you may be able to use argopy to get BGC Argo data, but there's no guarantee this would work at all or as expected ...
We're working on this and hope to roll out an argopy release this year with BGC support
Still need some patience here

Example of code to load BGC data, without argopy support:

from argopy import DataFetcher as ArgoDataFetcher
argo = ArgoDataFetcher(src='erddap', ds='bgc', mode='expert').float(5906217)
argo.data

4 replies

iuryt Apr 25, 2022
Author

Hi @gmaze ,
It is awesome that you are implementing this.
I understand that you mentioned that it has no support by now, but the example you gave returns error when access the data.

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
/tmp/ipykernel_129495/1974750946.py in <module>
      1 from argopy import DataFetcher as ArgoDataFetcher
      2 argo = ArgoDataFetcher(src='erddap', ds='bgc', mode='expert').float(5906217)
----> 3 argo.data

.
.
.
.

FileNotFoundError: https://www.ifremer.fr/erddap/tabledap/ArgoFloats-bio.nc?data_mode,latitude,longitude,position_qc,time,time_qc,direction,platform_number,cycle_number,config_mission_number,vertical_sampling_scheme,pres,temp,psal,doxy,pres_qc,temp_qc,psal_qc,doxy_qc,pres_adjusted,temp_adjusted,psal_adjusted,doxy_adjusted,pres_adjusted_qc,temp_adjusted_qc,psal_adjusted_qc,doxy_adjusted_qc,pres_adjusted_error,temp_adjusted_error,psal_adjusted_error,doxy_adjusted_error&platform_number=~"5906217"&distinct()&orderBy("time,pres")

It looks like this is happening because there is no "ArgoFloats-bio.nc" in the table. Is it?
https://erddap.ifremer.fr/erddap/tabledap/index.html?page=1&itemsPerPage=1000

gmaze Apr 26, 2022
Maintainer

That is probably because you're not using the last version of argopy, try to update to 0.1.11, otherwise this is something else.

You may also use the parallel option to make it go through:

argo = ArgoDataFetcher(src='erddap', ds='bgc', mode='expert', parallel=True).float(5906217)

iuryt Apr 28, 2022
Author

I am not relying on this for my research, but just by curiosity, I tried to run

argo = ArgoDataFetcher(src='erddap', ds='bgc', mode='expert', parallel=True).float(5906217)
argo.data

which returned:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/tmp/ipykernel_47196/1814280123.py in <module>
      1 argo = ArgoDataFetcher(src='erddap', ds='bgc', mode='expert', parallel=True).float(5906217)
----> 2 argo.data

/autofs/nas1/home/isimoesdesousa/programs/mambaforge/envs/pangeo-notebook/lib/python3.9/site-packages/argopy/fetchers.py in data(self)
    248         """
    249         if not isinstance(self._data, xr.Dataset):
--> 250             self.load()
    251         return self._data
    252 

/autofs/nas1/home/isimoesdesousa/programs/mambaforge/envs/pangeo-notebook/lib/python3.9/site-packages/argopy/fetchers.py in load(self, force, **kwargs)
    561         if not self._loaded or force:
    562             # Fetch measurements:
--> 563             self._data = self.to_xarray(**kwargs)
    564             # Next 2 lines must come before ._index because to_index(full=False) calls back on .load() to read .data
    565             self._request = self.__repr__()  # Save definition of loaded data

/autofs/nas1/home/isimoesdesousa/programs/mambaforge/envs/pangeo-notebook/lib/python3.9/site-packages/argopy/fetchers.py in to_xarray(self, **kwargs)
    424                 % ",".join(self.Fetchers.keys())
    425             )
--> 426         xds = self.fetcher.to_xarray(**kwargs)
    427         xds = self.postproccessor(xds)
    428 

/autofs/nas1/home/isimoesdesousa/programs/mambaforge/envs/pangeo-notebook/lib/python3.9/site-packages/argopy/data_fetchers/erddap_data.py in to_xarray(self, errors)
    468         else:
    469             try:
--> 470                 ds = self.fs.open_mfdataset(
    471                     self.uri, method=self.parallel_method, progress=self.progress, errors=errors
    472                 )

/autofs/nas1/home/isimoesdesousa/programs/mambaforge/envs/pangeo-notebook/lib/python3.9/site-packages/argopy/stores/filesystems.py in open_mfdataset(self, urls, max_workers, method, progress, concat, concat_dim, preprocess, preprocess_opts, errors, *args, **kwargs)
    632                 return results
    633         elif len(failed) == len(urls):
--> 634             raise ValueError("Errors happened with all URLs, this could be due to an internal impossibility to read returned content.")
    635         else:
    636             raise DataNotFound(urls)

ValueError: Errors happened with all URLs, this could be due to an internal impossibility to read returned content.

I also tried specifying the region (tried a region that I know there are some BioArgos) with

bgc = ArgoDataFetcher(src='erddap', ds='bgc', mode='expert').region([-38, -15, 50, 62, 0, 1000, '2000-01', '2022-05'])
bgc.data

and it returned the following error:

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
/tmp/ipykernel_47196/2648734437.py in <module>
      1 bgc = ArgoDataFetcher(src='erddap', ds='bgc', mode='expert').region([-38, -15, 50, 62, 0, 1000, '2000-01', '2022-05'])
----> 2 bgc.data

/autofs/nas1/home/isimoesdesousa/programs/mambaforge/envs/pangeo-notebook/lib/python3.9/site-packages/argopy/fetchers.py in data(self)
    248         """
    249         if not isinstance(self._data, xr.Dataset):
--> 250             self.load()
    251         return self._data
    252 

/autofs/nas1/home/isimoesdesousa/programs/mambaforge/envs/pangeo-notebook/lib/python3.9/site-packages/argopy/fetchers.py in load(self, force, **kwargs)
    561         if not self._loaded or force:
    562             # Fetch measurements:
--> 563             self._data = self.to_xarray(**kwargs)
    564             # Next 2 lines must come before ._index because to_index(full=False) calls back on .load() to read .data
    565             self._request = self.__repr__()  # Save definition of loaded data

/autofs/nas1/home/isimoesdesousa/programs/mambaforge/envs/pangeo-notebook/lib/python3.9/site-packages/argopy/fetchers.py in to_xarray(self, **kwargs)
    424                 % ",".join(self.Fetchers.keys())
    425             )
--> 426         xds = self.fetcher.to_xarray(**kwargs)
    427         xds = self.postproccessor(xds)
    428 

/autofs/nas1/home/isimoesdesousa/programs/mambaforge/envs/pangeo-notebook/lib/python3.9/site-packages/argopy/data_fetchers/erddap_data.py in to_xarray(self, errors)
    456             if len(self.uri) == 1:
    457                 try:
--> 458                     ds = self.fs.open_dataset(self.uri[0])
    459                 except ClientResponseError as e:
    460                     raise ErddapServerError(e.message)

/autofs/nas1/home/isimoesdesousa/programs/mambaforge/envs/pangeo-notebook/lib/python3.9/site-packages/argopy/stores/filesystems.py in open_dataset(self, url, *args, **kwargs)
    424         """
    425         try:
--> 426             data = self.fs.cat_file(url)
    427         except aiohttp.ClientResponseError as e:
    428             if e.status == 413:

/autofs/nas1/home/isimoesdesousa/programs/mambaforge/envs/pangeo-notebook/lib/python3.9/site-packages/fsspec/asyn.py in wrapper(*args, **kwargs)
     89     def wrapper(*args, **kwargs):
     90         self = obj or args[0]
---> 91         return sync(self.loop, func, *args, **kwargs)
     92 
     93     return wrapper

/autofs/nas1/home/isimoesdesousa/programs/mambaforge/envs/pangeo-notebook/lib/python3.9/site-packages/fsspec/asyn.py in sync(loop, func, timeout, *args, **kwargs)
     69         raise FSTimeoutError from return_result
     70     elif isinstance(return_result, BaseException):
---> 71         raise return_result
     72     else:
     73         return return_result

/autofs/nas1/home/isimoesdesousa/programs/mambaforge/envs/pangeo-notebook/lib/python3.9/site-packages/fsspec/asyn.py in _runner(event, coro, result, timeout)
     23         coro = asyncio.wait_for(coro, timeout=timeout)
     24     try:
---> 25         result[0] = await coro
     26     except Exception as ex:
     27         result[0] = ex

/autofs/nas1/home/isimoesdesousa/programs/mambaforge/envs/pangeo-notebook/lib/python3.9/site-packages/fsspec/implementations/http.py in _cat_file(self, url, start, end, **kwargs)
    224         async with session.get(url, **kw) as r:
    225             out = await r.read()
--> 226             self._raise_not_found_for_status(r, url)
    227         return out
    228 

/autofs/nas1/home/isimoesdesousa/programs/mambaforge/envs/pangeo-notebook/lib/python3.9/site-packages/fsspec/implementations/http.py in _raise_not_found_for_status(self, response, url)
    205         """
    206         if response.status == 404:
--> 207             raise FileNotFoundError(url)
    208         response.raise_for_status()
    209 

FileNotFoundError: [https://www.ifremer.fr/erddap/tabledap/ArgoFloats-synthetic-BGC.nc?parameter_data_mode,latitude,longitude,position_qc,time,time_qc,direction,platform_number,cycle_number,config_mission_number,pres,temp,psal,cndc,doxy,beta_backscattering,fluorescence_chla,bbp,turbidity,cp,chla,cdom,nitrate,pres_qc,temp_qc,psal_qc,cndc_qc,doxy_qc,beta_backscattering_qc,fluorescence_chla_qc,bbp_qc,turbidity_qc,cp_qc,chla_qc,cdom_qc,nitrate_qc,pres_adjusted,temp_adjusted,psal_adjusted,cndc_adjusted,doxy_adjusted,beta_backscattering_adjusted,fluorescence_chla_adjusted,bbp_adjusted,turbidity_adjusted,cp_adjusted,chla_adjusted,cdom_adjusted,nitrate_adjusted,pres_adjusted_qc,temp_adjusted_qc,psal_adjusted_qc,cndc_adjusted_qc,doxy_adjusted_qc,beta_backscattering_adjusted_qc,fluorescence_chla_adjusted_qc,bbp_adjusted_qc,turbidity_adjusted_qc,cp_adjusted_qc,chla_adjusted_qc,cdom_adjusted_qc,nitrate_adjusted_qc,pres_adjusted_error,temp_adjusted_error,psal_adjusted_error,cndc_adjusted_error,doxy_adjusted_error,beta_backscattering_adjusted_error,fluorescence_chla_adjusted_error,bbp_adjusted_error,turbidity_adjusted_error,cp_adjusted_error,chla_adjusted_error,cdom_adjusted_error,nitrate_adjusted_error&longitude>=-38&longitude<=-15&latitude>=50&latitude<=62&pres>=0&pres<=1000&time>=946684800.0&time<=1651363200.0&distinct()&orderBy](https://www.ifremer.fr/erddap/tabledap/ArgoFloats-synthetic-BGC.nc?parameter_data_mode,latitude,longitude,position_qc,time,time_qc,direction,platform_number,cycle_number,config_mission_number,pres,temp,psal,cndc,doxy,beta_backscattering,fluorescence_chla,bbp,turbidity,cp,chla,cdom,nitrate,pres_qc,temp_qc,psal_qc,cndc_qc,doxy_qc,beta_backscattering_qc,fluorescence_chla_qc,bbp_qc,turbidity_qc,cp_qc,chla_qc,cdom_qc,nitrate_qc,pres_adjusted,temp_adjusted,psal_adjusted,cndc_adjusted,doxy_adjusted,beta_backscattering_adjusted,fluorescence_chla_adjusted,bbp_adjusted,turbidity_adjusted,cp_adjusted,chla_adjusted,cdom_adjusted,nitrate_adjusted,pres_adjusted_qc,temp_adjusted_qc,psal_adjusted_qc,cndc_adjusted_qc,doxy_adjusted_qc,beta_backscattering_adjusted_qc,fluorescence_chla_adjusted_qc,bbp_adjusted_qc,turbidity_adjusted_qc,cp_adjusted_qc,chla_adjusted_qc,cdom_adjusted_qc,nitrate_adjusted_qc,pres_adjusted_error,temp_adjusted_error,psal_adjusted_error,cndc_adjusted_error,doxy_adjusted_error,beta_backscattering_adjusted_error,fluorescence_chla_adjusted_error,bbp_adjusted_error,turbidity_adjusted_error,cp_adjusted_error,chla_adjusted_error,cdom_adjusted_error,nitrate_adjusted_error&longitude%3E=-38&longitude%3C=-15&latitude%3E=50&latitude%3C=62&pres%3E=0&pres%3C=1000&time%3E=946684800.0&time%3C=1651363200.0&distinct()&orderBy)("time,pres")

As I said, I am not relying on this for my research and just posted here because it might help somehow the development of this part.

gmaze Apr 29, 2022
Maintainer

Ok, so if you wish to debug why you can't get these data while I can, this discussion must be moved to issues,
raise a 👍🏻 if you want me to do so

Otherwise, I think I answered your question with a simple "be patient" 😉
Thanks for pointing again this important need for argopy support of BGC data

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BioArgo #211

{{title}}

Replies: 1 comment 4 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

BioArgo #211

iuryt Apr 22, 2022

Replies: 1 comment · 4 replies

gmaze Apr 25, 2022 Maintainer

iuryt Apr 25, 2022 Author

gmaze Apr 26, 2022 Maintainer

iuryt Apr 28, 2022 Author

gmaze Apr 29, 2022 Maintainer

iuryt
Apr 22, 2022

Replies: 1 comment 4 replies

gmaze
Apr 25, 2022
Maintainer

iuryt Apr 25, 2022
Author

gmaze Apr 26, 2022
Maintainer

iuryt Apr 28, 2022
Author

gmaze Apr 29, 2022
Maintainer