Accessing ATL11

[whyjz]

Hello folks! I am running the following code and get the error message as follows. Does anyone know what went wrong? Thanks a lot! 🙏

region_a = ipx.Query("ATL11", [-87, 74.7, -85, 75.4], ['2018-09-01','2022-01-01'])
region_a.order_granules() 

We are already authenticated with NASA EDL
Total number of data order requests is  1  for  22  granules.
Data request  1  of  1  is submitting to NSIDC
order ID:  5000004558591
Initial status of your order request at NSIDC is:  processing
Your order status is still  processing  at NSIDC. Please continue waiting... this may take a few moments.
Your order is:  failed
NSIDC provided these error messages:
['All granules failed. 22 granules reported processing messages.',
 'P0Y0M0DT0H0M3.205S',
 'ICESAT2',
 '282375161:InternalError-An internal error occured. Exit code -11',
 '282376530:InternalError-An internal error occured. Exit code -11',
 '282375465:InternalError-An internal error occured. Exit code -11',
 '282375494:InternalError-An internal error occured. Exit code -11',
 '282374724:InternalError-An internal error occured. Exit code -11',
 '282380635:InternalError-An internal error occured. Exit code -11',
 '282387079:InternalError-An internal error occured. Exit code -11',
 '282377403:InternalError-An internal error occured. Exit code -11',
 '282385276:InternalError-An internal error occured. Exit code -11',
 '282386380:InternalError-An internal error occured. Exit code -11',
 '282380237:InternalError-An internal error occured. Exit code -11',
 '282386733:InternalError-An internal error occured. Exit code -11',
 '282378352:InternalError-An internal error occured. Exit code -11',
 '282386464:InternalError-An internal error occured. Exit code -11',
 '282380960:InternalError-An internal error occured. Exit code -11',
 '282386790:InternalError-An internal error occured. Exit code -11',
 '282392310:InternalError-An internal error occured. Exit code -11',
 '282391643:InternalError-An internal error occured. Exit code -11',
 '282388378:InternalError-An internal error occured. Exit code -11',
 '282388171:InternalError-An internal error occured. Exit code -11',
 '282391897:InternalError-An internal error occured. Exit code -11',
 '282392103:InternalError-An internal error occured. Exit code -11']
Request failed.
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[34], line 2
      1 region_a = ipx.Query("ATL11", [-87, 74.7, -85, 75.4], ['2018-09-01','2022-01-01'])
----> 2 region_a.order_granules() 

File /srv/conda/envs/notebook/lib/python3.10/site-packages/icepyx/core/query.py:1062, in Query.order_granules(self, verbose, subset, email, **kwargs)
   1051         self._granules.place_order(
   1052             tempCMRparams,
   1053             self.reqparams,
   (...)
   1058             geom_filepath=self._spatial._geom_file,
   1059         )
   1061 else:
-> 1062     self._granules.place_order(
   1063         self.CMRparams,
   1064         self.reqparams,
   1065         self.subsetparams(**kwargs),
   1066         verbose,
   1067         subset,
   1068         session=self.session,
   1069         geom_filepath=self._spatial._geom_file,
   1070     )

File /srv/conda/envs/notebook/lib/python3.10/site-packages/icepyx/core/granules.py:465, in Granules.place_order(self, CMRparams, reqparams, subsetparams, verbose, subset, session, geom_filepath)
    463 # --- Output the final orderIDs
    464 with open(order_fn, "w") as fid:
--> 465     json.dump({"orderIDs": self.orderIDs}, fid)
    467 return self.orderIDs

AttributeError: 'Granules' object has no attribute 'orderIDs'

[andypbarrett]

Hi Whyjay

NSIDC Ops took a look at the logs but do not have any more information than you are seeing here. I've copied their message to me here

These kinds of errors usually occur as a result of a mismatch between the user's subset area and the actual data in the file, or that data being all "fill" values rather than actual data. Your best bet is to look at the data and see if the request makes sense - i.e the temporal and spatial extent actually contains data from the files. As a start, I attempted an identical request without the temporal constraint and it completed successfully, so that's the most likely culprit.

I suggest running the query without the temporal constraint to see if you can recreate their result. Then maybe check a few of the ATL11 files to see if there is non-missing data.

Andy Barrett
NSIDC

[whyjz]

Thanks @andypbarrett for the response!

I checked the ATL11 files I manually downloaded and confirmed that there are actual data, so to me it's likely because a temporal constraint is not accepted during the ATL-11 query process.

However, a temporal constraint is required to query ATL-11 data if using Icepyx. I did several attempts and could not find a way to search ATL-11 without the temporal constraints. I wonder if there's a way to rewire the query step...

[CryoKatOnIce]

I have a similar issue with ATL06 and using a simple shapefile for Greenland, only some data are download (some months in the time range), but when checked with OpenAltimetry the data for missing months exists. Also, I know from using previous versions that the data for that time period exists. Manual download would be a chore. Any ideas on how to solve this?

[JessicaS11]

Hey @whyjz and @CryoKatOnIce! An interesting issue you've raised here (and sorry for the slow response - GitHub didn't send me a notification that the discussion was started!)

@andypbarrett thanks for digging in on the NSIDC side.

I suggest running the query without the temporal constraint to see if you can recreate their result.

I've been digging into the icepyx code and I suspect the issue is - as Andy suggests - that the temporal parameter is submitted to NSIDC - when I manually edit a url to remove the time piece it provides links to download the data granules without error. However, in the process of trying to come up with a workaround (e.g. region_a.CMRparams.pop("temporal"), in between the Query and order_granules), I discovered that somehow the spatial parameter is actually being submitted twice, using the CMR and EGI keys (see #488). If anyone here is able to dig into #488 it may help us brainstorm a fix (sooner) for the ATL11 case (one initial thought is to check the 'temporalSubsetting' value as seen in region_a.show_custom_options() against the about-to-be submitted parameters to see if any will be invalid).

@CryoKatOnIce, could you share the code you're using to submit your order? It would be helpful to recreate your issue in order to dig into it (and since ATL06 DOES have temporal subsetting, my naive guess is that you may be facing a different problem).

[JessicaS11]

@whyjz I implemented a fix for this issue in #515 (which is ready for review). You'll still have to include a temporal parameter when you create your query object, but neither of the temporal keywords will be submitted to NSIDC so you should be able to order and download data without any issues.

@andypbarrett Do you know if this is a concern for any other ICESat-2 products?