Unity Catalog Support

[dhirschfeld]

I'm opening this to discuss the state of Unity Catalog support in delta-rs.

In my ideal world I'd be able get a DeltaTable object from Unity Catalog, with permissions controlled by UC access policies.

[dhirschfeld]

xref:

• fix: remove the get_data_catalog() function #1941

[dhirschfeld]

Extract the session credentials from your cluster

• https://kb.databricks.com/en_US/notebooks/access-s3-temp-session-token

[rtyler]

I have had a number of discussions with Databricks on the topic. The 🦀 code has support for table lookup but it's kind of useless IMHO without credential vending support from the Unity APIs. Basically a user today would need to have an access token for Unity and their own separate storage credentials for accessing the table in object storage.

What is missing is a public API from Databricks which would allow the actor running delta-rs to retrieve credentials for use in accessing storage.

So yes, I would like to see this too, but I'm marking this on-hold until Databricks provides us the necessary REST APIs for Unity Catalog

[dhirschfeld]

Thanks for the info @rtyler. I'll continue to monitor this thread in the hope that something changes on the Databricks side...

[wmoreiraa]

Hi sir, @rtyler. Did Databricks gave any updates on the last months on this topic? I think we'all on databricks ecosystem are waiting for this.

[nicornk]

https://github.com/duckdb/uc_catalog/blob/4a2ba143eb8686becf69f759caa731742c2d7c81/src/uc_api.cpp#L120

There seems to be an API now to get temporary storage credentials?

[ion-elgreco]

Those APIs aren't public yet if you check the docs. But it's promising

[ion-elgreco]

Actually they seems already to be working : ) I'll start working on it soon ^^

[nicornk]

I also found zero google results on the APIs, just wanted to share this here.

[ion-elgreco]

No luck:

{'error_code': 'UNAUTHENTICATED',
 'message': "Request to generate access credential for table '<redacted>' from outside of Databricks Unity Catalog enabled compute environment is denied for security. Please contact Databricks support for integrations with Unity Catalog.",
 'details': [{'@type': 'type.googleapis.com/google.rpc.ErrorInfo',
   'reason': 'UNITY_CATALOG_EXTERNAL_GENERATE_TABLE_CREDENTIALS_DENIED',
   'domain': 'unity-catalog.databricks.com',
   'metadata': {'tableId': '<redacted>'}},
  {'@type': 'type.googleapis.com/google.rpc.RequestInfo',
   'request_id': '<redacted>',
   'serving_data': ''}]}