Connect to azure datalake gen2

[fermff]

I'm trying to connect to Azure Datalake Gen2. I read in doc that abfss scheme is supported. However, it fails. Is there something I'm missing?

Configuration

Environment Variables

AZURE_STORAGE_ACCOUNT_NAME=xxxxxxstadatalake
AZURE_STORAGE_CLIENT_ID=xxxxxxxxxxxxxxxx
AZURE_STORAGE_CLIENT_SECRET=xxxxxxxxxxxxxxxxxxx
AZURE_STORAGE_TENANT_ID=xxxxxxxxx

YAML

tables:

• name: mytable
  uri: abfss://mycontainer/mg/cli/profile/mytable
  option:
  format: delta

Error Message

Failed to create Roapi context: Whatever { source: Some(TableError { source: Io { source: InvalidUriScheme { scheme: "abfss" } } }), message: "Failed to load table", backtrace: Backtrace(()) }

[fermff]

It seems to me that abfss is not supported? Line 56 of columnq.src.io.mod.rs

[fermff]

I've been digging a bit around. If I add abfss to the list of possible azure types it works, but only if we don't load table into memory.

If I try to load a table into memory I get an error because deltable schema does not match rowbatch schema (the difference is in the nullable attribute of some fields... being correct the ones considered inside RowBatch).

Is there any possibility to specify nullable values in the configuration Yaml?

Cheers