Added support for dynamically getting delta table features #428
Conversation
| @@ -0,0 +1,3 @@ | |||
| { | |||
There was a problem hiding this comment.
Can you update the .gitignore with .vscode/* as well?
| InternalSchema.MetadataValue.MICROS); | ||
| metadata = DEFAULT_TIMESTAMP_PRECISION_METADATA; | ||
| break; | ||
| case "timestamp_ntz": |
There was a problem hiding this comment.
The test class should also get a small update https://github.com/apache/incubator-xtable/blob/main/core/src/test/java/org/apache/xtable/delta/TestDeltaSchemaExtractor.java
| @@ -268,9 +281,23 @@ private void commitTransaction() { | |||
| } | |||
|
|
|||
| private Map<String, String> getConfigurationsForDeltaSync() { | |||
|
|
|||
There was a problem hiding this comment.
We can test this here
I would expect we can add the timestamp_ntz and see that the delta table has the correct versions set for min reader/writer?
| // a private variable to of deltaTable | ||
|
|
||
| // limit the results to the attributes needed | ||
| Dataset<Row> record = deltaTable.detail().select("minWriterVersion", "minReaderVersion"); |
There was a problem hiding this comment.
Can you get this information from deltaLog.snapshot().metadata()?
There was a problem hiding this comment.
I dont think so. I looked through the api for the deltaLog.snapshot().metadata(), and did not find a referance to it. The only place i found it at was here https://docs.delta.io/latest/delta-utility.html
The test creates a new df with its partition field set as the timestamp ntz type. IF this is set, then the table should be upgraded to a reader and writer version of 3,7.
| @@ -57,7 +57,7 @@ | |||
| import org.junit.jupiter.params.ParameterizedTest; | |||
| import org.junit.jupiter.params.provider.Arguments; | |||
| import org.junit.jupiter.params.provider.MethodSource; | |||
|
|
|||
| import org.apache.spark.sql.delta.DeltaConfigs; | |||
There was a problem hiding this comment.
@ForeverAngry Can you remove these changes to this file if they are no longer required?
Important Read
To address: Issue #354
What is the purpose of the pull request
This pull request implements dynamically capturing the min reader and writer version of a target delta table, based on its table details.
Brief change log
Verify this pull request
This pull request is already covered by existing tests, i think...
This is my first PR to an open source project - i was a bit unsure of how to do the unit test for this change. Any pointers or insight would be welcomed :) !