Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support nullable types when reading Avro files [HZ-2507] #24840

Merged
merged 4 commits into from
Jul 28, 2023

Conversation

burakgok
Copy link
Contributor

Specifically, the following statements are affected:

CREATE MAPPING <name> TYPE File OPTIONS ('format'='avro', ...)
TABLE(AVRO_FILE(...))

Fixes HZ-2507

Checklist:

  • Labels (Team:, Type:, Source:, Module:) and Milestone set
  • Label Add to Release Notes or Not Release Notes content set
  • Request reviewers if possible
  • Send backports/forwardports if fix needs to be applied to past/future releases

Specifically, the following statements are affected:
  CREATE MAPPING <name> TYPE File OPTIONS ('format'='avro', ...)
  TABLE(AVRO_FILE(...))
@burakgok burakgok added this to the 5.4.0 milestone Jun 19, 2023
@burakgok burakgok requested a review from k-jamroz June 19, 2023 10:09
@burakgok burakgok self-assigned this Jun 19, 2023
@burakgok burakgok added the SQL-only Use when changes are only in hazelcast-sql module label Jun 20, 2023
@k-jamroz
Copy link
Contributor

k-jamroz commented Jul 3, 2023

Is it a breaking change?

@burakgok burakgok requested a review from frant-hartm July 5, 2023 15:22
@burakgok
Copy link
Contributor Author

This change just enables Avro file mappings and AVRO_FILE function to read unions of a primitive and null. It doesn't break the existing capability of reading primitives.

Comment on lines +190 to +193
@Test
public void test_tableFunction_nullableFields() {
test_tableFunction(AVRO_NULLABLE_FILE);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are missing test reading nulls. This reads nullable fields, which contain values.

# Conflicts:
#	hazelcast-sql/src/main/java/com/hazelcast/jet/sql/impl/connector/keyvalue/KvMetadataAvroResolver.java
#	hazelcast-sql/src/test/java/com/hazelcast/jet/sql/impl/connector/kafka/SqlAvroTest.java
@burakgok burakgok removed the SQL-only Use when changes are only in hazelcast-sql module label Jul 27, 2023
@burakgok burakgok force-pushed the fix/5.4/read-nullable-avro-types branch from 1b281bc to fba7e40 Compare July 27, 2023 15:33
@hazelcast hazelcast deleted a comment from hz-devops-test Jul 27, 2023
@burakgok burakgok merged commit d180dd5 into hazelcast:master Jul 28, 2023
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants