Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] STRING Schema topic sink failed #183

Open
ethqunzhong opened this issue Dec 13, 2022 · 2 comments
Open

[Bug] STRING Schema topic sink failed #183

ethqunzhong opened this issue Dec 13, 2022 · 2 comments

Comments

@ethqunzhong
Copy link

I tried to run iceberg sink demo follow with pulsar-io-lakehouse sink docs. it fail commit record because getSchema result unexcepted.

Describe the bug
my test flow shows below:

  1. create topic & produce data
    first, I produce lots of data to test topic persistent://public/default/iceberg_test by Flink-connector.
    image

message format like:
22061772,1670896138459,mmdc-bigdata-test,11.156.128.57,jobmanager,11.156.128.75,2022-12-13 09:48:58,29
and set topic-schema with bin/pulsar-admin schemas upload command.
therefore,test-topic schema show below:
image
2. Run the lakehouse sink connector
image
logs shows sink iceberg failed with schema exception.
image
there are two question:

  1. why getSchemaType result different from these two ways:
    image
  2. record.getSchema().getSchemaInfo().getSchemaDefinition()=null
    image so records will skiped in sinkWriter.run
    image
    and i found that in getSchemaDefinition
    image
    if SchemaType=STRING/BYTES, it's SchemaDefinition will always be null cause sink failed.

Environment

  • Pulsar version: 2.9.3
  • Deployment: On-premises cluster
  • pulsar-io-lakehouse-connector version: 2.9.3.16

4 broker & 1 function-worker (run as a separate process in separate machines.)

@ethqunzhong
Copy link
Author

@hangc0276 @zymap @danpi PTAL.

@danpi
Copy link
Contributor

danpi commented Dec 15, 2022

@hangc0276 @zymap @danpi PTAL.

OK, I will take a look.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants