-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SUPPORT] HiveSyncTool failure - Unable to create a _ro
table when writing data
#11254
Comments
shubhamn21
changed the title
[SUPPORT] Fails to create a
[SUPPORT] HiveSyncTool failure - Unable to create a May 17, 2024
_ro
table hive when writing table_ro
table when writing data
@shubhamn21 Please Provide Writer configurations. |
Here it is:
|
I think it may have to do something with AWSGlue compatibility. The documentation said that it is only support upto hudi 0.12.1. As a workaround - I am using |
Closing this as it is no longer an issue. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the problem you faced
Unable to write a hudi table to aws hadoop emr setup.
From the error it seems that it is failing while creating a metadata table (with suffix
_ro
) with hive/glue. Am I missing a setting with hive to allow it create Null type tables? Are there alternative solutions?To Reproduce
Steps to reproduce the behavior:
df.write.format("hudi")\ .mode('append') \ .options(**options)\ .partitionBy("kafka_topic", "event_dt") \ .saveAsTable('db_name.snimbalkar_test_table')
Expected behavior
Creates and stores table.
Environment Description
Hudi version : 0.13.1
Spark version : 3.30
Hive version :
Hadoop version : 3.2.1
Storage (HDFS/S3/GCS..) : EMRFS
Running on Docker? (yes/no) : no
Additional context
Add any other context about the problem here.
Stacktrace
The text was updated successfully, but these errors were encountered: