-
Notifications
You must be signed in to change notification settings - Fork 510
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[scio-core](feature) Add readFiles and readFilesWithPath apis #5350
Conversation
desiredBundleSizeBytes: Long, | ||
directoryTreatment: DirectoryTreatment, | ||
compression: Compression |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cannot give default since we have overload.
* @param compression | ||
* Reads files using the given [[org.apache.beam.sdk.io.Compression]]. | ||
*/ | ||
def readFilesWithPath[A: Coder]( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It'd be nice to provide variants for this API for the various IO types to alleviate the user burden of knowing about the underlying beam api and needing to construct the various sources.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added to Text and Avro.
I think we should also add the readFiles
variant for other file IOs.
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #5350 +/- ##
==========================================
- Coverage 62.69% 61.05% -1.64%
==========================================
Files 301 303 +2
Lines 10848 10901 +53
Branches 773 737 -36
==========================================
- Hits 6801 6656 -145
- Misses 4047 4245 +198 ☔ View full report in Codecov by Sentry. |
Add new
readFiles
api based on splittableFileBasedSource
Add new
readFilesWithPath
propagating source file path along with element emitted by the sourcerename
readFiles
toreadTextFiles
for lines apiadd
readAvroGenericFiles
andreadAvroSpecificFiles
apis +WithPath
api variants