Feature for Daffodil DFDL Data querying #2835
Assignees
Labels
doc-impacting
PRs that affect the documentation
enhancement
PRs that add a new functionality to Drill
new-format
New Format Plugin
Comments
cgivre
added
enhancement
|
Very excited to see this! DFDL + Drill is really a great combo. |
mbeckerle
added a commit
to mbeckerle/drill
that referenced
this issue
Oct 11, 2023
POMs changed. New format-daffodil module created (by copy/rename on format-xml module) Maven seems to be happy dependency-wise, but we won't know until we try to compile actual java code using daffodil APIs. apache#2835
mbeckerle
added a commit
to mbeckerle/drill
that referenced
this issue
Oct 11, 2023
mbeckerle
added a commit
to mbeckerle/drill
that referenced
this issue
Dec 22, 2023
3.7.0-SNAPSHOT of Daffodil which has metadata support we're using. New format-daffodil module created Still uses absolute paths for the schemaFileURI. (which is cheating. Wouldn't work in a true distributed drill environment.) We have yet to work out how to enable Drill to provide access for DFDL schemas in XML form with include/import to be resolved. The input data stream is, however, being accessed in the proper Drill manner. Gunzip happened automatically. Nice. Note: Fix boxed Boolean vs. boolean problem. Don't use boxed primitives in Format config objects. Test show this works for data as complex as having nested repeating sub-records. These DFDL types are supported: - int - long - short - byte - boolean - double - float (does not work. Bug DAFFODIL-2367) - hexBinary - string apache#2835
mbeckerle
added a commit
to mbeckerle/drill
that referenced
this issue
Dec 22, 2023
3.7.0-SNAPSHOT of Daffodil which has metadata support we're using. New format-daffodil module created Still uses absolute paths for the schemaFileURI. (which is cheating. Wouldn't work in a true distributed drill environment.) We have yet to work out how to enable Drill to provide access for DFDL schemas in XML form with include/import to be resolved. The input data stream is, however, being accessed in the proper Drill manner. Gunzip happened automatically. Nice. Note: Fix boxed Boolean vs. boolean problem. Don't use boxed primitives in Format config objects. Test show this works for data as complex as having nested repeating sub-records. These DFDL types are supported: - int - long - short - byte - boolean - double - float (does not work. Bug DAFFODIL-2367) - hexBinary - string apache#2835
|
PR updated: #2836
Not done: Distribution of schemas (or compiled schemas) across drill's computation fabric. Testing of nillability, and real queries against realistic DFDL schemas are needed. |
mbeckerle
added a commit
to mbeckerle/drill
that referenced
this issue
Apr 27, 2024
Requires Daffodil version 3.7.0 or higher. New format-daffodil module created Still uses absolute paths for the schemaFileURI. (which is cheating. Wouldn't work in a true distributed drill environment.) We have yet to work out how to enable Drill to provide access for DFDL schemas in XML form with include/import to be resolved. The input data stream is, however, being accessed in the proper Drill manner. Gunzip happened automatically. Nice. Note: Fix boxed Boolean vs. boolean problem. Don't use boxed primitives in Format config objects. Test show this works for data as complex as having nested repeating sub-records. These DFDL types are supported: - int - long - short - byte - boolean - double - float (does not work. Bug DAFFODIL-2367) - hexBinary - string apache#2835
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Use DFDL language to describe data, and then enable Drill to query that data immediately by way of Apache Daffodil's DFDL implementation.
(Creating this so I have a ticket number to cite in commits/PRs)
The text was updated successfully, but these errors were encountered: