Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Serializing bigdecimal for thrift/parquet #57

Open
xrl opened this issue Apr 19, 2019 · 0 comments
Open

Serializing bigdecimal for thrift/parquet #57

xrl opened this issue Apr 19, 2019 · 0 comments

Comments

@xrl
Copy link

xrl commented Apr 19, 2019

I am retrieving data from a postgres database with diesel. The table's schema is tagged with a Numeric meta-type, which I deserializing in to BigDecimal:

// diesel table definition
table! {
    invoices (id) {
        id -> Int4,
        commission_rate -> Nullable<Numeric>
    }
}

// target rust struct
use bigdecimal::BigDecimal;
use parquet::record::RecordWriter;

#[derive(Queryable, Debug, ParquetRecordWriter)]
pub struct Invoice {
    pub id: i32,
    commission_rate: Option<BigDecimal>,
}

but now I'm trying to figure out what parquet wants me to write out. The parquet logical types have this bit of info about byte representations:

    /// A decimal value.
    /// This may be used to annotate binary or fixed primitive types. The
    /// underlying byte array stores the unscaled value encoded as two's
    /// complement using big-endian byte order (the most significant byte is the
    /// zeroth element).
    ///
    /// This must be accompanied by a (maximum) precision and a scale in the
    /// SchemaElement. The precision specifies the number of digits in the decimal
    /// and the scale stores the location of the decimal point. For example 1.23
    /// would have precision 3 (3 total digits) and scale 2 (the decimal point is
    /// 2 digits over).
    DECIMAL,

I guess the pressing question is, how can I get the big-endian byte representation out of a BigDecimal? Looking over the implementation and it's got a lot of support for data types in but not so much out. Would the project be open to a PR addressing this need?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant