GH-41502: [Python] Fix reading column index with decimal values

[jrversteegh]

Fix for #41502

Convert pandas "decimal" to "object" in numpy.

• GitHub Issue: [Python] pandas roundtrip failing with decimal in column index #41502

[github-actions]

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - Contributing Overview

[github-actions]

⚠️ GitHub issue #41502 has been automatically assigned in GitHub to PR creator.

[AlenkaF]

Hi @jrversteegh, thank you for the contribution!

I think this is an elegant solution. Not sure if this was discussed before or not, can't find any similar issue on our issue tracker. I am sure @jorisvandenbossche will know straight away if this change fits or not.

From my side, a test needs to be added in python/pyarrow/tests/parquet/test_pandas.py.

[jorisvandenbossche]

That indeed looks like a good fix.

The error itself should already happen with just a roundtrip from pandas->pyarrow->pandas (without parquet), so you can add a test for this in python/pyarrow/tests/test_pandas.py.

[jrversteegh]

@AlenkaF

From my side, a test needs to be added in python/pyarrow/tests/parquet/test_pandas.py.

Thanks for that suggestion. I tried, but this issue appears more involved than I expected. It looks like pyarrow expects column names to be strings. If not, it converts them (in turn because the parquet format expects this?).
My fix avoids the exception, but still converts the column names from decimal to string, which is better, but still undesirable. I'll have to see whether there is something I can do about that. To be continued.

[jrversteegh]

@AlenkaF @jorisvandenbossche I've added a test and restored the decimal index from strings. This looks like a bit of a kludge. I think it's because both numpy and pandas don't understand Decimal. It's just an object, like any other, so you might expect similar issues for any other object type used as a column index. However, Decimal is not just any object and it does make some sense to use as an index, just like dates, so I feel it's a warranted addition. Let me know what you think.