Skip to content
This repository has been archived by the owner on Nov 16, 2023. It is now read-only.

Tests failing on main for apple silicon #85

Closed
brancz opened this issue Mar 3, 2022 · 2 comments · Fixed by #109
Closed

Tests failing on main for apple silicon #85

brancz opened this issue Mar 3, 2022 · 2 comments · Fixed by #109
Assignees
@achille-roussel
Labels
bug Something isn't working enhancement Improve a feature that already exists

Comments

@brancz
Copy link
Contributor

brancz commented Mar 3, 2022

One of the latest patches appear to have broken main to build on apple silicon, here's what I get running go test from main, which previously passed just fine.

first_name > C:0 D:0 R:0 V:Han
first_name > C:0 D:0 R:0 V:Leia
first_name > C:0 D:0 R:0 V:Luke
last_name > C:1 D:0 R:0 V:Solo
last_name > C:1 D:0 R:0 V:Skywalker
first_name > C:0 D:0 R:0 V:Han
first_name > C:0 D:0 R:0 V:Leia
first_name > C:0 D:0 R:0 V:Luke
last_name > C:1 D:0 R:0 V:Skywalker
last_name > C:1 D:0 R:0 V:Solo
last_name > C:1 D:0 R:0 V:Skywalker
last_name > C:1 D:0 R:0 V:Skywalker
name > C:0 D:0 R:0 V:http_request_total
name > C:0 D:0 R:0 V:http_request_total
name > C:0 D:0 R:0 V:http_request_total
name > C:0 D:0 R:0 V:http_request_total
name > C:0 D:1 R:0 V:Dmitriy Ryaboy
name > C:0 D:1 R:1 V:Chris Aniszczyk
name > C:0 D:0 R:0 V:<null>
name > C:0 D:0 R:0 V:http_request_total
name > C:0 D:0 R:0 V:http_request_total
name > C:0 D:0 R:0 V:http_request_total
phoneNumber > C:1 D:2 R:0 V:555 987 6543
phoneNumber > C:1 D:1 R:1 V:<null>
phoneNumber > C:1 D:0 R:0 V:<null>
name > C:0 D:0 R:0 V:http_request_total
name > C:0 D:1 R:0 V:Dmitriy Ryaboy
name > C:0 D:1 R:1 V:Chris Aniszczyk
name > C:0 D:0 R:0 V:<null>
owner > C:2 D:0 R:0 V:Julien Le Dem
phoneNumber > C:1 D:2 R:0 V:555 987 6543
phoneNumber > C:1 D:1 R:1 V:<null>
phoneNumber > C:1 D:0 R:0 V:<null>
name > C:0 D:0 R:0 V:http_request_total
owner > C:2 D:0 R:0 V:A. Nonymous
name > C:0 D:0 R:0 V:http_request_total
owner > C:2 D:0 R:0 V:Julien Le Dem
timestamp > C:1 D:0 R:0 V:1639444033
timestamp > C:1 D:0 R:0 V:1639444058
owner > C:2 D:0 R:0 V:A. Nonymous
timestamp > C:1 D:0 R:0 V:1639444085
timestamp > C:1 D:0 R:0 V:1639444093
timestamp > C:1 D:0 R:0 V:1639444101
timestamp > C:1 D:0 R:0 V:1639444108
timestamp > C:1 D:0 R:0 V:1639444133
timestamp > C:1 D:0 R:0 V:1639444137
timestamp > C:1 D:0 R:0 V:1639444141
timestamp > C:1 D:0 R:0 V:1639444144
value > C:2 D:0 R:0 V:100
value > C:2 D:0 R:0 V:0
value > C:2 D:0 R:0 V:42
value > C:2 D:0 R:0 V:1
value > C:2 D:0 R:0 V:2
value > C:2 D:0 R:0 V:5
value > C:2 D:0 R:0 V:4
value > C:2 D:0 R:0 V:5
value > C:2 D:0 R:0 V:6
value > C:2 D:0 R:0 V:10
ownerPhoneNumbers > C:3 D:1 R:0 V:555 123 4567
ownerPhoneNumbers > C:3 D:1 R:1 V:555 666 1337
ownerPhoneNumbers > C:3 D:0 R:0 V:<null>
ownerPhoneNumbers > C:3 D:1 R:0 V:555 123 4567
ownerPhoneNumbers > C:3 D:1 R:1 V:555 666 1337
ownerPhoneNumbers > C:3 D:0 R:0 V:<null>
--- FAIL: TestWriter (0.03s)
    --- FAIL: TestWriter/example_from_the_twitter_blog_(v1) (0.60s)
        writer_test.go:428: 
            row group 0 
            --------------------------------------------------------------------------------
            contacts:          
            .name:              BINARY UNCOMPRESSED DO:0 FPO:4 SZ:120/120/1,00 VC:3 [more]...
            .phoneNumber:       BINARY SNAPPY DO:0 FPO:124 SZ:100/96/0,96 VC:3 ENC [more]...
            owner:              BINARY ZSTD DO:0 FPO:224 SZ:98/80/0,82 VC:2 ENC:DE [more]...
            ownerPhoneNumbers:  BINARY GZIP DO:0 FPO:322 SZ:166/116/0,70 VC:3 ENC: [more]...
            org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Mac and os.arch=aarch64
            	at org.xerial.snappy.SnappyLoader.findNativeLibrary(SnappyLoader.java:361)
            	at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:195)
            	at org.xerial.snappy.SnappyLoader.loadSnappyApi(SnappyLoader.java:167)
            	at org.xerial.snappy.Snappy.init(Snappy.java:69)
            	at org.xerial.snappy.Snappy.<clinit>(Snappy.java:46)
            	at org.apache.parquet.hadoop.codec.SnappyDecompressor.decompress(SnappyDecompressor.java:62)
            	at org.apache.parquet.hadoop.codec.NonBlockedDecompressorStream.read(NonBlockedDecompressorStream.java:51)
            	at java.base/java.io.DataInputStream.readFully(DataInputStream.java:201)
            	at java.base/java.io.DataInputStream.readFully(DataInputStream.java:172)
            	at org.apache.parquet.bytes.BytesInput$StreamBytesInput.toByteArray(BytesInput.java:286)
            	at org.apache.parquet.tools.command.DumpCommand$1.visit(DumpCommand.java:279)
            	at org.apache.parquet.tools.command.DumpCommand$1.visit(DumpCommand.java:265)
            	at org.apache.parquet.column.page.DataPageV1.accept(DataPageV1.java:120)
            	at org.apache.parquet.tools.command.DumpCommand.dump(DumpCommand.java:265)
            	at org.apache.parquet.tools.command.DumpCommand.dump(DumpCommand.java:195)
            	at org.apache.parquet.tools.command.DumpCommand.execute(DumpCommand.java:148)
            	at org.apache.parquet.tools.Main.main(Main.java:223)
            org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Mac and os.arch=aarch64
            
        writer_test.go:429: exit status 1
    --- FAIL: TestWriter/timeseries_with_delta_encoding (0.65s)
        writer_test.go:435: 
            --- want.txt
            +++ got.txt
            @@ -1,8 +1,8 @@
             row group 0
             --------------------------------------------------------------------------------
            -name:       BINARY UNCOMPRESSED DO:4 FPO:45 SZ:101/101/1.00 VC:10 ENC: [more]...
            -timestamp:  INT64 UNCOMPRESSED DO:0 FPO:105 SZ:278/278/1.00 VC:10 ENC: [more]...
            -value:      DOUBLE UNCOMPRESSED DO:0 FPO:383 SZ:220/220/1.00 VC:10 ENC:PLAIN [more]...
            +name:       BINARY UNCOMPRESSED DO:4 FPO:45 SZ:101/101/1,00 VC:10 ENC: [more]...
            +timestamp:  INT64 UNCOMPRESSED DO:0 FPO:105 SZ:278/278/1,00 VC:10 ENC: [more]...
            +value:      DOUBLE UNCOMPRESSED DO:0 FPO:383 SZ:220/220/1,00 VC:10 ENC:PLAIN [more]...
             
                 name TV=10 RL=0 DL=0 DS: 1 DE:PLAIN
                 ----------------------------------------------------------------------------
    --- FAIL: TestWriter/page_v2_with_dictionary_encoding (0.71s)
        writer_test.go:435: 
            --- want.txt
            +++ got.txt
            @@ -1,7 +1,7 @@
             row group 0
             --------------------------------------------------------------------------------
            -first_name:  BINARY ZSTD DO:4 FPO:55 SZ:86/77/0.90 VC:3 ENC:RLE_DICTIONARY,PLAIN [more]...
            -last_name:   BINARY ZSTD DO:0 FPO:90 SZ:163/136/0.83 VC:3 ENC:DELTA_BYTE_ARRAY [more]...
            +first_name:  BINARY ZSTD DO:4 FPO:55 SZ:86/77/0,90 VC:3 ENC:RLE_DICTIONARY,PLAIN [more]...
            +last_name:   BINARY ZSTD DO:0 FPO:90 SZ:163/136/0,83 VC:3 ENC:DELTA_BYTE_ARRAY [more]...
             
                 first_name TV=3 RL=0 DL=0 DS: 3 DE:PLAIN
                 ----------------------------------------------------------------------------
    --- FAIL: TestWriter/page_v1_with_dictionary_encoding (0.74s)
        writer_test.go:435: 
            --- want.txt
            +++ got.txt
            @@ -1,7 +1,7 @@
             row group 0
             --------------------------------------------------------------------------------
            -first_name:  BINARY ZSTD DO:4 FPO:55 SZ:90/72/0.80 VC:3 ENC:PLAIN,RLE_DICTIONARY [more]...
            -last_name:   BINARY ZSTD DO:0 FPO:94 SZ:148/121/0.82 VC:3 ENC:DELTA_BYTE_ARRAY [more]...
            +first_name:  BINARY ZSTD DO:4 FPO:55 SZ:90/72/0,80 VC:3 ENC:PLAIN,RLE_DICTIONARY [more]...
            +last_name:   BINARY ZSTD DO:0 FPO:94 SZ:148/121/0,82 VC:3 ENC:DELTA_BYTE_ARRAY [more]...
             
                 first_name TV=3 RL=0 DL=0 DS: 3 DE:PLAIN
                 ----------------------------------------------------------------------------
    --- FAIL: TestWriter/example_from_the_twitter_blog_(v2) (0.78s)
        writer_test.go:428: 
            row group 0 
            --------------------------------------------------------------------------------
            contacts:          
            .name:              BINARY UNCOMPRESSED DO:0 FPO:4 SZ:114/114/1,00 VC:3 [more]...
            .phoneNumber:       BINARY SNAPPY DO:0 FPO:118 SZ:94/90/0,96 VC:3 ENC: [more]...
            owner:              BINARY ZSTD DO:0 FPO:212 SZ:108/90/0,83 VC:2 ENC:D [more]...
            ownerPhoneNumbers:  BINARY GZIP DO:0 FPO:320 SZ:159/109/0,69 VC:3 ENC: [more]...
            
                contacts.name TV=3 RL=1 DL=1
                ----------------------------------------------------------------------------
                page 0:  DLE:RLE RLE:RLE VLE:DELTA_LENGTH_BYTE_ARRAY ST:[no stats  [more]... VC:2
                page 1:  DLE:RLE RLE:RLE VLE:DELTA_LENGTH_BYTE_ARRAY ST:[no stats  [more]... VC:1
            
                contacts.phoneNumber TV=3 RL=1 DL=2
                ----------------------------------------------------------------------------
                page 0:  DLE:RLE RLE:RLE VLE:DELTA_LENGTH_BYTE_ARRAY ST:[no stats  [more]... VC:2
                page 1:  DLE:RLE RLE:RLE VLE:DELTA_LENGTH_BYTE_ARRAY ST:[no stats  [more]... VC:1
            
                owner TV=2 RL=0 DL=0
                ----------------------------------------------------------------------------
                page 0:  DLE:RLE RLE:RLE VLE:DELTA_LENGTH_BYTE_ARRAY ST:[no stats  [more]... VC:1
                page 1:  DLE:RLE RLE:RLE VLE:DELTA_LENGTH_BYTE_ARRAY ST:[no stats  [more]... VC:1
            
                ownerPhoneNumbers TV=3 RL=1 DL=1
                ----------------------------------------------------------------------------
                page 0:  DLE:RLE RLE:RLE VLE:DELTA_LENGTH_BYTE_ARRAY ST:[no stats  [more]... VC:2
                page 1:  DLE:RLE RLE:RLE VLE:DELTA_LENGTH_BYTE_ARRAY ST:[no stats  [more]... VC:1
            
            BINARY contacts.name 
            --------------------------------------------------------------------------------
            *** row group 1 of 1, values 1 to 3 *** 
            value 1: R:0 D:1 V:Dmitriy Ryaboy
            value 2: R:1 D:1 V:Chris Aniszczyk
            value 3: R:0 D:0 V:<null>
            
            BINARY contacts.phoneNumber 
            --------------------------------------------------------------------------------
            org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Mac and os.arch=aarch64
            	at org.xerial.snappy.SnappyLoader.findNativeLibrary(SnappyLoader.java:361)
            	at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:195)
            	at org.xerial.snappy.SnappyLoader.loadSnappyApi(SnappyLoader.java:167)
            	at org.xerial.snappy.Snappy.init(Snappy.java:69)
            	at org.xerial.snappy.Snappy.<clinit>(Snappy.java:46)
            	at org.apache.parquet.hadoop.codec.SnappyDecompressor.decompress(SnappyDecompressor.java:62)
            	at org.apache.parquet.hadoop.codec.NonBlockedDecompressorStream.read(NonBlockedDecompressorStream.java:51)
            	at java.base/java.io.DataInputStream.readFully(DataInputStream.java:201)
            	at java.base/java.io.DataInputStream.readFully(DataInputStream.java:172)
            	at org.apache.parquet.bytes.BytesInput$StreamBytesInput.toByteArray(BytesInput.java:286)
            	at org.apache.parquet.bytes.BytesInput.toByteBuffer(BytesInput.java:237)
            	at org.apache.parquet.bytes.BytesInput.toInputStream(BytesInput.java:246)
            	at org.apache.parquet.column.impl.ColumnReaderBase.readPageV2(ColumnReaderBase.java:698)
            	at org.apache.parquet.column.impl.ColumnReaderBase.access$400(ColumnReaderBase.java:57)
            	at org.apache.parquet.column.impl.ColumnReaderBase$3.visit(ColumnReaderBase.java:628)
            	at org.apache.parquet.column.impl.ColumnReaderBase$3.visit(ColumnReaderBase.java:620)
            	at org.apache.parquet.column.page.DataPageV2.accept(DataPageV2.java:192)
            	at org.apache.parquet.column.impl.ColumnReaderBase.readPage(ColumnReaderBase.java:620)
            	at org.apache.parquet.column.impl.ColumnReaderBase.checkRead(ColumnReaderBase.java:594)
            	at org.apache.parquet.column.impl.ColumnReaderBase.consume(ColumnReaderBase.java:735)
            	at org.apache.parquet.column.impl.ColumnReaderImpl.consume(ColumnReaderImpl.java:30)
            	at org.apache.parquet.column.impl.ColumnReaderImpl.<init>(ColumnReaderImpl.java:47)
            	at org.apache.parquet.column.impl.ColumnReadStoreImpl.getColumnReader(ColumnReadStoreImpl.java:82)
            	at org.apache.parquet.tools.command.DumpCommand.dump(DumpCommand.java:312)
            	at org.apache.parquet.tools.command.DumpCommand.dump(DumpCommand.java:225)
            	at org.apache.parquet.tools.command.DumpCommand.execute(DumpCommand.java:148)
            	at org.apache.parquet.tools.Main.main(Main.java:223)
            org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Mac and os.arch=aarch64
            
        writer_test.go:429: exit status 1
FAIL
exit status 1
FAIL	github.com/segmentio/parquet-go	2.098s
@achille-roussel
Copy link
Contributor

This appears to be caused by snappy-java not having a native build for arm64:

org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Mac and os.arch=aarch64

I searched for similar issues but could not find much. This issue is probably worth reporting to the maintainers of parquet-tools or snappy-java.

@achille-roussel
Copy link
Contributor

A similar issue was reported here xerial/snappy-java#302

@achille-roussel achille-roussel added the enhancement Improve a feature that already exists label Mar 15, 2022
@Pryz Pryz mentioned this issue Mar 21, 2022
Merged
@achille-roussel achille-roussel self-assigned this Jun 21, 2022
@achille-roussel achille-roussel added the bug Something isn't working label Jun 21, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@achille-roussel achille-roussel

Labels
bug Something isn't working enhancement Improve a feature that already exists
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix test on arm64 segmentio/parquet-go
2 participants
@achille-roussel @brancz