Unsupported machine

[Peter2121]

To reproduce

QuestDB 7.4.3 built from sources in FreeBSD jail hosted on Ampere powered VM (AARCH64) in Oracle Cloud.
Openjdk 11.0.22
Apache Maven 3.9.6
Trying to start QuestDB got an error:

java.lang.UnsatisfiedLinkError: /tmp/libquestdb9744066241967681339.so: /tmp/libquestdb9744066241967681339.so: unsupported machine (Possible cause: can't load AMD 64-bit .so on a AARCH64-bit platform)
        at java.base/java.lang.ClassLoader$NativeLibrary.load0(Native Method)
        at java.base/java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2450)
        at java.base/java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2506)
        at java.base/java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2705)
        at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2635)
        at java.base/java.lang.Runtime.load0(Runtime.java:768)
        at java.base/java.lang.System.load(System.java:1854)
        at io.questdb@7.4.3-SNAPSHOT/io.questdb.std.Os.loadLib(Os.java:221)
        at io.questdb@7.4.3-SNAPSHOT/io.questdb.std.Os.loadLib(Os.java:199)
        at io.questdb@7.4.3-SNAPSHOT/io.questdb.std.Os.<clinit>(Os.java:266)
        at io.questdb@7.4.3-SNAPSHOT/io.questdb.Bootstrap.<clinit>(Bootstrap.java:696)
        at io.questdb@7.4.3-SNAPSHOT/io.questdb.ServerMain.<init>(ServerMain.java:74)
        at io.questdb@7.4.3-SNAPSHOT/io.questdb.ServerMain.main(ServerMain.java:168)
Exception in thread "main" java.lang.NoClassDefFoundError: Could not initialize class io.questdb.std.Os
        at io.questdb@7.4.3-SNAPSHOT/io.questdb.log.LogFactory.<clinit>(LogFactory.java:1152)
        at io.questdb@7.4.3-SNAPSHOT/io.questdb.ServerMain.main(ServerMain.java:171)

Probably, something need to be rebuilt on current arch, but I cannot find any document about it.

QuestDB version:

QuestDB 7.4.3-SNAPSHOT

OS, in case of Docker specify Docker and the Host OS:

FreeBSD 14.0-RELEASE-p6 (jail)

File System, in case of Docker specify Host File System:

UFS

Full Name:

Peter TKATCHENKO

Affiliation:

Flytrace

Have you followed Linux, MacOs kernel configuration steps to increase Maximum open files and Maximum virtual memory areas limit?

• Yes, I have

Additional context

No response

[bluestreak01]

building from source is a good start! Perhaps you could also have a look into: https://github.com/questdb/questdb/blob/master/.github/workflows/rebuild_native_libs.yml to build on aarch64 FreeBSD?

I will help you along if anything else is missing for this platform!

[Peter2121]

Thanks for the hint.
I could build the binaries, then I replaced them in jar file (in both armfreebsd and freebsd), and I could start questdb.
If I rebuild the jar - I get the same error on start. It seems that some build files need to be fixed.
I'll do more tests ASAP.

[Peter2121]

Unfortunately, something works bad on this platform.
I can create table and inject data (exactly at the same way I've done it on amd64). But when I try to query the data - WHERE using symbol column never matches. It matches for c1<>'', but it never matches for c1='test'. I tested using web interface and using PG client. The data looks good if I send SELECT * ... query.
Any ideas?

Example:

CREATE TABLE 'test' (sym1 symbol, comment text, num double);
INSERT INTO test VALUES('strtr', 'sdflk lkjls mlkmlk', 6.225);
SELECT * FROM test;
SELECT * FROM test WHERE sym1='strtr';

The first SELECT returns one row, the second returns no data.

[bluestreak01]

Would you mind opening a PR even if it does not work? We can take a look at what’s going on with the exact code you are testing

[Peter2121]

I have not look closely at your build system (yet), all the modifications I've done to get it running are manual hacks. So, no PR for the moment. I'll try to run a release and not the dev snapshot to see if the query problem is present in release code.

[Peter2121]

I'm trying to build the release 7.4.2 at the same manner, but it does not start.
I could rebuild C++ library (Rust library does not exist in this version?), after rebuild of jar the library is present in io/questdb/bin/armfreebsd. But if I try to launch the jar I get the following error:

Error occurred during initialization of boot layer
java.lang.module.FindException: Module io.questdb not found

[nwoolmer]

Hi @Peter2121, any more info/logs could be helpful to debug this. If the hacks can be done via code, it'd be great to get a PR open to make it easier to collaborate and fix this.

Also, some build reworking has been merged to current master, this moves the goalposts on fixing this.

Unfortunately, something works bad on this platform. I can create table and inject data (exactly at the same way I've done it on amd64). But when I try to query the data - WHERE using symbol column never matches. It matches for c1<>'', but it never matches for c1='test'. I tested using web interface and using PG client. The data looks good if I send SELECT * ... query. Any ideas?

Example:

CREATE TABLE 'test' (sym1 symbol, comment text, num double);
INSERT INTO test VALUES('strtr', 'sdflk lkjls mlkmlk', 6.225);
SELECT * FROM test;
SELECT * FROM test WHERE sym1='strtr';

The first SELECT returns one row, the second returns no data.

Are you able to try a different FS on the box? Whilst UFS may work, its not on the supported list: https://questdb.io/docs/deployment/capacity-planning/#storage-and-filesystem

This is one less variable to remove from the equation. You might see logs like this on startup otherwise:

-> UNSUPPORTED (SYSTEM COULD BE UNSTABLE)"

The description of odd query behaviour could be related to this. Your example worked fine on latest master with macOS ARM.

[Peter2121]

Hi @nwoolmer

QuestDB is installed inside a jail, so I'm not sure it can see the details of the filesystem. The root of host system (VM) is on UFS.
Note that I successfully run QuestDB 7.4.0 on a physical server (inside a jail too) using amd64 CPU and ZFS filesystem. No issues seen.
AFAIK, all 'modern' installations of FreeBSD use ZFS. My ARM VM in Oracle Cloud is on UFS only because the 'modern' image is pending, and reformatting root FS it a little bit difficult ;)
When ZFS based image of FreeBSD will be available - the VM will be reinstalled using this image (on ZFS). There are too many advantages using ZFS on FreeBSD servers to continue using UFS today.

As about logs...

Startup:

2024-05-08T20:31:51.971745Z A server-main Config:
2024-05-08T20:31:51.971790Z A server-main  - http.enabled : true
2024-05-08T20:31:51.971825Z A server-main  - tcp.enabled  : true
2024-05-08T20:31:51.971851Z A server-main  - pg.enabled   : true
2024-05-08T20:31:51.971864Z A server-main  - attach partition suffix: .attachable
2024-05-08T20:31:51.971889Z A server-main  - open database [id=6113340196020290196.-4501280100660552094]
2024-05-08T20:31:51.972042Z A server-main  - db root: [path=/var/db/questdb/db, magic=0x35] -> SUPPORTED
2024-05-08T20:31:51.972100Z A server-main  - backup root: NOT SET
2024-05-08T20:31:51.972136Z A server-main  - snapshot root: [path=/var/db/questdb/snapshot, magic=0x00] -> UNSUPPORTED (SYSTEM COULD BE UNSTABLE)
2024-05-08T20:31:51.972170Z A server-main  - sql copy input root: NOT SET
2024-05-08T20:31:51.972188Z A server-main  - sql copy input worker root: NOT SET
2024-05-08T20:31:51.989843Z A server-main  - SQL JIT compiler mode: on

Query:

2024-05-08T20:33:05.668535Z I http-server connected [ip=192.168.213.131, fd=113]
2024-05-08T20:33:05.675587Z I i.q.g.SqlCompilerImpl parse [fd=113, thread=13, q=SELECT * FROM TEST WHERE sym1='strtr']
2024-05-08T20:33:05.677289Z I i.q.g.SqlCompilerImpl plan [q=`select-choose sym1, comment, num from (select [sym1, comment, num] from TEST where sym1 = 'strtr')`, fd=113]
2024-05-08T20:33:05.689764Z I i.q.g.SqlCodeGenerator JIT enabled for (sub)query [tableName=TEST, fd=113]
2024-05-08T20:33:05.690346Z I i.q.c.h.p.JsonQueryProcessorState [113] execute-new [skip: 0, stop: 1000]
2024-05-08T20:33:05.691265Z I i.q.c.TableReader open partition /var/db/questdb/db/test/default [rowCount=1, partitionNameTxn=-1, transientRowCount=1, partitionIndex=0, partition
Count=1]
2024-05-08T20:33:05.696715Z I i.q.c.h.p.JsonQueryProcessorState [113] timings [compiler: 15059834, count: 0, execute: 22400871, q=`SELECT * FROM TEST WHERE sym1='strtr'`]
2024-05-08T20:33:05.696903Z I i.q.c.h.p.JsonQueryProcessor all sent [fd=113, lastRequestBytesSent=510, nCompletedRequests=15, totalBytesSent=409876]
2024-05-08T20:33:05.697120Z I http-server scheduling disconnect [fd=113, reason=12]
2024-05-08T20:33:05.697306Z I http-server disconnected [ip=192.168.213.131, fd=113, src=queue]
2024-05-08T20:33:05.765415Z I http-server connected [ip=192.168.213.131, fd=113]
2024-05-08T20:33:05.765689Z I i.q.c.h.p.StaticContentProcessor [113] incoming [url=/6437d5f4c14e01aaf4801aef3909151b.woff]
2024-05-08T20:33:05.766627Z I http-server scheduling disconnect [fd=113, reason=12]
2024-05-08T20:33:05.766679Z I http-server disconnected [ip=192.168.213.131, fd=113, src=queue]

[nwoolmer]

Hi @Peter2121,

From the logs, you can see that the database is reporting you're using an unsupported file system, so there's a risk of instability. Whether it detects UFS or ZFS isn't clear.

However, its planned to add ZFS support, possibly in the next open-source release. So it might be worth revisiting then and try to build again with the revised build system.

Otherwise we can start a PR to debug this issue.

Thanks!

[Peter2121]

Hi @nwoolmer
Probably, I was not clear.
The problem seems to be CPU arch specific. Another instance on amd64/ZFS works correctly, I have the problem with arm64/UFS instance. So, I don't understand how ZFS support can help here.

[Peter2121]

I tested a bit more. Only symbol columns have this problem. string and double data are queried correctly.
What is the difference in processing queries with string data and with symbol data?

[nwoolmer]

Symbols are dictionary encoded, so the column vector is integral and the strings stored once in the symbol table.

In open source, ZFS is prone to corrupting data under hybrid loads. Other users have seen issues with symbol table on ZFS storage. Whilst you haven't noticed an issue before with your other deployment, hybrid loads could break it.

Therefore when ZFS is supported in open source, that removes one source of potential instability.

The example you gave earlier worked fine on NTFS, when I tried it.

[Peter2121]

Finally, I could get it working on arm64/UFS instance. All I needed was INDEX (sym1) in CREATE TABLE.
It's strange.

[Peter2121]

@nwoolmer,
Please, leave this ticket open and put here some words when ZFS will be production-ready.

[bluestreak01]

ZFS is on master now and we aim to release this week