dbeaver/dbeaver#23390 Support REAL_VECTOR type in HANA plugin #23391

stefanuhrig · 2024-04-10T15:16:13Z

The new vector data type REAL_VECTOR was introduced with HANA Cloud Database QRC 1/2024. Details about the new type are available in the SAP HANA Database Vector Engine Guide.

HANA's JDBC driver natively supports that type starting with version 2.21.5.

This change introduces a new value handler so that vectors are displayed like arrays. Furthermore, the column type modifiers are adapted to display vector dimension constraints.

The new vector data type REAL_VECTOR was introduced with HANA Cloud Database QRC 1/2024. Details about the new type are available in the SAP HANA Database Vector Engine Guide. HANA's JDBC driver natively supports that type starting with version 2.21.5. This change introduces a new value handler so that vectors are displayed like arrays. Furthermore, the column type modifiers are adapted to display vector dimension constraints.

LonwoLonwo · 2024-04-11T10:27:18Z

Hello @stefanuhrig

Thanks for your contribution.
We will look into it.

LonwoLonwo · 2024-04-11T12:26:08Z

Please use a meaningful branch name next time.

LonwoLonwo · 2024-04-11T12:27:24Z

Please provide more info for our QA Team.
Screens of your working feature also will be very helpful.

LonwoLonwo · 2024-04-11T12:28:32Z

plugins/org.jkiss.dbeaver.ext.hana/src/org/jkiss/dbeaver/ext/hana/model/HANADataSource.java

@@ -72,6 +73,14 @@ protected DBPDataSourceInfo createDataSourceInfo(DBRProgressMonitor monitor, @No
        return info;
    }

+    @Override
+    public DBPDataKind resolveDataKind(String typeName, int valueType) {
+        if ("REAL_VECTOR".equalsIgnoreCase(typeName)) {


Please create a constant in the HanaConstants class for the "REAL_VECTOR"

stefanuhrig · 2024-04-11T13:17:12Z

@LonwoLonwo

Which additional info do you require?

You can find the full guide to the new HANA feature at https://help.sap.com/docs/hana-cloud-database/sap-hana-cloud-sap-hana-database-vector-engine-guide/sap-hana-cloud-sap-hana-database-vector-engine-guide.

To shortly summarize:

A new datatype has been introduced into SAP HANA Cloud. This datatype has been designed for vector embeddings, i.e. high-dimensional vectors consisting of floating-point numbers. Vector embeddings play an important role in generative AI contexts.

Without this feature, vectors are displayed in their binary format in DBeaver and look like data garbage. With this feature, they are displayed like arrays, and the elements of the vectors can be edited, too.

If you have any further questions or require any further information, please reach out.

Here is a screenshot of the new feature:

This is how it looks without this new feature:

LonwoLonwo · 2024-04-12T10:10:57Z

...iss.dbeaver.ext.hana/src/org/jkiss/dbeaver/ext/hana/model/data/HANAValueHandlerProvider.java

@@ -33,6 +33,8 @@ public class HANAValueHandlerProvider implements DBDValueHandlerProvider {
    public DBDValueHandler getValueHandler(DBPDataSource dataSource, DBDFormatSettings preferences,
            DBSTypedObject typedObject) {
        switch (typedObject.getTypeName()) {
+        case "REAL_VECTOR":


You didn't add a constant here

And in other places.

@LonwoLonwo
Thanks for your feedback.

I did not use a constant here because there are hard-coded datatype names below.

I considered the following options:

Use the constant and make the code look inconsistent.

Introduce constants for ST_Geometry and ST_Point, which would add refactorings not related to this feature.

Use the hard-coded datatype name.

I opted for 3 but can change it. What's your proposal?

Please also add constants for ST_Geometry and ST_Point.
(2)

LonwoLonwo · 2024-04-12T10:59:29Z

Thanks for the explanation of the future.

LonwoLonwo · 2024-04-15T09:36:33Z

plugins/org.jkiss.dbeaver.ext.hana/src/org/jkiss/dbeaver/ext/hana/model/HANAConstants.java

@@ -24,4 +24,9 @@ public class HANAConstants {

    // pseudo schema for PUBLIC SYNONYMs
    public static final String SCHEMA_PUBLIC = "PUBLIC";
+
+    // Data type names
+    public static final String DATATYPENAME_REAL_VECTOR = "REAL_VECTOR";


Suggested change

public static final String DATATYPENAME_REAL_VECTOR = "REAL_VECTOR";

public static final String DATA_TYPE_NAME_REAL_VECTOR = "REAL_VECTOR";

LonwoLonwo · 2024-04-15T09:36:44Z

plugins/org.jkiss.dbeaver.ext.hana/src/org/jkiss/dbeaver/ext/hana/model/HANAConstants.java

+
+    // Data type names
+    public static final String DATATYPENAME_REAL_VECTOR = "REAL_VECTOR";
+    public static final String DATATYPENAME_ST_GEOMETRY = "ST_GEOMETRY";


Suggested change

public static final String DATATYPENAME_ST_GEOMETRY = "ST_GEOMETRY";

public static final String DATA_TYPE_NAME_ST_GEOMETRY = "ST_GEOMETRY";

LonwoLonwo · 2024-04-15T09:36:57Z

plugins/org.jkiss.dbeaver.ext.hana/src/org/jkiss/dbeaver/ext/hana/model/HANAConstants.java

+    // Data type names
+    public static final String DATATYPENAME_REAL_VECTOR = "REAL_VECTOR";
+    public static final String DATATYPENAME_ST_GEOMETRY = "ST_GEOMETRY";
+    public static final String DATATYPENAME_ST_POINT = "ST_POINT";


Suggested change

public static final String DATATYPENAME_ST_POINT = "ST_POINT";

public static final String DATA_TYPE_NAME_ST_POINT = "ST_POINT";

Change data type constant prefix from DATATYPENAME to DATA_TYPE_NAME.

stefanuhrig · 2024-04-19T10:14:57Z

@LonwoLonwo

Is there something else I need to take care of so that this change can be merged or is it just a matter of time?

Matvey16 · 2024-05-14T13:53:53Z

@stefanuhrig Is there a wasy to get an instance of HANA Cloud Database for thesting. We only have on premise HANA for our tests and as I understand it doesn't support this feature. Also how can I get 2.21 jdbc driver? I only see 2.20 version on maven

stefanuhrig · 2024-05-15T08:32:32Z

@Matvey16

We have HANA Cloud instances for OSS testing and could grant access to you. We have one in the US West region and one in Germany. Which location is closer to you?

The 2.21 JDBC driver has not been released yet. Unfortunately, I am not allowed to share a preliminary version with you. I can ask the responsible colleagues for an estimate release date though.

Matvey16 · 2024-05-17T14:00:48Z

You can send me the credentials on matvei.baranov@dbeaver.com Germany would be better. But I want to ask if there is even a point to test or merge this without th newest driver? How would this data type be handled by the current driver?

stefanuhrig · 2024-05-22T13:24:10Z

@Matvey16

Both the driver and the HANA Cloud version supporting the new client/server protocol (HANA Cloud QRC 2/24) will probably be released end of June.

If either the driver or the database does not support the new client/server protocol, vectors will be reported to be VARBINARY data by the database. So, if a vector column would be directly selected, the result would look like in the second screenshot above.

I will send you the credentials for the HANA Cloud instance per mail.

stefanuhrig added 2 commits April 10, 2024 17:12

Fix codacity issue

190c4d3

LonwoLonwo added the external label Apr 11, 2024

LonwoLonwo linked an issue Apr 11, 2024 that may be closed by this pull request

Support REAL_VECTOR type in HANA plugin #23390

Open

LonwoLonwo reviewed Apr 11, 2024

View reviewed changes

Move data type name to HANAConstants class

2222543

LonwoLonwo reviewed Apr 12, 2024

View reviewed changes

Introduce constants for datatype names

075685d

LonwoLonwo reviewed Apr 15, 2024

View reviewed changes

Change data type constant name prefix

4c2b8de

Change data type constant prefix from DATATYPENAME to DATA_TYPE_NAME.

LonwoLonwo approved these changes Apr 15, 2024

View reviewed changes

E1izabeth approved these changes Apr 15, 2024

View reviewed changes

Merge branch 'devel' into devel

a3bd2e7

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dbeaver/dbeaver#23390 Support REAL_VECTOR type in HANA plugin #23391

dbeaver/dbeaver#23390 Support REAL_VECTOR type in HANA plugin #23391

stefanuhrig commented Apr 10, 2024

LonwoLonwo commented Apr 11, 2024

LonwoLonwo commented Apr 11, 2024

LonwoLonwo commented Apr 11, 2024

LonwoLonwo Apr 11, 2024

stefanuhrig commented Apr 11, 2024

LonwoLonwo Apr 12, 2024

LonwoLonwo Apr 12, 2024

stefanuhrig Apr 12, 2024

LonwoLonwo Apr 13, 2024

LonwoLonwo commented Apr 12, 2024

LonwoLonwo Apr 15, 2024

LonwoLonwo Apr 15, 2024

LonwoLonwo Apr 15, 2024

stefanuhrig commented Apr 19, 2024

Matvey16 commented May 14, 2024

stefanuhrig commented May 15, 2024

Matvey16 commented May 17, 2024 •

edited

stefanuhrig commented May 22, 2024

	public static final String DATATYPENAME_REAL_VECTOR = "REAL_VECTOR";
	public static final String DATA_TYPE_NAME_REAL_VECTOR = "REAL_VECTOR";

dbeaver/dbeaver#23390 Support REAL_VECTOR type in HANA plugin #23391

Are you sure you want to change the base?

dbeaver/dbeaver#23390 Support REAL_VECTOR type in HANA plugin #23391

Conversation

stefanuhrig commented Apr 10, 2024

LonwoLonwo commented Apr 11, 2024

LonwoLonwo commented Apr 11, 2024

LonwoLonwo commented Apr 11, 2024

Choose a reason for hiding this comment

stefanuhrig commented Apr 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LonwoLonwo commented Apr 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stefanuhrig commented Apr 19, 2024

Matvey16 commented May 14, 2024

stefanuhrig commented May 15, 2024

Matvey16 commented May 17, 2024 • edited

stefanuhrig commented May 22, 2024

Matvey16 commented May 17, 2024 •

edited