Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(sql): support usage of ALTER table SQL to convert column types #4413

Merged
merged 83 commits into from
May 15, 2024

Conversation

ideoma
Copy link
Collaborator

@ideoma ideoma commented Apr 17, 2024

Fixes #257

Support of syntax

CREATE TABLE tbl_name (a SYMBOL);

ALTER TABLE tbl_name ALTER COLUMN a TYPE STRING;
ALTER TABLE tbl_name ALTER COLUMN a TYPE VARCHAR;
ALTER TABLE tbl_name ALTER COLUMN a TYPE SYMBOL;

These column-type conversions are supported:

  • any combination of 2 from BYTE, SHORT, INT, LONG, FLOAT, DOUBLE, TIMESTAMP, BOOL, STRING, SYMBOL, VARCHAR
  • any combination of GUID, IPv4 and STRING, SYMBOL, VARCHAR

Not included (rarely used types):

  • Binary, GeoHash

The conversion SQL rewrites the column in a single transaction. WAL tables support concurrent writes without data loss while the operation is performed. This change allows ILP to send STRING-type columns (in the fields section) to SYMBOL columns.

  • Save column order in table _metadata that is used in SELECT * queries
  • Support of SQL syntax and create AlterOperation
  • Support of changing the column type as a single transaction in TableWriter. This will add the column of the new type, copy the data into it, drop the old column, and change the order of SELECT * columns in metadata.
  • Support of the column type change in WAL writers
  • Fuzz tests with column conversions
  • Support of the column type update in WAL ILP writing pipelines

@ideoma ideoma marked this pull request as draft April 18, 2024 12:52
@glasstiger glasstiger self-requested a review May 9, 2024 08:52
ideoma and others added 3 commits May 9, 2024 15:24
 Conflicts:
	core/src/main/resources/io/questdb/bin/darwin-amd64/libquestdb.dylib
	core/src/main/resources/io/questdb/bin/freebsd-amd64/libquestdb.so
	core/src/main/resources/io/questdb/bin/linux-amd64/libquestdb.so
	core/src/main/resources/io/questdb/bin/windows-amd64/libquestdb.dll
	core/src/main/resources/io/questdb/bin/windows-amd64/questdbr.dll
glasstiger
glasstiger previously approved these changes May 15, 2024
GitHub Actions - Rebuild Native Libraries and others added 4 commits May 15, 2024 14:56
…olumn-convert

# Conflicts:
#	core/src/main/resources/io/questdb/bin/linux-aarch64/libquestdb.so
#	core/src/main/resources/io/questdb/bin/linux-amd64/libquestdb.so
#	core/src/main/resources/io/questdb/bin/windows-amd64/libquestdb.dll
#	core/src/main/resources/io/questdb/bin/windows-amd64/questdbr.dll
@ideoma
Copy link
Collaborator Author

ideoma commented May 15, 2024

[PR Coverage check]

😍 pass : 1577 / 1670 (94.43%)

file detail

path covered line new line coverage
🔵 io/questdb/cairo/vm/MemoryFMCRImpl.java 0 2 00.00%
🔵 io/questdb/cairo/vm/NullMemoryMR.java 0 1 00.00%
🔵 io/questdb/griffin/ConvertersNative.java 0 2 00.00%
🔵 io/questdb/cutlass/http/DefaultHttpServerConfiguration.java 0 3 00.00%
🔵 io/questdb/cairo/vm/api/NullMemory.java 0 1 00.00%
🔵 io/questdb/cairo/vm/AbstractMemoryCR.java 3 11 27.27%
🔵 io/questdb/cairo/vm/MemoryCMRImpl.java 9 13 69.23%
🔵 io/questdb/cutlass/line/tcp/LineTcpParser.java 7 10 70.00%
🔵 io/questdb/cairo/wal/CopyWalSegmentUtils.java 30 37 81.08%
🔵 io/questdb/cutlass/line/tcp/LineTcpMeasurementEvent.java 9 11 81.82%
🔵 io/questdb/cairo/vm/MemoryCMARWImpl.java 13 15 86.67%
🔵 io/questdb/cairo/TableWriter.java 131 146 89.73%
🔵 io/questdb/griffin/engine/ops/AlterOperation.java 21 23 91.30%
🔵 io/questdb/cairo/wal/WalWriter.java 185 197 93.91%
🔵 io/questdb/cairo/TableUtils.java 45 48 93.75%
🔵 io/questdb/griffin/ConvertOperatorImpl.java 177 184 96.20%
🔵 io/questdb/cairo/ColumnTypeConverter.java 507 526 96.39%
🔵 io/questdb/griffin/engine/functions/cast/CastTimestampToBooleanFunctionFactory.java 2 2 100.00%
🔵 io/questdb/griffin/engine/functions/cast/CastIntToBooleanFunctionFactory.java 2 2 100.00%
🔵 io/questdb/griffin/engine/functions/cast/CastLongToBooleanFunctionFactory.java 2 2 100.00%
🔵 io/questdb/cutlass/line/tcp/LineWalAppender.java 19 19 100.00%
🔵 io/questdb/griffin/engine/functions/cast/CastDoubleToShortFunctionFactory.java 2 2 100.00%
🔵 io/questdb/cairo/TableReader.java 30 30 100.00%
🔵 io/questdb/cairo/wal/seq/SequencerMetadataService.java 2 2 100.00%
🔵 io/questdb/griffin/engine/ops/AlterOperationBuilder.java 5 5 100.00%
🔵 io/questdb/griffin/engine/functions/cast/CastDoubleToIntFunctionFactory.java 1 1 100.00%
🔵 io/questdb/cutlass/http/processors/LineHttpProcessorState.java 1 1 100.00%
🔵 io/questdb/cairo/wal/WalWriterMetadata.java 4 4 100.00%
🔵 io/questdb/cairo/vm/MemoryCMORImpl.java 2 2 100.00%
🔵 io/questdb/cairo/TableReaderMetadata.java 99 99 100.00%
🔵 io/questdb/cutlass/line/tcp/TableUpdateDetails.java 30 30 100.00%
🔵 io/questdb/griffin/engine/functions/cast/CastDoubleToTimestampFunctionFactory.java 1 1 100.00%
🔵 io/questdb/griffin/engine/functions/cast/CastFloatToDateFunctionFactory.java 1 1 100.00%
🔵 io/questdb/cairo/wal/SegmentColumnRollSink.java 30 30 100.00%
🔵 io/questdb/griffin/engine/functions/cast/CastDoubleToLongFunctionFactory.java 1 1 100.00%
🔵 io/questdb/cairo/wal/seq/SequencerMetadata.java 4 4 100.00%
🔵 io/questdb/griffin/engine/functions/cast/CastBooleanToFloatFunctionFactory.java 2 2 100.00%
🔵 io/questdb/cairo/TableReaderMetadataTransitionIndex.java 17 17 100.00%
🔵 io/questdb/griffin/engine/functions/cast/CastBooleanToDoubleFunctionFactory.java 2 2 100.00%
🔵 io/questdb/cairo/TableReaderMetadataColumn.java 6 6 100.00%
🔵 io/questdb/cairo/vm/api/MemoryCR.java 2 2 100.00%
🔵 io/questdb/griffin/PurgingOperator.java 1 1 100.00%
🔵 io/questdb/cairo/TxReader.java 1 1 100.00%
🔵 io/questdb/cutlass/line/tcp/LineTcpConnectionContext.java 1 1 100.00%
🔵 io/questdb/griffin/engine/functions/cast/CastDateToBooleanFunctionFactory.java 2 2 100.00%
🔵 io/questdb/griffin/engine/functions/cast/CastFloatToByteFunctionFactory.java 2 2 100.00%
🔵 io/questdb/griffin/engine/functions/cast/CastFloatToIntFunctionFactory.java 1 1 100.00%
🔵 io/questdb/cairo/DatabaseSnapshotAgentImpl.java 1 1 100.00%
🔵 io/questdb/cairo/CursorPrinter.java 2 2 100.00%
🔵 io/questdb/griffin/engine/functions/cast/CastFloatToBooleanFunctionFactory.java 2 2 100.00%
🔵 io/questdb/cairo/DdlListener.java 1 1 100.00%
🔵 io/questdb/griffin/engine/functions/cast/CastDoubleToBooleanFunctionFactory.java 2 2 100.00%
🔵 io/questdb/griffin/SqlCompilerImpl.java 132 132 100.00%
🔵 io/questdb/griffin/engine/functions/cast/CastFloatToTimestampFunctionFactory.java 1 1 100.00%
🔵 io/questdb/cairo/vm/MemoryPMARImpl.java 6 6 100.00%
🔵 io/questdb/cairo/DynamicTableReaderMetadata.java 3 3 100.00%
🔵 io/questdb/griffin/engine/functions/cast/CastDoubleToByteFunctionFactory.java 2 2 100.00%
🔵 io/questdb/griffin/engine/functions/cast/CastDoubleToFloatFunctionFactory.java 2 2 100.00%
🔵 io/questdb/griffin/engine/functions/cast/CastFloatToShortFunctionFactory.java 2 2 100.00%
🔵 io/questdb/griffin/engine/functions/cast/CastDoubleToDateFunctionFactory.java 1 1 100.00%
🔵 io/questdb/cairo/ColumnPurgeOperator.java 5 5 100.00%
🔵 io/questdb/cairo/StringTypeDriver.java 2 2 100.00%
🔵 io/questdb/griffin/engine/functions/cast/CastFloatToLongFunctionFactory.java 1 1 100.00%

@ideoma ideoma merged commit 22d2984 into master May 15, 2024
23 of 24 checks passed
@ideoma ideoma deleted the feat-column-convert branch May 15, 2024 16:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Core Related to storage, data type, etc. SQL Issues or changes relating to SQL execution
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feature: Change type of a column from SQL
5 participants