Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Cloud Spanner driver implementation #8730

Merged
merged 40 commits into from
Apr 12, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
a6b3729
working on Cloud Spanner driver implementation
AlexMesser Mar 5, 2022
07525cc
working on DDL synchronization
AlexMesser Mar 12, 2022
77930a3
working on DDL synchronization
AlexMesser Mar 14, 2022
4021e6c
fixed failing test
AlexMesser Mar 14, 2022
c2bd6b3
working on VIEW implementation
AlexMesser Mar 14, 2022
fd6afd7
Merge branch 'master' into cloud-spanner
AlexMesser Mar 14, 2022
0a55b6c
fixed query parameters
AlexMesser Mar 14, 2022
4196567
lint
AlexMesser Mar 15, 2022
85734ae
added transaction support;
AlexMesser Mar 15, 2022
5db26e9
fixed column types
AlexMesser Mar 19, 2022
c566c42
Merge remote-tracking branch 'origin/master' into cloud-spanner
AlexMesser Mar 19, 2022
ab52d92
fixes after merge
AlexMesser Mar 19, 2022
e6506fd
prettier
AlexMesser Mar 19, 2022
5719771
added support for generated columns
AlexMesser Mar 19, 2022
a72d0b7
added escaping for distinct alias
AlexMesser Mar 19, 2022
0502e83
working on generated columns
AlexMesser Mar 26, 2022
e7727b5
changed failing test
AlexMesser Apr 2, 2022
83cb986
updated tests for Spanner;
AlexMesser Apr 2, 2022
fd490c6
updated tests for Spanner;
AlexMesser Apr 2, 2022
7dadc2a
Merge branch 'master' into cloud-spanner
AlexMesser Apr 2, 2022
698d7cc
updated tests for Spanner;
AlexMesser Apr 4, 2022
ec5c822
fixed failing test
AlexMesser Apr 4, 2022
cfe3560
fixed failing test
AlexMesser Apr 4, 2022
0906f65
fixing failing tests
AlexMesser Apr 6, 2022
86caac7
fixing failing tests
AlexMesser Apr 6, 2022
08f6d30
fixing failing tests
AlexMesser Apr 6, 2022
e978273
added support for typeorm-generated uuid;
AlexMesser Apr 6, 2022
130dfd0
fixing failing tests
AlexMesser Apr 6, 2022
513bc70
fixing failing tests
AlexMesser Apr 6, 2022
1fc30d0
fixing failing tests
AlexMesser Apr 7, 2022
a1ccc68
fixing failing tests
AlexMesser Apr 7, 2022
b6d158b
fixing failing tests
AlexMesser Apr 7, 2022
fb11c60
Merge branch 'master' into cloud-spanner
AlexMesser Apr 7, 2022
47feebe
fixing failing tests
AlexMesser Apr 7, 2022
e3a7a29
debugging failing test
AlexMesser Apr 7, 2022
8a65709
debugging failing test
AlexMesser Apr 7, 2022
2dda13e
fixed bug in @PrimaryColumn decorator
AlexMesser Apr 7, 2022
2bf3bd7
fixed failing tests
AlexMesser Apr 11, 2022
c6283d0
fixed VIEW functionality;
AlexMesser Apr 11, 2022
3bb96ba
updated docs
AlexMesser Apr 11, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/ISSUE_TEMPLATE/bug-report.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,7 @@ assignees: ''
| `postgres` | no |
| `react-native` | no |
| `sap` | no |
| `spanner` | no |
| `sqlite` | no |
| `sqlite-abstract` | no |
| `sqljs` | no |
Expand Down
1 change: 1 addition & 0 deletions .github/ISSUE_TEMPLATE/feature-request.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@ assignees: ''
| `postgres` | no |
| `react-native` | no |
| `sap` | no |
| `spanner` | no |
| `sqlite` | no |
| `sqlite-abstract` | no |
| `sqljs` | no |
Expand Down
37 changes: 33 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -214,12 +214,41 @@ await timber.remove()
- for **SAP Hana**

```
npm i @sap/hana-client
npm i hdb-pool
npm install @sap/hana-client
npm install hdb-pool
```

_SAP Hana support made possible by the sponsorship of [Neptune Software](https://www.neptune-software.com/)._

- for **Google Cloud Spanner**

```
npm install @google-cloud/spanner --save
```

Provide authentication credentials to your application code
by setting the environment variable `GOOGLE_APPLICATION_CREDENTIALS`:

```shell
# Linux/macOS
export GOOGLE_APPLICATION_CREDENTIALS="KEY_PATH"

# Windows
set GOOGLE_APPLICATION_CREDENTIALS=KEY_PATH

# Replace KEY_PATH with the path of the JSON file that contains your service account key.
```

To use Spanner with the emulator you should set `SPANNER_EMULATOR_HOST` environment variable:

```shell
# Linux/macOS
export SPANNER_EMULATOR_HOST=localhost:9010

# Windows
set SPANNER_EMULATOR_HOST=localhost:9010
```

- for **MongoDB** (experimental)

`npm install mongodb@^3.6.0 --save`
Expand Down Expand Up @@ -255,7 +284,7 @@ npx typeorm init --name MyProject --database postgres
```

Where `name` is the name of your project and `database` is the database you'll use.
Database can be one of the following values: `mysql`, `mariadb`, `postgres`, `cockroachdb`, `sqlite`, `mssql`, `oracle`, `mongodb`,
Database can be one of the following values: `mysql`, `mariadb`, `postgres`, `cockroachdb`, `sqlite`, `mssql`, `sap`, `spanner`, `oracle`, `mongodb`,
`cordova`, `react-native`, `expo`, `nativescript`.

This command will generate a new project in the `MyProject` directory with the following files:
Expand Down Expand Up @@ -553,7 +582,7 @@ AppDataSource.initialize()

We are using Postgres in this example, but you can use any other supported database.
To use another database, simply change the `type` in the options to the database type you are using:
`mysql`, `mariadb`, `postgres`, `cockroachdb`, `sqlite`, `mssql`, `oracle`, `cordova`, `nativescript`, `react-native`,
`mysql`, `mariadb`, `postgres`, `cockroachdb`, `sqlite`, `mssql`, `oracle`, `sap`, `spanner`, `cordova`, `nativescript`, `react-native`,
`expo`, or `mongodb`.
Also make sure to use your own host, port, username, password and database settings.

Expand Down
8 changes: 8 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,14 @@ services:
- default
- typeorm

# google cloud spanner
spanner:
image: alexmesser/spanner-emulator
container_name: "typeorm-spanner"
ports:
- "9010:9010"
- "9020:9020"

# sap hana (works only on linux)
# hanaexpress:
# image: "store/saplabs/hanaexpress:2.00.040.00.20190729.1"
Expand Down
2 changes: 1 addition & 1 deletion docs/data-source-options.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ Different RDBMS-es have their own specific options.

- `type` - RDBMS type. You must specify what database engine you use.
Possible values are:
"mysql", "postgres", "cockroachdb", "sap", "mariadb", "sqlite", "cordova", "react-native", "nativescript", "sqljs", "oracle", "mssql", "mongodb", "aurora-mysql", "aurora-postgres", "expo", "better-sqlite3", "capacitor".
"mysql", "postgres", "cockroachdb", "sap", "spanner", "mariadb", "sqlite", "cordova", "react-native", "nativescript", "sqljs", "oracle", "mssql", "mongodb", "aurora-mysql", "aurora-postgres", "expo", "better-sqlite3", "capacitor".
This option is **required**.

- `extra` - Extra options to be passed to the underlying driver.
Expand Down
4 changes: 4 additions & 0 deletions docs/entities.md
Original file line number Diff line number Diff line change
Expand Up @@ -337,6 +337,10 @@ or
`timestamp with local time zone`, `interval year to month`, `interval day to second`, `bfile`, `blob`, `clob`,
`nclob`, `rowid`, `urowid`

### Column types for `spanner`

`bool`, `int64`, `float64`, `numeric`, `string`, `json`, `bytes`, `date`, `timestamp`, `array`

### `enum` column type

`enum` column type is supported by `postgres` and `mysql`. There are various possible column definitions:
Expand Down
17 changes: 12 additions & 5 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -74,12 +74,15 @@
"postgresql-orm",
"mariadb",
"mariadb-orm",
"spanner",
"sqlite",
"sqlite-orm",
"sql-server",
"sql-server-orm",
"oracle",
"oracle-orm"
"oracle-orm",
"cloud-spanner",
"cloud-spanner-orm"
],
"devDependencies": {
"@types/app-root-path": "^1.2.4",
Expand Down Expand Up @@ -121,6 +124,7 @@
"mysql2": "^2.2.5",
"pg": "^8.5.1",
"pg-query-stream": "^4.0.0",
"prettier": "^2.5.1",
"redis": "^3.1.1",
"remap-istanbul": "^0.13.0",
"rimraf": "^3.0.2",
Expand All @@ -131,10 +135,10 @@
"sqlite3": "^5.0.2",
"ts-node": "^10.7.0",
"typeorm-aurora-data-api-driver": "^2.0.0",
"typescript": "^4.6.2",
"prettier": "^2.5.1"
"typescript": "^4.6.2"
},
"peerDependencies": {
"@google-cloud/spanner": "^5.18.0",
"@sap/hana-client": "^2.11.14",
"better-sqlite3": "^7.1.2",
"hdb-pool": "^0.1.6",
Expand All @@ -153,6 +157,9 @@
"typeorm-aurora-data-api-driver": "^2.0.0"
},
"peerDependenciesMeta": {
"@google-cloud/spanner": {
"optional": true
},
"@sap/hana-client": {
"optional": true
},
Expand Down Expand Up @@ -208,6 +215,7 @@
"buffer": "^6.0.3",
"chalk": "^4.1.0",
"cli-highlight": "^2.1.11",
"date-fns": "^2.28.0",
"debug": "^4.3.3",
"dotenv": "^16.0.0",
"glob": "^7.2.0",
Expand All @@ -218,8 +226,7 @@
"tslib": "^2.3.1",
"uuid": "^8.3.2",
"xml2js": "^0.4.23",
"yargs": "^17.3.1",
"date-fns": "^2.28.0"
"yargs": "^17.3.1"
},
"scripts": {
"test": "rimraf ./build && tsc && mocha --file ./build/compiled/test/utils/test-setup.js --bail --recursive --timeout 60000 ./build/compiled/test",
Expand Down
2 changes: 1 addition & 1 deletion sample/sample1-simple-entity/entity/Post.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import { Generated } from "../../../src/decorator/Generated"

@Entity("sample01_post")
export class Post {
@PrimaryColumn("integer")
@PrimaryColumn()
@Generated()
id: number

Expand Down
14 changes: 13 additions & 1 deletion src/cache/DbQueryResultCache.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import { QueryRunner } from "../query-runner/QueryRunner"
import { Table } from "../schema-builder/table/Table"
import { QueryResultCache } from "./QueryResultCache"
import { QueryResultCacheOptions } from "./QueryResultCacheOptions"
import { v4 as uuidv4 } from "uuid"

/**
* Caches query result into current database, into separate table called "query-result-cache".
Expand Down Expand Up @@ -80,7 +81,10 @@ export class DbQueryResultCache implements QueryResultCache {
type: driver.normalizeType({
type: driver.mappedDataTypes.cacheId,
}),
generationStrategy: "increment",
generationStrategy:
driver.options.type === "spanner"
? "uuid"
: "increment",
isGenerated: true,
},
{
Expand Down Expand Up @@ -256,6 +260,14 @@ export class DbQueryResultCache implements QueryResultCache {

await qb.execute()
} else {
// Spanner does not support auto-generated columns
if (
this.connection.driver.options.type === "spanner" &&
!insertedValues.id
) {
insertedValues.id = uuidv4()
}

// otherwise insert
await queryRunner.manager
.createQueryBuilder()
Expand Down
2 changes: 2 additions & 0 deletions src/data-source/DataSourceOptions.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ import { SapConnectionOptions } from "../driver/sap/SapConnectionOptions"
import { AuroraPostgresConnectionOptions } from "../driver/aurora-postgres/AuroraPostgresConnectionOptions"
import { BetterSqlite3ConnectionOptions } from "../driver/better-sqlite3/BetterSqlite3ConnectionOptions"
import { CapacitorConnectionOptions } from "../driver/capacitor/CapacitorConnectionOptions"
import { SpannerConnectionOptions } from "../driver/spanner/SpannerConnectionOptions"

/**
* DataSourceOptions is an interface with settings and options for specific DataSource.
Expand All @@ -37,3 +38,4 @@ export type DataSourceOptions =
| ExpoConnectionOptions
| BetterSqlite3ConnectionOptions
| CapacitorConnectionOptions
| SpannerConnectionOptions
1 change: 1 addition & 0 deletions src/decorator/Index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,7 @@ export function Index(
unique: options && options.unique ? true : false,
spatial: options && options.spatial ? true : false,
fulltext: options && options.fulltext ? true : false,
nullFiltered: options && options.nullFiltered ? true : false,
parser: options ? options.parser : undefined,
sparse: options && options.sparse ? true : false,
background: options && options.background ? true : false,
Expand Down
9 changes: 7 additions & 2 deletions src/decorator/columns/PrimaryColumn.ts
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,13 @@ export function PrimaryColumn(
return function (object: Object, propertyName: string) {
// normalize parameters
let type: ColumnType | undefined
if (typeof typeOrOptions === "string") {
type = typeOrOptions
if (
typeof typeOrOptions === "string" ||
typeOrOptions === String ||
typeOrOptions === Boolean ||
typeOrOptions === Number
) {
type = typeOrOptions as ColumnType
} else {
options = Object.assign({}, <PrimaryColumnOptions>typeOrOptions)
}
Expand Down
9 changes: 9 additions & 0 deletions src/decorator/options/IndexOptions.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,15 @@ export interface IndexOptions {
*/
fulltext?: boolean

/**
* NULL_FILTERED indexes are particularly useful for indexing sparse columns, where most rows contain a NULL value.
* In these cases, the NULL_FILTERED index can be considerably smaller and more efficient to maintain than
* a normal index that includes NULL values.
*
* Works only in Spanner.
*/
nullFiltered?: boolean

/**
* Fulltext parser.
* Works only in MySQL.
Expand Down
4 changes: 4 additions & 0 deletions src/driver/DriverFactory.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ import { DataSource } from "../data-source/DataSource"
import { SapDriver } from "./sap/SapDriver"
import { BetterSqlite3Driver } from "./better-sqlite3/BetterSqlite3Driver"
import { CapacitorDriver } from "./capacitor/CapacitorDriver"
import { SpannerDriver } from "./spanner/SpannerDriver"

/**
* Helps to create drivers.
Expand Down Expand Up @@ -65,6 +66,8 @@ export class DriverFactory {
return new AuroraPostgresDriver(connection)
case "capacitor":
return new CapacitorDriver(connection)
case "spanner":
return new SpannerDriver(connection)
default:
throw new MissingDriverError(type, [
"aurora-mysql",
Expand All @@ -85,6 +88,7 @@ export class DriverFactory {
"sap",
"sqlite",
"sqljs",
"spanner",
])
}
}
Expand Down
19 changes: 19 additions & 0 deletions src/driver/spanner/SpannerConnectionCredentialsOptions.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
/**
* Spanner specific connection credential options.
*/
export interface SpannerConnectionCredentialsOptions {
/**
* Connection url where perform connection to.
*/
readonly instanceId?: string

/**
* Database host.
*/
readonly projectId?: string

/**
* Database host port.
*/
readonly databaseId?: string
}