Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: gajus/slonik
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v30.0.0
Choose a base ref
...
head repository: gajus/slonik
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: v30.1.0
Choose a head ref
  • 3 commits
  • 27 files changed
  • 1 contributor

Commits on Aug 5, 2022

  1. docs: split library comparison

    @gajus
    gajus committed Aug 5, 2022
    Copy the full SHA
    6db9857 View commit details

  2. docs: clean up documentation

    @gajus
    gajus committed Aug 5, 2022
    Copy the full SHA
    aed170d View commit details

  3. feat: add runtime result validation using zod (#375)

    @gajus
    gajus authored Aug 5, 2022
    Copy the full SHA
    2948a7b View commit details
82 changes: 35 additions & 47 deletions .README/ABOUT_SLONIK.md
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
## About Slonik

### Battle-Tested

Slonik began as a collection of utilities designed for working with [`node-postgres`](https://github.com/brianc/node-postgres). We continue to use `node-postgres` as it provides a robust foundation for interacting with PostgreSQL. However, what once was a collection of utilities has since grown into a framework that abstracts repeating code patterns, protects against unsafe connection handling and value interpolation, and provides rich debugging experience.
Slonik began as a collection of utilities designed for working with [`node-postgres`](https://github.com/brianc/node-postgres). It continues to use `node-postgres` driver as it provides a robust foundation for interacting with PostgreSQL. However, what once was a collection of utilities has since grown into a framework that abstracts repeating code patterns, protects against unsafe connection handling and value interpolation, and provides a rich debugging experience.

Slonik has been [battle-tested](https://medium.com/@gajus/lessons-learned-scaling-postgresql-database-to-1-2bn-records-month-edc5449b3067) with large data volumes and queries ranging from simple CRUD operations to data-warehousing needs.

@@ -16,14 +18,12 @@ Read: [The History of Slonik, the PostgreSQL Elephant Logo](https://www.vertabel

Among the primary reasons for developing Slonik, was the motivation to reduce the repeating code patterns and add a level of type safety. This is primarily achieved through the methods such as `one`, `many`, etc. But what is the issue? It is best illustrated with an example.

Suppose the requirement is to write a method that retrieves a resource ID given values defining (what we assume to be) a unique constraint. If we did not have the aforementioned convenience methods available, then it would need to be written as:
Suppose the requirement is to write a method that retrieves a resource ID given values defining (what we assume to be) a unique constraint. If we did not have the aforementioned helper methods available, then it would need to be written as:

```js
```ts
import {
sql
} from 'slonik';
import type {
DatabaseConnection
sql,
type DatabaseConnection
} from 'slonik';

type DatabaseRecordIdType = number;
@@ -45,20 +45,18 @@ const getFooIdByBar = async (connection: DatabaseConnection, bar: string): Promi

return fooResult[0].id;
};

```

`oneFirst` method abstracts all of the above logic into:

```js
```ts
const getFooIdByBar = (connection: DatabaseConnection, bar: string): Promise<DatabaseRecordIdType> => {
return connection.oneFirst(sql`
SELECT id
FROM foo
WHERE bar = ${bar}
`);
};

```

`oneFirst` throws:
@@ -67,11 +65,11 @@ const getFooIdByBar = (connection: DatabaseConnection, bar: string): Promise<Dat
* `DataIntegrityError` if query returns multiple rows
* `DataIntegrityError` if query returns multiple columns

This becomes particularly important when writing routines where multiple queries depend on the previous result. Using methods with inbuilt assertions ensures that in case of an error, the error points to the original source of the problem. In contrast, unless assertions for all possible outcomes are typed out as in the previous example, the unexpected result of the query will be fed to the next operation. If you are lucky, the next operation will simply break; if you are unlucky, you are risking data corruption and hard to locate bugs.
In the absence of helper methods, the overhead of repeating code becomes particularly visible when writing routines where multiple queries depend on the proceeding query results. Using methods with inbuilt assertions ensures that in case of an error, the error points to the source of the problem. In contrast, unless assertions for all possible outcomes are typed out as in the previous example, the unexpected result of the query will be fed to the next operation. If you are lucky, the next operation will simply break; if you are unlucky, you are risking data corruption and hard-to-locate bugs.

Furthermore, using methods that guarantee the shape of the results, allows us to leverage static type checking and catch some of the errors even before they executing the code, e.g.
Furthermore, using methods that guarantee the shape of the results allows us to leverage static type checking and catch some of the errors even before executing the code, e.g.

```js
```ts
const fooId = await connection.many(sql`
SELECT id
FROM foo
@@ -82,7 +80,6 @@ await connection.query(sql`
DELETE FROM baz
WHERE foo_id = ${fooId}
`);

```

Static type check of the above example will produce a warning as the `fooId` is guaranteed to be an array and binding of the last query is expecting a primitive value.
@@ -93,8 +90,8 @@ Slonik only allows to check out a connection for the duration of the promise rou

The primary reason for implementing _only_ this connection pooling method is because the alternative is inherently unsafe, e.g.

```js
// Note: This example is using unsupported API.
```ts
// This is not valid Slonik API

const main = async () => {
const connection = await pool.connect();
@@ -103,15 +100,14 @@ const main = async () => {

await connection.release();
};

```

In this example, if `SELECT foo()` produces an error, then connection is never released, i.e. the connection remains to hang.
In this example, if `SELECT foo()` produces an error, then connection is never released, i.e. the connection hangs indefinitely.

A fix to the above is to ensure that `connection#release()` is always called, i.e.

```js
// Note: This example is using unsupported API.
```ts
// This is not valid Slonik API

const main = async () => {
const connection = await pool.connect();
@@ -126,32 +122,29 @@ const main = async () => {

return lastExecutionResult;
};

```

Slonik abstracts the latter pattern into `pool#connect()` method.

```js
```ts
const main = () => {
return pool.connect((connection) => {
return connection.query(sql`SELECT foo()`);
});
};

```

Connection is always released back to the pool after the promise produced by the function supplied to `connect()` method is either resolved or rejected.
Using this pattern, we guarantee that connection is always released as soon as the `connect()` routine resolves or is rejected.

### Protecting against unsafe transaction handling

Just like in the [unsafe connection handling](#protecting-against-unsafe-connection-handling) described above, Slonik only allows to create a transaction for the duration of the promise routine supplied to the `connection#transaction()` method.
Just like in the [unsafe connection handling](#protecting-against-unsafe-connection-handling) example, Slonik only allows to create a transaction for the duration of the promise routine supplied to the `connection#transaction()` method.

```js
```ts
connection.transaction(async (transactionConnection) => {
await transactionConnection.query(sql`INSERT INTO foo (bar) VALUES ('baz')`);
await transactionConnection.query(sql`INSERT INTO qux (quux) VALUES ('quuz')`);
});

```

This pattern ensures that the transaction is either committed or aborted the moment the promise is either resolved or rejected.
@@ -160,33 +153,32 @@ This pattern ensures that the transaction is either committed or aborted the mom

[SQL injections](https://en.wikipedia.org/wiki/SQL_injection) are one of the most well known attack vectors. Some of the [biggest data leaks](https://en.wikipedia.org/wiki/SQL_injection#Examples) were the consequence of improper user-input handling. In general, SQL injections are easily preventable by using parameterization and by restricting database permissions, e.g.

```js
// Note: This example is using unsupported API.
```ts
// This is not valid Slonik API

connection.query('SELECT $1', [
userInput
]);

```

In this example, the query text (`SELECT $1`) and parameters (value of the `userInput`) are passed to the PostgreSQL server where the parameters are safely substituted into the query. This is a safe way to execute a query using user-input.
In this example, the query text (`SELECT $1`) and parameters (`userInput`) are passed separately to the PostgreSQL server where the parameters are safely substituted into the query. This is a safe way to execute a query using user-input.

The vulnerabilities appear when developers cut corners or when they do not know about parameterization, i.e. there is a risk that someone will instead write:

```js
// Note: This example is using unsupported API.
```ts
// This is not valid Slonik API

connection.query('SELECT \'' + userInput + '\'');

```

As evident by the history of the data leaks, this happens more often than anyone would like to admit. This is especially a big risk in Node.js community, where predominant number of developers are coming from frontend and have not had training working with RDBMSes. Therefore, one of the key selling points of Slonik is that it adds multiple layers of protection to prevent unsafe handling of user-input.
As evident by the history of the data leaks, this happens more often than anyone would like to admit. This security vulnerability is especially a significant risk in Node.js community, where a predominant number of developers are coming from frontend and have not had training working with RDBMSes. Therefore, one of the key selling points of Slonik is that it adds multiple layers of protection to prevent unsafe handling of user input.

To begin with, Slonik does not allow to run plain-text queries.
To begin with, Slonik does not allow running plain-text queries.

```js
connection.query('SELECT 1');
```ts
// This is not valid Slonik API

connection.query('SELECT 1');
```

The above invocation would produce an error:
@@ -195,23 +187,21 @@ The above invocation would produce an error:
This means that the only way to run a query is by constructing it using [`sql` tagged template literal](https://github.com/gajus/slonik#slonik-value-placeholders-tagged-template-literals), e.g.

```js
```ts
connection.query(sql`SELECT 1`);

```

To add a parameter to the query, user must use [template literal placeholders](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals#Description), e.g.

```js
```ts
connection.query(sql`SELECT ${userInput}`);

```

Slonik takes over from here and constructs a query with value bindings, and sends the resulting query text and parameters to the PostgreSQL. As `sql` tagged template literal is the only way to execute the query, it adds a strong layer of protection against accidental unsafe user-input handling due to limited knowledge of the SQL client API.
Slonik takes over from here and constructs a query with value bindings, and sends the resulting query text and parameters to PostgreSQL. There is no other way of passing parameters to the query – this adds a strong layer of protection against accidental unsafe user input handling due to limited knowledge of the SQL client API.

As Slonik restricts user's ability to generate and execute dynamic SQL, it provides helper functions used to generate fragments of the query and the corresponding value bindings, e.g. [`sql.identifier`](#sqlidentifier), [`sql.join`](#sqljoin) and [`sql.unnest`](#sqlunnest). These methods generate tokens that the query executor interprets to construct a safe query, e.g.

```js
```ts
connection.query(sql`
SELECT ${sql.identifier(['foo', 'a'])}
FROM (
@@ -228,7 +218,6 @@ connection.query(sql`
) foo(a, b, c)
WHERE foo.b IN (${sql.join(['c1', 'a2'], sql`, `)})
`);

```

This (contrived) example generates a query equivalent to:
@@ -241,9 +230,8 @@ FROM (
($4, $5, $6)
) foo(a, b, c)
WHERE foo.b IN ($7, $8)

```

That is executed with the parameters provided by the user.
This query is executed with the parameters provided by the user.

To sum up, Slonik is designed to prevent accidental creation of queries vulnerable to SQL injections.
55 changes: 51 additions & 4 deletions .README/DEBUGGING.md
View file
Original file line number Diff line number Diff line change
@@ -16,10 +16,57 @@ Note: Requires [`slonik-interceptor-query-logging`](https://github.com/gajus/slo

Enabling `captureStackTrace` configuration will create a stack trace before invoking the query and include the stack trace in the logs, e.g.

```json
{"context":{"package":"slonik","namespace":"slonik","logLevel":20,"executionTime":"357 ms","queryId":"01CV2V5S4H57KCYFFBS0BJ8K7E","rowCount":1,"sql":"SELECT schedule_cinema_data_task();","stackTrace":["/Users/gajus/Documents/dev/applaudience/data-management-program/node_modules/slonik/dist:162:28","/Users/gajus/Documents/dev/applaudience/data-management-program/node_modules/slonik/dist:314:12","/Users/gajus/Documents/dev/applaudience/data-management-program/node_modules/slonik/dist:361:20","/Users/gajus/Documents/dev/applaudience/data-management-program/node_modules/slonik/dist/utilities:17:13","/Users/gajus/Documents/dev/applaudience/data-management-program/src/bin/commands/do-cinema-data-tasks.js:59:21","/Users/gajus/Documents/dev/applaudience/data-management-program/src/bin/commands/do-cinema-data-tasks.js:590:45","internal/process/next_tick.js:68:7"],"values":[]},"message":"query","sequence":4,"time":1540915127833,"version":"1.0.0"}
{"context":{"package":"slonik","namespace":"slonik","logLevel":20,"executionTime":"66 ms","queryId":"01CV2V5SGS0WHJX4GJN09Z3MTB","rowCount":1,"sql":"SELECT cinema_id \"cinemaId\", target_data \"targetData\" FROM cinema_data_task WHERE id = ?","stackTrace":["/Users/gajus/Documents/dev/applaudience/data-management-program/node_modules/slonik/dist:162:28","/Users/gajus/Documents/dev/applaudience/data-management-program/node_modules/slonik/dist:285:12","/Users/gajus/Documents/dev/applaudience/data-management-program/node_modules/slonik/dist/utilities:17:13","/Users/gajus/Documents/dev/applaudience/data-management-program/src/bin/commands/do-cinema-data-tasks.js:603:26","internal/process/next_tick.js:68:7"],"values":[17953947]},"message":"query","sequence":5,"time":1540915127902,"version":"1.0.0"}

```tson
{
"context": {
"package": "slonik",
"namespace": "slonik",
"logLevel": 20,
"executionTime": "357 ms",
"queryId": "01CV2V5S4H57KCYFFBS0BJ8K7E",
"rowCount": 1,
"sql": "SELECT schedule_cinema_data_task();",
"stackTrace": [
"/node_modules/slonik/dist:162:28",
"/node_modules/slonik/dist:314:12",
"/node_modules/slonik/dist:361:20",
"/node_modules/slonik/dist/utilities:17:13",
"/src/bin/commands/do-cinema-data-tasks.js:59:21",
"/src/bin/commands/do-cinema-data-tasks.js:590:45",
"internal/process/next_tick.js:68:7"
],
"values": []
},
"message": "query",
"sequence": 4,
"time": 1540915127833,
"version": "1.0.0"
}
{
"context": {
"package": "slonik",
"namespace": "slonik",
"logLevel": 20,
"executionTime": "66 ms",
"queryId": "01CV2V5SGS0WHJX4GJN09Z3MTB",
"rowCount": 1,
"sql": "SELECT cinema_id \"cinemaId\", target_data \"targetData\" FROM cinema_data_task WHERE id = ?",
"stackTrace": [
"/node_modules/slonik/dist:162:28",
"/node_modules/slonik/dist:285:12",
"/node_modules/slonik/dist/utilities:17:13",
"/src/bin/commands/do-cinema-data-tasks.js:603:26",
"internal/process/next_tick.js:68:7"
],
"values": [
17953947
]
},
"message": "query",
"sequence": 5,
"time": 1540915127902,
"version": "1.0.0"
}
```

Use [`@roarr/cli`](https://github.com/gajus/roarr-cli) to pretty-print the output.
19 changes: 7 additions & 12 deletions .README/ERROR_HANDLING.md
View file
Original file line number Diff line number Diff line change
@@ -2,7 +2,7 @@

All Slonik errors extend from `SlonikError`, i.e. You can catch Slonik specific errors using the following logic.

```js
```ts
import {
SlonikError
} from 'slonik';
@@ -14,24 +14,23 @@ try {
// This error is thrown by Slonik.
}
}

```

### Original `node-postgres` error

When error originates from `node-postgres`, the original error is available under `originalError` property.

This propery is exposed for debugging purposes only. Do not use it for conditional checks – it can change.
This property is exposed for debugging purposes only. Do not use it for conditional checks – it can change.

If you require to extract meta-data about a specific type of error (e.g. contraint violation name), raise a GitHub issue describing your use case.
If you require to extract meta-data about a specific type of error (e.g. constraint violation name), raise a GitHub issue describing your use case.

### Handling `BackendTerminatedError`

`BackendTerminatedError` is thrown when the backend is terminated by the user, i.e. [`pg_terminate_backend`](https://www.postgresql.org/docs/current/functions-admin.html#FUNCTIONS-ADMIN-SIGNAL).

`BackendTerminatedError` must be handled at the connection level, i.e.

```js
```ts
await pool.connect(async (connection0) => {
try {
await pool.connect(async (connection1) => {
@@ -55,7 +54,6 @@ await pool.connect(async (connection0) => {
}
}
});

```

### Handling `CheckIntegrityConstraintViolationError`
@@ -70,7 +68,7 @@ await pool.connect(async (connection0) => {

To handle the case where the data result does not match the expectations, catch `DataIntegrityError` error.

```js
```ts
import {
DataIntegrityError
} from 'slonik';
@@ -86,7 +84,6 @@ try {
throw error;
}
}

```

### Handling `ForeignKeyIntegrityConstraintViolationError`
@@ -97,7 +94,7 @@ try {

To handle the case where query returns less than one row, catch `NotFoundError` error.

```js
```ts
import {
NotFoundError
} from 'slonik';
@@ -115,7 +112,6 @@ try {
if (row) {
// row.foo is the result of the `foo` column value of the first row.
}

```

### Handling `NotNullIntegrityConstraintViolationError`
@@ -128,7 +124,7 @@ if (row) {

It should be safe to use the same connection if `StatementCancelledError` is handled, e.g.

```js
```ts
await pool.connect(async (connection0) => {
await pool.connect(async (connection1) => {
const backendProcessId = await connection1.oneFirst(sql`SELECT pg_backend_pid()`);
@@ -148,7 +144,6 @@ await pool.connect(async (connection0) => {
}
});
});

```

### Handling `StatementTimeoutError`
Loading