Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Refactor client to support custom connections #163

Closed
wants to merge 2 commits into from

Conversation

slvrtrn
Copy link
Contributor

@slvrtrn slvrtrn commented May 29, 2023

Summary

Refactor the client to support custom connections.

NB: I haven't implemented connections other than the existing Node.js one yet; the first attempt (browser) is planned as a follow-up. This is just the necessary restructuring before we can make some progress there.

Related issues:

The code now is a small monorepo, divided into several workspaces

  • common (platform-independent types, data/query formatting, and other utils)
  • client (the main client with all the config parsing and validation, without the connection layer, depends on the common workspace)
  • node connection (the actual Node.js implementation, which depends on the client workspace)

The idea is that we can create new connections (fetch/axios/whatever) as just a new workspace and run CI using all of them one by one (run all tests with node connections, run a subset of tests for the browser, etc).

This also means we will have multiple NPM packages - the base client itself without a connection implementation included (replacing current @clickhouse/client) and at least one package per connection (@clickhouse/client-node, @clickhouse/client-fetch). The end user will have to install only one, though. This PR does not address the release routine (need to address the imports/paths, adjust GitHub Actions, etc)

Once it is released (when at least one new connection is made), the changes for the existing Node.js users should be minimal, just a simple change of import (createClient will come from @clickhouse/client-node instead of @clickhouse/client).

For example, with @clickhouse/client-node:

import { createClient } from '@clickhouse/client-node'
const client = createClient({
  host: getFromEnv('CLICKHOUSE_HOST'),
  password: getFromEnv('CLICKHOUSE_PASSWORD'),
})

See the workspace source for more details.

However, if just @clickhouse/client is added as a dependency, it is now possible to implement a custom version of the Connection interface and pass it with client configuration options.

For example:

const client = new ClickHouseClient({
  host: 'http://<host>:<port>',
  connection: (config) => new MyCustomConnection(config),
})

Where MyCustomConnection implements

export interface Connection {
  ping(): Promise<boolean>
  close(): Promise<void>
  query(params: BaseQueryParams): Promise<QueryResult>
  exec(params: BaseQueryParams): Promise<QueryResult>
  insert(params: InsertParams): Promise<InsertResult>
}

See connection.ts for more details.

Currently, we have stream and Buffer as leftover Node.js-specific types in the common module. However, there are polyfills available:

Again, as a follow-up, I will investigate what is possible for the browser or, for example, Cloudflare workers since these are the common use cases.

Checklist

  • Unit and integration tests covering the common scenarios were added
  • A human-readable description of the changes was provided to include in CHANGELOG
  • For significant changes, documentation in https://github.com/ClickHouse/clickhouse-docs was updated with further explanations or tutorials

@slvrtrn slvrtrn requested a review from mshustov May 29, 2023 20:33
@slvrtrn slvrtrn changed the base branch from main to 0.1.0 May 29, 2023 20:37
@slvrtrn
Copy link
Contributor Author

slvrtrn commented May 29, 2023

Changed the base branch from main to 0.1.0 cause I expect that it will take several PRs before we have it ready.

@slvrtrn
Copy link
Contributor Author

slvrtrn commented Jun 8, 2023

Closing this in favor of #165

@slvrtrn slvrtrn closed this Jun 8, 2023
@slvrtrn slvrtrn deleted the connections-plugins branch November 13, 2023 09:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant