Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support direct shard database operation routing in Spring JDBC #31506

Conversation

meedbek
Copy link
Contributor

@meedbek meedbek commented Oct 26, 2023

This PR is a follow up to #30988.
Goal is to add support for direct database operation routing in sharded databases in Spring JDBC.

This functionality is achieved by acquiring a shard connection through the JDBC API
DataSource.createConnectionBuilder().shardingKey(key).build().

In this PR we will focus on one of the two approaches that were introduced in the previous PR.

We introduce a DataSource adapter class that applies sharding keys to every standard getConnection() call. The sharding keys are specified through a ShardingKeyProvider object.

Here's an example of how to use it:

ShardingKeyDataSourceAdapter dataSourceAdapter = new ShardingKeyDataSourceAdapter(dataSource);

dataSourceAdapter.setShardingKeyProvider(new ShardingKeyProvider() {
    public ShardingKey getShardingKey() throws SQLException {
        // Example: The sharding key is derived from the user name.
        String name = SecurityContextHolder.getContext().getAuthentication().getName();
        return dataSource.createShardingKeyBuilder().subKey(name, JDBCType.VARCHAR).build();
    }
    
    public ShardingKey getSuperShardingKey() throws SQLException {
        return null;
    }
});

JdbcTemplate shardingJdbcTemplate = new JdbcTemplate(dataSourceAdapter);

// The SQL query will be executed directly in the shard corresponding 
// to the sharding key returned from the ShardingKeyProvider.getShardingKey() method.
shardingJdbcTemplate.execute(SQL);

Mohamed Lahyane (Anir) and others added 4 commits August 3, 2023 11:04
This commit introduces a DataSource proxy, that changes the behavior of the getConnection method to use the `createConnectionBuilder()` api to acquire direct shard connections. The shard connection is acquired by specifying a `ShardingKey` that is correspondent to the wanted shard.

The sharding key can be specified either by setting the thread bound sharding key value, which will be used later by the `getConnection()` method, or by defining a `ShardingKeyProvider` which is used in the `getConnection` method to get the ShardingKey.

If both ways are used at the same time, the thread-bound sharding key is the one that will be used.

The goal of binding the sharding key to the current thread, is that we want to specify the sharding key at one level, and want it to be used later in another level, without the need to propagate it, for example set the thread-bound sharding key, and execute a `JdbcTemplate.query(SQL)` method that that will call `DataSource.getConnection()`, and as the sharding key is bound to the thread, it can be acquired to get the shard connection.
@spring-projects-issues spring-projects-issues added the status: waiting-for-triage An issue we've not yet triaged or decided on label Oct 26, 2023
@meedbek meedbek changed the title Support direct shard routing through key provider Support direct shard database operation routing in Spring JDBC Oct 26, 2023
@snicoll
Copy link
Member

snicoll commented Oct 26, 2023

@meedbek thanks for the PR, but I think it would only be actionable if we decide to merge #30988, which we haven't. Am I missing something?

@meedbek meedbek closed this Oct 26, 2023
@meedbek meedbek reopened this Oct 26, 2023
@meedbek
Copy link
Contributor Author

meedbek commented Oct 26, 2023

@meedbek thanks for the PR, but I think it would only be actionable if we decide to merge #30988, which we haven't. Am I missing something?

I am taking over @Anir-ln work on the feature that's why I opened a new PR.
I am focusing on one side of the feature for now, I will look into the rest later.

@snicoll
Copy link
Member

snicoll commented Oct 26, 2023

What do you mean by "taking over"? You mean replacing? If so, then there should be some discussion to close the other PR then.

@meedbek
Copy link
Contributor Author

meedbek commented Oct 26, 2023

What do you mean by "taking over"? You mean replacing? If so, then there should be some discussion to close the other PR then.

@snicoll thank you for the reply, it's done. The other PR has been closed.

@jeandelavarene
Copy link

Hallo @jhoeller, we would like to have a meeting with you to review the comments you made in the older PR #30988 regarding transactions. Would that be possible? If so, how to proceed? Thank you.

@jhoeller jhoeller added in: data Issues in data modules (jdbc, orm, oxm, tx) type: enhancement A general enhancement and removed status: waiting-for-triage An issue we've not yet triaged or decided on labels Nov 14, 2023
@jhoeller jhoeller self-assigned this Nov 14, 2023
@jhoeller jhoeller added this to the 6.1.x milestone Nov 14, 2023
@jhoeller jhoeller modified the milestones: 6.1.x, 6.1.2 Nov 21, 2023
@jhoeller jhoeller merged commit e4e2224 into spring-projects:main Dec 8, 2023
3 checks passed
jhoeller added a commit that referenced this pull request Dec 8, 2023
Also moves ShardingKeyProvider to datasource package and declares getSuperShardingKey as default method.

Closes gh-31795
See gh-31506
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
in: data Issues in data modules (jdbc, orm, oxm, tx) type: enhancement A general enhancement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants