Couchbase SDK 3, Spring Data Couchbase 4 Integration #15645

gkysaad · 2021-07-14T04:24:07Z

This PR implements a fix for and is related to the following Issues/PRs: #14184, #11845, #13770, #13544, #12300

This adds support for this issue: #14745 and was tested with the latest stable version of Couchbase (6.6.2) and uses Spring Data Couchbase 4 (Couchbase Java SDK 3).

I have used @mraible's fix-couchbase branch here and in PR #14184 as a starting point for my work on this.

Some notes:

I have disabled a couple of the integration tests as they were flaky (but worked fine when the endpoint was tested locally). These tests are: UserResourceIT.getAllUsers() and PublicUserResourceIT.getAllAuthorities().
Some of the other tests are flaky as well, but pass the majority of the time (and are random), so rerunning the pipeline usually makes it pass. I can disable these as well if I pinpoint any that fail more frequently.
This fix is for the non-reactive Couchbase implementation, so Vue, React, and Angular tests were re-enabled, but not Webflux
@N1qlJoin support in Spring Data Couchbase 4 seems to be broken or disabled - there is little documentation or info about it online as well, so I've switched over the way document relationships are handled to use the normal @Field annotations to embed the documents. I will create a separate issue in case anyone would like to look into this further.

Please make sure the below checklist is followed for Pull Requests.

All continuous integration tests are green
Tests are added where necessary
The JDL part is updated if necessary
jhipster-online is updated if necessary
Documentation is added/updated where necessary
Coding Rules & Commit Guidelines as per our CONTRIBUTING.md document are followed

When you are still working on the PR, consider converting it to Draft (bellow reviewers) and adding skip-ci label, you can still see CI build result at your branch.

CLAassistant · 2021-07-14T04:26:42Z

All committers have signed the CLA.

DanielFran · 2021-07-14T12:56:58Z

@gkysaad Can you please sign the cla?

Tcharl · 2021-07-20T19:39:24Z

Whow, it looks like there's a lot of code to achieve in order to consume couchbase with spring

DanielFran · 2021-07-27T07:11:07Z

@gkysaad Can you fix the conflicts please?

gkysaad · 2021-08-05T22:52:00Z

Apologies for the delay on the conflicts/CLA - should be able to finish that up soon

tchlyah · 2021-08-08T13:20:46Z

Hello,

I tried to generate an app to test this, but I keep getting this error after generating some files, so the app is not fully generated.

ERROR! Callback called multiple times
Error [ERR_MULTIPLE_CALLBACK]: Callback called multiple times
    at NodeError (node:internal/errors:371:5)
    at onFinish (node:internal/streams/writable:667:37)
    at processTicksAndRejections (node:internal/process/task_queues:82:21)
    at runNextTicks (node:internal/process/task_queues:65:3)
    at listOnTimeout (node:internal/timers:526:9)
    at processTimers (node:internal/timers:500:7) {
  code: 'ERR_MULTIPLE_CALLBACK'
}

any one has an idea ?

tchlyah · 2021-08-10T15:23:15Z

Hello,

I tried to generate an app to test this, but I keep getting this error after generating some files, so the app is not fully generated.

ERROR! Callback called multiple times
Error [ERR_MULTIPLE_CALLBACK]: Callback called multiple times
    at NodeError (node:internal/errors:371:5)
    at onFinish (node:internal/streams/writable:667:37)
    at processTicksAndRejections (node:internal/process/task_queues:82:21)
    at runNextTicks (node:internal/process/task_queues:65:3)
    at listOnTimeout (node:internal/timers:526:9)
    at processTimers (node:internal/timers:500:7) {
  code: 'ERR_MULTIPLE_CALLBACK'
}

any one has an idea ?

Nevermind, the issue was about using a non LTS version of node, so I installed back the latest LTS version (14.17.4), and now it is working.

…ory and add fix

mraible · 2021-08-12T15:26:24Z

@gkysaad Can you please resolve conflicts?

gkysaad · 2021-08-13T02:21:53Z

Signed the CLA, working on the rebase

tchlyah

This seems to be a huge work, thanks for your help. Can you check out my comment please?

...tity-server/templates/src/main/java/package/domain/Entity.java.jhi.spring_data_couchbase.ejs

generators/entity-server/templates/src/test/java/package/web/rest/EntityResourceIT.java.ejs

generators/server/templates/src/main/java/package/repository/AuthorityRepository.java.ejs

generators/server/templates/src/test/java/package/web/rest/TestUtil.java.ejs

generators/server/templates/build.gradle.ejs

generators/entity-server/templates/src/main/java/package/repository/EntityRepository.java.ejs

tchlyah · 2021-08-13T15:43:50Z

...ver/templates/src/main/java/package/config/couchbase/CustomCouchbaseRepositoryQuery.java.ejs

+import org.springframework.data.repository.core.NamedQueries;
+import <%= packageName %>.config.couchbase.CustomN1qlRepositoryQueryExecutor;
+
+public class CustomCouchbaseRepositoryQuery extends CouchbaseRepositoryQuery {


CouchbaseRepositoryQuery is deprecated. Can you please explain exactly what the problem is that forces you to implement all this custom classes ?

This was already there on the branch that I started working from, so I'm not exactly sure. However, I didn't want to remove it so as to not break anything that depended on it. If you have any suggestions for removing or changing it, I can make those changes.

After testing without it, it seems that there is some broken tests in AccountResourceIT due to this error:

java.lang.IllegalArgumentException: Authentication object cannot be null

I think we can find a more concise solution for that. But leave it for now. I'll try to look at it ASAP.

Added Couchbase tests to Github Actions and ensured they work with latest changes in the code

Only the non-reactive implementation of Couchbase was upgraded to the latest compatible SDK version

…serRepository.java.ejs Co-authored-by: Marcelo Shima <marceloshima@gmail.com>

mshima · 2021-08-23T13:01:09Z

561e5f3 is conflicting with 228719d

generators/entity-server/templates/src/test/java/package/web/rest/EntityResourceIT.java.ejs

...tity-server/templates/src/main/java/package/domain/Entity.java.jhi.spring_data_couchbase.ejs

…itory/EntityRepository.java.ejs

gkysaad · 2021-08-25T01:05:16Z

561e5f3 is conflicting with 228719d

It seems like there aren't any common files between those commits and everything looks fine on my end. The 561e5f3 commit was made to revert back to snake_case for the user initialization and the 228719d commit changes all the Couchbase code for the entities and users to use snake_case as well.

This reverts commit 561e5f3

This reverts commit 228719d

mshima

There are 3 missing tasks:

Or we can disable the tests for now and merge as it is, so work can continue in main repository.

generators/server/templates/src/main/java/package/service/UserService.java.ejs

gkysaad · 2021-08-26T05:01:18Z

There are 3 missing tasks:

Couchbase SDK 3, Spring Data Couchbase 4 Integration #15645 (comment)

Couchbase SDK 3, Spring Data Couchbase 4 Integration #15645 (comment)

Make tests succeed.

Or we can disable the tests for now and merge as it is, so work can continue in main repository.

I've completed the 2 tasks for those comments. As for the tests, when I made the PR, all the tests were passing and the functionality seemed to be working. There was some flakiness with some of the tests where they would not pass once but then pass when rerun. I disabled 2 that were consistently failing, but had working functionality. I think we should merge it as it is for now, then make a GitHub issue after the merge to look into the test flakiness.

generators/entity-server/files.js

...entity-server/templates/src/main/java/package/repository/EntityRepository_couchbase.java.ejs

# Conflicts: # generators/server/files.js

mshima · 2021-08-26T15:03:54Z

I will cleanup the PR to merge.

mshima · 2021-08-26T17:15:08Z

Merging.

couchbase isn’t fully functional yet
current main is completely broken
improves modularization
conflicting too often

gkysaad mentioned this pull request Jul 14, 2021

Jhipster 7.0.1 and couchbase 7 #14745

Closed

gkysaad force-pushed the main branch from 7b35f0f to 8aea850 Compare July 14, 2021 04:36

mraible mentioned this pull request Aug 5, 2021

Upgrade Couchbase support for Spring Boot 2.4 #14184

Closed

5 tasks

gkysaad added 10 commits August 10, 2021 23:11

Update pom.xml, react tests, and prompts for Couchbase

2fc531c

Switch CouchbasePagingAndSortingRepository to PagingAndSortingReposit…

58bb457

…ory and add fix

Add gradle dependency fix [react]

ce838a4

Update Field import to use correct library for Spring Data 4.x

f6f5f77

Couchbase generator setup

815bb46

update gitignore

f917fd8

add couchbase searchEngine to test yo-rc.json

ffdbbf3

fix typo

858637b

Spring Boot 4.x update for Couchbase imports

68094ce

Fix jhipster#13544

1e46bfb

tchlyah suggested changes Aug 13, 2021

View reviewed changes

gkysaad added 7 commits August 13, 2021 14:07

Fix jhipster#14184

20adcc7

Fix jhipster#12300

fb874d1

Test cases fixes and disable flaky ones

a763e01

Fix jhipster#14745

c1a284e

Fix couchmove, FTS, and users

91d325b

Setup GitHub Actions to work with couchbase tests

f374ec1

Added Couchbase tests to Github Actions and ensured they work with latest changes in the code

Fix prompt so Couchbase is only available as non-reactive

0ecf443

Only the non-reactive implementation of Couchbase was upgraded to the latest compatible SDK version

gkysaad and others added 4 commits August 22, 2021 23:27

Update generators/server/templates/src/main/java/package/repository/U…

70b49d8

…serRepository.java.ejs Co-authored-by: Marcelo Shima <marceloshima@gmail.com>

Update generators/server/templates/src/main/java/package/repository/U…

d4990a2

…serRepository.java.ejs Co-authored-by: Marcelo Shima <marceloshima@gmail.com>

Revert users initialization to snake_case

561e5f3

Refactor PersistentTokenRepository

f02de93

gkysaad force-pushed the main branch from a2357cb to f02de93 Compare August 23, 2021 04:03

tchlyah suggested changes Aug 23, 2021

View reviewed changes

Update generators/entity-server/templates/src/main/java/package/repos…

67d3a7b

…itory/EntityRepository.java.ejs

gkysaad added 4 commits August 24, 2021 21:13

Remove wait time from EntityResourceIT

c40befb

Revert "Revert users initialization to snake_case"

5ebf471

This reverts commit 561e5f3

Revert "Add back Couchbase custom @field key specification"

e51b9e0

This reverts commit 228719d

Remove custom key from all Couchbase fields (change to camelCase)

7c952c1

mshima reviewed Aug 25, 2021

View reviewed changes

generators/server/templates/src/main/java/package/service/UserService.java.ejs Outdated Show resolved Hide resolved

gkysaad added 2 commits August 25, 2021 09:46

Remove Couchbase code from PersistentTokenRepository

7e980e3

Refactor EntityRepository to EntityRepository_couchbase

cfdc6c0

Tcharl reviewed Aug 26, 2021

View reviewed changes

generators/entity-server/files.js Outdated Show resolved Hide resolved

Tcharl reviewed Aug 26, 2021

View reviewed changes

...entity-server/templates/src/main/java/package/repository/EntityRepository_couchbase.java.ejs Outdated Show resolved Hide resolved

Tcharl reviewed Aug 26, 2021

View reviewed changes

...entity-server/templates/src/main/java/package/repository/EntityRepository_couchbase.java.ejs Outdated Show resolved Hide resolved

mshima added 6 commits August 26, 2021 11:15

Merge remote-tracking branch 'upstream/main' into couchbase

5e92e18

# Conflicts: # generators/server/files.js

Lint fix.

3c0854d

Fixup EntityRepository/couchbase

aa15e52

Disable couchbase tests.

c63aedf

Don't set default language.

13dfcfc

Update snapshot.

e8a0d56

mshima merged commit 3a2e16c into jhipster:main Aug 26, 2021

pascalgrimaud added this to the 7.2.0 milestone Sep 11, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Couchbase SDK 3, Spring Data Couchbase 4 Integration #15645

Couchbase SDK 3, Spring Data Couchbase 4 Integration #15645

gkysaad commented Jul 14, 2021

CLAassistant commented Jul 14, 2021 •

edited

DanielFran commented Jul 14, 2021

Tcharl commented Jul 20, 2021

DanielFran commented Jul 27, 2021

gkysaad commented Aug 5, 2021

tchlyah commented Aug 8, 2021

tchlyah commented Aug 10, 2021

mraible commented Aug 12, 2021

gkysaad commented Aug 13, 2021

tchlyah left a comment

tchlyah Aug 13, 2021

gkysaad Aug 19, 2021

tchlyah Aug 19, 2021

mshima commented Aug 23, 2021

gkysaad commented Aug 25, 2021

mshima left a comment

gkysaad commented Aug 26, 2021

mshima commented Aug 26, 2021

mshima commented Aug 26, 2021

Couchbase SDK 3, Spring Data Couchbase 4 Integration #15645

Couchbase SDK 3, Spring Data Couchbase 4 Integration #15645

Conversation

gkysaad commented Jul 14, 2021

CLAassistant commented Jul 14, 2021 • edited

DanielFran commented Jul 14, 2021

Tcharl commented Jul 20, 2021

DanielFran commented Jul 27, 2021

gkysaad commented Aug 5, 2021

tchlyah commented Aug 8, 2021

tchlyah commented Aug 10, 2021

mraible commented Aug 12, 2021

gkysaad commented Aug 13, 2021

tchlyah left a comment

Choose a reason for hiding this comment

tchlyah Aug 13, 2021

Choose a reason for hiding this comment

gkysaad Aug 19, 2021

Choose a reason for hiding this comment

tchlyah Aug 19, 2021

Choose a reason for hiding this comment

mshima commented Aug 23, 2021

gkysaad commented Aug 25, 2021

mshima left a comment

Choose a reason for hiding this comment

gkysaad commented Aug 26, 2021

mshima commented Aug 26, 2021

mshima commented Aug 26, 2021

CLAassistant commented Jul 14, 2021 •

edited