Skip to content

Commit

Permalink
feat: add Connection interface (#1374)
Browse files Browse the repository at this point in the history
* manage dependencies

* manage dependencies

* update pagination logic

* Minor cleanup

* Updated the bufferSize

* Added testBQResultSetPagination

* refactored processQueryResponseResults

* updated testBQResultSetPagination to use generated table

* implemented a child thread for fetching next page

* Adding processQueryResponseResults2 (Independent pageFetcher and producer threads)

* Adding processQueryResponseResults2 (Independent pageFetcher and producer threads)

* nit: correct Javadoc

* Added getPageCacheSize for dynamically computing the size of pageCache

* Fixed NPE @ getPageCacheSize

* Refactored - Renamed processQueryResponseResults methods

* Refactored - exception handling for processQueryResponseResults

* Added testConnectionCancel for testing ConnectionImpl.cancel

* Added ConnectionImpl.cancel implementation and the interrupt logic in processQueryResponseResults

* Lint

* Commented testConnectionCancel - Need to update the Cancel Method logic

* Implemented processQueryResponseResults_thread_pooled

* nit: rename a few vars

* Uncommented testConnectionCancel

* Cleaned processQueryResponseResults. Removed unused methods. Implemented ThreadPool

* Cleaned processQueryResponseResults. Removed unused methods. Implemented ThreadPool

* add sessions support and BigQueryResultSetStats interface and impl

* add BigQueryResultSetStats to getQueryResults path; refactor jobs.get rpc call method

* feat: add support for BI Engine Statistics

fixes b/205146044

* Updated pageCacheSize and buffer's size between a MIN and MAX bound.

* 🦉 Updates from OwlBot

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* Refactored processQueryResponseResults to avoid code duplicity. Added upper bound for the buffer size

* Implemented processGetQueryResponseResults(results, jobId) & Refactored the code

* Implemented BigQueryDryRunResultImpl

* implemented dryRun

* Implemented execute select logic when the FastQuery is not Supported

* added null check at useReadAPI

* Added BQResultSet Test cases

* code cleanup and refactoring

* add import

* additional lint

* 🦉 Updates from OwlBot

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* add test requirements

* add dryRun implementation

* Added schema check @ testConnectionImplDryRun

* Added schema check @ testConnectionImplDryRun

* Added Java Doc, Removed commented code of processGetQueryResults

* Added SetUp and testExecuteSelect

* Exposing processQueryResponseResults for testing

* Exposing processQueryResponseResults for testing

* Added testFastQueryMultiplePages and testCancel

* Added testQueryDryRun

* Exposing Async methods for testing

* Exposing Async methods for testing

* Exposing tableDataListRpc for testing

* Exposed tableDataListRpc for testing

* Added testcases for the 3 async methods

* Exposing getQueryResultsFirstPage, isFastQuerySupported and getSubsequentQueryResultsWithJob for testing

* Exposing getQueryResultsFirstPage, isFastQuerySupported and getSubsequentQueryResultsWithJob for testing

* Added testGetQueryResultsFirstPage and testLegacyQuerySinglePage testcases + Minor refactor

* Deleted TODO

* Exposed getDestinationTable and tableDataList for testing

* Exposed getDestinationTable and tableDataList for testing

* Added testFastQueryLongRunning testcase

* Added testLegacyQueryMultiplePages

* Added BigqueryStorage and Arrow's dependencies

* Added draft impl for highThroughPutRead

* resolve dependency issues

* 🦉 Updates from OwlBot

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* Added Attribute to for ArrowSchema

* updated highThroughPutRead

* Revert "Added Attribute to for ArrowSchema"

This reverts commit 3297e4d.

* composed BigQueryResultSetImpl with all the params

* Added parsing logic for ArrowVectors

* Update processRows

* updated next methods and getters for 3 data types

* Added translateAndThrow(Exception ex)

* Refactored processRows, added exception handling

* Added arrow-memory-netty

* Updated highThroughPutRead - Code fix and refactoring

* clean up Connection.java and remove methods meant only for testing from the public interface. Update ConnectionImplTest.java to test against ConnectionImpl instead of Connection.

* Updated to StandardSQLTypeName

* Added Arrow specific logic for primitive getters

* Added Arrow specific parsing logic for primitive getters

* Added Arrow specific parsing logic for additional datatypes

* Modified the getter's logic for parsing arrow

* Added Interrupt logic for Arrow processing

* Added Arrow Support in ResultSet Methods

* Removed typecasting from processRows method

* Added Json typecasting logic

* Updated Arrow dependency to latest

* Added UseReadAPI property

* Updated useReadAPI

* Updated useReadAPI

* feat: add Interval type support (#1844)

* feat: add Interval type support

Fixes b/208051516

* 🦉 Updates from OwlBot

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* add threeten-extra PeriodDuration support

* add unit test coverage

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>

* Reformat

* remove unused declacred dependency: org.apache.arrow:arrow-memory-netty:jar:7.0.0:runtime

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* fixed getDate

* format

* Added containsIntervalType check

* Added containsIntervalType check

* add back the arrow-netty dependency

* add a rule to ignore org.apache.arrow:arrow-memory-netty in maven-dependency-plugin

* Fixed getTime

* Updated getString

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* Added Nightly Integration test for testing Read API integration

* Added Nightly Integration test for testing Read API integration

* Added Nightly Integration test for testing Read API integration

* Modified Nightly Integration test

* Fixed NPE during bulk read

* Fixed class cast exception @ getLong

* Added @BetaApi Annotation

* Added @BetaApi Annotation

* Exposed methods for unit testing

* Added more unit testing

* fixed getString to handle LocalDateTime

* Added ConnImplBenchmark

* Added instructions for running ConnImplBenchmark

* Updated Java Docs

* add IT TODOs

* Fixed getInt for JSONField types (Int returned as Text)

* Added JsonType tests

* Added test - testForAllDataTypeValues

* Added translateAndThrowBigQuerySQLException

* Added Exception Handling for executeSelect methods

* Added testInvalidQuery

* Added testMultipleRuns

* Added testPositionalParams

* updated positional query

* update logic to not use Read API when QueryParameters are passed since it's not supported by the Read API

* update support for query parameters

deps: update dependency com.google.oauth-client:google-oauth-client-jetty to v1.33.3 (#1978)

[![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [com.google.oauth-client:google-oauth-client-jetty](https://togithub.com/googleapis/google-oauth-java-client) | `1.33.2` -> `1.33.3` | [![age](https://badges.renovateapi.com/packages/maven/com.google.oauth-client:google-oauth-client-jetty/1.33.3/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/maven/com.google.oauth-client:google-oauth-client-jetty/1.33.3/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/maven/com.google.oauth-client:google-oauth-client-jetty/1.33.3/compatibility-slim/1.33.2)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/maven/com.google.oauth-client:google-oauth-client-jetty/1.33.3/confidence-slim/1.33.2)](https://docs.renovatebot.com/merge-confidence/) |

---

<details>
<summary>googleapis/google-oauth-java-client</summary>

[Compare Source](https://togithub.com/googleapis/google-oauth-java-client/compare/v1.33.2...v1.33.3)

</details>

---

📅 **Schedule**: At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, click this checkbox.

---

This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/java-bigquery).

deps: update dependency com.google.oauth-client:google-oauth-client-java6 to v1.33.3 (#1977)

[![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [com.google.oauth-client:google-oauth-client-java6](https://togithub.com/googleapis/google-oauth-java-client) | `1.33.2` -> `1.33.3` | [![age](https://badges.renovateapi.com/packages/maven/com.google.oauth-client:google-oauth-client-java6/1.33.3/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/maven/com.google.oauth-client:google-oauth-client-java6/1.33.3/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/maven/com.google.oauth-client:google-oauth-client-java6/1.33.3/compatibility-slim/1.33.2)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/maven/com.google.oauth-client:google-oauth-client-java6/1.33.3/confidence-slim/1.33.2)](https://docs.renovatebot.com/merge-confidence/) |

---

<details>
<summary>googleapis/google-oauth-java-client</summary>

[Compare Source](https://togithub.com/googleapis/google-oauth-java-client/compare/v1.33.2...v1.33.3)

</details>

---

📅 **Schedule**: At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, click this checkbox.

---

This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/java-bigquery).

* chore(main): release 2.10.8 (#1979)

:robot: I have created a release *beep* *boop*
---


### [2.10.8](v2.10.7...v2.10.8) (2022-04-14)


### Dependencies

* update dependency com.google.oauth-client:google-oauth-client-java6 to v1.33.3 ([#1977](#1977)) ([b084791](b084791))
* update dependency com.google.oauth-client:google-oauth-client-jetty to v1.33.3 ([#1978](#1978)) ([a24ce6e](a24ce6e))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

* chore(main): release 2.10.9-SNAPSHOT (#1980)

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

* deps: update dependency com.google.cloud:native-image-support to v0.13.1 (#1982)

[![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [com.google.cloud:native-image-support](https://togithub.com/googleapis/java-core) | `0.12.11` -> `0.13.1` | [![age](https://badges.renovateapi.com/packages/maven/com.google.cloud:native-image-support/0.13.1/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/maven/com.google.cloud:native-image-support/0.13.1/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/maven/com.google.cloud:native-image-support/0.13.1/compatibility-slim/0.12.11)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/maven/com.google.cloud:native-image-support/0.13.1/confidence-slim/0.12.11)](https://docs.renovatebot.com/merge-confidence/) |

---

### Configuration

📅 **Schedule**: At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Renovate will not automatically rebase this PR, because other commits have been found.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, click this checkbox.

---

This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/java-bigquery).

* chore(deps): update dependency com.google.cloud:google-cloud-bigquery to v2.10.8 (#1981)

[![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [com.google.cloud:google-cloud-bigquery](https://togithub.com/googleapis/java-bigquery) | `2.10.7` -> `2.10.8` | [![age](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-bigquery/2.10.8/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-bigquery/2.10.8/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-bigquery/2.10.8/compatibility-slim/2.10.7)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-bigquery/2.10.8/confidence-slim/2.10.7)](https://docs.renovatebot.com/merge-confidence/) |

---

### Release Notes

<details>
<summary>googleapis/java-bigquery</summary>

### [`v2.10.8`](https://togithub.com/googleapis/java-bigquery/blob/HEAD/CHANGELOG.md#&#8203;2108-httpsgithubcomgoogleapisjava-bigquerycomparev2107v2108-2022-04-14)

[Compare Source](https://togithub.com/googleapis/java-bigquery/compare/v2.10.7...v2.10.8)

</details>

---

### Configuration

📅 **Schedule**: At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Renovate will not automatically rebase this PR, because other commits have been found.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, click this checkbox.

---

This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/java-bigquery).

* deps: update dependency com.google.cloud:google-cloud-shared-dependencies to v2.10.0 (#1983)

[![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [com.google.cloud:google-cloud-shared-dependencies](https://togithub.com/googleapis/java-shared-dependencies) | `2.9.0` -> `2.10.0` | [![age](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-shared-dependencies/2.10.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-shared-dependencies/2.10.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-shared-dependencies/2.10.0/compatibility-slim/2.9.0)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-shared-dependencies/2.10.0/confidence-slim/2.9.0)](https://docs.renovatebot.com/merge-confidence/) |

---

### Release Notes

<details>
<summary>googleapis/java-shared-dependencies</summary>

### [`v2.10.0`](https://togithub.com/googleapis/java-shared-dependencies/blob/HEAD/CHANGELOG.md#&#8203;2100-httpsgithubcomgoogleapisjava-shared-dependenciescomparev290v2100-2022-04-15)

[Compare Source](https://togithub.com/googleapis/java-shared-dependencies/compare/v2.9.0...v2.10.0)

##### Dependencies

-   google-cloud-core 2.6.0 ([#&#8203;668](https://togithub.com/googleapis/java-shared-dependencies/issues/668)) ([22f4d18](https://togithub.com/googleapis/java-shared-dependencies/commit/22f4d1809cbb9848174b3569ab527e4bef00d443))
-   reverting protobuf to 3.19.4 ([#&#8203;657](https://togithub.com/googleapis/java-shared-dependencies/issues/657)) ([8501e6d](https://togithub.com/googleapis/java-shared-dependencies/commit/8501e6d842c10d2370bbd5d5246070134336bddd))
-   update dependency com.fasterxml.jackson:jackson-bom to v2.13.2.20220328 ([#&#8203;646](https://togithub.com/googleapis/java-shared-dependencies/issues/646)) ([7bfd6d7](https://togithub.com/googleapis/java-shared-dependencies/commit/7bfd6d7073859d1955b91b368c6713a72ffa14b6))
-   update dependency com.google.api-client:google-api-client-bom to v1.34.0 ([#&#8203;662](https://togithub.com/googleapis/java-shared-dependencies/issues/662)) ([1b8e378](https://togithub.com/googleapis/java-shared-dependencies/commit/1b8e378fe0ccf2a28c759868caaf5ba593a95728))
-   update dependency com.google.errorprone:error_prone_annotations to v2.12.1 ([#&#8203;652](https://togithub.com/googleapis/java-shared-dependencies/issues/652)) ([1cc80ee](https://togithub.com/googleapis/java-shared-dependencies/commit/1cc80ee984ebcad9bc2a95e2f28c0a49fe924b37))
-   update dependency com.google.errorprone:error_prone_annotations to v2.13.0 ([#&#8203;669](https://togithub.com/googleapis/java-shared-dependencies/issues/669)) ([61b7834](https://togithub.com/googleapis/java-shared-dependencies/commit/61b78341b34a251722be4805a6bdd895cd64836c))
-   update dependency com.google.http-client:google-http-client-bom to v1.41.6 ([#&#8203;654](https://togithub.com/googleapis/java-shared-dependencies/issues/654)) ([140ef40](https://togithub.com/googleapis/java-shared-dependencies/commit/140ef405bc17ed83f5ce177df59affca14fbe49c))
-   update dependency com.google.http-client:google-http-client-bom to v1.41.7 ([#&#8203;658](https://togithub.com/googleapis/java-shared-dependencies/issues/658)) ([f6f93e5](https://togithub.com/googleapis/java-shared-dependencies/commit/f6f93e5b9172c9684623c4c148e4a8fe2fae1e94))
-   update dependency com.google.oauth-client:google-oauth-client-bom to v1.33.2 ([#&#8203;655](https://togithub.com/googleapis/java-shared-dependencies/issues/655)) ([20cd9ed](https://togithub.com/googleapis/java-shared-dependencies/commit/20cd9eda112c96836a5ab7485a4247ed2bc0edb8))
-   update dependency com.google.oauth-client:google-oauth-client-bom to v1.33.3 ([#&#8203;663](https://togithub.com/googleapis/java-shared-dependencies/issues/663)) ([f011a46](https://togithub.com/googleapis/java-shared-dependencies/commit/f011a46c551dba16851b4f8c919c40452fc5d5c3))
-   update dependency com.google.protobuf:protobuf-bom to v3.20.0 ([#&#8203;651](https://togithub.com/googleapis/java-shared-dependencies/issues/651)) ([ad2ff73](https://togithub.com/googleapis/java-shared-dependencies/commit/ad2ff73207dd6493321c77d8eca0022baf13b4ce))
-   update dependency io.grpc:grpc-bom to v1.45.1 ([#&#8203;647](https://togithub.com/googleapis/java-shared-dependencies/issues/647)) ([38e46fc](https://togithub.com/googleapis/java-shared-dependencies/commit/38e46fc4f03af0a02f30ce4a2fa222c71797ae15))
-   update dependency org.checkerframework:checker-qual to v3.21.4 ([#&#8203;650](https://togithub.com/googleapis/java-shared-dependencies/issues/650)) ([125e80a](https://togithub.com/googleapis/java-shared-dependencies/commit/125e80ab2c3225a00c03f5ff5de94872ebb94303))
-   update gax.version to v2.15.0 ([#&#8203;649](https://togithub.com/googleapis/java-shared-dependencies/issues/649)) ([c7f32a6](https://togithub.com/googleapis/java-shared-dependencies/commit/c7f32a68b14520104432282ac9598643700162eb))
-   update gax.version to v2.16.0 ([#&#8203;664](https://togithub.com/googleapis/java-shared-dependencies/issues/664)) ([caaf941](https://togithub.com/googleapis/java-shared-dependencies/commit/caaf941643af04295f5527a0144587d7bf040862))
-   update google.common-protos.version to v2.8.1 ([#&#8203;656](https://togithub.com/googleapis/java-shared-dependencies/issues/656)) ([df4a4a2](https://togithub.com/googleapis/java-shared-dependencies/commit/df4a4a2130a3cdb2965ea42962d1ea6a85506ba7))
-   update google.common-protos.version to v2.8.2 ([#&#8203;659](https://togithub.com/googleapis/java-shared-dependencies/issues/659)) ([b499e2b](https://togithub.com/googleapis/java-shared-dependencies/commit/b499e2bc99506d48d26e35bf6e68c09409ce8b11))
-   update google.common-protos.version to v2.8.3 ([#&#8203;660](https://togithub.com/googleapis/java-shared-dependencies/issues/660)) ([461081c](https://togithub.com/googleapis/java-shared-dependencies/commit/461081c0cf73057c1f6e07fc573453ad467a60ae))
-   update iam.version to v1.3.0 ([#&#8203;648](https://togithub.com/googleapis/java-shared-dependencies/issues/648)) ([6670c4f](https://togithub.com/googleapis/java-shared-dependencies/commit/6670c4f61fcf075c543bfd148eea49796e0662ce))
-   update iam.version to v1.3.1 ([#&#8203;661](https://togithub.com/googleapis/java-shared-dependencies/issues/661)) ([cc8fbe6](https://togithub.com/googleapis/java-shared-dependencies/commit/cc8fbe6eae03341c2ece7d3356febc843a74a812))

</details>

---

### Configuration

📅 **Schedule**: At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Renovate will not automatically rebase this PR, because other commits have been found.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, click this checkbox.

---

This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/java-bigquery).

* chore(deps): update dependency com.google.cloud:google-cloud-bigqueryconnection to v2.2.3 (#1986)

* test(deps): update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.7.3 (#1987)

* chore(main): release 2.10.9 (#1984)

:robot: I have created a release *beep* *boop*
---


### [2.10.9](v2.10.8...v2.10.9) (2022-04-16)


### Dependencies

* update dependency com.google.cloud:google-cloud-shared-dependencies to v2.10.0 ([#1983](#1983)) ([50ac31c](50ac31c))
* update dependency com.google.cloud:native-image-support to v0.13.1 ([#1982](#1982)) ([04f4679](04f4679))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

* chore(main): release 2.10.10-SNAPSHOT (#1988)

:robot: I have created a release *beep* *boop*
---


### Updating meta-information for bleeding-edge SNAPSHOT release.

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

* chore(deps): update dependency com.google.cloud:google-cloud-bigquery to v2.10.9 (#1989)

[![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [com.google.cloud:google-cloud-bigquery](https://togithub.com/googleapis/java-bigquery) | `2.10.8` -> `2.10.9` | [![age](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-bigquery/2.10.9/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-bigquery/2.10.9/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-bigquery/2.10.9/compatibility-slim/2.10.8)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-bigquery/2.10.9/confidence-slim/2.10.8)](https://docs.renovatebot.com/merge-confidence/) |

---

### Release Notes

<details>
<summary>googleapis/java-bigquery</summary>

### [`v2.10.9`](https://togithub.com/googleapis/java-bigquery/blob/HEAD/CHANGELOG.md#&#8203;2109-httpsgithubcomgoogleapisjava-bigquerycomparev2108v2109-2022-04-16)

[Compare Source](https://togithub.com/googleapis/java-bigquery/compare/v2.10.8...v2.10.9)

</details>

---

### Configuration

📅 **Schedule**: At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Renovate will not automatically rebase this PR, because other commits have been found.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, click this checkbox.

---

This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/java-bigquery).

* deps: update dependency com.google.cloud:google-cloud-storage to v2.6.1 (#1991)

[![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [com.google.cloud:google-cloud-storage](https://togithub.com/googleapis/java-storage) | `2.6.0` -> `2.6.1` | [![age](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-storage/2.6.1/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-storage/2.6.1/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-storage/2.6.1/compatibility-slim/2.6.0)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-storage/2.6.1/confidence-slim/2.6.0)](https://docs.renovatebot.com/merge-confidence/) |

---

### Release Notes

<details>
<summary>googleapis/java-storage</summary>

### [`v2.6.1`](https://togithub.com/googleapis/java-storage/blob/HEAD/CHANGELOG.md#&#8203;261-httpsgithubcomgoogleapisjava-storagecomparev260v261-2022-04-15)

[Compare Source](https://togithub.com/googleapis/java-storage/compare/v2.6.0...v2.6.1)

</details>

---

### Configuration

📅 **Schedule**: At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, click this checkbox.

---

This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/java-bigquery).

* deps: update dependency com.google.cloud:google-cloud-bigtable to v2.6.2 (#1990)

[![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [com.google.cloud:google-cloud-bigtable](https://togithub.com/googleapis/java-bigtable) | `2.6.1` -> `2.6.2` | [![age](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-bigtable/2.6.2/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-bigtable/2.6.2/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-bigtable/2.6.2/compatibility-slim/2.6.1)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-bigtable/2.6.2/confidence-slim/2.6.1)](https://docs.renovatebot.com/merge-confidence/) |

---

### Release Notes

<details>
<summary>googleapis/java-bigtable</summary>

### [`v2.6.2`](https://togithub.com/googleapis/java-bigtable/blob/HEAD/CHANGELOG.md#&#8203;262-httpsgithubcomgoogleapisjava-bigtablecomparev261v262-2022-04-15)

[Compare Source](https://togithub.com/googleapis/java-bigtable/compare/v2.6.1...v2.6.2)

</details>

---

### Configuration

📅 **Schedule**: At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, click this checkbox.

---

This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/java-bigquery).

* chore(main): release 2.10.10 (#1992)

:robot: I have created a release *beep* *boop*
---

* update dependency com.google.cloud:google-cloud-bigtable to v2.6.2 ([#1990](#1990)) ([4c1cb4c](4c1cb4c))
* update dependency com.google.cloud:google-cloud-storage to v2.6.1 ([#1991](#1991)) ([e02bf31](e02bf31))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

chore(main): release 2.10.11-SNAPSHOT (#1993)

:robot: I have created a release *beep* *boop*
---

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

* fix dependencies.sh

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* update query parameters for dryRun

* Made Label optional by making it a vararg

* Made Label optional by making it a vararg

* Removed null labels

* Renamed cancel to close

* Refactored testConnectionCancel

* Renamed cancel() to close()

* Renamed testCancel

* Added testPositionalParams

* Replaced all the shutdownNows with MoreExecutors.shutdownAndAwaitTermination()

* Using TimeUnit.DAYS.toMillis for conversion from Days to Millis

* deleted outdated comment

* nit update

* update based on comments

* updated to using a public dataset

* updated cancel() to close()

* using constants for clientConnectionConfiguration

* using constants for clientConnectionConfiguration

* updated Warmup iterations to 1

* updated dataset

* Fix for testConnectionClose IT

* update labels and close method

🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Made Label optional by making it a vararg

Made Label optional by making it a vararg

Removed null labels

Renamed cancel to close

Refactored testConnectionCancel

Renamed cancel() to close()

Renamed testCancel

Added testPositionalParams

Replaced all the shutdownNows with MoreExecutors.shutdownAndAwaitTermination()

* update based on comments

update based on feedback

Using TimeUnit.DAYS.toMillis for conversion from Days to Millis

deleted outdated comment

nit update

update based on comments

updated to using a public dataset

updated cancel() to close()

using constants for clientConnectionConfiguration

using constants for clientConnectionConfiguration

updated Warmup iterations to 1

combine commits

updated dataset

Fix for testConnectionClose IT

update based on comments

fix lint

* Refactored code - Replaced Tuple<Map<String, Object>, Boolean> with Row

* Added class Row and Refactored code - Replaced Tuple<Map<String, Object>, Boolean> with Row

* Added overloaded constructor with just message as an argument

* refactored and added BigQuerySQLException in next()

* refactored and added throwing BigQuerySQLException

* Added fieldvalue null checks

* modified close

* modified close

* modified getLong

* address feedback

deleted outdated comment

update based on comments

updated to using a public dataset

updated cancel() to close()

using constants for clientConnectionConfiguration

using constants for clientConnectionConfiguration

updated Warmup iterations to 1

updated dataset

Fix for testConnectionClose IT

* update based on comments

update based on feedback

Using TimeUnit.DAYS.toMillis for conversion from Days to Millis

deleted outdated comment

nit update

update based on comments

updated to using a public dataset

updated cancel() to close()

using constants for clientConnectionConfiguration

using constants for clientConnectionConfiguration

updated Warmup iterations to 1

combine commits

updated dataset

Fix for testConnectionClose IT

update based on comments

fix lint

* chore(deps): update dependency com.google.cloud:google-cloud-bigquery to v2.10.10 (#1995)

* chore(deps): update dependency com.google.cloud:google-cloud-bigquery to v2.10.10

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>

* feat: next release from main branch is 2.10.9 (#1996)

* build(deps): update dependency com.google.cloud:google-cloud-shared-config to v1.3.3 (#2000)

* test(deps): update dependency org.mockito:mockito-core to v4.5.0 (#1999)

* test(deps): update dependency org.mockito:mockito-core to v4.5.1 (#2003)

* build(deps): update dependency org.apache.maven.plugins:maven-javadoc-plugin to v3.4.0 (#2002)

* build(deps): update dependency org.sonatype.plugins:nexus-staging-maven-plugin to v1.6.13 (#2004)

[![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [org.sonatype.plugins:nexus-staging-maven-plugin](http://www.sonatype.com/) ([source](https://togithub.com/sonatype/nexus-maven-plugins)) | `1.6.12` -> `1.6.13` | [![age](https://badges.renovateapi.com/packages/maven/org.sonatype.plugins:nexus-staging-maven-plugin/1.6.13/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/maven/org.sonatype.plugins:nexus-staging-maven-plugin/1.6.13/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/maven/org.sonatype.plugins:nexus-staging-maven-plugin/1.6.13/compatibility-slim/1.6.12)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/maven/org.sonatype.plugins:nexus-staging-maven-plugin/1.6.13/confidence-slim/1.6.12)](https://docs.renovatebot.com/merge-confidence/) |

---

### Release Notes

<details>
<summary>sonatype/nexus-maven-plugins</summary>

### [`v1.6.13`](https://togithub.com/sonatype/nexus-maven-plugins/compare/release-1.6.12...release-1.6.13)

[Compare Source](https://togithub.com/sonatype/nexus-maven-plugins/compare/release-1.6.12...release-1.6.13)

</details>

---

### Configuration

📅 **Schedule**: At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, click this checkbox.

---

This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/java-bigquery).

* chore(deps): update dependency com.google.cloud:google-cloud-bigqueryconnection to v2.2.4 (#2006)

[![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [com.google.cloud:google-cloud-bigqueryconnection](https://togithub.com/googleapis/java-bigqueryconnection) | `2.2.3` -> `2.2.4` | [![age](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-bigqueryconnection/2.2.4/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-bigqueryconnection/2.2.4/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-bigqueryconnection/2.2.4/compatibility-slim/2.2.3)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-bigqueryconnection/2.2.4/confidence-slim/2.2.3)](https://docs.renovatebot.com/merge-confidence/) |

---

### Release Notes

<details>
<summary>googleapis/java-bigqueryconnection</summary>

### [`v2.2.4`](https://togithub.com/googleapis/java-bigqueryconnection/blob/HEAD/CHANGELOG.md#&#8203;224-httpsgithubcomgoogleapisjava-bigqueryconnectioncomparev223v224-2022-04-21)

[Compare Source](https://togithub.com/googleapis/java-bigqueryconnection/compare/v2.2.3...v2.2.4)

</details>

---

### Configuration

📅 **Schedule**: At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, click this checkbox.

---

This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/java-bigquery).

* chore(deps): update dependency com.google.cloud:libraries-bom to v25.2.0 (#2008)

* chore(deps): update dependency com.google.cloud:libraries-bom to v25.2.0

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>

* docs(sample): remove unused dependency and add setup instructions (#2010)

* build(deps): update dependency org.apache.maven.plugins:maven-project-info-reports-plugin to v3.3.0 (#2011)

* chore: IT clean-up (#2012)

Fixes #1924
Per discussion in b/227623980

* build(deps): update dependency com.google.cloud:google-cloud-shared-config to v1.4.0 (#2013)

* address feedback

Refactored code - Replaced Tuple<Map<String, Object>, Boolean> with Row

Added class Row and Refactored code - Replaced Tuple<Map<String, Object>, Boolean> with Row

Added overloaded constructor with just message as an argument

refactored and added BigQuerySQLException in next()

address feedback

refactored and added throwing BigQuerySQLException

Added fieldvalue null checks

modified close

modified close

modified getLong

* add clarification to dryRun query params

* Fixed BigQuerySQLException in the Integration Tests

* Added line comment

* Updated getInt to include check for Long

* Added testReadAPIIterationAndOrder IT

* Add testReadAPIConnectionMultiClose, Removed ReadClientConnectionConfiguration

* Refactored for removal of ReadClientConnectionConfiguration

* Added more properties and default values

* Changed the type of preFetchedRowLimit from Long to Integer

* Updated method signature - Changed the type of preFetchedRowLimit from Long to Integer

* Refactor - Removed the usage of ReadClientConnectionConfiguration

* Refactor - Removed the usage of ReadClientConnectionConfiguration

* Changed the type of preFetchedRowLimit to Integer

* Refactor due to Change in the type of preFetchedRowLimit to Integer

* lint

* some cleanup and corrections

* nit

* mark createConnection as Beta

* disable graalvm IT until it works with Apache Arrow

* Added instance check for IntegerArrayField

* Added Connection createConnection()

* Added testIterateAndOrderDefaultConnSettings IT

* Added Connection createConnection(). Marked createConnection(@nonnull ConnectionSettings connectionSettings)

* Added com.google.cloud.bigquery.Connection createConnection()

* add IT and some clean up

* add org.checkerframework:checker-qual:jar:3.21.4:compile as a dependency to fix dependency check failure

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* provide BigQueryResultStats in dryRun results

* Fixed testQueryDryRun UT failure - Added jobConfigurationQuery

Co-authored-by: Prashant Mishra <prash_mi@yahoo.com>
Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Co-authored-by: WhiteSource Renovate <bot@renovateapp.com>
Co-authored-by: Prashant Mishra <inbox.prashant@gmail.com>
Co-authored-by: Mridula <66699525+mpeddada1@users.noreply.github.com>
  • Loading branch information
7 people committed May 6, 2022
1 parent b9fa786 commit 3804275
Show file tree
Hide file tree
Showing 31 changed files with 6,381 additions and 6 deletions.
6 changes: 5 additions & 1 deletion .gitignore
Expand Up @@ -6,4 +6,8 @@ __pycache__
.settings
.classpath
.DS_Store

.diff.txt
.new-list.txt
.org-list.txt
SimpleBenchmarkApp/src/main/java/com/google/cloud/App.java
.flattened-pom.xml
4 changes: 2 additions & 2 deletions .kokoro/dependencies.sh
Expand Up @@ -67,13 +67,13 @@ function completenessCheck() {
msg "Generating dependency list using original pom..."
# Excluding commons-codec,commons-logging from the comparison as a temp fix
# Explanation and issue filed in maven-dependency-plugin: https://issues.apache.org/jira/browse/MDEP-737
mvn dependency:list -f pom.xml -DexcludeArtifactIds=commons-codec,commons-logging -DincludeScope=runtime -Dsort=true | grep '\[INFO] .*:.*:.*:.*:.*' | sed -e s/\\s--\\smodule.*// >.org-list.txt
mvn dependency:list -f pom.xml -DexcludeArtifactIds=commons-codec,commons-logging,grpc-googleapis -DincludeScope=runtime -Dsort=true | grep '\[INFO] .*:.*:.*:.*:.*' | sed -e s/\\s--\\smodule.*// >.org-list.txt

# Output dep list generated using the flattened pom (only 'compile' and 'runtime' scopes)
msg "Generating dependency list using flattened pom..."
# Excluding commons-codec,commons-logging from the comparison as a temp fix
# Explanation and issue filed in maven-dependency-plugin: https://issues.apache.org/jira/browse/MDEP-737
mvn dependency:list -f .flattened-pom.xml -DexcludeArtifactIds=commons-codec,commons-logging -DincludeScope=runtime -Dsort=true | grep '\[INFO] .*:.*:.*:.*:.*' >.new-list.txt
mvn dependency:list -f .flattened-pom.xml -DexcludeArtifactIds=commons-codec,commons-logging,grpc-googleapis -DincludeScope=runtime -Dsort=true | grep '\[INFO] .*:.*:.*:.*:.*' >.new-list.txt

# Compare two dependency lists
msg "Comparing dependency lists..."
Expand Down
7 changes: 7 additions & 0 deletions benchmark/README.md
Expand Up @@ -19,3 +19,10 @@ To run a benchmark jar, run the following command
cd benchmark
java -jar target/benchmark.jar
```

To run ConnImplBenchmark, run the following command
```
# Run from benchmark directory
cd benchmark
java -jar target/benchmark.jar com.google.cloud.bigquery.ConnImplBenchmark
```
@@ -0,0 +1,186 @@
/*
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package com.google.cloud.bigquery;

import java.io.IOException;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.concurrent.TimeUnit;
import java.util.logging.Level;
import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Fork;
import org.openjdk.jmh.annotations.Measurement;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.OutputTimeUnit;
import org.openjdk.jmh.annotations.Param;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.Setup;
import org.openjdk.jmh.annotations.State;
import org.openjdk.jmh.annotations.Warmup;
import org.openjdk.jmh.infra.Blackhole;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;

@Fork(value = 1)
@BenchmarkMode(Mode.AverageTime)
@Warmup(iterations = 1)
@Measurement(iterations = 3)
@State(Scope.Benchmark)
@OutputTimeUnit(TimeUnit.MILLISECONDS)
public class ConnImplBenchmark {
@Param({"500000", "1000000", "10000000", "100000000"}) // 500K, 1M, 10M, and 100M
public int rowLimit;

private ConnectionSettings connectionSettingsReadAPIEnabled, connectionSettingsReadAPIDisabled;
private long numBuffRows = 100000L;
private final String DATASET = "new_york_taxi_trips";
private final String QUERY =
"SELECT * FROM bigquery-public-data.new_york_taxi_trips.tlc_yellow_trips_2017 LIMIT %s";
public static final long NUM_PAGE_ROW_CNT_RATIO =
10; // ratio of [records in the current page :: total rows] to be met to use read API
public static final long NUM_MIN_RESULT_SIZE =
200000; // min number of records to use to ReadAPI with

@Setup
public void setUp() throws IOException {
java.util.logging.Logger.getGlobal().setLevel(Level.ALL);
ReadClientConnectionConfiguration clientConnectionConfiguration;

clientConnectionConfiguration =
ReadClientConnectionConfiguration.newBuilder()
.setTotalToPageRowCountRatio(NUM_PAGE_ROW_CNT_RATIO)
.setMinResultSize(NUM_MIN_RESULT_SIZE)
.setBufferSize(numBuffRows)
.build();

connectionSettingsReadAPIEnabled =
ConnectionSettings.newBuilder()
.setDefaultDataset(DatasetId.of(DATASET))
.setNumBufferedRows(numBuffRows) // page size
.setPriority(
QueryJobConfiguration.Priority
.INTERACTIVE) // DEFAULT VALUE - so that isFastQuerySupported returns false
.setReadClientConnectionConfiguration(clientConnectionConfiguration)
.setUseReadAPI(true) // enable read api
.build();
connectionSettingsReadAPIDisabled =
ConnectionSettings.newBuilder()
.setDefaultDataset(DatasetId.of(DATASET))
.setNumBufferedRows(numBuffRows) // page size
.setPriority(
QueryJobConfiguration.Priority
.INTERACTIVE) // so that isFastQuerySupported returns false
.setReadClientConnectionConfiguration(clientConnectionConfiguration)
.setUseReadAPI(false) // disable read api
.build();
}

@Benchmark
public void iterateRecordsUsingReadAPI(Blackhole blackhole)
throws InterruptedException, BigQuerySQLException {
Connection connectionReadAPIEnabled =
BigQueryOptions.getDefaultInstance()
.getService()
.createConnection(connectionSettingsReadAPIEnabled);
String selectQuery = String.format(QUERY, rowLimit);
long hash = 0L;
try {
BigQueryResultSet bigQueryResultSet = connectionReadAPIEnabled.executeSelect(selectQuery);
hash = getResultHash(bigQueryResultSet);
} catch (Exception e) {
e.printStackTrace();
} finally {
connectionReadAPIEnabled.close(); // IMP to kill the bg workers
}
blackhole.consume(hash);
}

@Benchmark
public void iterateRecordsWithoutUsingReadAPI(Blackhole blackhole)
throws InterruptedException, BigQuerySQLException {
Connection connectionReadAPIDisabled =
BigQueryOptions.getDefaultInstance()
.getService()
.createConnection(connectionSettingsReadAPIDisabled);
String selectQuery = String.format(QUERY, rowLimit);
long hash = 0L;
try {
BigQueryResultSet bigQueryResultSet = connectionReadAPIDisabled.executeSelect(selectQuery);
hash = getResultHash(bigQueryResultSet);
} catch (Exception e) {
e.printStackTrace();
} finally {
connectionReadAPIDisabled.close(); // IMP to kill the bg workers
}
blackhole.consume(hash);
}

// Hashes all the 20 columns of all the rows
private long getResultHash(BigQueryResultSet bigQueryResultSet) throws SQLException {
ResultSet rs = bigQueryResultSet.getResultSet();
long hash = 0L;
int cnt = 0;
System.out.print("\n Running");
while (rs.next()) {
hash += rs.getString("vendor_id") == null ? 0 : rs.getString("vendor_id").hashCode();
hash +=
rs.getString("pickup_datetime") == null ? 0 : rs.getString("pickup_datetime").hashCode();
hash +=
rs.getString("dropoff_datetime") == null
? 0
: rs.getString("dropoff_datetime").hashCode();
hash += rs.getLong("passenger_count");
hash += rs.getDouble("trip_distance");
hash += rs.getDouble("pickup_longitude");
hash += rs.getDouble("pickup_latitude");
hash += rs.getString("rate_code") == null ? 0 : rs.getString("rate_code").hashCode();
hash +=
rs.getString("store_and_fwd_flag") == null
? 0
: rs.getString("store_and_fwd_flag").hashCode();
hash += rs.getDouble("dropoff_longitude");
hash += rs.getDouble("dropoff_latitude");
hash += rs.getString("payment_type") == null ? 0 : rs.getString("payment_type").hashCode();
hash += rs.getDouble("fare_amount");
hash += rs.getDouble("extra");
hash += rs.getDouble("mta_tax");
hash += rs.getDouble("tip_amount");
hash += rs.getDouble("tolls_amount");
hash += rs.getDouble("imp_surcharge");
hash += rs.getDouble("total_amount");
hash +=
rs.getString("pickup_location_id") == null
? 0
: rs.getString("pickup_location_id").hashCode();
hash +=
rs.getString("dropoff_location_id") == null
? 0
: rs.getString("dropoff_location_id").hashCode();
if (++cnt % 100000 == 0) { // just to indicate the progress while long running benchmarks
System.out.print(".");
}
}
return hash;
}

public static void main(String[] args) throws Exception {
Options opt = new OptionsBuilder().include(ConnImplBenchmark.class.getSimpleName()).build();
new Runner(opt).run();
}
}
35 changes: 35 additions & 0 deletions google-cloud-bigquery/clirr-ignored-differences.xml
@@ -0,0 +1,35 @@
<?xml version="1.0" encoding="UTF-8"?>
<!-- see http://www.mojohaus.org/clirr-maven-plugin/examples/ignored-differences.html -->
<differences>
<!-- TODO: REMOVE AFTER RELEASE -->
<difference>
<differenceType>7012</differenceType>
<className>com/google/cloud/bigquery/BigQuery</className>
<method>com.google.cloud.bigquery.Connection createConnection(com.google.cloud.bigquery.ConnectionSettings)</method>
</difference>
<difference>
<differenceType>7012</differenceType>
<className>com/google/cloud/bigquery/BigQuery</className>
<method>com.google.cloud.bigquery.Connection createConnection()</method>
</difference>
<difference>
<differenceType>7012</differenceType>
<className>com/google/cloud/bigquery/spi/v2/BigQueryRpc</className>
<method>com.google.api.services.bigquery.model.Job createJobForQuery(com.google.api.services.bigquery.model.Job)</method>
</difference>
<difference>
<differenceType>7012</differenceType>
<className>com/google/cloud/bigquery/spi/v2/BigQueryRpc</className>
<method>com.google.api.services.bigquery.model.Job getQueryJob(java.lang.String, java.lang.String, java.lang.String)</method>
</difference>
<difference>
<differenceType>7012</differenceType>
<className>com/google/cloud/bigquery/spi/v2/BigQueryRpc</className>
<method>com.google.api.services.bigquery.model.GetQueryResultsResponse getQueryResultsWithRowLimit(java.lang.String, java.lang.String, java.lang.String, java.lang.Integer)</method>
</difference>
<difference>
<differenceType>7012</differenceType>
<className>com/google/cloud/bigquery/spi/v2/BigQueryRpc</className>
<method>com.google.api.services.bigquery.model.TableDataList listTableDataWithRowLimit(java.lang.String, java.lang.String, java.lang.String, java.lang.Integer, java.lang.String)</method>
</difference>
</differences>
20 changes: 20 additions & 0 deletions google-cloud-bigquery/pom.xml
Expand Up @@ -46,6 +46,10 @@
<groupId>org.checkerframework</groupId>
<artifactId>checker-compat-qual</artifactId>
</dependency>
<dependency>
<groupId>org.checkerframework</groupId>
<artifactId>checker-qual</artifactId>
</dependency>
<dependency>
<groupId>com.google.auth</groupId>
<artifactId>google-auth-library-oauth2-http</artifactId>
Expand Down Expand Up @@ -97,6 +101,21 @@
</dependency>

<!-- Test dependencies -->
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-datacatalog</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.google.api.grpc</groupId>
<artifactId>proto-google-cloud-datacatalog-v1</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-storage</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
Expand All @@ -105,6 +124,7 @@
<dependency>
<groupId>com.google.truth</groupId>
<artifactId>truth</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
Expand Down

1 comment on commit 3804275

@Mahajayanthi
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yty

Please sign in to comment.