Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Stackdriver batch limit for TimeSeries #88

Merged

Conversation

TigerHe7
Copy link
Contributor

@TigerHe7 TigerHe7 commented Jun 8, 2020

Description:

  • Adds a batch limit of 200 to TimeSeries sent due to limit in Stackdriver backend.

Smaller changes:

  • Adds helper function for partitioning batches
  • Fixes misleading comment in transform.ts
  • Fixes parameter mismatch error being throw by metricDescriptors spy

Previous implementation in OpenCensus:
census-instrumentation/opencensus-node#644

Fixes #87

@mayurkale22 mayurkale22 self-requested a review June 8, 2020 19:27
@mayurkale22
Copy link
Contributor

/cc @dancalvert-google @kintel

@mayurkale22
Copy link
Contributor

Did you get a chance to verify these changes against real instance?

Alss, please fix the lint.

Copy link
Contributor Author

@TigerHe7 TigerHe7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some comments for clarity and for a question I had

timeSeries,
MAX_BATCH_EXPORT_SIZE
)) {
await this._sendTimeSeries(batchedTimeSeries);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added an await here for the async code in _sendTimeSeries to be properly called and spied on in tests.

I was wondering if there was a preferred testing practice so we don't have to wait here, or if it's ok to wait so we can handle if an error is thrown?

@@ -99,7 +99,7 @@ function transformValueType(valueType: OTValueType): ValueType {
}

/**
* Converts metric's timeseries to a list of TimeSeries, so that metric can be
* Converts metric's timeseries to a TimeSeries, so that metric can be
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the comment here since a single TimeSeries is being returned. I think the discrepancy was made when this code was copied over.

@@ -69,7 +71,7 @@ describe('MetricExporter', () => {

metricDescriptors = sinon.spy(
/* tslint:disable no-any */
(request: any, callback: (err: Error | null) => void): any => {
(request: any, params: any, callback: (err: Error | null) => void): any => {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See spied on code on line 187 in packages/opentelemetry-cloud-monitoring-exporter/src/monitoring.ts

@codecov
Copy link

codecov bot commented Jun 8, 2020

Codecov Report

Merging #88 into master will increase coverage by 4.21%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #88      +/-   ##
==========================================
+ Coverage   88.53%   92.74%   +4.21%     
==========================================
  Files           9       10       +1     
  Lines         253      262       +9     
  Branches       43       43              
==========================================
+ Hits          224      243      +19     
+ Misses         29       19      -10     
Impacted Files Coverage Δ
...lemetry-cloud-monitoring-exporter/src/transform.ts 93.10% <ø> (ø)
...emetry-cloud-monitoring-exporter/src/monitoring.ts 86.07% <100.00%> (+13.70%) ⬆️
...entelemetry-cloud-monitoring-exporter/src/utils.ts 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f522e68...ea93f51. Read the comment docs.

@TigerHe7 TigerHe7 changed the title Adding batch limit, unit test, and fixes to unit tests fix: Stackdriver batch limit for TimeSeries Jun 8, 2020
@TigerHe7
Copy link
Contributor Author

TigerHe7 commented Jun 9, 2020

@mayurkale22 Did not get to testing against a real instance but will look into that now

Copy link

@nlehrer nlehrer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Just one comment about array mutation.

import { TimeSeries } from './types';

/** Returns an array with arrays of the given size. */
export function partitionList(list: TimeSeries[], chunkSize: number) {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be better to copy the array first, since this will mutate the original array passed into the function. E.g.
const listCopy === [...list];

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the feedback, updated!


import { TimeSeries } from './types';

/** Returns an array with arrays of the given size. */
Copy link

@nlehrer nlehrer Jun 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this could be more clear, like maybe "Returns the given array partitioned into arrays of the given size."

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated to "Returns the minimum number of arrays of max size chunkSize, partitioned from the given array"

@mayurkale22 mayurkale22 added the bug Something isn't working label Jun 10, 2020
@mayurkale22 mayurkale22 requested a review from nlehrer June 16, 2020 06:56
@mayurkale22 mayurkale22 merged commit 2134f00 into GoogleCloudPlatform:master Jun 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add Batch Limit for Stackdriver Exporter
3 participants