Allowing routes to specify the idle socket timeout #72362

kobelb · 2020-07-17T23:20:16Z

This will allow Fleet to configure their routes to have an idle socket timeout that is longer than the global HTTP server's idle socket timeout.

Testing this became quite the challenge...

Fake timers don't work with net.Socket#setTimeout. Using the fake timers has no affect on the socket timeout behaviors.
Using real timers within unit tests is a bad idea, because they're a frequent source of flakiness. This is further complicated by what I consider to be a bug in hapi where the "socket timeout" must be larger than the "payload timeout", and the payload timeout's default is 10 seconds. This would require us to have at least a 10 second period of idleness in our tests, and I'd prefer to not have to do so.
net.Socket#timeout does exist, and it's being consumed within Node.js itself. However, it's not documented. I've opened up doc: add net.Socket#timeout nodejs/node#34543 to document this property, so we can take advantage of it in our tests and not have to rely on real timers.

This PR only allows the route definition to override the "idle socket timeout", it does not allow the route definition to override the "payload timeout" or the "response timeout" as we haven't had any requests for this behavior. This interacts interestingly with the bug in hapi because the idle socket timeout must be larger than the "payload timeout" which defaults to 10 seconds. Until this bug is resolved, that means that the "idle socket timeout" must be larger than 10 seconds, or Hapi will throw an error when the route is defined. I'd prefer to not have to add our own validation for the idle socket timeout, as this will have to be removed when the hapi bug is resolved; however, it's somewhat confusing for the consumers because we leak details about hapi.

…timeout

roncohen · 2020-07-23T18:58:45Z

hey @kobelb. Any chance to move this forward? let us know how we can help

kobelb · 2020-07-23T19:19:22Z

hey @kobelb. Any chance to move this forward? let us know how we can help

I was waiting to catch-up with @joshdover, who returns from PTO shortly, to discuss testing strategies. I'm hopeful we'll be able to move this forward shortly.

kobelb · 2020-07-29T14:09:41Z

#73103 merged prior to this PR, causing conflicts... 👀

kobelb · 2020-07-29T15:20:36Z

@restrry The work that was done as part of https://github.com/elastic/kibana/pull/73103/files conflicts with the approach in this PR. I saw some comments about you hesitating to allow route definitions to specify the "payload timeout", "response timeout" and the "idle socket timeout". However, Fleet would now like to specify the "idle socket timeout". This will cause a conflict with the workaround implemented here to deal with what I consider a bug in hapi.

If the user specifies either the "payload timeout" and the "response timeout", we can increment the "idle socket timeout" by a millisecond to work around this. However, if they specify the "idle socket timeout" in addition to these settings, we're going to potentially violate Hapi's rules and leak the fact that we're using Hapi to consumers.

The only other option I've thought of is to leave it up to the consumer to ensure these timeouts are configured without violating Hapi's rules, but this seems even worse.

mshustov · 2020-07-30T07:13:19Z

However, Fleet would now like to specify the "idle socket timeout". This will cause a conflict with the workaround implemented here to deal with what I consider a bug in hapi.

My point was that we are okay to support this when necessary, so it's not a problem to extend the definition of the route config.

However, if they specify the "idle socket timeout" in addition to these settings, we're going to potentially violate Hapi's rules and leak the fact that we're using Hapi to consumers.

We can add this validation on the platform level as a temporary measurement to buy some time to contribute to Hapi to fix the problem. Another option is to call request.socket.setTimeout manually on the platform level to avoid relying on Hapi logic.

Using real timers within unit tests is a bad idea, because they're a frequent source of flakiness. This is further complicated by what I consider to be a bug in hapi where the "socket timeout" must be larger than the "payload timeout", and the payload timeout's default is 10 seconds.

Introducing the payload timeout option in #73103 allows us to reduce this time. We struggle with the Hapi logic, but:

we can get rid of this limitation when Hapi bug fixed
it's okay for the platform tests to be coupled to Hapi. The fleet might not test this logic if it's covered by the platform.

This will allow Fleet to configure their routes to have an idle socket timeout that is longer than the global HTTP server's idle socket timeout.

Is there an issue to understand the use-case?

kibanamachine · 2020-07-30T08:54:26Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 6ca6708
Pipeline Steps (look for red circles / failed steps)
Interpreting CI Failures

Failed CI Steps

Test Failures

Kibana Pipeline / kibana-oss-agent / Chrome UI Functional Tests.test/functional/apps/context/_date_nanos·js.context app context view for date_nanos displays predessors - anchor - successors in right order

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 16 times on tracked branches: https://github.com/elastic/kibana/issues/58815

[00:00:00]       │
[00:00:12]         └-: context app
[00:00:12]           └-> "before all" hook
[00:00:12]           └-> "before all" hook
[00:00:12]             │ info [logstash_functional] Loading "mappings.json"
[00:00:12]             │ info [logstash_functional] Loading "data.json.gz"
[00:00:12]             │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093237568296121] [logstash-2015.09.22] creating index, cause [api], templates [], shards [1]/[0]
[00:00:13]             │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093237568296121] current.health="GREEN" message="Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[logstash-2015.09.22][0]]])." previous.health="YELLOW" reason="shards started [[logstash-2015.09.22][0]]"
[00:00:13]             │ info [logstash_functional] Created index "logstash-2015.09.22"
[00:00:13]             │ debg [logstash_functional] "logstash-2015.09.22" settings {"index":{"analysis":{"analyzer":{"url":{"max_token_length":"1000","tokenizer":"uax_url_email","type":"standard"}}},"number_of_replicas":"0","number_of_shards":"1"}}
[00:00:13]             │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093237568296121] [logstash-2015.09.20] creating index, cause [api], templates [], shards [1]/[0]
[00:00:13]             │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093237568296121] current.health="GREEN" message="Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[logstash-2015.09.20][0]]])." previous.health="YELLOW" reason="shards started [[logstash-2015.09.20][0]]"
[00:00:13]             │ info [logstash_functional] Created index "logstash-2015.09.20"
[00:00:13]             │ debg [logstash_functional] "logstash-2015.09.20" settings {"index":{"analysis":{"analyzer":{"url":{"max_token_length":"1000","tokenizer":"uax_url_email","type":"standard"}}},"number_of_replicas":"0","number_of_shards":"1"}}
[00:00:13]             │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093237568296121] [logstash-2015.09.21] creating index, cause [api], templates [], shards [1]/[0]
[00:00:13]             │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093237568296121] current.health="GREEN" message="Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[logstash-2015.09.21][0]]])." previous.health="YELLOW" reason="shards started [[logstash-2015.09.21][0]]"
[00:00:13]             │ info [logstash_functional] Created index "logstash-2015.09.21"
[00:00:13]             │ debg [logstash_functional] "logstash-2015.09.21" settings {"index":{"analysis":{"analyzer":{"url":{"max_token_length":"1000","tokenizer":"uax_url_email","type":"standard"}}},"number_of_replicas":"0","number_of_shards":"1"}}
[00:00:22]             │ info progress: 7520
[00:00:28]             │ info [logstash_functional] Indexed 4633 docs into "logstash-2015.09.22"
[00:00:28]             │ info [logstash_functional] Indexed 4757 docs into "logstash-2015.09.20"
[00:00:28]             │ info [logstash_functional] Indexed 4614 docs into "logstash-2015.09.21"
[00:00:28]             │ info [visualize] Loading "mappings.json"
[00:00:28]             │ info [visualize] Loading "data.json"
[00:00:28]             │ info [o.e.c.m.MetadataDeleteIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093237568296121] [.kibana_1/CDx02xpJQuWDkMHhSniTYQ] deleting index
[00:00:28]             │ info [visualize] Deleted existing index [".kibana_1"]
[00:00:28]             │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093237568296121] [.kibana] creating index, cause [api], templates [], shards [1]/[1]
[00:00:28]             │ info [visualize] Created index ".kibana"
[00:00:28]             │ debg [visualize] ".kibana" settings {"index":{"number_of_replicas":"1","number_of_shards":"1"}}
[00:00:28]             │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093237568296121] [.kibana/I_B2Ovu0T8yeR0q7NvW6Cw] update_mapping [_doc]
[00:00:28]             │ info [visualize] Indexed 12 docs into ".kibana"
[00:00:29]             │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093237568296121] [.kibana/I_B2Ovu0T8yeR0q7NvW6Cw] update_mapping [_doc]
[00:00:29]             │ debg Migrating saved objects
[00:00:29]             │ proc [kibana]   log   [07:33:59.285] [info][savedobjects-service] Creating index .kibana_2.
[00:00:29]             │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093237568296121] [.kibana_2] creating index, cause [api], templates [], shards [1]/[1]
[00:00:29]             │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093237568296121] updating number_of_replicas to [0] for indices [.kibana_2]
[00:00:29]             │ proc [kibana]   log   [07:33:59.384] [info][savedobjects-service] Reindexing .kibana to .kibana_1
[00:00:29]             │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093237568296121] [.kibana_1] creating index, cause [api], templates [], shards [1]/[1]
[00:00:29]             │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093237568296121] updating number_of_replicas to [0] for indices [.kibana_1]
[00:00:29]             │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093237568296121] [.tasks] creating index, cause [auto(task api)], templates [], shards [1]/[1]
[00:00:29]             │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093237568296121] updating number_of_replicas to [0] for indices [.tasks]
[00:00:29]             │ info [o.e.t.LoggingTaskListener] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093237568296121] 851 finished with response BulkByScrollResponse[took=97.9ms,timed_out=false,sliceId=null,updated=0,created=12,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]]
[00:00:29]             │ info [o.e.c.m.MetadataDeleteIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093237568296121] [.kibana/I_B2Ovu0T8yeR0q7NvW6Cw] deleting index
[00:00:29]             │ proc [kibana]   log   [07:33:59.854] [info][savedobjects-service] Migrating .kibana_1 saved objects to .kibana_2
[00:00:29]             │ proc [kibana]   log   [07:33:59.881] [error][savedobjects-service] Error: Unable to migrate the corrupt Saved Object document index-pattern:test_index*. To prevent Kibana from performing a migration on every restart, please delete or fix this document by ensuring that the namespace and type in the document's id matches the values in the namespace and type fields.
[00:00:29]             │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093237568296121] [.kibana_2/GTMHRa_AR--UOUx-7dwHKQ] update_mapping [_doc]
[00:00:29]             │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093237568296121] [.kibana_2/GTMHRa_AR--UOUx-7dwHKQ] update_mapping [_doc]
[00:00:29]             │ proc [kibana]   log   [07:34:00.015] [info][savedobjects-service] Pointing alias .kibana to .kibana_2.
[00:00:29]             │ proc [kibana]   log   [07:34:00.094] [info][savedobjects-service] Finished in 813ms.
[00:00:29]             │ debg applying update to kibana config: {"accessibility:disableAnimations":true,"dateFormat:tz":"UTC"}
[00:00:29]             │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093237568296121] [.kibana_2/GTMHRa_AR--UOUx-7dwHKQ] update_mapping [_doc]
[00:00:31]             │ debg replacing kibana config doc: {"defaultIndex":"logstash-*"}
[00:00:32]             │ debg navigating to discover url: http://localhost:6121/app/discover#/
[00:00:32]             │ debg navigate to: http://localhost:6121/app/discover#/
[00:00:32]             │ debg browser[INFO] http://localhost:6121/app/discover?_t=1596094442441#/ 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:00:32]             │
[00:00:32]             │ debg browser[INFO] http://localhost:6121/bundles/app/core/bootstrap.js 42:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:00:32]             │ debg ... sleep(700) start
[00:00:33]             │ debg ... sleep(700) end
[00:00:33]             │ debg returned from get, calling refresh
[00:00:33]             │ debg browser[INFO] http://localhost:6121/app/discover?_t=1596094442441#/ 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:00:33]             │
[00:00:33]             │ debg browser[INFO] http://localhost:6121/bundles/app/core/bootstrap.js 42:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:00:33]             │ debg currentUrl = http://localhost:6121/app/discover#/
[00:00:33]             │          appUrl = http://localhost:6121/app/discover#/
[00:00:33]             │ debg TestSubjects.find(kibanaChrome)
[00:00:33]             │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:00:34]             │ debg browser[INFO] http://localhost:6121/34906/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js 452:106112 "INFO: 2020-07-30T07:34:04Z
[00:00:34]             │        Adding connection to http://localhost:6121/elasticsearch
[00:00:34]             │
[00:00:34]             │      "
[00:00:34]             │ debg ... sleep(501) start
[00:00:35]             │ debg ... sleep(501) end
[00:00:35]             │ debg in navigateTo url = http://localhost:6121/app/discover#/
[00:00:35]             │ debg TestSubjects.exists(statusPageContainer)
[00:00:35]             │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="statusPageContainer"]') with timeout=2500
[00:00:37]             │ debg --- retry.tryForTime error: [data-test-subj="statusPageContainer"] is not displayed
[00:02:03]           └-: context view for date_nanos
[00:02:03]             └-> "before all" hook
[00:02:03]             └-> "before all" hook
[00:02:03]               │ info [date_nanos] Loading "mappings.json"
[00:02:03]               │ info [date_nanos] Loading "data.json"
[00:02:03]               │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093237568296121] [date-nanos] creating index, cause [api], templates [], shards [1]/[0]
[00:02:03]               │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093237568296121] current.health="GREEN" message="Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[date-nanos][0]]])." previous.health="YELLOW" reason="shards started [[date-nanos][0]]"
[00:02:03]               │ info [date_nanos] Created index "date-nanos"
[00:02:03]               │ debg [date_nanos] "date-nanos" settings {"index":{"number_of_replicas":"0","number_of_shards":"1"}}
[00:02:03]               │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093237568296121] [date-nanos/dFcqzMk4Qm-qSKCoc2ukHg] update_mapping [_doc]
[00:02:03]               │ info [date_nanos] Indexed 9 docs into "date-nanos"
[00:02:03]               │ info [date_nanos] Indexed 2 docs into ".kibana"
[00:02:03]               │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093237568296121] [.kibana_2/GTMHRa_AR--UOUx-7dwHKQ] update_mapping [_doc]
[00:02:03]               │ debg Migrating saved objects
[00:02:03]               │ proc [kibana]   log   [07:35:33.621] [info][savedobjects-service] Creating index .kibana_3.
[00:02:03]               │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093237568296121] [.kibana_3] creating index, cause [api], templates [], shards [1]/[1]
[00:02:03]               │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093237568296121] updating number_of_replicas to [0] for indices [.kibana_3]
[00:02:03]               │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093237568296121] current.health="GREEN" message="Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.kibana_3][0]]])." previous.health="YELLOW" reason="shards started [[.kibana_3][0]]"
[00:02:03]               │ proc [kibana]   log   [07:35:33.709] [info][savedobjects-service] Migrating .kibana_2 saved objects to .kibana_3
[00:02:03]               │ proc [kibana]   log   [07:35:33.719] [error][savedobjects-service] Error: Unable to migrate the corrupt Saved Object document index-pattern:test_index*. To prevent Kibana from performing a migration on every restart, please delete or fix this document by ensuring that the namespace and type in the document's id matches the values in the namespace and type fields.
[00:02:03]               │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093237568296121] [.kibana_3/7ZOniGewS1u-M8G71aaOHQ] update_mapping [_doc]
[00:02:03]               │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093237568296121] [.kibana_3/7ZOniGewS1u-M8G71aaOHQ] update_mapping [_doc]
[00:02:03]               │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093237568296121] [.kibana_3/7ZOniGewS1u-M8G71aaOHQ] update_mapping [_doc]
[00:02:03]               │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093237568296121] [.kibana_3/7ZOniGewS1u-M8G71aaOHQ] update_mapping [_doc]
[00:02:03]               │ proc [kibana]   log   [07:35:33.897] [info][savedobjects-service] Pointing alias .kibana to .kibana_3.
[00:02:03]               │ proc [kibana]   log   [07:35:33.958] [info][savedobjects-service] Finished in 339ms.
[00:02:03]               │ debg replacing kibana config doc: {"defaultIndex":"date-nanos"}
[00:02:04]               │ debg applying update to kibana config: {"context:defaultSize":"1","context:step":"3"}
[00:02:05]             └-> displays predessors - anchor - successors in right order 
[00:02:05]               └-> "before each" hook: global before each
[00:02:05]               │ debg browser.get(http://localhost:6121/app/discover#/context/date-nanos/AU_x3-TaGFA8no6Qj999Z?_a=(columns:!('@message')))
[00:02:05]               │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:02:05]               │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:02:05]               │ debg browser[INFO] http://localhost:6121/app/discover?_t=1596094535724#/context/date-nanos/AU_x3-TaGFA8no6Qj999Z?_a=(columns:!(%27@message%27)) 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:02:05]               │
[00:02:05]               │ debg browser[INFO] http://localhost:6121/bundles/app/core/bootstrap.js 42:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:02:07]               │ debg browser[INFO] http://localhost:6121/34906/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js 452:106112 "INFO: 2020-07-30T07:35:36Z
[00:02:07]               │        Adding connection to http://localhost:6121/elasticsearch
[00:02:07]               │
[00:02:07]               │      "
[00:02:07]               │ debg TestSubjects.find(successorsLoadMoreButton)
[00:02:07]               │ debg Find.findByCssSelector('[data-test-subj="successorsLoadMoreButton"]') with timeout=10000
[00:02:07]               │ debg TestSubjects.find(predecessorsLoadMoreButton)
[00:02:07]               │ debg Find.findByCssSelector('[data-test-subj="predecessorsLoadMoreButton"]') with timeout=10000
[00:02:07]               │ debg --- retry.try error: loading context rows
[00:02:07]               │ERROR browser[SEVERE] http://localhost:6121/internal/search/es - Failed to load resource: the server responded with a status of 400 (Bad Request)
[00:02:07]               │ERROR browser[SEVERE] http://localhost:6121/internal/search/es - Failed to load resource: the server responded with a status of 400 (Bad Request)
[00:02:08]               │ERROR browser[SEVERE] http://localhost:6121/34906/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js 413:78739 Error: Bad Request
[00:02:08]               │          at Fetch._callee3$ (http://localhost:6121/34906/bundles/core/core.entry.js:34:108535)
[00:02:08]               │          at l (http://localhost:6121/34906/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:155138)
[00:02:08]               │          at Generator._invoke (http://localhost:6121/34906/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:154891)
[00:02:08]               │          at Generator.forEach.e.<computed> [as next] (http://localhost:6121/34906/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:155495)
[00:02:08]               │          at fetch_asyncGeneratorStep (http://localhost:6121/34906/bundles/core/core.entry.js:34:101676)
[00:02:08]               │          at _next (http://localhost:6121/34906/bundles/core/core.entry.js:34:101992) "Possibly unhandled rejection: {\"request\":{},\"response\":{},\"body\":{\"statusCode\":400,\"error\":\"Bad Request\",\"message\":\"[illegal_argument_exception] date[+49714927-06-25T16:26:39.999Z] is after 2262-04-11T23:47:16.854775807 and cannot be stored in nanosecond resolution\",\"attributes\":{\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"date[+49714927-06-25T16:26:39.999Z] is after 2262-04-11T23:47:16.854775807 and cannot be stored in nanosecond resolution\"}],\"type\":\"search_phase_execution_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":[{\"shard\":0,\"index\":\"date-nanos\",\"node\":\"G5yrxC_5S86gUB1C90AUhQ\",\"reason\":{\"type\":\"illegal_argument_exception\",\"reason\":\"date[+49714927-06-25T16:26:39.999Z] is after 2262-04-11T23:47:16.854775807 and cannot be stored in nanosecond resolution\"}}],\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"date[+49714927-06-25T16:26:39.999Z] is after 2262-04-11T23:47:16.854775807 and cannot be stored in nanosecond resolution\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"date[+49714927-06-25T16:26:39.999Z] is after 2262-04-11T23:47:16.854775807 and cannot be stored in nanosecond resolution\"}}}}},\"name\":\"Error\",\"req\":\"...\",\"res\":\"...\"}"
[00:02:08]               │ debg TestSubjects.find(successorsLoadMoreButton)
[00:02:08]               │ debg Find.findByCssSelector('[data-test-subj="successorsLoadMoreButton"]') with timeout=10000
[00:02:08]               │ debg TestSubjects.find(predecessorsLoadMoreButton)
[00:02:08]               │ debg Find.findByCssSelector('[data-test-subj="predecessorsLoadMoreButton"]') with timeout=10000
[00:02:08]               │ debg ... sleep(1000) start
[00:02:09]               │ debg ... sleep(1000) end
[00:02:09]               │ debg TestSubjects.find(docTable)
[00:02:09]               │ debg Find.findByCssSelector('[data-test-subj="docTable"]') with timeout=10000
[00:02:09]               │ info Taking screenshot "/dev/shm/workspace/kibana/test/functional/screenshots/failure/context app context view for date_nanos displays predessors - anchor - successors in right order .png"
[00:02:09]               │ info Current URL is: http://localhost:6121/app/discover#/context/date-nanos/AU_x3-TaGFA8no6Qj999Z?_a=(columns:!(%27@message%27))
[00:02:09]               │ info Saving page source to: /dev/shm/workspace/kibana/test/functional/failure_debug/html/context app context view for date_nanos displays predessors - anchor - successors in right order .html
[00:02:09]               └- ✖ fail: context app context view for date_nanos displays predessors - anchor - successors in right order 
[00:02:09]               │       Error: expected [ 'Sep 18, 2019 @ 06:50:12.999999999-3' ] to sort of equal [ 'Sep 18, 2019 @ 06:50:13.000000000-2',
[00:02:09]               │   'Sep 18, 2019 @ 06:50:12.999999999-3',
[00:02:09]               │   'Sep 19, 2015 @ 06:50:13.0001000011' ]
[00:02:09]               │       + expected - actual
[00:02:09]               │ 
[00:02:09]               │        [
[00:02:09]               │       +  "Sep 18, 2019 @ 06:50:13.000000000-2"
[00:02:09]               │          "Sep 18, 2019 @ 06:50:12.999999999-3"
[00:02:09]               │       +  "Sep 19, 2015 @ 06:50:13.0001000011"
[00:02:09]               │        ]
[00:02:09]               │       
[00:02:09]               │       at Assertion.assert (packages/kbn-expect/expect.js:100:11)
[00:02:09]               │       at Assertion.eql (packages/kbn-expect/expect.js:244:8)
[00:02:09]               │       at Context.<anonymous> (test/functional/apps/context/_date_nanos.js:57:33)
[00:02:09]               │       at process._tickCallback (internal/process/next_tick.js:68:7)
[00:02:09]               │ 
[00:02:09]               │

Stack Trace

{ Error: expected [ 'Sep 18, 2019 @ 06:50:12.999999999-3' ] to sort of equal [ 'Sep 18, 2019 @ 06:50:13.000000000-2',
  'Sep 18, 2019 @ 06:50:12.999999999-3',
  'Sep 19, 2015 @ 06:50:13.0001000011' ]
    at Assertion.assert (packages/kbn-expect/expect.js:100:11)
    at Assertion.eql (packages/kbn-expect/expect.js:244:8)
    at Context.<anonymous> (test/functional/apps/context/_date_nanos.js:57:33)
    at process._tickCallback (internal/process/next_tick.js:68:7)
  actual: '[\n  "Sep 18, 2019 @ 06:50:12.999999999-3"\n]',
  expected:
   '[\n  "Sep 18, 2019 @ 06:50:13.000000000-2"\n  "Sep 18, 2019 @ 06:50:12.999999999-3"\n  "Sep 19, 2015 @ 06:50:13.0001000011"\n]',
  showDiff: true }

Kibana Pipeline / kibana-xpack-agent / X-Pack API Integration Tests.x-pack/test/api_integration/apis/endpoint/resolver·ts.apis Endpoint plugin Resolver related alerts route endpoint events should return details for the root node

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

[00:00:00]       │
[00:00:00]         └-: apis
[00:00:00]           └-> "before all" hook
[00:13:24]           └-: Endpoint plugin
[00:13:24]             └-> "before all" hook
[00:13:24]             └-> "before all" hook
[00:13:31]               │ info [o.e.x.s.a.r.TransportPutRoleAction] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093207871198888] updated role [fleet_enroll]
[00:13:31]               │ info [o.e.x.s.a.u.TransportPutUserAction] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093207871198888] updated user [fleet_enroll]
[00:13:33]             └-: Resolver
[00:13:33]               └-> "before all" hook
[00:13:33]               └-> "before all" hook
[00:13:33]                 │ info [endpoint/resolver/api_feature] Loading "mappings.json"
[00:13:33]                 │ info [endpoint/resolver/api_feature] Loading "data.json.gz"
[00:13:33]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093207871198888] [endgame-4.21.0-000001] creating index, cause [api], templates [], shards [5]/[0]
[00:13:34]                 │ info [endpoint/resolver/api_feature] Created index "endgame-4.21.0-000001"
[00:13:34]                 │ debg [endpoint/resolver/api_feature] "endgame-4.21.0-000001" settings {"index":{"lifecycle":{"name":"endgame_policy-4.21.0","rollover_alias":"endgame-4.21.0"},"mapping":{"ignore_malformed":"true","total_fields":{"limit":"10000"}},"number_of_replicas":"0","number_of_shards":"5","refresh_interval":"5s"}}
[00:13:34]                 │ info [endpoint/resolver/api_feature] Indexed 156 docs into "endgame-4.21.0-000001"
[00:13:34]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093207871198888] [.ds-logs-endpoint.events.process-default-000001] creating index, cause [initialize_data_stream], templates [logs-endpoint.events.process], shards [1]/[1]
[00:13:34]                 │ info [o.e.c.m.MetadataCreateDataStreamService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093207871198888] adding data stream [logs-endpoint.events.process-default]
[00:13:34]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093207871198888] [.ds-logs-endpoint.alerts-default-000001] creating index, cause [initialize_data_stream], templates [logs-endpoint.alerts], shards [1]/[1]
[00:13:34]                 │ info [o.e.c.m.MetadataCreateDataStreamService] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093207871198888] adding data stream [logs-endpoint.alerts-default]
[00:13:34]                 │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093207871198888] moving index [.ds-logs-endpoint.events.process-default-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [logs]
[00:13:34]                 │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093207871198888] moving index [.ds-logs-endpoint.alerts-default-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [logs]
[00:13:34]                 │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093207871198888] moving index [.ds-logs-endpoint.events.process-default-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [logs]
[00:13:34]               └-: related alerts route
[00:13:34]                 └-> "before all" hook
[00:13:34]                 └-: endpoint events
[00:13:34]                   └-> "before all" hook
[00:13:34]                   └-> should not find any alerts
[00:13:34]                     └-> "before each" hook: global before each
[00:13:34]                     │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-16-tests-xl-1596093207871198888] moving index [.ds-logs-endpoint.alerts-default-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [logs]
[00:13:34]                     └- ✓ pass  (19ms) "apis Endpoint plugin Resolver related alerts route endpoint events should not find any alerts"
[00:13:34]                   └-> should return details for the root node
[00:13:34]                     └-> "before each" hook: global before each
[00:13:34]                     └- ✖ fail: apis Endpoint plugin Resolver related alerts route endpoint events should return details for the root node
[00:13:34]                     │      Error: expected undefined to be truthy
[00:13:34]                     │       at Assertion.assert (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:100:11)
[00:13:34]                     │       at Assertion.ok (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:122:8)
[00:13:34]                     │       at Function.ok (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:531:15)
[00:13:34]                     │       at forEach (test/api_integration/apis/endpoint/resolver.ts:175:13)
[00:13:34]                     │       at Array.forEach (<anonymous>)
[00:13:34]                     │       at compareArrays (test/api_integration/apis/endpoint/resolver.ts:170:10)
[00:13:34]                     │       at Context.it (test/api_integration/apis/endpoint/resolver.ts:280:11)
[00:13:34]                     │ 
[00:13:34]                     │

Stack Trace

Error: expected undefined to be truthy
    at Assertion.assert (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:100:11)
    at Assertion.ok (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:122:8)
    at Function.ok (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:531:15)
    at forEach (test/api_integration/apis/endpoint/resolver.ts:175:13)
    at Array.forEach (<anonymous>)
    at compareArrays (test/api_integration/apis/endpoint/resolver.ts:170:10)
    at Context.it (test/api_integration/apis/endpoint/resolver.ts:280:11)

Kibana Pipeline / x-pack-intake-agent / X-Pack Jest Tests.x-pack/plugins/uptime/public/components/common/charts/__tests__.PingHistogram component renders the component without errors

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 8 times on tracked branches: https://github.com/elastic/kibana/issues/73522

Stack Trace

Error: expect(received).toMatchSnapshot()

Snapshot name: `PingHistogram component renders the component without errors 1`

- Snapshot  - 1
+ Received  + 1

@@ -3,11 +3,11 @@
      class="euiTitle euiTitle--xsmall"
    >
      Pings over time
    </h2>,
    <div
-     aria-label="Bar Chart showing uptime status over time from a year ago to a year ago."
+     aria-label="Bar Chart showing uptime status over time from 2 years ago to 2 years ago."
      style="height:100%;opacity:1;transition:opacity 0.2s"
    >
      <div
        class="echChart"
      >
    at Object.it (/dev/shm/workspace/kibana/x-pack/plugins/uptime/public/components/common/charts/__tests__/ping_histogram.test.tsx:58:23)
    at Promise (/dev/shm/workspace/kibana/node_modules/jest-circus/build/utils.js:198:28)
    at new Promise (<anonymous>)
    at callAsyncCircusFn (/dev/shm/workspace/kibana/node_modules/jest-circus/build/utils.js:162:10)
    at _callCircusTest (/dev/shm/workspace/kibana/node_modules/jest-circus/build/run.js:205:40)
    at process._tickCallback (internal/process/next_tick.js:68:7)

and 1 more failures, only showing the first 3.

Build metrics

✅ unchanged

History

💔 Build #65191 failed 6ca6708
💔 Build #62971 failed 6cc4fc1
💔 Build #62786 failed 9ce7c48

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

roncohen · 2020-07-31T19:13:30Z

@restrry the use case is that Elastic Agent connects to Kibana and does long-polling . It does long-polling because it means Elastic Agent can react to user-initiated changes immediately and at the same time, we reduce the number of requests that Kibana would need to respond to if we were doing regular polling. Increasing the socket timeout greatly increases the number of Elastic Agents each Kibana instance can handle.

Can we help move this forward somehow?

mshustov · 2020-08-03T12:52:30Z

@roncohen How soon you need it? As I can see @kobelb is going back tomorrow

roncohen · 2020-08-10T09:25:24Z

@restrry @kobelb aiming for 7.10 for this, right?

kobelb · 2020-08-10T14:01:04Z

@restrry @kobelb aiming for 7.10 for this, right?

Yup. However, we'll likely go with #73730 instead of this PR.

mshustov · 2020-08-19T13:16:51Z

@kobelb can we close PR as #73730 merged?

kobelb · 2020-08-19T14:01:06Z

@kobelb can we close PR as #73730 merged?

Yup, absolutely!

kobelb added 3 commits July 17, 2020 10:39

Allowing routes to specify the idle socket timeout

9ce7c48

Adding unit-tests which rely on non-public properties by abusing any

5aba5fc

Merge remote-tracking branch 'upstream/master' into core/idle-socket-…

6cc4fc1

…timeout

roncohen mentioned this pull request Jul 21, 2020

[Ingest Manager/Cloud] Fleet: Set good settings for Fleet in Cloud #72612

Closed

Add link to the nodejs/node issue documenting net.Socket:timeout

6ca6708

kobelb mentioned this pull request Jul 29, 2020

Allow routes to specify the idle socket timeout in addition to the payload timeout #73730

Merged

ph requested a review from jfsiii July 30, 2020 01:00

kobelb closed this Aug 19, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allowing routes to specify the idle socket timeout #72362

Allowing routes to specify the idle socket timeout #72362

kobelb commented Jul 17, 2020 •

edited

roncohen commented Jul 23, 2020

kobelb commented Jul 23, 2020

kobelb commented Jul 29, 2020

kobelb commented Jul 29, 2020

mshustov commented Jul 30, 2020 •

edited

kibanamachine commented Jul 30, 2020

Standard Out

Stack Trace

Standard Out

Stack Trace

Standard Out

Stack Trace

roncohen commented Jul 31, 2020

mshustov commented Aug 3, 2020

roncohen commented Aug 10, 2020

kobelb commented Aug 10, 2020

mshustov commented Aug 19, 2020

kobelb commented Aug 19, 2020

Allowing routes to specify the idle socket timeout #72362

Allowing routes to specify the idle socket timeout #72362

Conversation

kobelb commented Jul 17, 2020 • edited

roncohen commented Jul 23, 2020

kobelb commented Jul 23, 2020

kobelb commented Jul 29, 2020

kobelb commented Jul 29, 2020

mshustov commented Jul 30, 2020 • edited

kibanamachine commented Jul 30, 2020

💔 Build Failed

Failed CI Steps

Test Failures

Standard Out

Stack Trace

Standard Out

Stack Trace

Standard Out

Stack Trace

Build metrics

History

roncohen commented Jul 31, 2020

mshustov commented Aug 3, 2020

roncohen commented Aug 10, 2020

kobelb commented Aug 10, 2020

mshustov commented Aug 19, 2020

kobelb commented Aug 19, 2020

kobelb commented Jul 17, 2020 •

edited

mshustov commented Jul 30, 2020 •

edited