Support for multiple instances behind load balancer

[lautarodragan]

Running multiple instances of the node behind a load balancer could lead to race conditions.

node/src/Storage/ClaimController.ts

Lines 111 to 117 in b1ffa5e

	const pipe = pipeP(
	this.findEntryToDownload,
	this.updateEntryAttempts,
	this.downloadEntryClaim,
	this.updateEntryPairs,
	this.publishEntryDownload
	)

To ensure there are no race conditions we need to use findAndModify and ensure the proper indexes are created.

TODO:

• Find and list all places in code that could lead to race conditions
• List all indexes we're missing

[geoffturk]

Assigning to @warrenv.

[WesleyCharlesBlake]

@lautarodragan and @wzalazar : would we need to replicate this across to frost-api as well?

[lautarodragan]

@WesleyCharlesBlake we already have four instances of Frost API behind an LB in production and they have been working fine 😎

Frost API has much less state in general (since it delegates most operations to the node), but it'd be nice to do a thorough check just to be on the safe side.

[WesleyCharlesBlake]

Awesome! Was just making sure. But yes will need a thorough audit in frost-api . So for in k8s it's been working well.

[geoffturk]

Reassigning to @kennylavender. Please sync with Warren if you need any input.

[geoffturk]

Moving this back over to you @warrenv to keep Kenny focused on API and work/claim split. Thanks.

[geoffturk]

Running multiple instances of the node behind a load balancer could lead to race conditions.

Is this something we can be testing out in the new QA environment right now? (@WesleyCharlesBlake @lautarodragan)

[WesleyCharlesBlake]

yeah simple enough to test out, just need to increase the replica count for each node configuration from 1 to n (where n is the ideal amount needed to run optimally)