This document describes a standardized load and performance testing suite that can be used to gauge the performance and resiliency of an ILPv4 Connector under load.
Testing a connector involves several facets of the overall Interledger payment protocol stack. For example, it's useful to measure each of the following metrics:
- Fulfilled Packets-per-second.
- Rejected Packets-per-second.
- Unauthenticated Packets-per-second.
The fulfill/reject metrics should be measured using different payment paths with varying characteristics such as FX-rate conversions, payment path lengths, and other variables inherent in various topologies.
This harness utilizes four different topologies for its testing:
-
Single Connector with Loopback Links
This topology contains a single connector with 2 loopback links. The first link always fulfills; the second link always rejects with aT02 Peer Busyresponse to simulate a peer link that is being throttled. -
Single Connector with SPSP Peer
This topology contains a single connector with 2 child links that support [SPSP](https://github.com/interledger/rfcs /blob/master/0009-simple-payment-setup-protocol/0009-simple-payment-setup-protocol.md). -
Two Connectors with ILP-over-HTTP
This type of topology involves only a two connectors, each with a single ILP-over-HTTP link. -
4 Connector Multihop
This topology simulates four connectors (3 hops) with each operating ILP-over-HTTP links to simulate three total currency conversions.
This topology contains a single connector with 2 loopback
links. The first link always fulfills; the second link always rejects with a T02 Peer Busy response to simulate a peer link that is being throttled.
┌──────────────────┐ ┌─────────────────────────┐
│ │ │Account: `lt-lb-ingress` │
│ Load Test Sender │──────▶│ (Type: ILP-over-HTTP) │
│ │ └─────────────────────────┘
└──────────────────┘ │
┌──────────────┘
│
▽
┌───────────────────────┐
│ │
│ ILPv4 Connector │
│ https://jc.ilpv4.dev │
│ │
└───────────────────────┘
│
┌────────────┴────────────────────┐
▽ ▽
┌─────────────────────────┐ ┌─────────────────────────┐
│Account: lt-lb-fulfiller │ │ Account: lt-lb-rejector │
│ (Type: Loopback) │ │ (Type: Loopback) │
└─────────────────────────┘ └─────────────────────────┘
To create the lt-lb-ingress account, which is used for all ingress into the connector, execute the following command:
curl --location --request POST 'https://jc.ilpv4.dev/accounts' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'Authorization: Basic YWRtaW46cGFzc3dvcmQ=' \
--data-raw '{
"accountId": "lt-lb-ingress",
"accountRelationship": "CHILD",
"linkType": "ILP_OVER_HTTP",
"assetCode": "USD",
"assetScale": "4",
"customSettings": {
"ilpOverHttp.incoming.auth_type": "SIMPLE",
"ilpOverHttp.incoming.simple.auth_token": "shh",
"ilpOverHttp.outgoing.auth_type": "SIMPLE",
"ilpOverHttp.outgoing.simple.auth_token": "shh",
"ilpOverHttp.outgoing.url": "https://money.ilpv4.dev/accounts/lt-lb-ingress/ilp"
}
}'
To create the lt-lb-fulfiller account, execute the following command:
curl --location --request POST 'https://jc.ilpv4.dev/accounts' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'Authorization: Basic YWRtaW46cGFzc3dvcmQ=' \
--data-raw '{
"accountId": "lt-lb-fulfiller",
"accountRelationship": "CHILD",
"linkType": "LOOPBACK",
"assetCode": "XRP",
"assetScale": "9"
}'
To create the lt-lb-rejector account, execute the follwoing command:
curl --location --request POST 'https://jc.ilpv4.dev/accounts' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'Authorization: Basic YWRtaW46cGFzc3dvcmQ=' \
--data-raw '{
"accountId": "lt-lb-rejector",
"accountRelationship": "CHILD",
"linkType": "LOOPBACK",
"assetCode": "XRP",
"assetScale": "9",
"customSettings": {
"simulatedRejectErrorCode":"T02"
}
}'
This topology contains a single connector with 2 ILP-over-HTTP links that excercise actual outbound network connectivity from the Connector. Both links connect to the an SPSP receiver that fulfills or rejects depending on the condition in the packet created using SPSP and STREAM.
┌──────────────────┐ ┌─────────────────────────┐
│ │ │ Account: │
│ Load Test Sender │──────▶│ `lt-spsp-ingress` │
│ │ └─────────────────────────┘
└──────────────────┘ │
┌──────────────┘
│
▽
┌───────────────────────┐
│ │
│ ILPv4 Connector │
│ https://jc.ilpv4.dev │
│ │
└───────────────────────┘
│
┌────────────┴────────────────────┐
▽ ▽
┌─────────────────────────┐ ┌─────────────────────────┐
│ Account: `lt-spsp-rs` │ │ Account: `lt-spsp-java` │
│ (Type: ILP-over-HTTP) │ │ (Type: ILP-over-HTTP) │
└─────────────────────────┘ └─────────────────────────┘
│ │
▽ ▽
┌───────────────────────┐ ┌───────────────────────┐
│ SPSP Server │ │ SPSP Server │
│https://rc3.xpring.dev │ │ https://jc.ilpv4.dev │
└───────────────────────┘ └───────────────────────┘
To create the lt-ingress account, which is used for all ingress into the connector, execute the following command:
curl --location --request POST 'https://jc.ilpv4.dev/accounts' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'Authorization: Basic YWRtaW46cGFzc3dvcmQ=' \
--data-raw '{
"accountId": "lt-spsp-ingress",
"accountRelationship": "CHILD",
"linkType": "ILP_OVER_HTTP",
"assetCode": "USD",
"assetScale": "4",
"customSettings": {
"ilpOverHttp.incoming.auth_type": "SIMPLE",
"ilpOverHttp.incoming.simple.auth_token": "shh",
"ilpOverHttp.outgoing.auth_type": "SIMPLE",
"ilpOverHttp.outgoing.simple.auth_token": "shh",
"ilpOverHttp.outgoing.url": "https://money.ilpv4.dev/accounts/lt-spsp-ingress/ilp"
}
}'
To create the lt-spsp-java account, execute the following command:
curl --location --request POST 'https://jc.ilpv4.dev/accounts' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'Authorization: Basic YWRtaW46cGFzc3dvcmQ=' \
--data-raw '{
"accountId": "lt-spsp-java",
"accountRelationship": "CHILD",
"linkType": "ILP_OVER_HTTP",
"assetCode": "XRP",
"assetScale": "9",
"customSettings": {
"ilpOverHttp.incoming.auth_type": "SIMPLE",
"ilpOverHttp.incoming.simple.auth_token": "shh",
"ilpOverHttp.outgoing.auth_type": "SIMPLE",
"ilpOverHttp.outgoing.simple.auth_token": "shh",
"ilpOverHttp.outgoing.url": "https://jc.ilpv4.dev/accounts/lt-spsp-java/ilp"
}
}'
To create the lt-spsp-rs account, execute the follwoing command:
curl --location --request POST 'https://jc.ilpv4.dev/accounts' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'Authorization: Basic YWRtaW46cGFzc3dvcmQ=' \
--data-raw '{
"accountId": "lt-spsp-rs",
"accountRelationship": "CHILD",
"linkType": "ILP_OVER_HTTP",
"assetCode": "XRP",
"assetScale": "9",
"customSettings": {
"ilpOverHttp.incoming.auth_type": "SIMPLE",
"ilpOverHttp.incoming.simple.auth_token": "shh",
"ilpOverHttp.outgoing.auth_type": "SIMPLE",
"ilpOverHttp.outgoing.simple.auth_token": "shh",
"ilpOverHttp.outgoing.url": "https://jc.ilpv4.dev/accounts/lt-spsp-rs/ilp"
}
}'
Coming soon.
Coming soon.
Load tests are executed via Google Cloud Platform in a setup that largely reflects the AWS Fargate product.
The load tests are contained within a configurable Docker container called interledger4j/ilp-performance. This
container has an executable jar inside of it and is based on Alpine Java 8 at the time of this document being written.
This is due to our use of the Gatling framework which is Scala based and seems to have indigestion when it comes to
later versions of Java.
-e SIMULATION="simulation name"specifies the test to be run-e CONCURRENCY=numbernumber of threads or users-e RAMP_UP=durationinterval to add new users-e THROUGHPUT=number-e HOLD_FOR=durationhold the current throughput for a given duration.-v /path/to/results:resultsmounts a volume to the location where Gatling writes reports. In GCP cases/path/to/resultswill likely be a GCP Storage Bucket.
This description is a bottom up account of the process we're using, rather than top down.
One of the tricky bits to get this all working was that we don't want to have a ton of VMs kicking about after a load test is run because it's just a waste of money: the tests only run for so long and then the VM isn't useful anymore. To accomplish this, we needed the VM to be able to tear itself down at the end of its execution. This is accomplished via a service account that has the following role attached to it:
name: ilp-performance-teardown
permissions:
compute.disks.delete
compute.instances.delete
compute.instances.deleteAccessConfig
The service account will need the ilp-performance-teardown role as well as the following roles:
name: ilp-performance-self-destruct
roles:
ilp-performance-teardown
Logs Writer
Storage Admin
Adding Logs Writer gives access to the gcloud logging write command. We end up needing Storage Admin to be able
to mount a Storage Bucket within the VM to a path that allows for publishing of the Gatling reports.
Nothing particularly fancy about this: we just need some storage bucket to write our reports to. The only specific setting that is (currently) necessary is that the bucket be publicly viewable in order to see the reports render.
The following roles need to be set for the member allUsers:
Storage Legacy Bucket Reader
Storage Object Viewer
The fancier bit happens during machine creation since we end up mounting the bucket via gcsfuse to a path the Docker
container will write the results to.
The JavaScript for this function can be found in the gcp-resources/cloud-function/index.js file.
We rely on a Google Cloud function that will spawn a virtual machine configured with Docker and the container we seek to run. The machine created runs with the service account specified above which allows it to run a command to kill itself after the job completes.
The function accepts a JSON payload to allow configuration of the load test. Below is an example object that contains the field names we care about for executing a test.
{
"simulation": "SomeSimulation",
"concurrency": "number",
"rampUp": "number of seconds",
"holdFor": "number of seconds",
"throughput": "number"
}The job is triggered by being subscribed to a Pub/Sub topic that can receive messages detailing what job should be run and how it should be configured.
Most of the magic happens in the startup script, which does the following:
- Installs
gcsfuseand mounts the Storage Bucket so load test reports are available after the VM is removed - Installs
docker - Runs the load test container and waits for it to finish
- Shuts itself down at the end of the load test
Beyond that, the following should be configured:
- Currently we're using commands compatible with Debian GNU/Linux 9, but there's not a strict limitation of that distro (though the script would be incompatible with non apt-based distros)
- The service account needs to be the one you created with self-destruct capabilities
- You should use at least 2 CPUs
We didn't end up using a VM that allows for direct configuration of a Docker container due to complications with GCP's
Container Optimized OS. That OS requires running a command called toolbox to do things like issue gcloud commands,
but due to the requirement of needing to execute tasks within a shell script we abandoned seeking a solution because
toolbox essentially creates a new shell and we didn't see a clear way to pipe commands to it. We also didn't see
an especially clean way of installing gcsfuse on that version of the OS.
These two are totally intertwined with one another and should just be described together.
We have a Cloud Scheduler job set up that sends a message to a Pub/Sub topic containing details about the test to be executed. The plan at the present is to right jobs nightly (though they should be staggered to avoid resource contention), creating a VM for each job.
The job can be configured with a payload that allows for passing configuration for the load test.
https://medium.com/google-cloud/manage-google-compute-engine-with-node-js-eef8e7a111b4
https://stackoverflow.com/questions/32856043/mount-google-cloud-storage-bucket-to-instance
https://medium.com/google-cloud/scheduled-mirror-sync-sftp-to-gcs-b167d0eb487a
https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-debian-9