benchmark

Benchmark Tool Usage

In this directory is a tool that can be used to benchmark GAIA. It serves as multiple clients to send queries to gremlin server through the gremlin endpoint exposed by the engine, and report the performance numbers (e.g., latency, throughput, query results). The benchmark program sends mixed queries to the server by reading query templates from queries with filling the parameters in the query templates using substitution_parameters. The program uses a round-robin strategy to iterate all the enabled queries with corresponding parameters.

Repository contents

- config                                
    - interactive-benchmark.properties  // configurations for running benchmark
- data
    - substitution_parameters           // query parameter files using to fill the query templates
- queries                               // query templates including LDBC queries, K-hop queries and user-defined queries
- scripts
    - benchmark.sh                      // script for running benchmark
    - cal.py                            // script for calculating benchmark results
- src                                   // source code of benchmark program

Note: the queries here with the prefix ldbc_query are implementations of LDBC official interactive complex reads, and the corresponding parameters (factor 1) are generated by LDBC official tools.

Building

Build benchmark program using Maven:

mvn clean package

All the binary and queries would be packed into target/benchmark-0.0.1-SNAPSHOT-dist.tar.gz, and you can use deploy the package to anywhere could connect to the gremlin endpoint.

Running the benchmark

cd target
tar -xvf gaia-benchmark-0.0.1-SNAPSHOT-dist.tar.gz
cd gaia-benchmark-0.0.1-SNAPSHOT
vim config/interactive-benchmark.properties # specify the gremlin endpoint of your server and modify running configurations
chmod +x ./scripts/benchmark.sh 
./scripts/benchmark.sh                      # run the benchmark program

Benchmark reports numbers as following:

QueryName[LDBC_QUERY_1], Parameter[{firstName=John, personId=17592186223433}], ResultCount[87], ExecuteTimeMS[ 1266 ].
QueryName[LDBC_QUERY_12], Parameter[{tagClassName=Judge, personId=19791209469071}], ResultCount[0], ExecuteTimeMS[ 259 ].
QueryName[LDBC_QUERY_11], Parameter[{workFromYear=2001, personId=32985348901156, countryName=Bolivia}], ResultCount[0], ExecuteTimeMS[ 60 ].
QueryName[LDBC_QUERY_9], Parameter[{personId=10995116420051, maxDate=20121128080000000}], ResultCount[20], ExecuteTimeMS[ 55755 ].
QueryName[LDBC_QUERY_8], Parameter[{personId=67523}], ResultCount[20], ExecuteTimeMS[ 148 ].
QueryName[LDBC_QUERY_7], Parameter[{personId=26388279199350}], ResultCount[0], ExecuteTimeMS[ 10 ].
QueryName[LDBC_QUERY_6], Parameter[{personId=26388279148519, tagName=Vallabhbhai_Patel}], ResultCount[0], ExecuteTimeMS[ 12837 ].
QueryName[LDBC_QUERY_5], Parameter[{minDate=20120814080000000, personId=2199023436754}], ResultCount[0], ExecuteTimeMS[ 11268 ].
QueryName[LDBC_QUERY_3], Parameter[{durationDays=30, endDate=20110701080000000, countryXName=Mongolia, countryYName=Namibia, personId=8796093204429, startDate=20110601080000000}], ResultCount[20]
, ExecuteTimeMS[ 21474 ].
QueryName[LDBC_QUERY_2], Parameter[{personId=28587302394490, maxDate=20121128080000000}], ResultCount[20], ExecuteTimeMS[ 331 ].
query count: 10; execute time(ms): ...; qps: ...

User-defined Benchmarking Queries

Users can add their own benchmarking queries to queries as well as adding substitution parameters of queries to substitution_parameters. Note that the file name of user-defined query templates should follow the prefix custom_query or custom_constant_query. The difference between custom_query and custom_constant_query is that the latter has no corresponding parameters.