Update diagnostics service to version 1 #7059

farost · 2024-05-07T08:31:15Z

Usage and product changes

We introduce an updated version of diagnostics sent from a TypeDB server.

config.yml gets a new field deploymentID for the diagnostics section. This field should be used for collecting the data from multiple servers of a single TypeDB Cloud deployment.
The updated diagnostics data contains more information about the server resources and details for each separate database. More details can be found in the examples below.
For the JSON reporting, we calculated diffs between the current timestamp and the sinceTimestamp (the previous hour when the data had to be sent: it's updated even if we had errors sending the data for simplicity). For the Prometheus data, we send raw counts as Prometheus calculates diffs based on its queries and expects raw diagnostics from our side.
For the JSON monitoring, we show only the incrementing counters from the start of the server just as for the Prometheus diagnostics data (also available through the monitoring page). This way, the content is different from the reporting data.
The schema and data diagnostics about each specific database are sent only from the primary replica of a deployment at the moment of the diagnostics collection. The connection peak values diagnostics regarding a database are still reported by a non-primary replica if the database exists or there were established transactions within the last hour before the database had been deleted.
If the statistics reporting is turned off in the config, we send a totally safe part of the diagnostics data once to notify the server about the moment when the diagnostics were turned off. No user data is shared in this snapshot (see examples below). This action is performed only if the server is up for 1 hour (to avoid our CI tests report data), and only if the server has not successfully sent such a snapshot after turning the statistics reporting off the last time. If there is an error in sending this snapshot, the server will try again after a restart (no extra logic here).

Example diagnostics data for Prometheus (http://localhost:4104/metrics?format=prometheus):

# distribution: TypeDB Core
# version: 2.28.0
# os: Mac OS X x86_64 14.2.1

# TYPE server_resources_count gauge
server_resources_count{kind="memoryUsedInBytes"} 68160245760
server_resources_count{kind="memoryAvailableInBytes"} 559230976
server_resources_count{kind="diskUsedInBytes"} 175619862528
server_resources_count{kind="diskAvailableInBytes"} 1819598303232

# TYPE typedb_schema_data_count gauge
typedb_schema_data_count{database="212487319", kind="typeCount"} 74
typedb_schema_data_count{database="212487319", kind="entityCount"} 2891
typedb_schema_data_count{database="212487319", kind="relationCount"} 2466
typedb_schema_data_count{database="212487319", kind="attributeCount"} 5832
typedb_schema_data_count{database="212487319", kind="hasCount"} 13325
typedb_schema_data_count{database="212487319", kind="roleCount"} 7984
typedb_schema_data_count{database="212487319", kind="storageInBytes"} 2164793
typedb_schema_data_count{database="212487319", kind="storageKeyCount"} 94028
typedb_schema_data_count{database="3717486", kind="typeCount"} 5
typedb_schema_data_count{database="3717486", kind="entityCount"} 0
typedb_schema_data_count{database="3717486", kind="relationCount"} 0
typedb_schema_data_count{database="3717486", kind="attributeCount"} 0
typedb_schema_data_count{database="3717486", kind="hasCount"} 0
typedb_schema_data_count{database="3717486", kind="roleCount"} 0
typedb_schema_data_count{database="3717486", kind="storageInBytes"} 0
typedb_schema_data_count{database="3717486", kind="storageKeyCount"} 0

# TYPE typedb_attempted_requests_total counter
typedb_attempted_requests_total{kind="CONNECTION_OPEN"} 4
typedb_attempted_requests_total{kind="DATABASES_ALL"} 4
typedb_attempted_requests_total{kind="DATABASES_GET"} 4
typedb_attempted_requests_total{kind="SERVERS_ALL"} 4
typedb_attempted_requests_total{database="212487319", kind="DATABASES_CONTAINS"} 2
typedb_attempted_requests_total{database="212487319", kind="SESSION_OPEN"} 2
typedb_attempted_requests_total{database="212487319", kind="TRANSACTION_EXECUTE"} 70
typedb_attempted_requests_total{database="212487319", kind="SESSION_CLOSE"} 1
typedb_attempted_requests_total{database="3717486", kind="DATABASES_CONTAINS"} 2
typedb_attempted_requests_total{database="3717486", kind="SESSION_OPEN"} 2
typedb_attempted_requests_total{database="3717486", kind="TRANSACTION_EXECUTE"} 54
typedb_attempted_requests_total{database="3717486", kind="SESSION_CLOSE"} 1

# TYPE typedb_successful_requests_total counter
typedb_successful_requests_total{kind="CONNECTION_OPEN"} 4
typedb_successful_requests_total{kind="DATABASES_ALL"} 4
typedb_successful_requests_total{kind="DATABASES_GET"} 4
typedb_successful_requests_total{kind="SERVERS_ALL"} 4
typedb_successful_requests_total{kind="USER_TOKEN"} 8
typedb_successful_requests_total{database="212487319", kind="DATABASES_CONTAINS"} 2
typedb_successful_requests_total{database="212487319", kind="SESSION_OPEN"} 2
typedb_successful_requests_total{database="212487319", kind="TRANSACTION_EXECUTE"} 67
typedb_successful_requests_total{database="212487319", kind="SESSION_CLOSE"} 1
typedb_successful_requests_total{database="3717486", kind="DATABASES_CONTAINS"} 2
typedb_successful_requests_total{database="3717486", kind="SESSION_OPEN"} 2
typedb_successful_requests_total{database="3717486", kind="TRANSACTION_EXECUTE"} 47
typedb_successful_requests_total{database="3717486", kind="SESSION_CLOSE"} 1

# TYPE typedb_error_total counter
typedb_error_total{database="3717486", code="TYR03"} 5
typedb_error_total{database="3717486", code="TXN08"} 2

Example diagnostics JSON data from monitoring (http://localhost:4104/metrics?format=JSON):

{
  "version": 1,
  "deploymentID": "HTAOYJNSRYY2WOUR",
  "serverID": "HTAOYJNSRYY2WOUR",
  "distribution": "TypeDB Core",
  "timestamp": "2024-05-14T09:50:46",
  "server": {
    "version": "2.28.0",
    "uptimeInSeconds": 134,
    "os": {
      "name": "Mac OS X",
      "arch": "x86_64",
      "version": "14.2.1"
    },
    "memoryUsedInBytes": 68151644160,
    "memoryAvailableInBytes": 567832576,
    "diskUsedInBytes": 175619862528,
    "diskAvailableInBytes": 1819598303232
  },
  "load": [
    {
      "database": "212487319",
      "schema": {
        "typeCount": 74
      },
      "data": {
        "entityCount": 2891,
        "relationCount": 2466,
        "attributeCount": 5832,
        "hasCount": 13325,
        "roleCount": 7984,
        "storageInBytes": 2164793,
        "storageKeyCount": 94028
      }
    },
    {
      "database": "3717486",
      "schema": {
        "typeCount": 5
      },
      "data": {
        "entityCount": 0,
        "relationCount": 0,
        "attributeCount": 0,
        "hasCount": 0,
        "roleCount": 0,
        "storageInBytes": 0,
        "storageKeyCount": 0
      }
    }
  ],
  "actions": [
    {
      "name": "CONNECTION_OPEN",
      "attempted": 4,
      "successful": 4
    },
    {
      "name": "DATABASES_ALL",
      "attempted": 4,
      "successful": 4
    },
    {
      "name": "DATABASES_GET",
      "attempted": 4,
      "successful": 4
    },
    {
      "name": "SERVERS_ALL",
      "attempted": 4,
      "successful": 4
    },
    {
      "name": "DATABASES_CONTAINS",
      "database": "212487319",
      "attempted": 2,
      "successful": 2
    },
    {
      "name": "SESSION_OPEN",
      "database": "212487319",
      "attempted": 2,
      "successful": 2
    },
    {
      "name": "TRANSACTION_EXECUTE",
      "database": "212487319",
      "attempted": 70,
      "successful": 67
    },
    {
      "name": "SESSION_CLOSE",
      "database": "212487319",
      "attempted": 1,
      "successful": 1
    },
    {
      "name": "DATABASES_CONTAINS",
      "database": "3717486",
      "attempted": 2,
      "successful": 2
    },
    {
      "name": "SESSION_OPEN",
      "database": "3717486",
      "attempted": 2,
      "successful": 2
    },
    {
      "name": "TRANSACTION_EXECUTE",
      "database": "3717486",
      "attempted": 54,
      "successful": 47
    },
    {
      "name": "SESSION_CLOSE",
      "database": "3717486",
      "attempted": 1,
      "successful": 1
    }
  ],
  "errors": [
    {
      "code": "TYR03",
      "database": "3717486",
      "count": 5
    },
    {
      "code": "TXN08",
      "database": "3717486",
      "count": 2
    }
  ]
}

Example of diagnostics JSON data sent when the reporting flag is turned on:

{
  "version":1,
  "deploymentID":"HTAOYJNSRYY2WOUR",
  "serverID":"HTAOYJNSRYY2WOUR",
  "distribution":"TypeDB Core",
  "timestamp":"2024-05-14T09:50:36",
  "periodInSeconds":3600,
  "enabled":true,
  "server":{
    "version":"2.28.0",
    "uptimeInSeconds":124,
    "os":{
      "name":"Mac OS X",
      "arch":"x86_64",
      "version":"14.2.1"
    },
    "memoryUsedInBytes":68097245184,
    "memoryAvailableInBytes":622231552,
    "diskUsedInBytes":175624044544,
    "diskAvailableInBytes":1819594121216
  },
  "load":[
    {
      "database":"212487319",
      "schema":{
        "typeCount":74
      },
      "data":{
        "entityCount":2868,
        "relationCount":2449,
        "attributeCount":5816,
        "hasCount":13247,
        "roleCount":7927,
        "storageInBytes":2164793,
        "storageKeyCount":93379
      },
      "connection":{
        "schemaTransactionPeakCount":0,
        "readTransactionPeakCount":1,
        "writeTransactionPeakCount":1
      }
    },
    {
      "database":"3717486",
      "schema":{
        "typeCount":5
      },
      "data":{
        "entityCount":0,
        "relationCount":0,
        "attributeCount":0,
        "hasCount":0,
        "roleCount":0,
        "storageInBytes":0,
        "storageKeyCount":0
      },
      "connection":{
        "schemaTransactionPeakCount":0,
        "readTransactionPeakCount":2,
        "writeTransactionPeakCount":1
      }
    }
  ],
  "actions":[
    {
      "name":"CONNECTION_OPEN",
      "successful":2,
      "failed":0
    },
    {
      "name":"DATABASES_ALL",
      "successful":2,
      "failed":0
    },
    {
      "name":"DATABASES_GET",
      "successful":2,
      "failed":0
    },
    {
      "name":"SERVERS_ALL",
      "successful":2,
      "failed":0
    },
    {
      "name":"DATABASES_CONTAINS",
      "database":"212487319",
      "successful":1,
      "failed":0
    },
    {
      "name":"SESSION_OPEN",
      "database":"212487319",
      "successful":1,
      "failed":0
    },
    {
      "name":"TRANSACTION_EXECUTE",
      "database":"212487319",
      "successful":32,
      "failed":2
    },
    {
      "name":"SESSION_CLOSE",
      "database":"212487319",
      "successful":1,
      "failed":0
    },
    {
      "name":"DATABASES_CONTAINS",
      "database":"3717486",
      "successful":1,
      "failed":0
    },
    {
      "name":"SESSION_OPEN",
      "database":"3717486",
      "successful":1,
      "failed":0
    },
    {
      "name":"TRANSACTION_EXECUTE",
      "database":"3717486",
      "successful":27,
      "failed":4
    },
    {
      "name":"SESSION_CLOSE",
      "database":"3717486",
      "successful":1,
      "failed":0
    }
  ],
  "errors":[
    {
      "code":"TYR03",
      "database":"3717486",
      "count":3
    },
    {
      "code":"TXN08",
      "database":"3717486",
      "count":1
    }
  ]
}

Example of diagnostics JSON data sent once when the reporting flag is turned off:

{
  "version":1,
  "deploymentID":"HTAOYJNSRYY2WOUR",
  "serverID":"HTAOYJNSRYY2WOUR",
  "distribution":"TypeDB Core",
  "timestamp":"2024-05-14T10:03:53",
  "periodInSeconds":3600,
  "enabled":false,
  "server":{
    "version":"2.28.0"
  }
}

Implementation

There is no huge refactoring as it's planned to be a cleaner feature in the incoming 3.0.

server/common/Constants.java

common/diagnostics/Metrics.java

farost · 2024-05-07T09:14:32Z

common/diagnostics/Metrics.java

-        this.usage = new CurrentCounts();
-        this.userErrors = new UserErrorStatistics();
+    public void takeSnapshot() {
+        this.base.updateSinceTimestamp();


I added sinceTimestamp in the json data in the end as it was strange to see diffs in the diagnostics on the monitoring page. It's not used anywhere in the external code, but could potentially be in the future.

I thought the monitoring page was supposed to use the same data as Prometheus, with counters steadily ticking up?

I can do that, but I just didn't know about it. Let's confirm with anyone who might be interested in this page. It would make sense, I guess.

Made it as you proposed. WIll update the examples and description tomorrow's morning.

common/diagnostics/Metrics.java

common/diagnostics/StatisticReporter.java

database/CoreDatabaseManager.java

server/TransactionService.java

dmitrii-ubskii

Partial review

server/test/parameters/config/config-correct-optional-values.yml

test/integration/server/ParametersTest.java

dmitrii-ubskii · 2024-05-13T11:49:21Z

common/diagnostics/Metrics.java

I thought this was going to be renamed to smth like DiagnosticsStore?

I decided not to rename anything before 3.0 as it could become messy with multiple changes. We are used to the current naming and it will be easer to reimplement and rename it in 3.0 this way, but I'm open to renaming it if it won't disturb other engineers even more.

I don't think anyone would care, hardly anyone would refer to this class outside of significant changes in the service layer.

Makes sense. I'd like to leave it as Metrics for now as it's at least another anchor which could be shown when you search for metrics and diagnostics store is a kind of another entity we'd introduce with this renaming. But I wouldn't resist if you insisted that Metrics is a bad option to leave in the codebase.

dmitrii-ubskii · 2024-05-13T12:09:12Z

common/diagnostics/Metrics.java

-        this.usage = new CurrentCounts();
-        this.userErrors = new UserErrorStatistics();
+    public void takeSnapshot() {
+        this.base.updateSinceTimestamp();


I thought the monitoring page was supposed to use the same data as Prometheus, with counters steadily ticking up?

common/diagnostics/Metrics.java

dmitrii-ubskii · 2024-05-13T12:31:43Z

server/TypeDBService.java

            UUID sessionID = byteStringAsUUID(request.getSessionId());
            SessionService sessionSvc = sessionServices.get(sessionID);
            if (sessionSvc == null) throw TypeDBException.of(SESSION_NOT_FOUND, sessionID);
+            databaseName = sessionSvc.session().database().name();


Not as careful here as in TransactionService, eh?

Well, it's inside try block and it would be a valid throw if the request didn't contain any of session, database, or its name. That's what I thought.

Hmm but now if sessionSvc != null, but database for some reason is, sessionSvc.close() won't be called. I don't know if that's even possible, but it is changing the behaviour in that case.

You're right, I didn't pay enough attention to the business logic here, my bad. And having a throw because of this line after everything would be dumb as well. Will be careful here as well then...

Let's actually add a databaseName() getter to the session service so that we can at least have one fewer null check in these places.

Good idea, done.

common/diagnostics/Diagnostics.java

common/diagnostics/Metrics.java

database/CoreDatabaseManager.java

dmitrii-ubskii

Left a couple comments to address, but otherwise LGTM!

common/diagnostics/Metrics.java

… the id generations

… scheduling. Add disabled reporting case

…hour whatever it takes

…en in JSON. Refactor

farost added the priority: high label May 7, 2024

farost added this to the 2.29.0 milestone May 7, 2024

farost requested a review from dmitrii-ubskii May 7, 2024 08:31

farost requested a review from haikalpribadi as a code owner May 7, 2024 08:31

farost commented May 7, 2024

View reviewed changes

server/common/Constants.java Show resolved Hide resolved

farost commented May 7, 2024

View reviewed changes

common/diagnostics/Metrics.java Show resolved Hide resolved

farost commented May 7, 2024

View reviewed changes

common/diagnostics/Metrics.java Outdated Show resolved Hide resolved

farost commented May 7, 2024

View reviewed changes

common/diagnostics/StatisticReporter.java Outdated Show resolved Hide resolved

farost commented May 7, 2024

View reviewed changes

common/diagnostics/StatisticReporter.java Outdated Show resolved Hide resolved

farost commented May 7, 2024

View reviewed changes

database/CoreDatabaseManager.java Outdated Show resolved Hide resolved

farost commented May 7, 2024

View reviewed changes

server/TransactionService.java Outdated Show resolved Hide resolved

dmitrii-ubskii requested changes May 13, 2024

View reviewed changes

common/diagnostics/Diagnostics.java Outdated Show resolved Hide resolved

common/diagnostics/Diagnostics.java Outdated Show resolved Hide resolved

common/diagnostics/Metrics.java Outdated Show resolved Hide resolved

database/CoreDatabaseManager.java Show resolved Hide resolved

dmitrii-ubskii self-assigned this May 13, 2024

vaticle-bot requested a review from dmitrii-ubskii May 13, 2024 15:05

farost modified the milestones: 2.29.0, 2.28.1 May 14, 2024

dmitrii-ubskii approved these changes May 17, 2024

View reviewed changes

common/diagnostics/Metrics.java Outdated Show resolved Hide resolved

common/diagnostics/Metrics.java Outdated Show resolved Hide resolved

farost added 11 commits May 20, 2024 09:09

Add deployment ID to the config, add config tests

314d9b3

Get deployment ID from server ID in core

cd96b32

Fix deployment id generation

51f2a0f

Refactor generation for cloud usage

12e5643

Refactor ID storage

e2d3fd6

Add deployment id to diagnostics. Add parameters integration test for…

adc26d3

… the id generations

Refactor, remove excessive prints

8dea25b

Refactor deployment id generation and add exceptions

07aa0bb

Add data resets

066eba6

Add hardware stats, refactor stats collection. Fix statistic reporter…

737376e

… scheduling. Add disabled reporting case

Add peak current counts calculating for transactions

086bbb9

farost added 25 commits May 20, 2024 09:12

Remove outdated code

ae66c17

Fix initial minute calc. Add warn in case of unsucessful report

65cd71f

Add explicit nullable to databaseName arguments

03fa3fb

Remove test log

8f0cb4a

Fix snapshots

6b00528

Fix deleted databases behavior

0745639

Refactor loops

590137e

Add hashcode TODO

a2c2529

Refactor diagnostics json file and version

007edcb

Fix build

5f1d141

Add database names to more requests. Make snapshot being taken every …

11821a9

…hour whatever it takes

Clean imports and leave fixme refs

371d546

Rename URI constants

ced75b0

Fix disabled reporting issues with CI and failed requests

33b31a1

Fix checkstyle

6b2c6f8

Remove excessive constants

eb30dac

Remove todo

bb79e47

Remove todos and clean prints in tests

c3b2e0c

Add uptime

c24043e

Fix namings and excessive calls after a review

83e561b

Update JSON_API_VERSION -> DIAGNOSTICS_JSON_SCHEMA_VERSION

541955a

Refactor method calls after a review

1cc9161

Rename public mayStartMonitoring method

9ed743f

Make monitoring show historic data instead of interval-based diffs ev…

53c69a7

…en in JSON. Refactor

Change warn to trace in case of failed diagnostics push

f7f8132

farost force-pushed the metrics-updates branch from ab5e51f to f7f8132 Compare May 20, 2024 08:13

farost added 2 commits May 20, 2024 09:16

Fix code style for pr

76d92fd

Rename CoreDatabaseManager's diagnostics methods

6961c64

flyingsilverfin approved these changes May 20, 2024

View reviewed changes

flyingsilverfin merged commit 7a3cf41 into vaticle:development May 20, 2024
10 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update diagnostics service to version 1 #7059

Update diagnostics service to version 1 #7059

farost commented May 7, 2024 •

edited

farost May 7, 2024

dmitrii-ubskii May 13, 2024

farost May 13, 2024 •

edited

farost May 13, 2024

dmitrii-ubskii left a comment

dmitrii-ubskii May 13, 2024

farost May 13, 2024

dmitrii-ubskii May 13, 2024

farost May 13, 2024

dmitrii-ubskii May 13, 2024

dmitrii-ubskii May 13, 2024

farost May 13, 2024

dmitrii-ubskii May 13, 2024

farost May 13, 2024

dmitrii-ubskii May 13, 2024

farost May 13, 2024

dmitrii-ubskii left a comment

Update diagnostics service to version 1 #7059

Update diagnostics service to version 1 #7059

Conversation

farost commented May 7, 2024 • edited

Usage and product changes

Implementation

Choose a reason for hiding this comment

Choose a reason for hiding this comment

farost May 13, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmitrii-ubskii left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmitrii-ubskii left a comment

Choose a reason for hiding this comment

farost commented May 7, 2024 •

edited

farost May 13, 2024 •

edited