use URI statistics

[lxxxxxxk]

Hello,
I am currently starting in this way on version 2.5.3 and would like to use the URI statistics feature:
java -jar -Dpinpoint.zookeeper.address=localhost pinpoint-web-starter-boot-2.5.3.jar java -jar -Dpinpoint.zookeeper.address=localhost pinpoint-collector-starter-boot-2.5.3.jar

The service is started with:

java -javaagent:/root/pinpoint-agent-2.5.3/pinpoint-bootstrap.jar \ -Dpinpoint.agentId=xxxx \ -Dpinpoint.applicationName=xxxx \ -Dprofiler.transport.grpc.collector.ip=xxx \ -Dprofiler.collector.ip=xxx \ -jar xxxx.jar \

However, there's no data in the URI statistics, even though initialization and everything in Pinot is done, and the table structures are there, but no data is coming in. Can you advise if there's something wrong with my setup or if I need to start something extra? Thank you.

[ga-ram]

Any logs on pinpoint agent or pinpoint collector would be very helpful.

[lxxxxxxk]

Any logs on pinpoint agent or pinpoint collector would be very helpful.

Are there any keywords? Look, they are all related to INFO level logs.

[ga-ram]

I wanted to check if connection between agent and collector is well established without any errors, and if there's any successful transactions between them. Pinpoint configuration properties are also being logged so you may check that too.

[lxxxxxxk]

pinpoint-collector-starter-boot-2.5.3.jar

Thank you so much
I’m not sure if it’s because I didn’t recompile the package but used the pinpoint-collector-starter-boot-2.5.3.jar package downloaded from git.
Does it need to be rebuilt and compiled?

[ga-ram]

You don't need to compile the code. It should work with the distributed jar file.

[ga-ram]

Did you set pinpoint.metric.kafka.bootstrap.servers property for kafka which connects pinpoint collector and pinot?

[lxxxxxxk]

Did you set pinpoint.metric.kafka.bootstrap.servers property for kafka which connects pinpoint collector and pinot?

Where is pinot's kafka setting? Currently, my kafka is on this machine.

/usr/lib/jvm/jdk-11.0.20/bin/java -jar -Dpinpoint.zookeeper.address=localhost -Dspring.config.additional-location=/root/pinpoint/collector.properties pinpoint-collector-starter-boot-2.5.3.jar
这是我当前collector的启动方式 我在文件里指定了kafka地址 不知道这样是否正确
spring.profiles.active=release
pinpoint.zookeeper.address=localhost

pinpoint.pinot.jdbc.url=jdbc:pinot://localhost:9000
pinpoint.pinot.jdbc.username=--release-username--
pinpoint.pinot.jdbc.password=--release-password--

pinpoint.metric.kafka.bootstrap.servers=localhost:9092

pinpoint.collector.type=ALL

collector.stat.uri=true

kafka.uri.topic=url-stat

pinpoint.pinot.tenantId=DefaultTenant

[lxxxxxxk]

I checked the collector log and the Kafka related information looks normal.
04-26 15:57:36.036 [pcStat-Worker-6] INFO o.a.k.c.p.ProducerConfig : ProducerConfig values:
acks = 1
batch.size = 16384
bootstrap.servers = [localhost:9092]
buffer.memory = 33554432
client.dns.lookup = use_all_dns_ips
client.id = producer-1
compression.type = zstd
connections.max.idle.ms = 540000
delivery.timeout.ms = 120000
enable.idempotence = false
interceptor.classes = []
key.serializer = class org.apache.kafka.common.serialization.StringSerializer
linger.ms = 0
max.block.ms = 60000
max.in.flight.requests.per.connection = 5
max.request.size = 1048576
metadata.max.age.ms = 300000
metadata.max.idle.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
receive.buffer.bytes = 32768
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retries = 2147483647
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000

I’m not sure if it’s because of writing localhost or because kafka is not configured in Pinot.

[lxxxxxxk]

Did you set pinpoint.metric.kafka.bootstrap.servers property for kafka which connects pinpoint collector and pinot?

Could you please tell me which IP address should be used for kafka? Is it ok to write localhost?
I feel it is a problem with pinpot and kafka. I am not sure about the kafka configuration.

If not set, it uses the value for "listeners".

advertised.listeners=PLAINTEXT://localhost:9092

Do you need to write the external IP address of the host here?
thanks

[lxxxxxxk]

Did you set pinpoint.metric.kafka.bootstrap.servers property for kafka which connects pinpoint collector and pinot?

Thank you very much. It seems to be a kafka configuration problem. Now there is data in pinot, but there is still no data in web url_statisic.
The data inside and outside the table is

mdm-test-1 1.5 MDM-test-1 3 0 0 0 0 0 0 0 0 0 2150
The agent connection method is
-Dpinpoint.agentId=mdm-test-1
-Dpinpoint.applicationName=MDM-test-1 \

But I didn’t check it on the web
What could be the reason for this?

[ga-ram]

#10964 (comment)

pinpoint.pinot.jdbc.username=--release-username--
pinpoint.pinot.jdbc.password=--release-password--

you need to set pinpoint.pinot.jdbc.username and pinpoint.pinot.jdbc.password to your pinot username & password.

[lxxxxxxk]

#10964 (comment)

pinpoint.pinot.jdbc.username=--release-username--
pinpoint.pinot.jdbc.password=--release-password--

you need to set pinpoint.pinot.jdbc.username and pinpoint.pinot.jdbc.password to your pinot username & password.

OK, thank you. Now you can write without setting an account and password.
Currently, pinot has data, but it is not displayed on the web.
Can you take a look at it for me? Thanks

[ga-ram]

did you try with username and password configuration and it still didn't work?
was there any log messages when you tried to access pinot data?

[lxxxxxxk]

did you try with username and password configuration and it still didn't work? was there any log messages when you tried to access pinot data?

I haven't tried this yet
Currently my pinpot startup method is
nohup ./bin/pinot-admin.sh QuickStart -type batch > pinot.log 2>&1 &
But I found that after restarting, the historical data is gone. Is my startup method wrong?

[lxxxxxxk]

"I found the problem. The query is going with WHERE tenantId = 'pinpoint', but in my database, the default is tenantId = 'DefaultTenant'.

Looking at the table structure, it shows:

{ "OFFLINE": { "tableName": "uriStat_OFFLINE", "tableType": "OFFLINE", "segmentsConfig": { "timeType": "MILLISECONDS", "schemaName": "uriStat", "replication": "3", "retentionTimeUnit": "DAYS", "retentionTimeValue": "56", "timeColumnName": "timestamp", "minimizeDataMovement": false }, "tenants": { "broker": "DefaultTenant", "server": "DefaultTenant" }, "tableIndexConfig": { "noDictionaryColumns": ["tot0", "tot1", "tot2", "tot3", "tot4", "tot5", "tot6", "tot7", "fail0", "fail1", "fail2", "fail3", "fail4", "fail5", "fail6", "fail7"], "rangeIndexVersion": 2, "autoGeneratedInvertedIndex": false, "createInvertedIndexDuringSegmentGeneration": false, "sortedColumn": ["applicationName"], "bloomFilterColumns": ["tenantId", "serviceName", "applicationName", "agentId"], "loadMode": "MMAP", "enableDefaultStarTree": false, "aggregateMetrics": false, "enableDynamic StarTreeCreation": false, "nullHandlingEnabled": true, "optimizeDictionary": false, "optimizeDictionaryForMetrics": false, "noDictionarySizeRatioThreshold": 0.0 }, "metadata": {}, "ingestionConfig": { "segmentTimeValueCheck": true, "batchIngestionConfig": { "segmentIngestionType": "APPEND", "segmentIngestionFrequency": "DAILY", "consistentDataPush": false }, "continueOnError": false, "rowTimeValueCheck": false }, "isDimTable": false }, "REALTIME": { "tableName": "uriStat_REALTIME", "tableType": "REALTIME", "segmentsConfig": { "timeType": "MILLISECONDS", "schemaName": "uriStat", "retentionTimeUnit": "DAYS", "retentionTimeValue": "7", "replicasPerPartition": "1", "timeColumnName": "timestamp", "minimizeDataMovement": false }, "tenants": { "broker": "DefaultTenant", "server": "DefaultTenant" }, "tableIndexConfig": { "noDictionaryColumns": ["tot0", "tot1", "tot2", "tot3", "tot4", "tot5", "tot6", "tot7", "fail0", "fail1", "fail2", "fail3", "fail4", "fail5", "fail6", "fail7"], "rangeIndexVersion": 2, "autoGeneratedInvertedIndex": false, "createInvertedIndex DuringSegmentGeneration": false, "sortedColumn": ["applicationName"], "bloomFilterColumns": ["tenantId", "serviceName", "applicationName", "agentId"], "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.topic.name": "url-stat", "stream.kafka.decoder.class name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "stream.kafka.consumer.factory.class name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.broker.list": "localhost:9092", "realtime.segment.flush.threshold.rows": "0", "realtime.segment.flush.threshold.time": "24h", "realtime.segment.flush.threshold.segment.size": "64M", "stream.kafka.consumer.prop.auto.offset.reset": "smallest" }, "enableDefaultStarTree": false, "aggregateMetrics": false, "enableDynamicStarTreeCreation": false, "nullHandlingEnabled": true, "optimizeDictionary": false, "optimizeDictionaryForMetrics": false, "noDictionarySizeRatioThreshold": 0.0 }, "metadata": {}, "task": { "taskTypeConfigsMap": { "RealtimeToOfflineSegmentsTask": { "bucketTimePeriod": "4h", "bufferTimePeriod": "12h", "schedule": "0 0 * * * ?", "maxNumRecordsPerSegment": "1000000" } } }, "isDimTable": false } }

How do I update this? Can I set the default to 'pinpoint'?"

[lxxxxxxk]

"I found the problem. The query is going with WHERE tenantId = 'pinpoint', but in my database, the default is tenantId = 'DefaultTenant'.

Looking at the table structure, it shows:

{ "OFFLINE": { "tableName": "uriStat_OFFLINE", "tableType": "OFFLINE", "segmentsConfig": { "timeType": "MILLISECONDS", "schemaName": "uriStat", "replication": "3", "retentionTimeUnit": "DAYS", "retentionTimeValue": "56", "timeColumnName": "timestamp", "minimizeDataMovement": false }, "tenants": { "broker": "DefaultTenant", "server": "DefaultTenant" }, "tableIndexConfig": { "noDictionaryColumns": ["tot0", "tot1", "tot2", "tot3", "tot4", "tot5", "tot6", "tot7", "fail0", "fail1", "fail2", "fail3", "fail4", "fail5", "fail6", "fail7"], "rangeIndexVersion": 2, "autoGeneratedInvertedIndex": false, "createInvertedIndexDuringSegmentGeneration": false, "sortedColumn": ["applicationName"], "bloomFilterColumns": ["tenantId", "serviceName", "applicationName", "agentId"], "loadMode": "MMAP", "enableDefaultStarTree": false, "aggregateMetrics": false, "enableDynamic StarTreeCreation": false, "nullHandlingEnabled": true, "optimizeDictionary": false, "optimizeDictionaryForMetrics": false, "noDictionarySizeRatioThreshold": 0.0 }, "metadata": {}, "ingestionConfig": { "segmentTimeValueCheck": true, "batchIngestionConfig": { "segmentIngestionType": "APPEND", "segmentIngestionFrequency": "DAILY", "consistentDataPush": false }, "continueOnError": false, "rowTimeValueCheck": false }, "isDimTable": false }, "REALTIME": { "tableName": "uriStat_REALTIME", "tableType": "REALTIME", "segmentsConfig": { "timeType": "MILLISECONDS", "schemaName": "uriStat", "retentionTimeUnit": "DAYS", "retentionTimeValue": "7", "replicasPerPartition": "1", "timeColumnName": "timestamp", "minimizeDataMovement": false }, "tenants": { "broker": "DefaultTenant", "server": "DefaultTenant" }, "tableIndexConfig": { "noDictionaryColumns": ["tot0", "tot1", "tot2", "tot3", "tot4", "tot5", "tot6", "tot7", "fail0", "fail1", "fail2", "fail3", "fail4", "fail5", "fail6", "fail7"], "rangeIndexVersion": 2, "autoGeneratedInvertedIndex": false, "createInvertedIndex DuringSegmentGeneration": false, "sortedColumn": ["applicationName"], "bloomFilterColumns": ["tenantId", "serviceName", "applicationName", "agentId"], "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.topic.name": "url-stat", "stream.kafka.decoder.class name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "stream.kafka.consumer.factory.class name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.broker.list": "localhost:9092", "realtime.segment.flush.threshold.rows": "0", "realtime.segment.flush.threshold.time": "24h", "realtime.segment.flush.threshold.segment.size": "64M", "stream.kafka.consumer.prop.auto.offset.reset": "smallest" }, "enableDefaultStarTree": false, "aggregateMetrics": false, "enableDynamicStarTreeCreation": false, "nullHandlingEnabled": true, "optimizeDictionary": false, "optimizeDictionaryForMetrics": false, "noDictionarySizeRatioThreshold": 0.0 }, "metadata": {}, "task": { "taskTypeConfigsMap": { "RealtimeToOfflineSegmentsTask": { "bucketTimePeriod": "4h", "bufferTimePeriod": "12h", "schedule": "0 0 * * * ?", "maxNumRecordsPerSegment": "1000000" } } }, "isDimTable": false } }

How do I update this? Can I set the default to 'pinpoint'?"

I have solved this problem by configuring when starting the web
pinpoint.pinot.tenantId=DefaultTenant

[ga-ram]

DefaultTenant is used for Pinot cluster, and tenantId in WHERE tenantId = 'pinpoint' is the column name in uri stat table in Pinot (within DefaultTenant) . So I strongly doubt that changing pinpoint.pinot.tenantId to DefaultTenant will solve your problem in which pinpoint web doesn't show the data already successfully stored in your pinot database.
If the data is being stored in your pinot database but pinpoint web doesn't show any, there should be error logs for pinpoint web..