Difference TPC-C benchmark score between version 4.8 and 4.10

[mitsu-ko]

Hi,

Thank you for your quick fix the other day. I use HammerDB in PostgreSQL on cloud Instance many time. Recently, I noticed that HammerDB 4.10 TPC-C benchmark score indicates better score than version 4.8. I read release note, seems to nothing about benchmark performance improvements. However, other little improvement patches are pulled many times in latest version.

Is it okay to recognize that the performance has actually improved?　And, in the benchmark test of version 4.8, it was occasionally observed that benchmark score decreased when EBS performance was increased. Is it correct to assume that these have also been revised? please tell me developer's feeling.

Best regards,

[sm-shaw]

You will need to provide more details on the comparative test configuration and results that you observe. There are some minor changes, but nothing that is expected to bring performance changes in PostgreSQL. Only by looking at the test setup would it be possible to determine the cause for different performance.

[mitsu-ko]

Hi sm-shaw-san,

Thank you for your always quick response. I was relieved to hear that the main developer said there was no change in performance. This problem occurs when trying to increase the performance of just table data, not WAL. I guess that it might seem to be occurred by TPC-C benchmark specification which is very complex. Or I guess this is kernel performance problem. I'm very sorry about I can not tell you about detail test settings, because it's my patents pending. It will be opened in academic paper in the future.

If there is anything else, please let me consult with you. Thank you.

[sm-shaw]

The important aspect of database benchmarking is that you don't have to guess, guessing a kernel performance problem at the outset is not a good methodology. Always start with the database metrics, HammerDB includes a graphical metrics options which is the easiest way. Look in the table event and see what you are waiting on.

https://www.hammerdb.com/docs/ch07s05.html / https://www.hammerdb.com/blog/uncategorized/hammerdb-v4-3-new-features-pt1-graphical-metrics-for-postgresql

For Linux performance troubleshooting below the database, I would highly recommend LinuxKI https://github.com/HewlettPackard/LinuxKI

[mitsu-ko]

Hi sm-shaw san,

Thank you for various information. Very helpful for me. And sorry for my delay self introduction. I had developed PostgreSQL statical tools which is called for pg_statsinfo and pg_stat_reporter for 10 years ago. But I'm currently working at database research task.
https://www.slideshare.net/mitsukondo/bernd-27905220

I think that traditional database stats tools is very useful in general database tuning. However, innovative research tasks have to be needed unknow stats in kernel and database. It seems to crazy but I need it. If my paper is accepted in academic conference, I can show you my good idea for database. HammerDB is very useful tools in my work. I look forward to your continued support and best benchmark tool.

Best regards,

[sm-shaw]

I do very much agree. In advanced scenarios, both OS and hardware are essential to database performance and different databases can perform differently on different platforms. As noted LinuxKI https://github.com/HewlettPackard/LinuxKI is very good for bridging the gap between the database and the OS in Linux environments.

Of great importance is how you test. Take the (real world) example below. Anyone who tested these systems at only say 8, 30, 60 or 120 Virtual Users would all reach different conclusions. You have to test the system across all levels of utilisation to be able to fully analyse system performance and identify the point of peak performance as well as response times.