Upload HammerDB to Debian/Ubuntu for official distro inclusion

[ottok]

According to https://github.com/TPC-Council/HammerDB/blob/master/LICENSE the HammerDB software is GPL-3. Yet it does not seem to be included in almost any Linux distro. I see only ArchLinux at https://repology.org/project/hammerdb/versions

Are there any plans to package and submit HammerDB for Debian/Ubuntu?

[sm-shaw]

Great suggestion. Yes HammerDB is GPL-3. Packaging for Linux distros was raised in this Issue #38 and now is a good time for the TPC-OSS subcommittee to discuss this. In particular as MariaDB has now been broken out from MySQL #234 this will be included from v4.2, so that would be a good release for inclusion to start from.

[memmertoIBM]

Relevant references:

• https://wiki.debian.org/HowToPackageForDebian
• https://packaging.ubuntu.com/html/

[sm-shaw]

Discussed at TPC-OSS Subcommittee meeting 11th May 2021.
Currently HammerDB is packaged in installer/zip packages for Linux and Windows as both formats support both operating systems for ease of maintenance.
HammerDB does not install dependencies as a user of one database may not need the libraries for all of the other supported databases.
Additional packaging for Linux distributions in .deb and .rpm format is certainly welcome but as with ArchLinux requires additional help to do the packaging and maintain in-line with future releases.
Note Issue #38 is a similar Issue for RPM packaging.

[ottok]

I was thinking of doing this packaging myself (I am a Debian Developer) but the docs at https://www.hammerdb.com/docs/ nor the README.md in the project itself do not have any section on howto compile HammerDB from sources. I don't see any CI system either from where I could look up how compilation is done and or what dependencies are used. The packaging in Arch is from 2015 and has no inline comments whatsoever and it seems to just re-use existing binaries from Sourceforge and does not actually compile the project.

[sm-shaw]

Yes, this was on the original Issues list when moving over to the TPC listed in #36 and #37 and is definitely something that should be adressed to bring in all source code for all of the packages used on both Linux and Windows.

Now, I agree it is not clear where all of the dependent packages are from as the contents of bin and lib in the Release files are compiled separately. This is from a current manual build process rather than intentional obfuscation. it is highly desirable to be able to build all of the dependencies on both Linux and Windows from source automatically, so as a starting point I will provide an overview here of what packages are used and where they are located.

Historically the dependent packages have been built manually for a major release and in some cases there has been no configured environment for a build for a particular OS (eg.tkblt on Windows) - meaning e.g. there is a lot of hardcoded Makefiles taken from elsewhere in the environment needing work to bring all of this into GitHub. This does not necessarily mean that we cannot start with Linux first and add Windows later (or vice versa) - up to now the thought process is that both should be added at the same time. (and of course it didn't get done until now).

As a starting point to explain the dependencies - HammerDB uses Tcl/Tk as its glue language and everything in the directories src, modules as well as some of the packages in lib are in "pure Tcl" meaning that they are identical on both Linux and Windows. (This does not mean that they are necessarily always interpreted as "procs" are compiled into bytecode at runtime.)

So at the highest level HammerDB gets the source for Tcl/Tk from here https://sourceforge.net/projects/tcl/files/Tcl/ and then configures and compiles both Tcl and Tk (v4.X uses Tcl/Tk 8.6.10) giving us the base bin lib and include directories and the tclsh8.6 and wish8.6 binaries in bin. tclsh8.6 is used for the CLI and Web Services, wish8.6 includes Tk and is used for the GUI.

gcc is used to compile for Linux and the MSVC command line on Windows.

The Tcl compilation also creates the following packages in lib. Everything is unmodified except for tdbcodbc which is used for SQL Server. There are 2 modifications that HammerDB makes to this package, to fix a bug with SQLULEN and to add an Execdirect command that is missing. Note that tpdbcmysql and tdbcpostgres are not used by HammerDB.

itcl4.2.0
libtcl8.6.so
libtclstub8.6.a
libtk8.6.so
libtkstub8.6.a
pkgconfig
sqlite3.30.1.2
tcl8
tcl8.6
tclConfig.sh
tclooConfig.sh
tdbc1.1.1
tdbcmysql1.1.1
tdbcodbc1.1.1
tdbcpostgres1.1.1
thread2.8.5
tk8.6
tkConfig.sh

So then added to lib are the following "pure tcl" packages that are not compiled.

awthemes-9.3.1
clearlooks
redis0.1

clearlooks is the graphical theme used in the non-scalable form. awthemes is the scalable themes using SVG graphics and redis0.1 is the interface to the hidden redis workload. (I tried the compiled redis interface but it was slower than the one included.)

Then we have the database interfaces from these locations. We extensively updated db2tcl and all of the changes were submitted by pull request to the link below. mariatcl has been created from mysqltcl. The rest are unmodified.

db2tcl2.0.0 https://github.com/memmertoIBM/db2tcl for Db2
mariatcl0.1 https://github.com/Jiang-Hua/mariatcl for MariaDB
mysqltcl-3.052 http://www.xdobry.de/mysqltcl/ for MySQL
Oratcl4.6 https://sourceforge.net/projects/oratcl/ for Oracle 
pgtcl2.1.1 http://pgtclng.sourceforge.net/ for PostgreSQL
tdbcodbc1.1.1 included with Tcl but updated as above for SQL Server

All of these database interfaces require the client libraries of their respective database at compile time (except Oratcl) and at runtime. At runtime the librarycheck command checks if the dependencies are available for the database extensions to be loaded.

This leaves the following additional compiled extensions:

expect5.45.4 https://sourceforge.net/projects/expect/  - used for the CLI command prompt on Linux only. 
tcltls1.7.20 https://core.tcl-lang.org/tcltls/index - binding to OpenSSL currently for HTTP testing only on Linux only.
tkblt3.2 https://github.com/wjoye/tkblt - the BLT toolkit used for the graph for graphical metrics (currently Oracle and PostgreSQL)
tksvg0.5 https://github.com/oehhar/tksvg - Scalable graphics package to enable SVG graphics to be used. It is expected that this will be included with mainline Tcl. 

On windows there is an additional compiled extension:
Twapi-4.3.7 https://twapi.magicsplat.com/ - used throughout HammerDB on Windows

and for completeness on Windows the Microsoft Visual C runtime libraries are added to the bin directory.

It is definitely desirable to bring this source into the project and I will initiate the discussion on how this can be done so all of the dependencies can be compiled together. This should then make packaging more viable.

[ottok]

gcc is used to compile for Linux

How is gcc used? I don't see any Makefile in the project, so can't use the default make && make install that gcc projects typically use, nor due i see references to CMake or Ninja or other build systems.

[sm-shaw]

Yes that is correct so a build system such as CMake or Ninja has not been set up yet and yes this should be done. So at the moment it is a manual process. For example the first step of building is to get Tcl/Tk from here https://sourceforge.net/projects/tcl/files/Tcl/ and running configure generates the first Makefile here.

/tmp/tcl8.6.10/unix$ ls -1
aclocal.m4
config.log
config.status
configure
configure.in
dltest
installManPage
install-sh
ldAix
Makefile
Makefile.in
README
tclAppInit.c
tclConfig.h.in
tclConfig.sh
tclConfig.sh.in
tclLoadAix.c
...

So as a summary list what is currently manually compiled for Linux for version 4.2/4.3, and the source locations are as follows:

Tcl/Tk 8.6.10 https://sourceforge.net/projects/tcl/files/Tcl/
tdbcodbc1.1.1 included with Tcl but updated as above for SQL Server
---
db2tcl2.0.0 https://github.com/memmertoIBM/db2tcl 
mariatcl0.1 https://github.com/Jiang-Hua/mariatcl 
mysqltcl-3.052 http://www.xdobry.de/mysqltcl/ 
Oratcl4.6 https://sourceforge.net/projects/oratcl/ 
pgtcl2.1.1 http://pgtclng.sourceforge.net/ 
---
expect5.45.4 https://sourceforge.net/projects/expect
tcltls1.7.20 https://core.tcl-lang.org/tcltls/index
tkblt3.2 https://github.com/wjoye/tkblt
tksvg0.5 https://github.com/oehhar/tksvg

So the next step before packaging is for us to pull all the current source packages into the GitHub repository and set up a build system, so everything can be built at once. At the moment this is not the case, so the packages are manually compiled and the binaries made available in the Release files.

[sm-shaw]

Reference to a research paper CMake-Based Cross Platform Build System for Tcl/Tk showing similar steps to what is needed for #36 as a first step before packaging.

[ottok]

There is no (or little) need to waste time manually downloading tk/tcl libraries from Sourceforge when most of them are already in Debian. It is simply enough to define e.g. tcl-tls, tkblt-dev etc as build dependencies and get building.

Some examples:

• https://packages.debian.org/sid/amd64/tkblt-dev/filelist
• https://packages.debian.org/sid/amd64/tcl-tls/filelist
• https://packages.debian.org/sid/amd64/tcl-expect/filelist
• https://packages.debian.org/bookworm/all/tcl-awthemes/filelist
• https://packages.debian.org/sid/amd64/tk8.6-dev/filelist
• https://packages.debian.org/sid/amd64/tcl8.6-dev/filelist
• https://packages.debian.org/sid/amd64/tcl-itcl4/filelist
• https://packages.debian.org/sid/amd64/tcl8.6-tdbc/filelist
• https://packages.debian.org/sid/amd64/mysqltcl/filelist
• https://packages.debian.org/sid/amd64/libpgtcl/filelist

[sm-shaw]

Some of the packages such as tdbc have been modified, also there is oratcl and db2tcl with most of the database interfaces requiring the database libraries to be installed to be compiled.
Have also been looking at BAWT http://www.bawt.tcl3d.org/ (Build Automation With Tcl) is a configurable framework written in Tcl for building C/C++ based software libraries from source code without user interaction - This may be a solution to compile from source for both Linux and Windows.

[dannysauer]

I'm going to put in a small plug for the SUSE's Open Build Service here. It's fairly easy to define a project which runs your build steps and emits packages in a repository. I'm a big fan of that particular system because you get one free place which builds packages for Debian-derived systems, quite a few RPM-based systems, and Arch (and appropriate repos for each). Easy to hook into Github so your project gets rebuilt upon release as well. Using kiwi, one can also create AMIs, Docker containers, and whatever from the same job (they provide a docker registry), which is of potential interest in #107.

It's always a surprise to me that more people don't use that service. It's really quite nice, after a minor learning curve. :)

I'd offer to help, but like everyone else I'm not sure how much time I can devote. If I end up integrating hammerDB into an infrastructure CI system like I'd like to do long term, I'll want a package - so maybe I'll get on it at that point (if someone else hasn't done it before that).

[sm-shaw]

Hi, Yes, so this hasn't been done as yet but is definitely a key topic at the moment. At the moment the binaries in bin and lib are compiled manually, this is fast becoming unsustainable, so the main task is to bring all the dependencies into the project and enable an automated build. There is some complexity to this as the project supports both Windows and Linux so needs to be done for both. Nevertheless, the plan is to look into this ASAP.

[sm-shaw]

BAWT is now working as a solution to build all packages from source with one command.
So far Linux is enabled with Windows and final packaging of the full distribution yet to do.
Nevertheless, as a POC this is the best approach to automate the build from source for both Linux and Windows.

/Build-Linux.sh x64 Setup/HammerDB.bawt update
...
12:27:00 > Summary
           Setup file     : /opt/Bawt/Bawt-2.1.0/Setup/HammerDB.bawt
           Build directory: /opt/Bawt/BawtBuild/Linux/x64/Release/Build
           Architecture   : x64
           Compilers      : gcc
           Global stages  : Finalize
           #  : Library Name         Version    Build time      Stages
           ----------------------------------------------------------------------
             1: Tcl                  8.6.12     1.47 minutes    Clean Extract Configure Compile Distribute
             2: Tk                   8.6.12     0.34 minutes    Clean Extract Configure Compile Distribute
             3: awthemes             10.4.0     0.00 minutes    Clean Extract Configure Compile Distribute
             4: clearlooks           1.0        0.00 minutes    Clean Extract Configure Compile Distribute
             5: db2tcl               2.0.1      0.09 minutes    Clean Extract Configure Compile Distribute
             6: expect               5.45.4     0.12 minutes    Clean Extract Configure Compile Distribute
             7: libressl             2.6.4      0.94 minutes    Clean Extract Configure Compile Distribute
             8: mariatcl             0.1        0.04 minutes    Clean Extract Configure Compile Distribute
             9: mysqltcl             3.052      0.04 minutes    Clean Extract Configure Compile Distribute
            10: oratcl               4.6        0.05 minutes    Clean Extract Configure Compile Distribute
            11: redis                0.1        0.00 minutes    Clean Extract Configure Compile Distribute
            12: tcltls               1.7.22     0.40 minutes    Clean Extract Configure Compile Distribute
            13: tkblt                3.2.23     0.19 minutes    Clean Extract Configure Compile Distribute
            14: tksvg                0.6        0.05 minutes    Clean Extract Configure Compile Distribute
            15: Pgtcl                2.8.0      0.06 minutes    Clean Extract Configure Compile Distribute
           ----------------------------------------------------------------------
           Total: 3.78 minutes

[sm-shaw]

Pull Request #323 adds build automation for Linux, with instructions on how to run it. Windows will be done subsequently.