BIGTOP-4106. Upgrade GPDB to 7.1.0 with Python 3 enabled.

[iwasakims]

https://issues.apache.org/jira/browse/BIGTOP-4106

• Since we are dropping python2 from toolchain and bigtop-deploy, packaging stuff of GPDB must be updated to be assuming python 3.
• We are dropping package dependency on python in order to make users freely choose python on their deployment.
• Upgrading to gpdb 7 is required for dropping dependency on PyGreSQL.
• Since MAKELEVEL matters on generating errcodes.h, reseting the level by MAKELEVEL=0 is needed on invoking sub-make from top-level make of deb packaging.
• --backend_output was removed.
• checkpoint_segments was replaced with min_wal_size and max_wal_size

[iwasakims]

Packaging was succeeded on ubuntu-22.04 and openeuler-22.03. Not yet running smoke-tests.

[iwasakims]

looking into the cause of test error.

> Task :bigtop-tests:smoke-tests:gpdb:test

TestGpdb > testGpdb STANDARD_ERROR
    24/05/16 10:16:11 INFO lang.Object: Test GPDB column query

TestGpdb > testGpdb STANDARD_OUT
    Failed command: runuser -l gpadmin /home/gpadmin/test-master-db.sh
        error code: 127
        stdout: []
        stderr: [-bash: /home/gpadmin/test-master-db.sh: No such file or directory]

Gradle Test Executor 2 finished executing tests.

[iwasakims]

bigtop-deploy must be updated too.

Error: Could not find command '/usr/bin/python2'
Error: /Stage[main]/Gpdb::Common::Install_packages/Exec[install_pip]/returns: change from 'notrun' to ['0'] failed: Could not find command '/usr/bin/python2'
Notice: /Stage[main]/Gpdb::Common::Install_packages/Exec[install_python_packages]: Dependency Exec[install_pip] has failures: true
Warning: /Stage[main]/Gpdb::Common::Install_packages/Exec[install_python_packages]: Skipping because of failed dependencies
Warning: /Stage[main]/Gpdb::Common::Prepare_db_dirs/File[/home/gpadmin/init-db.sh]: Skipping because of failed dependencies
Warning: /Stage[main]/Gpdb::Common::Prepare_db_dirs/File[/home/gpadmin/stop-db.sh]: Skipping because of failed dependencies
Warning: /Stage[main]/Gpdb::Common::Prepare_db_dirs/Exec[create_master_db/data_gp/master/gpseg-1]: Skipping because of failed dependencies
Warning: /Stage[main]/Gpdb::Common::Prepare_db_dirs/File_line[allow all host connection/data_gp/master/gpseg-1]: Skipping because of failed dependencies
Warning: /Stage[main]/Gpdb::Common::Prepare_db_dirs/File_line[add 1 conf/data_gp/master/gpseg-1]: Skipping because of failed dependencies
Warning: /Stage[main]/Gpdb::Common::Prepare_db_dirs/File_line[add 2 conf/data_gp/master/gpseg-1]: Skipping because of failed dependencies
Warning: /Stage[main]/Gpdb::Common::Prepare_db_dirs/File_line[add 3 conf/data_gp/master/gpseg-1]: Skipping because of failed dependencies
Warning: /Stage[main]/Gpdb::Common::Prepare_db_dirs/File_line[add 4 conf/data_gp/master/gpseg-1]: Skipping because of failed dependencies
Warning: /Stage[main]/Gpdb::Common::Prepare_db_dirs/File[/data_gp/master/gpseg-1/gp_dbid]: Skipping because of failed dependencies
Warning: /Stage[main]/Gpdb::Common::Prepare_db_dirs/File[/data_gp/master/gpseg-1/gpssh.conf]: Skipping because of failed dependencies
Warning: /Stage[main]/Gpdb::Common::Prepare_db_dirs/Exec[create_master_db/data_gp/primary/gpseg0]: Skipping because of failed dependencies
Warning: /Stage[main]/Gpdb::Common::Prepare_db_dirs/File_line[allow all host connection/data_gp/primary/gpseg0]: Skipping because of failed dependencies
Warning: /Stage[main]/Gpdb::Common::Prepare_db_dirs/File_line[add 1 conf/data_gp/primary/gpseg0]: Skipping because of failed dependencies
Warning: /Stage[main]/Gpdb::Common::Prepare_db_dirs/File_line[add 2 conf/data_gp/primary/gpseg0]: Skipping because of failed dependencies
Warning: /Stage[main]/Gpdb::Common::Prepare_db_dirs/File_line[add 3 conf/data_gp/primary/gpseg0]: Skipping because of failed dependencies
Warning: /Stage[main]/Gpdb::Common::Prepare_db_dirs/File_line[add 4 conf/data_gp/primary/gpseg0]: Skipping because of failed dependencies
Warning: /Stage[main]/Gpdb::Common::Prepare_db_dirs/File[/data_gp/primary/gpseg0/gp_dbid]: Skipping because of failed dependencies
Warning: /Stage[main]/Gpdb::Common::Prepare_db_dirs/File[/data_gp/primary/gpseg0/gpssh.conf]: Skipping because of failed dependencies
Warning: /Stage[main]/Gpdb::Common::Start_master_node/File[/home/gpadmin/start-master-db-in-admin-mode.sh]: Skipping because of failed dependencies
Warning: /Stage[main]/Gpdb::Common::Start_master_node/File[/home/gpadmin/test-master-db.sh]: Skipping because of failed dependencies
...

[iwasakims]

smoke-tests of gpdb passed on ubuntu-22.04 (amd64) and openeuler-22.03 (aarch64)

[JiaLiangC]

https://github.com/greenplum-db/gpdb/blob/main/python-dependencies.txt It seems that GPDB relies on psutil, pyyaml, psycopg2. So, do we need to install these Python dependencies during deployment?

[iwasakims]

So, do we need to install these Python dependencies during deployment?

It is user's responsibility to set up Python environment now. Since the current coverage of the smoke-tests does not require the python modules, I'm not going to update bigtop-deploy here. Feel free to file another jira to add tests and update manifests if you want.

[iwasakims]

packaging and smoke-tests passed on centos-7, rockylinu-8 and debian-11. (x86_64)

[iwasakims]

Packaging on fedora-38 fails due to lack of Perl module?

Can't locate FindBin.pm in @INC (you may need to install the FindBin module) (@INC contains: ../../src/tools /usr/local/lib64/perl5/5.36 /usr/local/share/perl5/5.36 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5) at ../../src/tools/gen_keywordlist.pl line 35.
BEGIN failed--compilation aborted at ../../src/tools/gen_keywordlist.pl line 35.
make: *** [Makefile:131: kwlist_d.h] Error 2
make: *** [Makefile:45: all-common-recurse] Error 2
make: *** [GNUmakefile:11: all-src-recurse] Error 2
error: Bad exit status from /var/tmp/rpm-tmp.EPzLvR (%build)
    source_date_epoch_from_changelog set but %changelog is missing
    Bad exit status from /var/tmp/rpm-tmp.EPzLvR (%build)

[iwasakims]

packaging on fedora-38 succeeded but deployment failed. The cause seems to be hidden bug of the manifest. Puppet 8 on Fedora 38 looks stricter than older versions.

Notice: Scope(Class[Node_with_components]): Roles to deploy: [bigtop-utils, gpdb-master, gpdb-segment]
Notice: Scope(Class[Gpdb::Common::Prepare_db_dirs]): 81a5e2ba286b.bigtop.apache.org
Notice: Scope(Class[Gpdb::Common::Prepare_db_dirs]): 81a5e2ba286b.bigtop.apache.org
Notice: Scope(Class[Gpdb::Common::Start_master_node]): must make admin
Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Unknown variable: 'dbid'. (file: /bigtop-home/bigtop-deploy/puppet/modules/gpdb/manifests/init.pp, line: 229, column: 38) (file: /bigtop-home/bigtop-deploy/puppet/manifests/cluster.pp, line: 187) on node 81a5e2ba286b.bigtop.apache.org

[iwasakims]

smoke-tests passed on fedora-38.

[sekikn]

+1, tested on CentOS 7. Thanks @iwasakims.

$ curl -sL https://github.com/apache/bigtop/pull/1269.diff | git apply
$ sudo bigtop_toolchain/bin/puppetize.sh
$ ./gradlew toolchain
$ ./gradlew allclean gpdb-pkg repo -Dbuildwithdeps=true

...

BUILD SUCCESSFUL in 2m 4s
47 actionable tasks: 47 executed
$ cd docker/bigtop-puppet
$ ./build.sh trunk-centos-7
$ cd -                                                                                                                                                         
$ cd docker/bigtop-slaves
$ ./build.sh trunk-centos-7
$ cd -                                                                                                                                                         
$ cd provisioner/docker
$ ./docker-hadoop.sh -d -dcp -C config_centos-7.yaml -F docker-compose-cgroupv2.yml -L -G -k gpdb -s gpdb -c 1

...

:bigtop-tests:smoke-tests:gpdb:test (Thread[Execution worker for ':' Thread 2,5,main]) completed. Took 0.691 secs.

BUILD SUCCESSFUL in 21s

[iwasakims]

I merged this. Thanks, @sekikn.

[MacChen01]

We verified this PR, but compile failed in rockylinux 9
https://issues.apache.org/jira/browse/BIGTOP-4110

@iwasakims @sekikn

[iwasakims]

@MacChen01 looks like rokcylinux-9 needs the same fix with fedora-38. Since rokcylinux-9 is out of scope of Bigtop 3.3.0 release, I'll look into it later.

[MacChen01]

@iwasakims
yes, the code needs to be modified like fedora-38:
4289923