Some confusing issues with pid_one_command: /bin/systemd and postgresql service

[Pursu1tOfHapp1ness]

🗣️ Foreword

Hello!
In my case, I'd like to install sonarqube+postgresql as services with Chef instruments (chef-client and cookbooks). I started to develop such decision and faced obstacles with restarting/starting postgresql service. So, try to answer this question to you.

👻 Brief Description

So, after reading and understanding what kind of resources I need, start to create custom recipe, that it is:

postgresql_server_install 'Install PostgreSQL Server' do
  version node['e3s']['postgresql']['version']
  action  :install
end

service 'postgresql' do
  action :restart
end

postgresql_user 'Create defined user for Sonarqube' do
  create_user node['e3s']['sonarqube']['jdbc']['username']
  createdb    true
  password    node['e3s']['sonarqube']['jdbc']['password']
end

postgresql_database 'Create database for Sonarqube' do
  database  node['e3s']['sonarqube']['jdbc']['database_name']
  owner     node['e3s']['sonarqube']['jdbc']['username']
end

Variables:

default['e3s']['postgresql']['version'] = '12'
default['e3s']['sonarqube']['jdbc']['username'] = 'sonar'
default['e3s']['sonarqube']['jdbc']['password'] = 'sonar'
default['e3s']['sonarqube']['jdbc']['database_name'] = 'sonar'

.kitchen.yml:

driver:
  name: dokken
  chef_version: 14
  privileged: true

transport:
  name: dokken

provisioner:
  name: dokken
  data_bags_path: 'test/fixtures/data_bags'
  environments_path: 'test/fixtures/environments'
  nodes_path: 'test/fixtures/nodes'

verifier:
  name: inspec

platforms:
  - name: debian-10
    driver:
      image: dokken/debian-10
      pid_one_command: /bin/systemd
      intermediate_instructions:
        - RUN /usr/bin/apt-get update

suites:
  - name: sonarqube
    driver:
      ports:
        - '9000:9000'
    run_list:
      - recipe[e3s-sonarqube::default]
    verifier:
      inspec_tests:
        - test/integration/default

So, after kitchen is creating and start to converge, in logs I see:

* postgresql_server_install[Install PostgreSQL Server] action install
    * postgresql_client_install[Install PostgreSQL Client] action install
      * postgresql_repository[Add downloads.postgresql.org repository] action add
        * apt_update[] action periodic (up to date)
        * apt_package[apt-transport-https] action install (up to date)
        * apt_repository[postgresql_org_repository] action add
          * remote_file[/opt/kitchen/cache/https___download_postgresql_org_pub_repos_apt_ACCC4CF8_asc] action create
            - create new file /opt/kitchen/cache/https___download_postgresql_org_pub_repos_apt_ACCC4CF8_asc
            - update content in file /opt/kitchen/cache/https___download_postgresql_org_pub_repos_apt_ACCC4CF8_asc from none to 014406
            --- /opt/kitchen/cache/https___download_postgresql_org_pub_repos_apt_ACCC4CF8_asc	2021-02-12 11:46:32.444847746 +0000
            +++ /opt/kitchen/cache/.chef-https___download_postgresql_org_pub_repos_apt_ACCC4CF8_asc20210212-247-f8bzr3	2021-02-12 11:46:32.396847744 +0000
            @@ -1 +1,78 @@
            +-----BEGIN PGP PUBLIC KEY BLOCK-----
            xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
            +-----END PGP PUBLIC KEY BLOCK-----
            - change mode from '' to '0644'
          * execute[apt-key add /opt/kitchen/cache/https___download_postgresql_org_pub_repos_apt_ACCC4CF8_asc] action run
            - execute ["apt-key", "add", "/opt/kitchen/cache/https___download_postgresql_org_pub_repos_apt_ACCC4CF8_asc"]
          * execute[apt-cache gencaches] action run
            - execute ["apt-cache", "gencaches"]
          * execute[apt-cache gencaches] action nothing (skipped due to action :nothing)
          * apt_update[postgresql_org_repository] action nothing (skipped due to action :nothing)
          * file[/etc/apt/sources.list.d/postgresql_org_repository.list] action create
            - create new file /etc/apt/sources.list.d/postgresql_org_repository.list
            - update content in file /etc/apt/sources.list.d/postgresql_org_repository.list from none to d66463
            --- /etc/apt/sources.list.d/postgresql_org_repository.list	2021-02-12 11:46:34.475847831 +0000
            +++ /etc/apt/sources.list.d/.chef-postgresql_org_repository20210212-247-1bnpwik.list	2021-02-12 11:46:34.475847831 +0000
            @@ -1 +1,2 @@
            +deb      "https://download.postgresql.org/pub/repos/apt/" buster-pgdg main 12
            - change mode from '' to '0644'
            - change owner from '' to 'root'
            - change group from '' to 'root'
          * execute[apt-cache gencaches] action run
            - execute ["apt-cache", "gencaches"]
          * apt_update[postgresql_org_repository] action update
            - force update new lists of packages
            * directory[/var/lib/apt/periodic] action create (up to date)
            * directory[/etc/apt/apt.conf.d] action create (up to date)
            * file[/etc/apt/apt.conf.d/15update-stamp] action create_if_missing (up to date)
            * execute[apt-get -q update] action run
              - execute ["apt-get", "-q", "update"]
          
        
      
      * apt_package[postgresql-client-12] action install
        - install version 12.6-1.pgdg100+1 of package postgresql-client-12
    
    * apt_package[postgresql-common] action install
      - install version 225.pgdg100+1 of package postgresql-common
    * template[/etc/postgresql-common/createcluster.conf] action create
      - update content in file /etc/postgresql-common/createcluster.conf from 9b3107 to 573630
      --- /etc/postgresql-common/createcluster.conf	2021-02-12 11:46:51.642848550 +0000
      +++ /etc/postgresql-common/.chef-createcluster20210212-247-1r4ur2r.conf	2021-02-12 11:46:52.565848589 +0000
      @@ -17,7 +17,7 @@
       #waldir = '/var/lib/postgresql/wal/%v/%c/pg_wal'
       
       # Options to pass to initdb.
      -#initdb_options = ''
      +initdb_options = ''
       
       # The following options are copied into the new cluster's postgresql.conf:
       
    * apt_package[postgresql-12] action install
      - install version 12.6-1.pgdg100+1 of package postgresql-12
  
  * service[postgresql] action restart
 ================================================================================
    Error executing action `restart` on resource 'service[postgresql]'
    ================================================================================
    
    Mixlib::ShellOut::CommandTimeout
    --------------------------------
    Command timed out after 900s:
    Command exceeded allowed execution time, process terminated
    ---- Begin output of /bin/systemctl --system restart postgresql ----
    STDOUT: 
    STDERR: 
    ---- End output of /bin/systemctl --system restart postgresql ----
    Ran /bin/systemctl --system restart postgresql returned 
    
    Resource Declaration:
    ---------------------
    # In /opt/kitchen/cache/cookbooks/e3s-sonarqube/recipes/postgresql.rb
    
     15: service 'postgresql' do
     16:   action :restart
     17: end
     18: 
    
    Compiled Resource:
    ------------------
    # Declared in /opt/kitchen/cache/cookbooks/e3s-sonarqube/recipes/postgresql.rb:15:in `from_file'
    
    service("postgresql") do
      action [:restart]
      default_guard_interpreter :default
      service_name "postgresql"
      enabled nil
      running nil
      masked nil
      pattern "postgresql"
      declared_type :service
      cookbook_name "e3s-sonarqube"
      recipe_name "postgresql"
    end
    
    System Info:
    ------------
    chef_version=14.15.6
    platform=debian
    platform_version=10.8
    ruby=ruby 2.5.8p224 (2020-03-31 revision 67882) [x86_64-linux]
    program_name=/opt/chef/embedded/bin/chef-client
    executable=/opt/chef/embedded/bin/chef-client
    

Running handlers:
[2021-02-12T12:02:35+00:00] ERROR: Running exception handlers
Running handlers complete
[2021-02-12T12:02:35+00:00] ERROR: Exception handlers complete
Chef Client failed. 41 resources updated in 19 minutes 11 seconds
[2021-02-12T12:02:35+00:00] FATAL: Stacktrace dumped to /opt/kitchen/cache/chef-stacktrace.out
[2021-02-12T12:02:35+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2021-02-12T12:02:35+00:00] FATAL: Mixlib::ShellOut::CommandTimeout: service[postgresql] (e3s-sonarqube::postgresql line 15) had an error: Mixlib::ShellOut::CommandTimeout: Command timed out after 900s:
Command exceeded allowed execution time, process terminated
---- Begin output of /bin/systemctl --system restart postgresql ----
STDOUT: 
STDERR: 
---- End output of /bin/systemctl --system restart postgresql ----
Ran /bin/systemctl --system restart postgresql returned 
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>>     Converge failed on instance <sonarqube-debian-10>.  Please see .kitchen/logs/sonarqube-debian-10.log for more details
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

🥞 Cookbook version

Latest cookbook version of postgresql:
postgresql (8.2.1)
apt (>= 0.0.0)
yum-epel (>= 0.0.0)

👩‍🍳 Chef-Infra Version

Version of chef-client in my environment:

chef-client version: 14.15.6
delivery version: master (4b21ec7e07fdfa82e86aa80e4f2372dde8e368bb)
berks version: 7.0.8
kitchen version: 1.25.0
inspec version: 3.9.3

🎩 Platform details

CentOS Linux release 7.9.2009 (Core)

Steps To Reproduce

Steps to reproduce the behavior:

1. Use code as above I mentioned.
2. Use OS - CentOS 7
3. .kitchen.yml as above

🚓 Expected behavior

Expect to restart/reload postgresql service as normal service, without timeout issues.

➕ Additional context

Vice verse, if I use in .kitchen.yml next code:

.kitchen.yml:

driver:
  name: dokken
  chef_version: 14
  privileged: true

transport:
  name: dokken

provisioner:
  name: dokken
  data_bags_path: 'test/fixtures/data_bags'
  environments_path: 'test/fixtures/environments'
  nodes_path: 'test/fixtures/nodes'

verifier:
  name: inspec

platforms:
  - name: debian-10
    driver:
      image: dokken/debian-10

suites:
  - name: sonarqube
    driver:
      ports:
        - '9000:9000'
    run_list:
      - recipe[e3s-sonarqube::default]
    verifier:
      inspec_tests:
        - test/integration/default

Everything will be good. Thanks for answering!

[ramereth]

@Pursu1tOfHapp1ness does journalctl -u postgresql show for the output logs? Also you might look at other logs for the pg server that might give you some help. I'm not sure this is a problem with our cookbook or not, it likely might be something else related to including sonarqube. Thanks for reporting!

[Pursu1tOfHapp1ness]

Hey,

Sorry for such a long waiting..
Output logs of journalctl -u postgresql there:

journalctl -u postgresql
-- Logs begin at Fri 2021-05-21 11:20:14 UTC, end at Fri 2021-05-21 11:35:01 UTC. --
-- No entries --

Logs of systemctl status postgresql there:

systemctl status postgresql
● postgresql.service - PostgreSQL RDBMS
   Loaded: loaded (/lib/systemd/system/postgresql.service; enabled; vendor preset: enabled)
   Active: inactive (dead)

And maybe logs from /var/log/postgresql will be useful:

2021-05-21 11:23:09.316 UTC [5852] LOG:  starting PostgreSQL 12.7 (Debian 12.7-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2021-05-21 11:23:09.316 UTC [5852] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2021-05-21 11:23:09.317 UTC [5852] LOG:  could not bind IPv6 address "::1": Cannot assign requested address
2021-05-21 11:23:09.317 UTC [5852] HINT:  Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
2021-05-21 11:23:09.318 UTC [5852] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-05-21 11:23:09.336 UTC [5853] LOG:  database system was shut down at 2021-05-21 11:23:07 UTC
2021-05-21 11:23:09.341 UTC [5852] LOG:  database system is ready to accept connections

[ramereth]

It looks like there's another PG instance running on the server. Does that seem correct?