Exploratory debug symbols & core dump analysis for Agent (RPM)

[ncvicchi]

Related issue
#21736
#21739

Description

This issue aims to make a exploratory session of current #9913 phase 1 development for the Agent instance of the RPM package.

Verifications should be performed on the following issues to check end to end the process from generation of symbols to core dump analysis with them.

These verifications must be performed by a different collaborator than the originally assigned to the issue, and a full detail of procedures, logs and results must be provided.
Evidence of success must be provided as well.

Goals

• Verify that binary and debug symbols packages are correctly generated by following the current documentation
• Verify that binary & debug symbols packages are automatically uploaded to their designated locations.
• Verify that packages perform a successful installation by following the current documentation.
• Verify that the installed agent instance runs succesfully.
• Verify that a core dump is generated on simulated failure.
• Verify that debug symbols are suitable to debug/analyze the core dumps.
• Verify that the documentation used during all the exploratory is adequate, correct and complete.
• Verify that core dump generation can be enabled and disabled just by following the proper documentation.

DoD

• Packages for binaries and debug symbols are generated by following documentation. @aritosteles
• Packages are confirmed to be uploaded to their designated location. @jotacarma90
• Installation is tested and validated.
• Installed agent behaves as expected.
• A simulation of failure is performed and as a result a core dump is generated.
• Core dump is successfully analyzed by using the corresponding debug symbols.
• No documentation errors are found of left uncorrected. @aritosteles
• Core dump generation is validated to be enabled or disabled just by following the proper documentation.
• Extensive evidence and documentation of the exploratory is provided

Approval
DRI Name: @ncvicchi
Objective: Generate debug symbols

[aritosteles]

Redhat based distros do not generate core dump when a process is killed using the kill command. To achieve the desired result we leveraged a recently discovered bug described in #23354. We replicated the issue (steps described in this comment) and set up a 4.9 manager for the agent to connect and thus a core dump was generated. The whole test had to be restarted to ensure the instructions given in the documentation part of the issue are accurate.

[aritosteles]

In CentOS core dumps are not generated by kill -9 commands but are generated for kill -11, making it unnecessary to use an existing but like the one mentioned in the previous comment. Also, the core dump configuration steps differ from Ubuntu systems.

Final test completed:

1- Cloned wazuh/wazuh repository:

    git clone https://github.com/wazuh/wazuh.git
    checkout enhancement/9913..epic

2- Installed dependencies:

    Docker: https://documentation.wazuh.com/current/deployment-options/docker/docker-installation.html
    Gcc, Cmake, etc: https://documentation.wazuh.com/current/deployment-options/wazuh-from-sources/wazuh-server/index.html

3- Generated rpm packages:

./generate_package.sh -t agent --system rpm

4- Installed the manager and debug symbols:

sudo rpm -i wazuh-manager_4.9.0-0_x86_64_36f93d8.rpm
sudo rpm -i wazuh-manager-debuginfo_4.9.0-0_x86_64_36f93d8.rpm

5- Restarted Wazuh:

6- Enabled and configured coredump:

• Add the following settings in /etc/systemd/system.conf

DumpCore=yes
DefaultLimitCORE=infinity

• Add the following settings in /etc/sysctl.d/core.conf

kernel.core_pattern = /var/lib/coredumps/core-%e-pid%p-time%t
kernel.core_uses_pid = 1
fs.suid_dumpable = 2

• Create /var/lib/coredumps and give it the permissions 773.

• Reboot the system

• ulimit -c unlimited

• sysctl -p

• restart wazuh agent

7- Select and kill Wazuh processes. Check for core dump generation:

8- Gdb output (core dump files were renamed for readability):

[Dwordcito]

@jotacarma90 will check if the debug symbols are uploaded to s3

[aritosteles]

After fixing the workflow for DEB Agent, the workflow for RPM Agent fails. But it fails using master branch too so there is some external issue.

Workflow fail on master branch:
https://github.com/wazuh/wazuh-agent-packages/actions/runs/9325672777/job/25673051400