Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Endless processing / Task #failed: Analysis X #2103

Closed
wuuuw opened this issue May 6, 2024 · 26 comments
Closed

Endless processing / Task #failed: Analysis X #2103

wuuuw opened this issue May 6, 2024 · 26 comments

Comments

@wuuuw
Copy link

wuuuw commented May 6, 2024 •
edited

Prerequisites

  • [+] I am running the latest version
  • [+] I did read the README!
  • [+] I checked the documentation and found no answer
  • [-] I checked to make sure that this issue has not already been filed
    Yes, but it didn't help me Analysis timeout #2019
  • [+] I'm reporting the issue to the correct repository (for multi-repository projects)
  • [+] I have read and checked all configs (with all optional parts)

Expected Behavior

The web interface will complete the scanning and I can check the scan report

Current Behavior

In the web interface in the Recent window the status is processing and I cannot view the scanning report. But in the Dashboard window it is listed as completed. The terminal displays INFO: Task #5: analysis procedure completed

Steps to Reproduce

  1. poetry run python3 cuckoo.py
  2. load file in web-interface (Submit)
  3. just "processing" in Recent, but "completed" in Dashboard
  4. poetry run python utils/process.py -r 5 -d
  5. "reported" in Recent and Dashboard

Also tried sudo systemctl restart cape-processor
and this poetry run python cleaners.py --clean
sudo systemctl restart cape

Context

Question Answer
Git commit commit 2248ab0 (HEAD -> master)
OS version Ubuntu 22.04.4 LTS, Windows 10 21H2

Failure Logs

Terminal from poetry run python3 cuckoo.py

OPTIONAL! Missed dependency: pip3 install https://github.com/CAPESandbox/peepdf/archive/20eda78d7d77fc5b3b652ffc2d8a5b0af796e3dd.zip#egg=peepdf==0.4.2
OPTIONAL! Missed dependency: poetry run pip install -U git+https://github.com/DissectMalware/batch_deobfuscator
OPTIONAL! Missed dependency: poetry run pip install -U git+https://github.com/CAPESandbox/httpreplay
/usr/bin/tcpdump
2024-05-06 17:51:45,486 [lib.cuckoo.core.scheduler] INFO: Using "kvm" machine manager with max_analysis_count=0, max_machines_count=10, and max_vmstartup_count=5
2024-05-06 17:51:45,494 [lib.cuckoo.core.scheduler] INFO: Loaded 1 machine/s
2024-05-06 17:51:45,511 [lib.cuckoo.core.scheduler] INFO: Waiting for analysis tasks
2024-05-06 17:52:54,972 [lib.cuckoo.core.scheduler] INFO: Task #5: Starting analysis of FILE '/tmp/cuckoo-tmp/upload_atx723hk/b5e6dde637ff9dbc4dc8.exe'
2024-05-06 17:52:55,016 [lib.cuckoo.core.scheduler] INFO: Task #5: acquired machine win10 (label=win10, arch=x64, platform=windows)
2024-05-06 17:53:20,352 [lib.cuckoo.core.scheduler] INFO: Enabled route 'internet'.
/usr/bin/tcpdump
2024-05-06 17:53:20,414 [modules.auxiliary.sniffer] INFO: Started sniffer with PID 15295 (interface=virbr0, host=192.168.122.240, dump path=/opt/CAPEv2/storage/analyses/5/dump.pcap)
2024-05-06 17:53:20,435 [lib.cuckoo.core.guest] INFO: Task #5: Starting analysis on guest (id=win10, ip=192.168.122.240)
2024-05-06 17:53:20,461 [lib.cuckoo.core.guest] INFO: Task #5: Guest is running CAPE Agent 0.17 (id=win10, ip=192.168.122.240)
2024-05-06 17:53:26,773 [lib.cuckoo.core.guest] INFO: Task #5: Uploading script files to guest (id=win10, ip=192.168.122.240)
2024-05-06 17:53:32,047 [lib.cuckoo.core.guest] INFO: Task #failed: Analysis 5 (id=win10, ip=192.168.122.240)
2024-05-06 17:53:33,289 [lib.cuckoo.core.scheduler] INFO: Disabled route 'internet'
2024-05-06 17:53:33,343 [lib.cuckoo.core.scheduler] INFO: Task #5: analysis procedure completed
^C
Session terminated, killing shell... ...killed.

Terminal from poetry run python utils/process.py -r 5 -d

2024-05-06 18:37:18,932 [root] DEBUG: Importing modules...
OPTIONAL! Missed dependency: pip3 install https://github.com/CAPESandbox/peepdf/archive/20eda78d7d77fc5b3b652ffc2d8a5b0af796e3dd.zip#egg=peepdf==0.4.2
OPTIONAL! Missed dependency: poetry run pip install -U git+https://github.com/DissectMalware/batch_deobfuscator
OPTIONAL! Missed dependency: poetry run pip install -U git+https://github.com/CAPESandbox/httpreplay
2024-05-06 18:37:19,894 [capa.rules] DEBUG: reading rules from directory /opt/CAPEv2/data/capa-rules
2024-05-06 18:37:19,933 [capa.rules.cache] DEBUG: loading rule set from cache: /home/cape/.cache/capa/capa-fa8c13d6.cache
2024-05-06 18:37:19,993 [capa.loader] DEBUG: reading signatures from directory /opt/CAPEv2/data/flare-signatures
2024-05-06 18:37:19,994 [capa.loader] DEBUG: found signature file: /opt/CAPEv2/data/flare-signatures/1_flare_msvc_rtf_32_64.sig
2024-05-06 18:37:19,994 [capa.loader] DEBUG: found signature file: /opt/CAPEv2/data/flare-signatures/2_flare_msvc_atlmfc_32_64.sig
2024-05-06 18:37:19,994 [capa.loader] DEBUG: found signature file: /opt/CAPEv2/data/flare-signatures/3_flare_common_libs.sig
2024-05-06 18:37:19,996 [root] DEBUG: Imported "auxiliary" modules:
2024-05-06 18:37:19,996 [root] DEBUG: -- Sniffer 2024-05-06 18:37:19,996 [root] DEBUG: Imported "processing" modules: 2024-05-06 18:37:19,996 [root] DEBUG: |-- CAPE 2024-05-06 18:37:19,996 [root] DEBUG: |-- AnalysisInfo . 2024-05-06 18:37:20,039 [root] DEBUG: Imported "machinery" modules: 2024-05-06 18:37:20,039 [root] DEBUG: -- KVM
2024-05-06 18:37:20,039 [Task 5] [root] DEBUG: Processing task
2024-05-06 18:37:20,197 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Executing processing module "CAPE" on analysis at "/opt/CAPEv2/storage/analyses/5"
2024-05-06 18:37:20,218 [Task 5] [lib.cuckoo.common.objects] DEBUG: Initializing Yara...
2024-05-06 18:37:20,301 [Task 5] [lib.cuckoo.common.objects] DEBUG: |-- binaries AutoIT.yar
.
2024-05-06 18:37:20,591 [Task 5] [lib.cuckoo.common.objects] DEBUG: |-- CAPE xRAT.yar
2024-05-06 18:37:20,591 [Task 5] [lib.cuckoo.common.objects] DEBUG: |-- CAPE zgRAT.yar
2024-05-06 18:37:21,998 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Executing processing module "AnalysisInfo" on analysis at "/opt/CAPEv2/storage/analyses/5"
2024-05-06 18:37:22,046 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Processing module autoruns not found in configuration file
2024-05-06 18:37:22,047 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Executing processing module "BehaviorAnalysis" on analysis at "/opt/CAPEv2/storage/analyses/5"
2024-05-06 18:37:22,047 [Task 5] [modules.processing.behavior] DEBUG: Analysis results folder does not contain any file or injection was disabled
2024-05-06 18:37:22,047 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Executing processing module "Debug" on analysis at "/opt/CAPEv2/storage/analyses/5"
2024-05-06 18:37:22,051 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Executing processing module "NetworkAnalysis" on analysis at "/opt/CAPEv2/storage/analyses/5"
2024-05-06 18:37:22,055 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Executing processing module "Suricata" on analysis at "/opt/CAPEv2/storage/analyses/5"
2024-05-06 18:37:22,640 [Task 5] [modules.processing.suricata] DEBUG: pcapfile list: {'message': {'count': 0, 'files': []}, 'return': 'OK'} current pcap: {'message': '/opt/CAPEv2/storage/analyses/5/dump.pcap', 'return': 'OK'}
2024-05-06 18:37:27,644 [Task 5] [modules.processing.suricata] DEBUG: pcapfile list: {'message': {'count': 0, 'files': []}, 'return': 'OK'} current pcap: {'message': 'None', 'return': 'OK'}
2024-05-06 18:37:27,644 [Task 5] [modules.processing.suricata] DEBUG: Pcap not in list and not current pcap lets assume it's processed
2024-05-06 18:37:27,688 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Executing processing module "UrlAnalysis" on analysis at "/opt/CAPEv2/storage/analyses/5"
2024-05-06 18:37:27,689 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Executing processing module "script_log_processing" on analysis at "/opt/CAPEv2/storage/analyses/5"
2024-05-06 18:37:27,690 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Executing processing module "ProcessMemory" on analysis at "/opt/CAPEv2/storage/analyses/5"
2024-05-06 18:37:27,749 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Applying signature overlays for signatures: creates_exe
2024-05-06 18:37:27,752 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Running 252 evented signatures
2024-05-06 18:37:27,752 [Task 5] [lib.cuckoo.core.plugins] DEBUG: |-- packer_themida
.
2024-05-06 18:37:27,953 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Running signature "dharma_mutexes"
2024-05-06 18:37:27,953 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Running signature "ransomware_extensions"
/opt/CAPEv2/utils/../lib/cuckoo/common/abstracts.py:1058: FutureWarning: Possible nested set at position 5
exp = re.compile(pattern, re.IGNORECASE)
2024-05-06 18:37:27,962 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Running signature "ransomware_files"
2024-05-06 18:37:27,975 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Running signature "fonix_mutexes"
2024-05-06 18:37:27,975 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Running signature "gandcrab_mutexes"
.
2024-05-06 18:37:28,046 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Executing reporting module "CAPASummary"
2024-05-06 18:37:28,058 [Task 5] [lib.cuckoo.common.integrations.capa] INFO: FLARE CAPA -> No process data available
2024-05-06 18:37:28,060 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Executing reporting module "PCAP2CERT"
2024-05-06 18:37:28,062 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Executing reporting module "JsonDump"
2024-05-06 18:37:28,063 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Executing reporting module "MongoDB"
2024-05-06 18:37:28,078 [Task 5] [modules.reporting.mongodb] DEBUG: Deleted previous MongoDB data for Task 5
2024-05-06 18:37:28,235 [Task 5] [root] DEBUG: Finished processing task
@wuuuw
Copy link
Author

wuuuw commented May 6, 2024

cuckoo.conf
ip = 192.168.122.1
freespace = 45000

kvm.conf
machines = win10
interface = virbr0
label = win10
ip = 192.168.122.240
arch = x64

routing.conf
route = internet
internet = virbr0

@doomedraven
Copy link
Collaborator

hey, yes we suspect the bug is here https://github.com/kevoreilly/CAPEv2/pull/2036/files we probably will revert that

@wuuuw
Copy link
Author

wuuuw commented May 8, 2024 •
edited

@doomedraven
I need any working version of the sandbox.
(I don't get any information in behavioral and network analysis, none at all)

where could I get it?
image
image

@doomedraven
Copy link
Collaborator

you can revert that commit that i mention, i can-t guarantee that one cause the problem but is our suspicion. well is open source so you can help us dig into issue, is not our paid job so we have to first handle our job and then work on this

@doomedraven doomedraven mentioned this issue May 10, 2024
Closed
1 task
@kevoreilly
Copy link
Owner

I have been able to recreate this issue on an instance of mine, and found that reverting the changes in #2306 fixes the issue.

I haven't had time to look into why it is occurring. @qux-bbb perhaps you can help diagnose this issue?

@qux-bbb
Copy link
Contributor

qux-bbb commented May 10, 2024

I'll diagnose this issue in a few days.

@qux-bbb
Copy link
Contributor

qux-bbb commented May 11, 2024

@wuuuw It's not a bug, you run the wrong process command.
poetry run python utils/process.py -r 5 -d is not a right command for this situation.

  -d, --debug           Display debug messages
  -r, --report          Re-generate report

You should run process.py just like in cape-processor. service:

cd /opt/CAPEv2/utils/
poetry run python process.py -p7 auto -pt 900

@kevoreilly
Copy link
Owner

Er hang on. I use -d -r all the time in my daily work. It was working before, it needs to work.

@qux-bbb
Copy link
Contributor

qux-bbb commented May 11, 2024

Er hang on. I use -d -r all the time in my daily work. It was working before, it needs to work.

I want to know all the status about cape* services before you do anything.
You can run this command: systemctl status cape*.service

@qux-bbb
Copy link
Contributor

qux-bbb commented May 14, 2024

@wuuuw @kevoreilly If the cape-processor service is not running at the begining, the behavior is like that.
You guys can give me more info, then I can verify it.

@wuuuw
Copy link
Author

wuuuw commented May 16, 2024

@qux-bbb
● cape-web.service - CAPE WSGI app
Loaded: loaded (/lib/systemd/system/cape-web.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2024-05-16 17:06:54 UTC; 8min ago
Docs: https://github.com/kevoreilly/CAPEv2
Main PID: 1401 (python)
Tasks: 3 (limit: 6938)
Memory: 480.7M
CPU: 1min 6.531s
CGroup: /system.slice/cape-web.service
├─1401 /home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/bin/python manage.py runserver_plus 0.0.0.0:8000 --traceback --keep-meta-shutdown
└─2550 /home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/bin/python manage.py runserver_plus 0.0.0.0:8000 --traceback --keep-meta-shutdown

мая 16 17:07:04 nan-pc python3[2550]: Missed dependency flare-floss: poetry run pip install -U flare-floss
мая 16 17:07:04 nan-pc python3[2550]: System check identified no issues (3 silenced).
мая 16 17:07:04 nan-pc python3[2550]: You have 32 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): account, admin, auth, authtoken, cont>
мая 16 17:07:04 nan-pc python3[2550]: Run 'python manage.py migrate' to apply them.
мая 16 17:07:04 nan-pc python3[2550]: Django version 4.2.11, using settings 'web.settings'
мая 16 17:07:04 nan-pc python3[2550]: Development server is running at http://0.0.0.0:8000/
мая 16 17:07:04 nan-pc python3[2550]: Using the Werkzeug debugger (http://werkzeug.pocoo.org/)
мая 16 17:07:04 nan-pc python3[2550]: Quit the server with CONTROL-C.
мая 16 17:07:04 nan-pc python3[2550]: * Debugger is active!
мая 16 17:07:04 nan-pc python3[2550]: * Debugger PIN: 968-109-742

● cape-processor.service - CAPE report processor
Loaded: loaded (/lib/systemd/system/cape-processor.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) since Thu 2024-05-16 17:12:04 UTC; 2min 50s ago
Docs: https://github.com/kevoreilly/CAPEv2
Process: 3484 ExecStart=/usr/bin/python3 -m poetry run python process.py -p7 auto -pt 900 (code=exited, status=0/SUCCESS)
Main PID: 3484 (code=exited, status=0/SUCCESS)
CPU: 3.653s

● cape.service - CAPE
Loaded: loaded (/lib/systemd/system/cape.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2024-05-16 17:06:54 UTC; 8min ago
Docs: https://github.com/kevoreilly/CAPEv2
Main PID: 1402 (python)
Tasks: 58 (limit: 6938)
Memory: 266.0M
CPU: 6min 3.042s
CGroup: /system.slice/cape.service
└─1402 /home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/bin/python cuckoo.py

мая 16 17:06:57 nan-pc python3[1402]: _.-'.i__i__i.'
мая 16 17:06:57 nan-pc python3[1402]: """""""""
мая 16 17:06:57 nan-pc python3[1402]: Cuckoo Sandbox 2.4-CAPE
мая 16 17:06:57 nan-pc python3[1402]: www.cuckoosandbox.org
мая 16 17:06:57 nan-pc python3[1402]: Copyright (c) 2010-2015
мая 16 17:06:57 nan-pc python3[1402]: CAPE: Config and Payload Extraction
мая 16 17:06:57 nan-pc python3[1402]: github.com/kevoreilly/CAPEv2
мая 16 17:07:00 nan-pc python3[1402]: 2024-05-16 17:07:00,788 [lib.cuckoo.core.scheduler] INFO: Using "kvm" machine manager with max_analysis_count=0, max_machines_count=10, and max_vmstart>
мая 16 17:07:00 nan-pc python3[1402]: 2024-05-16 17:07:00,796 [lib.cuckoo.core.scheduler] INFO: Loaded 1 machine/s
мая 16 17:07:00 nan-pc python3[1402]: 2024-05-16 17:07:00,812 [lib.cuckoo.core.scheduler] INFO: Waiting for analysis tasks

● cape-rooter.service - CAPE rooter
Loaded: loaded (/lib/systemd/system/cape-rooter.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2024-05-16 17:06:54 UTC; 8min ago
Docs: https://github.com/kevoreilly/CAPEv2
Process: 922 ExecStartPre=/usr/bin/python3 -m poetry config cache-dir /opt/CAPEv2/.cache/pypoetry (code=exited, status=0/SUCCESS)
Main PID: 1398 (python)
Tasks: 1 (limit: 6938)
Memory: 28.1M
CPU: 1.912s
CGroup: /system.slice/cape-rooter.service
└─1398 /opt/CAPEv2/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/bin/python rooter.py -g cape

мая 16 17:06:52 nan-pc systemd[1]: Starting CAPE rooter...
мая 16 17:06:54 nan-pc systemd[1]: Started CAPE rooter.

@qux-bbb
Copy link
Contributor

qux-bbb commented May 17, 2024

@wuuuw Can your cape-processor.service be runing status?

@wuuuw
Copy link
Author

wuuuw commented May 17, 2024

~$ systemctl status cape-processor.service
● cape-processor.service - CAPE report processor
Loaded: loaded (/lib/systemd/system/cape-processor.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) since Fri 2024-05-17 10:10:03 UTC; 2min 33s ago
Docs: https://github.com/kevoreilly/CAPEv2
Process: 1378 ExecStart=/usr/bin/python3 -m poetry run python process.py -p7 auto -pt 900 (code=exited, status=0/SUCCESS)
Main PID: 1378 (code=exited, status=0/SUCCESS)
CPU: 5.530s

мая 17 10:10:03 nan-pc systemd[1]: cape-processor.service: Consumed 5.530s CPU time.

~$ sudo systemctl restart cape-processor.service

~$ systemctl status cape-processor.service
● cape-processor.service - CAPE report processor
Loaded: loaded (/lib/systemd/system/cape-processor.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) since Fri 2024-05-17 10:13:12 UTC; 11s ago
Docs: https://github.com/kevoreilly/CAPEv2
Process: 4295 ExecStart=/usr/bin/python3 -m poetry run python process.py -p7 auto -pt 900 (code=exited, status=0/SUCCESS)
Main PID: 4295 (code=exited, status=0/SUCCESS)
CPU: 3.281s

@qux-bbb
Copy link
Contributor

qux-bbb commented May 17, 2024

@wuuuw Can you run these commands and give me the output?

sudo systemctl stop cape-processor.service
cd /opt/CAPEv2/utils/
sudo su cape
/usr/bin/python3 -m poetry run python process.py -p7 auto -pt 900 -d

@qux-bbb
Copy link
Contributor

qux-bbb commented May 17, 2024

@AghaHannan You need to create a new issue.

Repository owner deleted a comment from AghaHannan May 17, 2024
@wuuuw
Copy link
Author

wuuuw commented May 17, 2024

@qux-bbb
2024-05-17 22:16:50,644 [root] DEBUG: Importing modules...
OPTIONAL! Missed dependency: pip3 install https://github.com/CAPESandbox/peepdf/archive/20eda78d7d77fc5b3b652ffc2d8a5b0af796e3dd.zip#egg=peepdf==0.4.2
OPTIONAL! Missed dependency: poetry run pip install -U git+https://github.com/DissectMalware/batch_deobfuscator
OPTIONAL! Missed dependency: poetry run pip install -U git+https://github.com/CAPESandbox/httpreplay
2024-05-17 22:16:51,639 [capa.rules] DEBUG: reading rules from directory /opt/CAPEv2/data/capa-rules
2024-05-17 22:16:51,685 [capa.rules.cache] DEBUG: loading rule set from cache: /home/cape/.cache/capa/capa-fa8c13d6.cache
2024-05-17 22:16:51,754 [capa.loader] DEBUG: reading signatures from directory /opt/CAPEv2/data/flare-signatures
2024-05-17 22:16:51,754 [capa.loader] DEBUG: found signature file: /opt/CAPEv2/data/flare-signatures/1_flare_msvc_rtf_32_64.sig
2024-05-17 22:16:51,754 [capa.loader] DEBUG: found signature file: /opt/CAPEv2/data/flare-signatures/2_flare_msvc_atlmfc_32_64.sig
2024-05-17 22:16:51,754 [capa.loader] DEBUG: found signature file: /opt/CAPEv2/data/flare-signatures/3_flare_common_libs.sig
2024-05-17 22:16:51,757 [root] DEBUG: Imported "auxiliary" modules:
2024-05-17 22:16:51,757 [root] DEBUG: -- Sniffer 2024-05-17 22:16:51,757 [root] DEBUG: Imported "processing" modules: 2024-05-17 22:16:51,757 [root] DEBUG: |-- CAPE 2024-05-17 22:16:51,757 [root] DEBUG: |-- AnalysisInfo 2024-05-17 22:16:51,757 [root] DEBUG: |-- Autoruns 2024-05-17 22:16:51,757 [root] DEBUG: |-- BehaviorAnalysis 2024-05-17 22:16:51,758 [root] DEBUG: |-- Debug 2024-05-17 22:16:51,758 [root] DEBUG: |-- NetworkAnalysis 2024-05-17 22:16:51,758 [root] DEBUG: |-- ProcessMemory 2024-05-17 22:16:51,758 [root] DEBUG: |-- script_log_processing 2024-05-17 22:16:51,758 [root] DEBUG: |-- Suricata 2024-05-17 22:16:51,758 [root] DEBUG: -- UrlAnalysis
2024-05-17 22:16:51,758 [root] DEBUG: Imported "signatures" modules:
2024-05-17 22:16:51,758 [root] DEBUG: |-- ClamAV
2024-05-17 22:16:51,758 [root] DEBUG: |-- KnownVirustotal

2024-05-17 22:16:51,809 [root] DEBUG: |-- LinuxReadsFiles
2024-05-17 22:16:51,809 [root] DEBUG: -- LinuxWritesFiles 2024-05-17 22:16:51,809 [root] DEBUG: Imported "reporting" modules: 2024-05-17 22:16:51,809 [root] DEBUG: |-- BinGraph 2024-05-17 22:16:51,809 [root] DEBUG: |-- CAPASummary 2024-05-17 22:16:51,809 [root] DEBUG: |-- JsonDump 2024-05-17 22:16:51,809 [root] DEBUG: |-- MongoDB 2024-05-17 22:16:51,809 [root] DEBUG: -- PCAP2CERT
2024-05-17 22:16:51,809 [root] DEBUG: Imported "feeds" modules:
2024-05-17 22:16:51,809 [root] DEBUG: -- AbuseCH_SSL 2024-05-17 22:16:51,809 [root] DEBUG: Imported "machinery" modules: 2024-05-17 22:16:51,809 [root] DEBUG: -- KVM
2024-05-17 22:16:51,810 [root] INFO: Processing analysis data
Traceback (most recent call last):
File "/opt/CAPEv2/utils/process.py", line 298, in autoprocess
with pebble.ProcessPool(max_workers=parallel, max_tasks=maxtasksperchild, initializer=init_worker) as pool:
File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/pebble/pool/process.py", line 61, in init
self._pool_manager = PoolManager(self._context, mp_context)
File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/pebble/pool/process.py", line 200, in init
self.worker_manager = WorkerManager(context.workers,
File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/pebble/pool/process.py", line 342, in init
self.pool_channel, self.workers_channel = channels(mp_context)
File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/pebble/pool/channel.py", line 34, in channels
WorkerChannel(read0, write1, (read1, write0), mp_context))
File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/pebble/pool/channel.py", line 86, in init
self.mutex = ChannelMutex(mp_context)
File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/pebble/pool/channel.py", line 132, in init
self.reader_mutex = mp_context.RLock()
File "/usr/lib/python3.10/multiprocessing/context.py", line 73, in RLock
return RLock(ctx=self.get_context())
File "/usr/lib/python3.10/multiprocessing/synchronize.py", line 187, in init
SemLock.init(self, RECURSIVE_MUTEX, 1, 1, ctx=ctx)
File "/usr/lib/python3.10/multiprocessing/synchronize.py", line 57, in init
sl = self._semlock = _multiprocessing.SemLock(
OSError: [Errno 12] Cannot allocate memory

@qux-bbb
Copy link
Contributor

qux-bbb commented May 18, 2024

@wuuuw Sorry, I pulled the latest code, but I can't reproduce that error.
Do all samples have that problem, or the special sample?
If the special sample has the problem, can you provide the sample?

@qux-bbb
Copy link
Contributor

qux-bbb commented May 18, 2024

@wuuuw Can you run the command and give me the output?

free -h

@doomedraven
Copy link
Collaborator

Oh looks like someone doesn't have enought ram to run processing

@wuuuw
Copy link
Author

wuuuw commented May 18, 2024

@qux-bbb
free -h
total used free shared buff/cache available
Mem: 5,7Gi 2,6Gi 982Mi 103Mi 2,2Gi 2,9Gi
Swap: 2,0Gi 0B 2,0Gi

@qux-bbb
Copy link
Contributor

qux-bbb commented May 18, 2024 •
edited

@wuuuw Can You try to install CAPE on a computer with more RAM, for example 16G?

@doomedraven
Copy link
Collaborator

i have added better message in those cases with ram b1df219

@qux-bbb
Copy link
Contributor

qux-bbb commented May 18, 2024

@doomedraven Thanks!
@kevoreilly Do you also have the error "Cannot allocate memory"?

@doomedraven
Copy link
Collaborator

ok i finally found some time to jump to this and test latest master on my side, i cant reproduce it.

@qux-bbb
Copy link
Contributor

qux-bbb commented May 20, 2024 •
edited

To reproduce that "Cannot allocate memory", you can do like this:

First, stop cape-processor.service:

sudo systemctl stop cape-processor.service

Second, use many RAM, my computer RAM is 16G, so I will use 10G(10240M):

mkdir /tmp/memory
sudo mount -t tmpfs -o size=10240M tmpfs /tmp/memory
dd if=/dev/zero of=/tmp/memory/block

Third, try to start process.py:

cd /opt/CAPEv2/utils/
sudo su cape
/usr/bin/python3 -m poetry run python process.py -p7 auto -pt 900 -d

Then you can get the error message.

Finally, restore your RAM:

rm /tmp/memory/block
sudo umount /tmp/memory
rm /tmp/memory -r

Maybe the issue can be closed.

@doomedraven
Copy link
Collaborator

yes, i can't reproduce, and the user side low hardware, we cant do nothing there

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@doomedraven @qux-bbb @kevoreilly @wuuuw