[physical] - Windows 10 - Hangs on Restarting after successful analysis

[wlmitra]

Windows 10 - Hangs on Restarting after successful analysis.

Recently I started to have fun with Capev2.

I am using the physical machinery, and physical machine (hp elite desk 800 g8).

When analysis is complete, the restore task is sent to Fog server, and the shutdown command is sent from cape server to cape agent via “/execute” route, and it does trigger well on the agent side; however, the machine hangs on a blue screen (not BSOD) with “Restarting” information and it keeps like that forever.

Remarks:
-I tried the newest capemon dlls (it did not fix the issue)
-I wrote a powershell script that is triggered before shutdown.exe which checks the status of WMI services (as those are restarted during analysis process) and look for any potentially suspended or non-responding processes or so … but all is always OK.
-When analysis is not in progress, the machine can reboot normally

Always when shutdown is called (via shutdown.exe -r -f -t 0 or via “Restart-Computer -Force”, the reboot hangs … this is the moment when I started to suspect that the issue might be related to capemon.

In hook_misc.c, we can see that capemon would hook system shutdown/reboot API calls:
Ref: https://github.com/kevoreilly/capemon/blob/a1f5bf8e9cb904f32da8ff4801691fbfef2f87e3/hook_misc.c#:~:text=InitiateSystemShutdownExW

Example:

HOOKDEF_NOTAIL(WINAPI, InitiateSystemShutdownExW,
In_opt LPWSTR lpMachineName,
In_opt LPWSTR lpMessage,
In DWORD dwTimeout,
In BOOL bForceAppsClosed,
In BOOL bRebootAfterShutdown,
In DWORD dwReason
) {
DWORD ret = 0;
LOQ_zero("system", "uuiiih", "MachineName", lpMachineName, "Message", lpMessage, "Timeout", dwTimeout, "ForceAppsClosed", bForceAppsClosed, "RebootAfterShutdown", bRebootAfterShutdown, "Reason", dwReason);
pipe("SHUTDOWN:");
return ret;
}

Following system shutdown/reboot functions are hooked in “hooks.c” with the same pipe function.
Ref: https://github.com/kevoreilly/capemon/blob/da546e668081089480e17755ec83c32cfd53e09f/hooks.c

#define HOOK_NOTAIL(library, funcname, numargs) {L###library, #funcname, NULL, NULL,
&New_##funcname, NULL, NULL, TRUE, FALSE, numargs, TRUE}

HOOK_NOTAIL(ntdll, NtShutdownSystem, 1),
HOOK_NOTAIL(ntdll, NtSetSystemPowerState, 3),
HOOK_NOTAIL(user32, ExitWindowsEx, 2),
HOOK_NOTAIL(advapi32, InitiateShutdownW, 5),
HOOK_NOTAIL(advapi32, InitiateSystemShutdownW, 5),
HOOK_NOTAIL(advapi32, InitiateSystemShutdownExW, 6),

Suspected Root Cause:

Initially I wasted lot of time and evidently missed the fact that new function to be called is declared within the hook itself (dumb me).
Hence all shutdown/reboot requests are sent to a named pipe, instead of being executed.

Once I was able to keep the session in state allowing to execute Process Explorer, and I was looking for CAPEMon DLLs.

It seems it always remains hooked in the same processes: lsass.exe, svchost.exe and in the process started by the sample (in this case procexp…)

Once I killed the svchost process, I was able to issue reboot command again which was successfully rebooting the machine.

I think the root cause is capemon, which eats the restart requests … For some reason the DLL was not unloaded from svchost which seems to be crucial for some reason to perform the reboot.

Note: Setting “terminate_processes = off to on” in cuckoo.conf + cape restart, does not fix the issue, it brings more problems … there were some errors related to agent or analyzer not being able to stop the process (in a loop until timeout)

This issue was not previously spotted I guess cause guys are mainly using VMs which are restored from snapshot , not rebooted…

I started to dig, and capemon would look for specific shutdown mutex… In log.c:
https://github.com/kevoreilly/capemon/blob/52377d9be0e3597bca4ee4a0d6bf1a3726d5a8f1/log.c#:~:text=static%20DWORD%20WINAPI-,_logwatcher_thread,-(LPVOID%20param)
…

if (is_shutting_down() == 0) {
pipe("CRITICAL:Logging thread was terminated!");
}

The function “is_shutting_down()”:
https://github.com/kevoreilly/capemon/blob/a4c6cde30c12ff35de8c79823542bd2c27224ae4/misc.c#:~:text=int%20is_shutting_down()
…
mutex_handle = OpenMutex(SYNCHRONIZE, FALSE, g_config.shutdown_mutex);
if (mutex_handle != NULL) {
log_flush();
CloseHandle(mutex_handle);
ret = 1;
}

The “g_config.shutdown_mutex” is assigned a value in config.c by “parse_config_line” function:
https://github.com/kevoreilly/capemon/blob/113b1a6d37ffd1f24d0337b361a62d69b90d6baa/config.c#:~:text=strcmp(key%2C-,%22shutdown%2Dmutex%22,-))%20%7B

else if (!strcmp(key, "shutdown-mutex")) {
strncpy(g_config.shutdown_mutex, value,
ARRAYSIZE(g_config.shutdown_mutex));
}

The “value”’s value is read from config file line by line in “read_config” function and processed/assigned to g_config object accordingly depending on the key (as above):

char buf[32768], config_fname[MAX_PATH], analyzer_path[MAX_PATH];
FILE *fp;

// look for the config in monitor directory
strncpy(analyzer_path, our_dll_path, strlen(our_dll_path));
PathRemoveFileSpec(analyzer_path); // remove filename
sprintf(config_fname, "%s\%u.ini", analyzer_path, GetCurrentProcessId());

fp = fopen(config_fname, "r");

....

memset(buf, 0, sizeof(buf));
while (fgets(buf, sizeof(buf), fp) != NULL)
{
// cut off the newline
char *p = strchr(buf, '\r');
if (p != NULL) *p = 0;
p = strchr(buf, '\n');
if (p != NULL) *p = 0;

  parse_config_line(buf);

}

From another hand, the cape analyzer.py creates the shutdown mutex:
https://github.com/kevoreilly/CAPEv2/blob/ab30a65ef1aa71a068a811b1501c57abe90598ad/analyzer/windows/analyzer.py#:~:text=KERNEL32.CreateMutexA

 KERNEL32.CreateMutexA(None, False, SHUTDOWN_MUTEX)
    log.info("Created shutdown mutex")

The mutex will be random per each execution SHUTDOWN_MUTEX = f"Global\{random_string(6, 10)}"

So I do not know why CAPEmon DLLS won’t unload, if shutdown mutex is present …

Thank you in advance for any recommendations.

P.S I will try with Windows 7 later on, but would prefer to have it with more decent system.

[github-actions]

@wlmitra: hello! 👋

This issue is being automatically closed because it does not follow the issue template.

This is open source project!
So please apreciate our time that we sacrify from other thing that we could enjoy, instead of asking boring things over and over.

[kevoreilly]

Interesting issue - thanks for the detailed report and apologies about the auto-close. It protects us from most lame issues but occasionally closes good ones.

It seems your research has led you in the same direction as I would expect - my initial reaction to reading 'blue screen' was to consider which system-critical processes capemon might be loaded in. I would recommend a few tests to see if we can't pin it down a bit further.

The first one is to test whether it's the hooks you mention or not. Try running a job with the following options: exclude-apis=NtShutdownSystem:NtSetSystemPowerState:ExitWindowsEx:InitiateShutdownW:InitiateSystemShutdownW:InitiateSystemShutdownExW:NtRaiseHardError

To make sure svchost processes aren't monitored you could try single-process=1.

To test to see if it's injection in lsass (although it shouldn't be as here only a handful of TLS encryption-specific hooks are used). Nonetheless to rule it out, temporarily delete the auxiliary module from analyzer/windows/modules/auxiliary/tlsdump.py

Let me know how you get on with these tests.

[wlmitra]

I modified the "exclude-apis" list by adding WTSShutdownSystem:

exclude-apis=NtShutdownSystem:NtSetSystemPowerState:ExitWindowsEx:InitiateShutdownW:InitiateSystemShutdownW:InitiateSystemShutdownExW:NtRaiseHardError:WTSShutdownSystem

--> The system hangs, in process explorer (started prior analysis with python.exe name to avoid injection, so still not hanged) i can see that capemon remains injected into processes despite analysis complete status
--> The python process hosting analyzer, and python process hosting agent both do not exit (They keep running)
--> I have crafted quickly a reboot utlity using unhooked function "WTSShutdownSystem", but this one ends up on hanging on Restarting as well ... i guess still the svchost.exe with injected module is preventing the reboot... maybe exclude-apis does not work ?

...

single-process=1
--> The system hangs on Restarting (I guess this time both python processes related to agent and analyzer either exited properly or killed by reboot or something else)
---> I can confirm if needed (tomorrow)

Removing analyzer/windows/modules/auxiliary/tlsdump.py + exclude-apis=... same as above,
--> the machine hangs on Restarting ... So the lsass injection seems not relevant to reboot process

Instead of shutdown.exe, i pointed to another non-existent exe file + cape restart and sent the submission again (hoping to be able to run process explorer and see injected processes).
--> In the result i can see that without tlsdump.py the lsass is not injected ... Only svchost and the process started by the sample

When i try to execute new task from procexplorer -> cmd -> powershell, right after running powershell everything hangs... hence i loose the posibility to try a manual restart (Start menu is not responsive)

---> Sometimes more processes are still injected, but not lsass.

The workaround again appears to be:

--> Issue reboot command
--> Kill the svchost.exe process having the capemon dll in modules ... but following powershell script cannot locate svchost prcoess...

Note: The file capemon_paths contains the list of x86 and x64 paths to capemon files (like data\blabla.dll)... i am outputing it from analyzer.py...

Get-Content "capemon_paths" | ForEach-Object {
  $InjectedModuleName = $_ -replace "dll\\",""; # The output would be in format .dll
  Write-Host "Looking for CAPEMon module: '$InjectedModuleName'"
  $Injected_Processes = Get-Process svchost -ErrorAction SilentlyContinue | where {$_.Modules -like "*($InjectedModuleName)"}
  $Injected_Processes | ForEach-Object {
        Write-Host "Killing: "$_;
        kill -Force $_.Id -ErrorAction Continue -WarningAction Continue;
  }
}

Note: The powershell interpreter was running from renamed powershell.exe to python.exe to avoid injection.

Conclusion:

• It would be nice to verify if exclude-apis actually work (Does it prevent from loading hooks to these functions?)
• It's crucial to make sure that all injected DLLs unload when shutdown mutex is populated (this is initial root cause in my opinion)

Is there anything else i can do to help?

Witold

-I won't test on Windows 7, this hardware is apparently not working well with Windows 7
-I need to figure out why process explorer (started prior analysis) can see the capemon modules and powershell (started after analysis) cannot

[wlmitra]

And yes almost forgot, the working code (it reboots the system without capemon) of small utility (must be executed as admin to get shutdown privilege) ... it's probably another function to hook in the future:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Runtime.InteropServices;

namespace shutdown
{
    class Program
    {
        private const int WTS_WSD_REBOOT = 0x00000004;
        private const int WTS_WSD_SHUTDOWN = 0x00000002;
        
        public static readonly IntPtr LocalServerHandle = IntPtr.Zero;

        [DllImport("wtsapi32.dll", SetLastError = true)]
        public static extern int WTSShutdownSystem(IntPtr hServer, int shutdownFlag);

        [DllImport("wtsapi32.dll", CharSet = CharSet.Auto, SetLastError = true)]
        public static extern IntPtr WTSOpenServer(string serverName);

        static void Main(string[] args)
        {
            IntPtr hServer = WTSOpenServer("127.0.0.1");
            WTSShutdownSystem(hServer, WTS_WSD_REBOOT);
        }
    }
}