Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to analyze ARM64 / ARM bare metal image #294

Open
amarkovytch opened this issue Mar 3, 2022 · 10 comments
Open

Unable to analyze ARM64 / ARM bare metal image #294

amarkovytch opened this issue Mar 3, 2022 · 10 comments

Comments

@amarkovytch
Copy link

Ghidra process is terminated while analyzing AARCH64 bare metal image (same result for ARM as well) :

Execution of Ghidra plugin failed: Process was terminated.
INFO REPORT: Save succeeded for file: /security.abs (HeadlessAnalyzer)
INFO REPORT: Post-analysis succeeded for file: /input/security.abs (HeadlessAnalyzer)
@Enkelmann
Copy link
Collaborator

My standard test file for bare-metal analyses still works and it is also an ARM binary. So I will need more information to debug this. Can you share the bare-metal-config file that you used? Ideally, can you also share a binary where the error occurs? Although I know that it is often not possible to share binaries.

@amarkovytch
Copy link
Author

If this is the same test from bare_metal_samples dir, it is a bin file (probably a raw memory dump). I am trying to execute with ELF.
Did you ever try to test cwe_checker with ELF ?
I've tried a couple of other ELFs (from the same project, some arm and some aarch64) with the same result.
Tried without bare metal config and tried with (though using bare metal config json with ELF seems pointless) . The results are same.

Anyway here is the json file I tried with (though as I pointed above the results were the same when the config file was not supplied) :

{
"_comment": "The CPU architecture of the chip. Valid values are those that Ghidra accepts as processor IDs.",
"processor_id": "AARCH64:LE:64:v8A",
"_comment_1": "The base address, where the contents of the binary would be mapped on the chip, as a hexadecimal number.",
"flash_base_address": "0x00010000",
"_comment_2": "The base address, of the RAM memory region as a hexadecimal number.",
"ram_base_address": "0x20000000",
"_comment_3": "The size of the RAM memory region (in bytes) as a hexadecimal number.",
"ram_size": "0x00030000"
}

@Enkelmann
Copy link
Collaborator

Hmm, your config file seems fine.

Yes, the bare-metal-config option is meant for raw memory dumps and the like. But you should not use it for ELF files (where it is unnecessary anyway). But if you have the same problem in any combination then it does not seem to be related to the bare-metal-config option anyway. Since I cannot reproduce your problem on my end, I can only guess that something is wrong with either your installation of Ghidra or the cwe_checker. You could try to cleanly remove both (use make uninstall for the cwe_checker) and then reinstall them. And maybe make sure that you use the newest master branch of the cwe_checker. The other possible reason I can think of could be an unsupported system on which you are running the cwe_checker. What OS are you using?

@amarkovytch
Copy link
Author

amarkovytch commented Mar 4, 2022 via email

@Enkelmann
Copy link
Collaborator

So the file format you mean is not the Executable and Linkable Format? Regardless, the cwe_checker should not crash when using the bare-metal-option on any file format, so it is still a bug in the cwe_checker.

Ok, your environment seems to be fine, the cwe_checker works for other files... So I have to guess again that it is a bug related to the particular files you try it with.
Unfortunately, we do not have a good way to produce debug information for errors occurring in the Ghidra plugin part of the cwe_checker. To generate more logs you would have to print them by modifying the source code in the get_project_from_ghidra function in src/caller/src/main.rs. But from my experience getting the complete output from Ghidra may still not be enough to correctly identify the issue.
If you happen to know what development environment was used to generate your samples, I could try to get in and build some simple example binaries with it and see whether they also trigger the bug. Otherwise I am somewhat at my wits end on how to help you without a sample to reproduce the error myself.

@amarkovytch
Copy link
Author

amarkovytch commented Mar 5, 2022 •
edited

So the file format you mean is not the Executable and Linkable Format? Regardless, the cwe_checker should not crash when using the bare-metal-option on any file format, so it is still a bug in the cwe_checker.

The file format is ELF, I only said that it is not a Linux application but rather standalone image compiled for the embedded target. Also, we already agreed that "bare-metal" is not an issue here, since same result occurs regardless whether this option is used.

I've added some prints into main.rs, just before Ghidra headless execution and immediately after. Here are the results :

andrey@andrey-n1 ~/R/p/cwe_checker_test [1]> cwe_checker security.abs
Executing Ghidra plugin: "/home/andrey/Reverse/ghidra_10.1.2_PUBLIC/support/analyzeHeadless" "/mnt/wslg/runtime-dir/cwe_checker" "PcodeExtractor_security.abs_1646509917984" "-import" "security.abs" "-postScript" "/home/andrey/.local/share/cwe_checker/ghidra/p_code_extractor/PcodeExtractor.java" "/mnt/wslg/runtime-dir/cwe_checker/pcode_1646509917984.pipe" "-scriptPath" "/home/andrey/.local/share/cwe_checker/ghidra/p_code_extractor" "-deleteProject" "-analysisTimeoutPerFile" "3600"

The output was: INFO Using log config file: jar:file:/home/andrey/Reverse/ghidra_10.1.2_PUBLIC/Ghidra/Framework/Generic/lib/Generic.jar!/generic.log4j.xml (LoggingInitialization)
INFO Using log file: /home/andrey/.ghidra/.ghidra_10.1.2_PUBLIC/application.log (LoggingInitialization)
INFO Loading user preferences: /home/andrey/.ghidra/.ghidra_10.1.2_PUBLIC/preferences (Preferences)
INFO Class search complete (1527 ms) (ClassSearcher)
INFO Initializing SSL Context (SSLContextInitializer)
INFO Initializing Random Number Generator... (SecureRandomFactory)
INFO Random Number Generator initialization complete: NativePRNGNonBlocking (SecureRandomFactory)
INFO Trust manager disabled, cacerts have not been set (ApplicationTrustManagerFactory)
INFO HEADLESS Script Paths:
/home/andrey/Reverse/ghidra_10.1.2_PUBLIC/Ghidra/Features/Base/ghidra_scripts
/home/andrey/Reverse/ghidra_10.1.2_PUBLIC/Ghidra/Features/FunctionID/ghidra_scripts
/home/andrey/Reverse/ghidra_10.1.2_PUBLIC/Ghidra/Features/BytePatterns/ghidra_scripts
/home/andrey/Reverse/ghidra_10.1.2_PUBLIC/Ghidra/Processors/8051/ghidra_scripts
/home/andrey/Reverse/ghidra_10.1.2_PUBLIC/Ghidra/Features/GnuDemangler/ghidra_scripts
/home/andrey/Reverse/ghidra_10.1.2_PUBLIC/Ghidra/Features/MicrosoftCodeAnalyzer/ghidra_scripts
/home/andrey/.local/share/cwe_checker/ghidra/p_code_extractor
/home/andrey/Reverse/ghidra_10.1.2_PUBLIC/Ghidra/Features/Python/ghidra_scripts
/home/andrey/Reverse/ghidra_10.1.2_PUBLIC/Ghidra/Features/FileFormats/ghidra_scripts
/home/andrey/Reverse/ghidra_10.1.2_PUBLIC/Ghidra/Features/VersionTracking/ghidra_scripts
/home/andrey/Reverse/ghidra_10.1.2_PUBLIC/Ghidra/Processors/DATA/ghidra_scripts
/home/andrey/Reverse/ghidra_10.1.2_PUBLIC/Ghidra/Processors/PIC/ghidra_scripts
/home/andrey/Reverse/ghidra_10.1.2_PUBLIC/Ghidra/Debug/Debugger-agent-dbgmodel-traceloader/ghidra_scripts
/home/andrey/Reverse/ghidra_10.1.2_PUBLIC/Ghidra/Features/Decompiler/ghidra_scripts
/home/andrey/Reverse/ghidra_10.1.2_PUBLIC/Ghidra/Debug/Debugger/ghidra_scripts (HeadlessAnalyzer)
INFO HEADLESS: execution starts (HeadlessAnalyzer)
INFO Creating temporary project: /mnt/wslg/runtime-dir/cwe_checker/PcodeExtractor_security.abs_1646509917984 (HeadlessAnalyzer)
INFO Creating project: /mnt/wslg/runtime-dir/cwe_checker/PcodeExtractor_security.abs_1646509917984 (DefaultProject)
INFO REPORT: Processing input files: (HeadlessAnalyzer)
INFO project: /mnt/wslg/runtime-dir/cwe_checker/PcodeExtractor_security.abs_1646509917984 (HeadlessAnalyzer)
INFO IMPORTING: /home/andrey/Reverse/playground/cwe_checker_test/security.abs (HeadlessAnalyzer)
INFO REPORT: Import succeeded with language "AARCH64:LE:64:v8A" and cspec "default" for file: /home/andrey/Reverse/playground/cwe_checker_test/security.abs (HeadlessAnalyzer)
INFO ANALYZING all memory and code: /home/andrey/Reverse/playground/cwe_checker_test/security.abs (HeadlessAnalyzer)
INFO Packed database cache: /tmp/andrey-Ghidra/packed-db-cache (PackedDatabaseCache)
INFO -----------------------------------------------------
AARCH64 ELF PLT Thunks 0.000 secs
ASCII Strings 1.352 secs
Apply Data Archives 1.361 secs
Basic Constant Reference Analyzer 25.987 secs
Call Convention ID 2.112 secs
Call-Fixup Installer 0.205 secs
Create Address Tables 1.650 secs
Create Function 0.058 secs
Data Reference 0.327 secs
Decompiler Switch Analysis 3.150 secs
Demangler GNU 4.411 secs
Disassemble 0.024 secs
Disassemble Entry Points 14.478 secs
Disassemble Entry Points - One Time 0.114 secs
Embedded Media 0.068 secs
External Entry References 0.071 secs
Function Start Search 0.490 secs
Function Start Search After Code 0.120 secs
Function Start Search After Data 0.129 secs
Function Start Search delayed - One Time 0.077 secs
Non-Returning Functions - Discovered 1.500 secs
Non-Returning Functions - Known 0.128 secs
Reference 0.563 secs
Shared Return Calls 0.463 secs
Stack 19.913 secs
Subroutine References 0.364 secs

 Total Time   79 secs

(AutoAnalysisManager)
INFO REPORT: Analysis succeeded for file: /home/andrey/Reverse/playground/cwe_checker_test/security.abs (HeadlessAnalyzer)
INFO SCRIPT: /home/andrey/.local/share/cwe_checker/ghidra/p_code_extractor/PcodeExtractor.java (HeadlessAnalyzer)
ERROR REPORT SCRIPT ERROR: ( /home/andrey/Reverse/playground/cwe_checker_test/security.abs ) /home/andrey/.local/share/cwe_checker/ghidra/p_code_extractor/PcodeExtractor.java : null (HeadlessAnalyzer) java.lang.NullPointerException
at internal.TermCreator.createVariable(TermCreator.java:192)
at internal.TermCreator.createDefTerm(TermCreator.java:177)
at PcodeExtractor.iteratePcode(PcodeExtractor.java:231)
at PcodeExtractor.analysePcodeBlockOfAssemblyInstruction(PcodeExtractor.java:200)
at PcodeExtractor.iterateInstructions(PcodeExtractor.java:142)
at PcodeExtractor.iterateBlocks(PcodeExtractor.java:110)
at PcodeExtractor.iterateFunctions(PcodeExtractor.java:86)
at PcodeExtractor.run(PcodeExtractor.java:54)
at ghidra.app.script.GhidraScript.executeNormal(GhidraScript.java:379)
at ghidra.app.script.GhidraScript.doExecute(GhidraScript.java:234)
at ghidra.app.script.GhidraScript.execute(GhidraScript.java:212)
at ghidra.app.util.headless.HeadlessAnalyzer.runScript(HeadlessAnalyzer.java:576)
at ghidra.app.util.headless.HeadlessAnalyzer.runScriptsList(HeadlessAnalyzer.java:909)
at ghidra.app.util.headless.HeadlessAnalyzer.analyzeProgram(HeadlessAnalyzer.java:1057)
at ghidra.app.util.headless.HeadlessAnalyzer.processFileWithImport(HeadlessAnalyzer.java:1550)
at ghidra.app.util.headless.HeadlessAnalyzer.processWithImport(HeadlessAnalyzer.java:1688)
at ghidra.app.util.headless.HeadlessAnalyzer.processWithImport(HeadlessAnalyzer.java:1753)
at ghidra.app.util.headless.HeadlessAnalyzer.processLocal(HeadlessAnalyzer.java:445)
at ghidra.app.util.headless.AnalyzeHeadless.launch(AnalyzeHeadless.java:121)
at ghidra.GhidraLauncher.launch(GhidraLauncher.java:59)
at ghidra.Ghidra.main(Ghidra.java:47)

INFO ANALYZING changes made by post scripts: /home/andrey/Reverse/playground/cwe_checker_test/security.abs (HeadlessAnalyzer)
INFO REPORT: Post-analysis succeeded for file: /home/andrey/Reverse/playground/cwe_checker_test/security.abs (HeadlessAnalyzer)
INFO REPORT: Save succeeded for file: /security.abs (HeadlessAnalyzer)

Execution of Ghidra plugin failed: Process was terminated.
INFO REPORT: Save succeeded for file: /security.abs (HeadlessAnalyzer)
INFO REPORT: Post-analysis succeeded for file: /home/andrey/Reverse/playground/cwe_checker_test/security.abs (HeadlessAnalyzer)
andrey@andrey-n1 ~/R/p/cwe_checker_test [101]>

@Enkelmann
Copy link
Collaborator

Ok, it seems that the crash is due to a varnode in Ghidra P-Code that is marked as a register but has no name. The next step in debugging this would be to find out for which assembly instruction this happened and look at the raw P-Code generated for that instruction. For this one could add some debug printing to the createDefTerm function in src/ghidra/p_code_extractor/internal/TermCreator.java. Again sorry that no internal debug printing mechanism for that sort of problem exists yet in the cwe_checker.

By the way, the reason why the cwe_checker crashes for all your examples is probably that they all contain the same specific assembly instruction that our P-Code-extractor cannot handle yet. As another side note: When you have bare-metal samples in the ELF file format you do not need the bare-metal-config option. This config only exists for providing information that we can also parse from the ELF headers if they are available.

@amarkovytch
Copy link
Author

amarkovytch commented Mar 8, 2022 •
edited

No worries, we will debug this together :)
And yes, I already guessed that bare-metal-config is used by Ghidra to parse the binary if the information is not available otherwise (i.e. in ELF).

I've added the following print to createDefTerm just before the call to createVariable :
System.out.printf("DEBUG: looking at instruction %s\n", PcodeBlockData.pcodeOp); return new Term<Def>(defTid, new Def(createVariable(PcodeBlockData.pcodeOp.getOutput()), createExpression(), PcodeBlockData.pcodeIndex));

Here is the log :

DEBUG: looking at instruction (register, 0x4048, 8) COPY (unique, 0x11f80, 8)
DEBUG: looking at instruction (unique, 0x6800, 8) COPY (register, 0x4098, 8)
DEBUG: looking at instruction (unique, 0x6800, 8) COPY (register, 0x4040, 8)
DEBUG: looking at instruction (register, 0x5000, 8) LOAD (const, 0x1b1, 8) , (unique, 0x6800, 8)
DEBUG: looking at instruction (register, 0x5008, 8) COPY (const, 0x0, 8)
ERROR REPORT SCRIPT ERROR: ( /home/andrey/Reverse/playground/cwe_checker_test/security.abs ) /home/andrey/.local/share/cwe_checker/ghidra/p_code_extractor/PcodeExtractor.java : null (HeadlessAnalyzer) java.lang.NullPointerException
at internal.TermCreator.createVariable(TermCreator.java:193)
at internal.TermCreator.createDefTerm(TermCreator.java:178)
at PcodeExtractor.iteratePcode(PcodeExtractor.java:231)
at PcodeExtractor.analysePcodeBlockOfAssemblyInstruction(PcodeExtractor.java:200)
at PcodeExtractor.iterateInstructions(PcodeExtractor.java:142)
at PcodeExtractor.iterateBlocks(PcodeExtractor.java:110)
at PcodeExtractor.iterateFunctions(PcodeExtractor.java:86)
at PcodeExtractor.run(PcodeExtractor.java:54)
at ghidra.app.script.GhidraScript.executeNormal(GhidraScript.java:379)
at ghidra.app.script.GhidraScript.doExecute(GhidraScript.java:234)
at ghidra.app.script.GhidraScript.execute(GhidraScript.java:212)
at ghidra.app.util.headless.HeadlessAnalyzer.runScript(HeadlessAnalyzer.java:576)
at ghidra.app.util.headless.HeadlessAnalyzer.runScriptsList(HeadlessAnalyzer.java:909)
at ghidra.app.util.headless.HeadlessAnalyzer.analyzeProgram(HeadlessAnalyzer.java:1057)
at ghidra.app.util.headless.HeadlessAnalyzer.processFileWithImport(HeadlessAnalyzer.java:1550)
at ghidra.app.util.headless.HeadlessAnalyzer.processWithImport(HeadlessAnalyzer.java:1688)
at ghidra.app.util.headless.HeadlessAnalyzer.processWithImport(HeadlessAnalyzer.java:1753)
at ghidra.app.util.headless.HeadlessAnalyzer.processLocal(HeadlessAnalyzer.java:445)
at ghidra.app.util.headless.AnalyzeHeadless.launch(AnalyzeHeadless.java:121)
at ghidra.GhidraLauncher.launch(GhidraLauncher.java:59)
at ghidra.Ghidra.main(Ghidra.java:47)

So the issue is with
DEBUG: looking at instruction (register, 0x5008, 8) COPY (const, 0x0, 8)

Could it be that this is a unique combination of parameters not currently supported ?

@Enkelmann
Copy link
Collaborator

Enkelmann commented Mar 9, 2022 •
edited

Thank you very much, this should be enough information for me to develop a fix! I do not know yet whether I can solve this with a quick fix or whether this needs some larger code refactoring to be properly solved, so no promises on how long it will take me to fix it.

The thing that is not supported by the cwe_checker is (register, 0x5008, 8). If I am not mistaken this addresses the upper 8 bytes of the q0 floating point register. The cwe_checker expects that each accessed sub-span of a register is a sub-register with a human-readable name (e.g. s0 is a sub-register of q0). But since the upper bytes of a floating point register do not make much sense, the upper bytes of q0 have no human-readable name, which currently leads to the crash in the cwe_checker that you experienced.

It would be interesting to know what instruction caused this P-Code to be generated. Normally I would expect an INT_ZEXT instruction to be generated instead of addressing an unnamed sub-register directly. So this could also be a bug in Ghidra (in addition to causing a bug in the cwe_checker). But the P-Code still makes sense, so I will have to assume that it is correct P-Code. If you want to look up what instruction caused all this you could debug-print the defTid, it contains the address of the corresponding assembly instruction.

@amarkovytch
Copy link
Author

amarkovytch commented Mar 9, 2022 •
edited

Hi, Enkelmann, here is the log you requested, I've added an additional print that outputs the address of the original code immediately after the previous debug line :

image

Here is the new output :

image

It looks that the original instruction got translated into 3 P-code instructions :

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Enkelmann @amarkovytch