Cross Translation Unit analysis fails to detect wrapped allocations

[sahil-time]

Hi folks,

• I use CodeChecker on the compile-commands.json
• The command that I use is:
  CodeChecker analyze compile_commands.json --enable sensitive --ctu --analyzers clangsa --jobs 16 --output <>
• However sometimes, it fails to detect wrapped allocations that are used and eventually free isnt called
• For instance, I have a libfoo.so which calls an allocation function in libbar.so, this allocation function is a wrapper over malloc. However a real leak in libfoo.so goes undetected.
• Is it because my command needs extra flags? or CTU cannot analyze all the paths?

[whisperity]

CTU needs source code to analyse. If your implementation is in third-party libraries without source code (i.e., not part of your project, not present in the compile_commands.json) then it won't know where to continue the analysis.

[sahil-time]

@whisperity

• my implementation is in the source code
• the compile_commands.json has the source files where the wrapped alloc is defined
• What I see is that for smaller functions and limited number of source files, CodeChecker does correct CTU analysis
• However, when I run it for a large number of files, it does NOT detect the leak
• Is there a max depth or something for the number of external TU's CodeChecker or ClangSA can scan?

It would be really helpful to get more information since we are completely moving our codebase SA to CodeChecker from Coverity :)

[whisperity]

There are several "budgets" (usually these are unitless scalar counters) internally in the Clang Static Analyzer, and if the budget for an action (such as loop unrolling, or inlining of functions, or the opening of external TUs) is exhausted, either that action will be skipped (and the analysis is continued as if the result of that action was "unknown") or the entire analysis of that call graph is stopped. Some of these budgets can be configured by passing the right combination of arguments directly to the Clang binary, but some of these budgets are hardcoded and need a custom build of Clang to have the changes apply. You can find the names and the meaning of such "engine flags" here: https://github.com/llvm/llvm-project/blob/main/clang/include/clang/StaticAnalyzer/Core/AnalyzerOptions.def. (I'm sure there is a command-line combination of --help flags that dump these as well, but I do not know the exact details.) There is something called ctu-max-nodes-pct and ctu-max-nodes-min, which fine-tune how many nodes (whatever a "node" is, likely these are basic blocks...?) are visited:

https://github.com/llvm/llvm-project/blob/e7a8dd9b0d419403fe1d8adeb177a4ec78e036cc/clang/include/clang/StaticAnalyzer/Core/AnalyzerOptions.def#L404-L414

The number of TUs loaded during an analysis is also limited by ctu-import-threshold and ctu-import-cpp-threshold. The reason behind this is very practical, and the usual speed-accuracy-memory-use-tradeoff problem: when you're analysing a main.cpp and importing a lib1.cpp, the AST for both main and lib1.cpp must be kept in memory during the merging process. A too high threshold, especially if the analysis is run in multiple parallel jobs, can exhaust memory very quickly.

https://github.com/llvm/llvm-project/blob/e7a8dd9b0d419403fe1d8adeb177a4ec78e036cc/clang/include/clang/StaticAnalyzer/Core/AnalyzerOptions.def#L353-L366

These flags you can query through CodeChecker like this: CodeChecker analyzers clangsa --analyzer-config and this will give you the list of flags your current Clang binary supports; and set them for an analysis like this: CodeChecker analyze ... --analyzer-config "clangsa:ctu-import-cpp-threshold=32" ....