Suboptimal discovery of project files

[dvz]

The initial scanning of project files may be suboptimal for projects with a large number of ignored files/directories within projectFiles:

<projectFiles>
    <directory name="." />
    <ignoreFiles>
        <directory name=".git" />
        <directory name="other-big-subdirectory" />
    </ignoreFiles>
</projectFiles>

This may manifest in a long delay between invoking the CLI and its first output, and may be responsible for the major part of Psalm's execution time when using cache.

• Ignored directories iterated over recursively
  The recursive iteration in:

  • FileProvider::getFilesInDir()
    (addressed by Improve project files discovery performance #7161)
  • FileFilter::loadFromArray()
    (addressed by Add explicit resolveSymlinks option for project directories #7163)

  doesn't take ignored locations into account, and all leaves (files) are unnecessarily processed in the loop.

• Repetitive discovery of project files
  The discovery of project files happens multiple times within the same search scope:

  • psalm/src/Psalm/Internal/Analyzer/ProjectAnalyzer.php

    Line 295 in 16a61a7

    $file_paths = $this->file_provider->getFilesInDir($dir_name, $file_extensions);

    
    (addressed by Improve project files discovery performance #7161)

  • psalm/src/Psalm/Internal/Analyzer/ProjectAnalyzer.php

    Line 1070 in 16a61a7

    $this->file_provider->getFilesInDir($dir_name, $file_extensions)

    
    (addressed by Fix parser cache files maintenance #7082)

  • psalm/src/Psalm/Internal/Analyzer/ProjectAnalyzer.php

    Line 1102 in 16a61a7

    $file_paths = $this->file_provider->getFilesInDir($dir_name, $file_extensions);

    
    (addressed by Improve project files discovery performance #7161)

  To improve performance, the list could be cached.

• Psalm cache directory not ignored
  Psalm's own cache directory is not excluded automatically when located within projectFiles, which may be unintuitive. When not ignored, it may significantly contribute to the execution time.

• Late filtering during file discovery
  FileProvider::getFilesInDir() already filters files according to extension, but other checks (like Config::isInProjectDirs() and Config::isInExtraDirs()) are done outside, by iterating over the list again.
  It may be beneficial to perform them in the first loop instead (e.g. using callbacks).

  (addressed by Add early file filtering to FileProvider::getFilesInDir() #7201)

[orklah]

Thanks for identifying those! don't hesitate to ping me or @weirdan if you have questions on how to proceed