Autoloader generation happens too late #7298

dalibor-matura · 2018-05-02T00:24:34Z

I have a project with a standard master branch and an update feature branch that contains some composer.json and composer.lock changes.

The main chagnes in my update branch are:

The ircmaxell/password-compat (version 1.0.4) is removed (https://github.com/ircmaxell/password_compat).
The dealerdirect/phpcodesniffer-composer-installer (version 0.4.4) is added (https://github.com/Dealerdirect/phpcodesniffer-composer-installer).

When I preform the following sequence of steps:

Switch to the master branch (no updates applied yet) and run the composer install.
Switch to the update branch and run the composer install.

I get the following Symfony exception:

[Symfony\Component\Process\Exception\ProcessFailedException]
...
Fatal error: require(): Failed opening required '/var/www/sites/my-project/vendor/composer/../ircmaxell/password-compat/lib/password.php' (include_path='/var/www/sites/my-project/vendor/phing/phing/classes:.:/usr/share/php') in /var/w  
  ww/sites/my-project/vendor/composer/autoload_real.php on line 70

I investigated it and found the cause:

In the fist step the Autoloader Files (/var/www/my-project/vendor/composer/autoload_files.php) contains the line with /ircmaxell/password-compat/lib/password.php requirement as the package ircmaxell/password-compat is installed.
In the second step when the composer install is running the package ircmaxell/password-compat is removed, but because the process is not finished yet, the Autoloader Files (/var/www/my-project/vendor/composer/autoload_files.php) are not regenerated as yet. The dealerdirect/phpcodesniffer-composer-installer package is installed and as it uses a Custom Installer: https://github.com/Dealerdirect/phpcodesniffer-composer-installer/blob/v0.4.4/composer.json#L44 as explained in here: https://getcomposer.org/doc/articles/custom-installers.md#composer-json, it runs the Autoloader that uses Autoloader Files, but the /var/www/sites/my-project/vendor/composer/../ircmaxell/password-compat/lib/password.php doesn't exists any more.

I'm looking for the solution:

This is causing my Travis CI build to fail (I was able to replicate it on my local environement as well) and I can't separate it into more fine grained steps as I need it in one deployment step to PROD.

I understand that the Autoloader regeneration happens at the end of composer install run, but it is too late.

The composer dump-autoload doesn't help as running it before composer install is no good and I don't know how to inject it into the composer install process after the ircmaxell/password-compat package is removed.

I may try some one time workaround by not allowing the dealerdirect/phpcodesniffer-composer-installer package to run its Custom Installer or to require the ircmaxell/password-compat and not removing it, even though it is not required any more and remove it later on.

None of the workarounds mentioned above is a clean solution that would work all the time.

Output of composer diagnose:

Checking composer.json: WARNING
require.acquia/lightning : exact version constraints (2.2.0) should be avoided if the package follows semantic versioning
require.drupal/core : exact version constraints (8.4.0) should be avoided if the package follows semantic versioning
require.drupal/entity_class_formatter : exact version constraints (1.x-dev) should be avoided if the package follows semantic versioning
require.drupal/entity_reference_display : exact version constraints (1.x-dev) should be avoided if the package follows semantic versioning
require.drupal/jsonapi : exact version constraints (1.10.0) should be avoided if the package follows semantic versioning
require.drupal/libraries : exact version constraints (3.x-dev) should be avoided if the package follows semantic versioning
require.drupal/maillog : exact version constraints (1.x-dev) should be avoided if the package follows semantic versioning
require.drupal/memcache : exact version constraints (2.x-dev) should be avoided if the package follows semantic versioning
require.drupal/migrate_plus : exact version constraints (4.0-beta1) should be avoided if the package follows semantic versioning
require.drupal/migrate_tools : exact version constraints (4.0-beta1) should be avoided if the package follows semantic versioning
require.drupal/modifiers : exact version constraints (1.x-dev) should be avoided if the package follows semantic versioning
require.drupal/modifiers_pack : exact version constraints (1.x-dev) should be avoided if the package follows semantic versioning
require.drupal/node_edit_protection : exact version constraints (1.x-dev) should be avoided if the package follows semantic versioning
require.drupal/page_manager : exact version constraints (4.0-beta2) should be avoided if the package follows semantic versioning
require.drupal/pathauto : exact version constraints (1.0.0) should be avoided if the package follows semantic versioning
require.drupal/r4032login : exact version constraints (1.x-dev) should be avoided if the package follows semantic versioning
require.drupal/security_review : unbound version constraints (*) should be avoided
require.drupal/simple_oauth : exact version constraints (3.3) should be avoided if the package follows semantic versioning
require.drupal/viewfield : exact version constraints (3.x-dev) should be avoided if the package follows semantic versioning
Checking platform settings: OK
Checking git settings: OK
Checking http connectivity to packagist: OK
Checking https connectivity to packagist: OK
Checking github.com rate limit: OK
Checking disk free space: OK
Checking pubkeys: 
Tags Public Key Fingerprint: 57815BA2 7E54DC31 7ECC7CC5 573090D0  87719BA6 8F3BB723 4E5D42D0 84A14642
Dev Public Key Fingerprint: 4AC45767 E5EC2265 2F0C1167 CBBB8A2B  0C708369 153E328C AD90147D AFE50952
OK
Checking composer version: OK
Composer version: 1.6.4
PHP version: 7.1.16-1+ubuntu16.04.1+deb.sury.org+1
PHP binary path: /usr/bin/php7.1

The text was updated successfully, but these errors were encountered:

dalibor-matura · 2018-05-02T00:26:49Z

My understanding it is likely a Composer bug or a gap in its workflow.

It is also possible that one of the packages: ircmaxell/password-compat, dealerdirect/phpcodesniffer-composer-installer is doing something wrong, but I do not know what otherwise I woudl raise an issue in their issue queue.

dalibor-matura · 2018-05-02T00:30:33Z

FYI: My project is Drupal site and I update from Drupal core 8.3.x to 8.4.x togethere with BLT and Acquia Lightning update, but this information should be irelevant to the problem I described above. The problem is pretty generic and in my understanding should happen for any combination of packages being removed together with added package(s) running their Custom Installers.

dalibor-matura · 2018-05-02T00:37:04Z

I'm running the latest stable version of Composer (1.6.4).

Plus I'm running the commands in my Travis CI before the composer install is run:

$ composer self-update
You are already using composer version 1.6.4 (stable channel).
$ composer validate --no-check-all --ansi
./composer.json is valid

dalibor-matura · 2018-05-02T02:06:34Z

Further investigation of composer install process:

I can see that the Uninstall operations are being moved before the Installe or Update operations: https://github.com/composer/composer/blob/1.6.4/src/Composer/Installer.php#L498

It would be probably sensible to run the dump-autoload also after the all Uninstall operations are finished. Maybe having it optional by a parameter.

P.S. The workaround having the ircmaxell/password-compat still required didn't help as there are other packages uninstalled, meaning that the package ircmaxell/password-compat is not doing anything wrong.

stof · 2018-05-02T10:59:52Z

The issue seems related to dealerdirect/phpcodesniffer-composer-installer. Can you give the full stack trace of the exception you are getting (run composer in verbose mode with -v for that) to help identifying what is getting wrong exactly ?

dalibor-matura · 2018-05-03T00:07:17Z

As a workaround I solved the problem by putting the:

"scripts": {
    "pre-package-install": "composer dump-autoload"
}

into the project's composer.json file with a plan to remove it in a next deployment step as it slows the composer install quite a lot.

From what I observed: When a package is removed or updated, the Autoloader files might get out of sync (containing path that doesn't exist anymore) and the Autloader is not udpated before the composer install process is finished. And thus a Custom Installer defined by a package might get into trouble as the Autoloader run will fail if it is out of sync. I haven't found any information that the package should regenerate Autoloader when using Custom Installer (https://getcomposer.org/doc/articles/custom-installers.md) so I woul still say it is a bug in a Composer workflow.

What I believe Composer should do is to keep track when a package using Autoloading (defining classes with Autoloader) is removed/updated and rebuild Autoloader in case soma package is going to use Custom Installer or any call that is going to use Autoloader.

Thanks @stof for your comment. I've already used a workaround and I do not have a time right now to go back and replicate the problem again, but the best log I have is here (the error happens at the time when the Custom Installer is run by dealerdirect/phpcodesniffer-composer-installer):

Call Stack:
      0.0002     366480   1. {main}() /home/travis/build/morpht/my-project/vendor/squizlabs/php_codesniffer/scripts/phpcs:0
      0.0025     585336   2. include_once('/home/travis/build/morpht/my-project/vendor/squizlabs/php_codesniffer/CodeSniffer/CLI.php') /home/travis/build/morpht/my-project/vendor/squizlabs/php_codesniffer/scripts/phpcs:19
      0.0026     587016   3. include_once('/home/travis/build/morpht/my-project/vendor/autoload.php') /home/travis/build/morpht/my-project/vendor/squizlabs/php_codesniffer/CodeSniffer/CLI.php:27
      0.0029     602768   4. ComposerAutoloaderInit21af4d8172481d466cdf9b498e03efb7::getLoader() /home/travis/build/morpht/my-project/vendor/autoload.php:7
      0.0087    1145880   5. composerRequire21af4d8172481d466cdf9b498e03efb7() /home/travis/build/morpht/my-project/vendor/composer/autoload_real.php:60



  Error Output:
  ================
  PHP Warning:  require(/home/travis/build/morpht/my-project/vendor/composer/../ircmaxell/password-compat/lib/password.php): failed to open stream: No such file or directory in /home/travis/build/morpht/my-project/vendor/composer/autolo
  ad_real.php on line 70
  PHP Stack trace:
  PHP   1. {main}() /home/travis/build/morpht/my-project/vendor/squizlabs/php_codesniffer/scripts/phpcs:0
  PHP   2. include_once() /home/travis/build/morpht/my-project/vendor/squizlabs/php_codesniffer/scripts/phpcs:19
  PHP   3. include_once() /home/travis/build/morpht/my-project/vendor/squizlabs/php_codesniffer/CodeSniffer/CLI.php:27
  PHP   4. ComposerAutoloaderInit21af4d8172481d466cdf9b498e03efb7::getLoader() /home/travis/build/morpht/my-project/vendor/autoload.php:7
  PHP   5. composerRequire21af4d8172481d466cdf9b498e03efb7() /home/travis/build/morpht/my-project/vendor/composer/autoload_real.php:60
  PHP Fatal error:  require(): Failed opening required '/home/travis/build/morpht/my-project/vendor/composer/../ircmaxell/password-compat/lib/password.php' (include_path='/home/travis/build/morpht/my-project/vendor/phing/phing/classes:.
  :/home/travis/.phpenv/versions/7.1.17/share/pear') in /home/travis/build/morpht/my-project/vendor/composer/autoload_real.php on line 70
  PHP Stack trace:
  PHP   1. {main}() /home/travis/build/morpht/my-project/vendor/squizlabs/php_codesniffer/scripts/phpcs:0
  PHP   2. include_once() /home/travis/build/morpht/my-project/vendor/squizlabs/php_codesniffer/scripts/phpcs:19
  PHP   3. include_once() /home/travis/build/morpht/my-project/vendor/squizlabs/php_codesniffer/CodeSniffer/CLI.php:27
  PHP   4. ComposerAutoloaderInit21af4d8172481d466cdf9b498e03efb7::getLoader() /home/travis/build/morpht/my-project/vendor/autoload.php:7
  PHP   5. composerRequire21af4d8172481d466cdf9b498e03efb7() /home/travis/build/morpht/my-project/vendor/composer/autoload_real.php:60


Exception trace:
 () at phar:///home/travis/.phpenv/versions/7.1.17/bin/composer/vendor/symfony/process/Process.php:221
 Symfony\Component\Process\Process->mustRun() at /home/travis/build/morpht/my-project/vendor/dealerdirect/phpcodesniffer-composer-installer/src/Plugin.php:183
 Dealerdirect\Composer\Plugin\Installers\PHPCodeSniffer\Plugin->loadInstalledPaths() at /home/travis/build/morpht/my-project/vendor/dealerdirect/phpcodesniffer-composer-installer/src/Plugin.php:121
 Dealerdirect\Composer\Plugin\Installers\PHPCodeSniffer\Plugin->init() at /home/travis/build/morpht/my-project/vendor/dealerdirect/phpcodesniffer-composer-installer/src/Plugin.php:103
 Dealerdirect\Composer\Plugin\Installers\PHPCodeSniffer\Plugin->activate() at phar:///home/travis/.phpenv/versions/7.1.17/bin/composer/src/Composer/Plugin/PluginManager.php:236
 Composer\Plugin\PluginManager->addPlugin() at phar:///home/travis/.phpenv/versions/7.1.17/bin/composer/src/Composer/Plugin/PluginManager.php:205
 Composer\Plugin\PluginManager->registerPackage() at phar:///home/travis/.phpenv/versions/7.1.17/bin/composer/src/Composer/Installer/PluginInstaller.php:63
 Composer\Installer\PluginInstaller->install() at phar:///home/travis/.phpenv/versions/7.1.17/bin/composer/src/Composer/Installer/InstallationManager.php:173
 Composer\Installer\InstallationManager->install() at phar:///home/travis/.phpenv/versions/7.1.17/bin/composer/src/Composer/Installer/InstallationManager.php:160
 Composer\Installer\InstallationManager->execute() at phar:///home/travis/.phpenv/versions/7.1.17/bin/composer/src/Composer/Installer.php:588
 Composer\Installer->doInstall() at phar:///home/travis/.phpenv/versions/7.1.17/bin/composer/src/Composer/Installer.php:227
 Composer\Installer->run() at phar:///home/travis/.phpenv/versions/7.1.17/bin/composer/src/Composer/Command/InstallCommand.php:119
 Composer\Command\InstallCommand->execute() at phar:///home/travis/.phpenv/versions/7.1.17/bin/composer/vendor/symfony/console/Command/Command.php:242
 Symfony\Component\Console\Command\Command->run() at phar:///home/travis/.phpenv/versions/7.1.17/bin/composer/vendor/symfony/console/Application.php:843
 Symfony\Component\Console\Application->doRunCommand() at phar:///home/travis/.phpenv/versions/7.1.17/bin/composer/vendor/symfony/console/Application.php:193
 Symfony\Component\Console\Application->doRun() at phar:///home/travis/.phpenv/versions/7.1.17/bin/composer/src/Composer/Console/Application.php:251
 Composer\Console\Application->doRun() at phar:///home/travis/.phpenv/versions/7.1.17/bin/composer/vendor/symfony/console/Application.php:117
 Symfony\Component\Console\Application->run() at phar:///home/travis/.phpenv/versions/7.1.17/bin/composer/src/Composer/Console/Application.php:100
 Composer\Console\Application->run() at phar:///home/travis/.phpenv/versions/7.1.17/bin/composer/bin/composer:58
 require() at /home/travis/.phpenv/versions/7.1.17/bin/composer:24

dalibor-matura · 2018-05-17T00:57:17Z

This is still biting us a lot. Can someone help us here?

We have troubles even with the workaround of running composer dump-autoload in pre-package-install script/hook.

I believe this is a genuine composer bug.

Seldaek · 2018-05-31T15:23:32Z

Composer does generate an in-memory autoloader for script/plugins when needed, but indeed we don't dump a new on disk autoloader until everything is complete. The only way to fix this here would be to re-dump after every installation like you do, which is really inefficient.

The issue here as I understand it is not that you use a custom installer, because that shouldn't load the autoloader from vendor dir, but rather that you are executing php-cs during installation, and php-cs does include the autoloader that's in vendor dir and in a stale state. Could you delay the php-cs execution until after the whole install including autoload dumping was done?

dalibor-matura · 2018-06-01T02:05:57Z

Hi @Seldaek,

Thank you for your reply, it brings some light here.

The dealerdirect/phpcodesniffer-composer-installer (https://github.com/Dealerdirect/phpcodesniffer-composer-installer) package is not mine, so I'm not able to change it like that, but I may raise an issue in their issue queue.

Do I understand it right that it is a bug in dealerdirect/phpcodesniffer-composer-installer as the custom installer used by it executes php-cs and it does include the autoloader that's in vendor dir and in a stale state? Meaning custom installers are not meant to use the autoloader.

Is there any documentation saying that: "custom installers are not meant to use the autoloader"? It whould help me to stress out that the dealerdirect/phpcodesniffer-composer-installer has a bug in its custom installer.

Seldaek · 2018-06-01T07:48:22Z

There is no documentation about that, but it breaks the composer lifecycle and as such is a bad idea. You can link them here.

That said, from a quick glance at the code it appears to only react to post-install-cmd and post-update-cmd events, which should happen late enough that the autoloader has been dumped. So I am wondering if you're using an outdated version of that plugin, or if you perhaps have done something else in your code to trigger this problem. Maybe I'm just missing something though.

dalibor-matura · 2018-06-01T08:40:05Z

I'm using the currently latest stable version 0.4.4 and from my experience the custom installer of dealerdirect/phpcodesniffer-composer-installer causes the error immediately after the package is udpated and not after the whole composer update process is finished.

I do not not understand the dealerdirect/phpcodesniffer-composer-installer code implementation of composer plugin fully, but the public method DependenciesChangedEvent called by the events you've mentioned (https://github.com/Dealerdirect/phpcodesniffer-composer-installer/blob/master/src/Plugin.php#L131) is being triggered also in the public static run method: https://github.com/Dealerdirect/phpcodesniffer-composer-installer/blob/master/src/Plugin.php#L87 and this method might be run when the plugin implementation is initialized, at lest I guess.

EDIT: I've created an issue in dealerdirect/phpcodesniffer-composer-installer: PHPCSStandards/composer-installer#49, as you can see below.

stof · 2018-07-10T14:00:58Z

@Seldaek I found the issue (and commented on the plugin issue). It does not only run phpcs on the post_*_cmd events, but also on plugin initialization

dalibor-matura mentioned this issue Jun 1, 2018

The composer plugin implementation seems to be breaking the composer lifecycle PHPCSStandards/composer-installer#49

Closed

greg-1-anderson mentioned this issue Jun 8, 2018

Error in post install script drupal-composer/drupal-scaffold#79

Closed

alcohol added the Support label Jul 10, 2018

Seldaek closed this as completed Jul 17, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Autoloader generation happens too late #7298

Autoloader generation happens too late #7298

dalibor-matura commented May 2, 2018

dalibor-matura commented May 2, 2018

dalibor-matura commented May 2, 2018

dalibor-matura commented May 2, 2018

dalibor-matura commented May 2, 2018

stof commented May 2, 2018

dalibor-matura commented May 3, 2018

dalibor-matura commented May 17, 2018

Seldaek commented May 31, 2018

dalibor-matura commented Jun 1, 2018

Seldaek commented Jun 1, 2018

dalibor-matura commented Jun 1, 2018 •

edited

stof commented Jul 10, 2018

Autoloader generation happens too late #7298

Autoloader generation happens too late #7298

Comments

dalibor-matura commented May 2, 2018

dalibor-matura commented May 2, 2018

dalibor-matura commented May 2, 2018

dalibor-matura commented May 2, 2018

dalibor-matura commented May 2, 2018

stof commented May 2, 2018

dalibor-matura commented May 3, 2018

dalibor-matura commented May 17, 2018

Seldaek commented May 31, 2018

dalibor-matura commented Jun 1, 2018

Seldaek commented Jun 1, 2018

dalibor-matura commented Jun 1, 2018 • edited

stof commented Jul 10, 2018

dalibor-matura commented Jun 1, 2018 •

edited