Do not use a shell in `proc_open()` if not really needed #5766

staabm · 2024-03-23T07:49:51Z

in my testing I can confirm on PHP 8.3 the process starts 1-2ms faster.
a hello world process isolation test on my machine runs in 46-47ms, so 1-2ms faster is not too bad actually.

requires sebastianbergmann/environment#72

staabm · 2024-03-23T14:56:09Z

@MasonM could you please double check whether this PR does not introduce regressions? could you run it against your existing test-suite (on non windows machines; I have a look into windows support when I get back to a win laptop)?

src/Util/PHP/DefaultPhpProcess.php

crrodriguez · 2024-03-24T17:52:02Z

Note that any advantage in performance is highly environment dependant. you might not see any difference with older PHP versions, old OS versions or any combination thereof.

MasonM

@staabm Okay, I manually applied this patch to our codebase. It works, but the performance improvement is indistinguishable from noise (which is going to be the case for basically anything that isn't a 5%+ improvement).

Before:

$ uname -a
Linux 485b0600476b 6.6.16-linuxkit #1 SMP Fri Feb 16 11:54:02 UTC 2024 aarch64 GNU/Linux

$ php -v
PHP 8.2.10 (cli) (built: Sep  4 2023 08:13:17) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.2.10, Copyright (c) Zend Technologies
    with Zend OPcache v8.2.10, Copyright (c), by Zend Technologies
    with Xdebug v3.2.1, Copyright (c) 2002-2023, by Derick Rethans

$ phpunit -c ./phpunit.xml-dist
PHPUnit 9.5.23 #StandWithUkraine

...........................................................    59 / 20179 (  0%)
<SNIP>
.                                                           20179 / 20179 (100%)

Time: 07:03.880, Memory: 776.00 MB

OK (20179 tests, 40817 assertions)

After:

$ phpunit -c ./phpunit.xml-dist
OK
PHPUnit 9.5.23 #StandWithUkraine

...........................................................    59 / 20179 (  0%)
<SNIP>
.                                                           20179 / 20179 (100%)

Time: 06:59.617, Memory: 776.00 MB

OK (20179 tests, 40817 assertions)

staabm · 2024-03-25T23:10:44Z

Thanks for testing @MasonM .

Since we are just improving frmework overhead even this few ms are nice as it summs up across all projects beeing run on CI with phpunit.

The forking change should improve a lot more but will take more work and time.

These changes cannot make a slow test fast. 😅

Thanks again

crrodriguez · 2024-03-25T23:28:45Z

@staabm Okay, I manually applied this patch to our codebase. It works, but the performance improvement is indistinguishable from noise (which is going to be the case for basically anything that isn't a 5%+ improvement).

Before:

$ uname -a
Linux 485b0600476b 6.6.16-linuxkit #1 SMP Fri Feb 16 11:54:02 UTC 2024 aarch64 GNU/Linux

$ php -v
PHP 8.2.10 (cli) (built: Sep  4 2023 08:13:17) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.2.10, Copyright (c) Zend Technologies
    with Zend OPcache v8.2.10, Copyright (c), by Zend Technologies
    with Xdebug v3.2.1, Copyright (c) 2002-2023, by Derick Rethans

$ phpunit -c ./phpunit.xml-dist
PHPUnit 9.5.23 #StandWithUkraine

...........................................................    59 / 20179 (  0%)
<SNIP>
.                                                           20179 / 20179 (100%)

Time: 07:03.880, Memory: 776.00 MB

OK (20179 tests, 40817 assertions)

After:

$ phpunit -c ./phpunit.xml-dist
OK
PHPUnit 9.5.23 #StandWithUkraine

...........................................................    59 / 20179 (  0%)
<SNIP>
.                                                           20179 / 20179 (100%)

Time: 06:59.617, Memory: 776.00 MB

OK (20179 tests, 40817 assertions)

That PHP is too old. most gains will be with 8.3 vs older releases. again it is highly environment dependant.

staabm · 2024-03-26T16:21:26Z

interessting find.. as soon as I add a sleep after $this->process() the failling tests start to succeed on windows:

diff --git a/src/Util/PHP/DefaultPhpProcess.php b/src/Util/PHP/DefaultPhpProcess.php
index b85e31048..df228024a 100644
--- a/src/Util/PHP/DefaultPhpProcess.php
+++ b/src/Util/PHP/DefaultPhpProcess.php
@@ -115,6 +115,7 @@ protected function runProcess(string $job, array $settings): array

         if ($job) {
             $this->process($pipes[0], $job);
+            sleep(1);
         }

         fclose($pipes[0]);

crrodriguez · 2024-03-26T17:30:43Z

interessting find.. as soon as I add a sleep after $this->process() the failling tests start to succeed on windows:

diff --git a/src/Util/PHP/DefaultPhpProcess.php b/src/Util/PHP/DefaultPhpProcess.php
index b85e31048..df228024a 100644
--- a/src/Util/PHP/DefaultPhpProcess.php
+++ b/src/Util/PHP/DefaultPhpProcess.php
@@ -115,6 +115,7 @@ protected function runProcess(string $job, array $settings): array

         if ($job) {
             $this->process($pipes[0], $job);
+            sleep(1);
         }

         fclose($pipes[0]);

so, there is a race condition.. output comes but not fast enough.. unfortunately at least according to the manual you can't stream_select on things returned by proc_open on windows.. :-|

staabm · 2024-03-27T08:17:03Z

my local testing suggests on windows we are now having 25-30ms less overhead per isolated-test:
php ./phpunit tests/end-to-end/regression/5764/5764.phpt --configuration tests/end-to-end/regression/5764/phpunit.xml --do-not-cache-result

overall end-to-end this means

after

mstaab@NB-COMPLEX-61 MINGW64 /c/dvl/Workspace/phpunit (no-shell)
$ php ./phpunit --testsuite end-to-end
PHPUnit 10.5.15-11-gfb8de2dcb by Sebastian Bergmann and contributors.

Runtime:       PHP 8.2.12
Configuration: C:\dvl\Workspace\phpunit\phpunit.xml

...............................................................  63 / 468 ( 13%)
...........SSSSSSS........................S..................S. 126 / 468 ( 26%)
.............................................S................. 189 / 468 ( 40%)
.............................S................................. 252 / 468 ( 53%)
............................................................... 315 / 468 ( 67%)
.............................................................S. 378 / 468 ( 80%)
...................................S.......S................... 441 / 468 ( 94%)
...S.......................                                     468 / 468 (100%)

Time: 01:39.220, Memory: 8.00 MB

OK, but some tests were skipped!
Tests: 468, Assertions: 453, Skipped: 15.

before

mstaab@NB-COMPLEX-61 MINGW64 /c/dvl/Workspace/phpunit (10.5)
$ php ./phpunit --testsuite end-to-end
PHPUnit 10.5.15-10-g406a2b48d by Sebastian Bergmann and contributors.

Runtime:       PHP 8.2.12
Configuration: C:\dvl\Workspace\phpunit\phpunit.xml

...............................................................  63 / 468 ( 13%)
...........SSSSSSS........................S..................S. 126 / 468 ( 26%)
.............................................S................. 189 / 468 ( 40%)
.............................S................................. 252 / 468 ( 53%)
............................................................... 315 / 468 ( 67%)
........................................................S....S. 378 / 468 ( 80%)
...................................S.......S................... 441 / 468 ( 94%)
...S.......................                                     468 / 468 (100%)

Time: 01:57.334, Memory: 8.00 MB

OK, but some tests were skipped!
Tests: 468, Assertions: 452, Skipped: 16.

=> ~25% faster on windows 🚀

$ php -v
PHP 8.2.12 (cli) (built: Oct 24 2023 21:15:35) (NTS Visual C++ 2019 x64)
Copyright (c) The PHP Group
Zend Engine v4.2.12, Copyright (c) Zend Technologies

as windows is no longer using a tempfile after this PR, we could now remove the whole tempfile logic which simplifies the implementation further. this could be done here in a separate commit or in a separate PR

codecov · 2024-03-28T05:46:52Z

Codecov Report

Attention: Patch coverage is 61.90476% with 8 lines in your changes are missing coverage. Please review.

Project coverage is 90.14%. Comparing base (f8941b6) to head (5c2cc5b).

❗ Current head 5c2cc5b differs from pull request most recent head eeddef5. Consider uploading reports for the commit eeddef5 to get more accurate results

Files	Patch %	Lines
src/Util/PHP/AbstractPhpProcess.php	53.33%	7 Missing ⚠️
src/Util/PHP/WindowsPhpProcess.php	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               10.5    #5766      +/-   ##
============================================
+ Coverage     90.09%   90.14%   +0.04%     
+ Complexity     6447     6443       -4     
============================================
  Files           680      680              
  Lines         19557    19543      -14     
============================================
- Hits          17620    17617       -3     
+ Misses         1937     1926      -11

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

staabm · 2024-03-28T08:45:00Z

did some more testing. I locally applied this PRs changes to the main-branch and used it to run tests of Roave/BetterReflection

I can see a 8% end-to-end improvement when running the test-suite.

I also used this same patched phpunit@main and run the phpunit 11 tests several times.
I can see the phpunit 11 testsuite is running ~15 seconds faster then before (so a simliar improvemt I already outlined a few comments above for the phpunit 10.x suite):

with this PR

$ php ./phpunit --testsuite end-to-end
PHPUnit 11.1-gba8e3adc4 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.2.12
Configuration: C:\dvl\Workspace\phpunit\phpunit.xml

...............................................................  63 / 498 ( 12%)
...............................S.........................S..... 126 / 498 ( 25%)
.............S..............................................S.. 189 / 498 ( 37%)
.................................................S............. 252 / 498 ( 50%)
............................................................... 315 / 498 ( 63%)
............................................................... 378 / 498 ( 75%)
...................S....................................S...... 441 / 498 ( 88%)
.S...............................S.......................       498 / 498 (100%)

Time: 01:50.388, Memory: 8.00 MB

OK, but some tests were skipped!
Tests: 498, Assertions: 489, Skipped: 9.

before

mstaab@NB-COMPLEX-61 MINGW64 /c/dvl/Workspace/phpunit (main)
$ php ./phpunit --testsuite end-to-end
PHPUnit 11.1-gb4a469305 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.2.12
Configuration: C:\dvl\Workspace\phpunit\phpunit.xml

...............................................................  63 / 498 ( 12%)
...............................S.........................S..... 126 / 498 ( 25%)
.............S..............................................S.. 189 / 498 ( 37%)
.................................................S............. 252 / 498 ( 50%)
............................................................... 315 / 498 ( 63%)
............................................................... 378 / 498 ( 75%)
..............S....S....................................S...... 441 / 498 ( 88%)
.S...............................S.......................       498 / 498 (100%)

Time: 02:06.768, Memory: 8.00 MB

OK, but some tests were skipped!
Tests: 498, Assertions: 488, Skipped: 10.

all testing done on windows11 with

$ php -v
PHP 8.2.12 (cli) (built: Oct 24 2023 21:15:35) (NTS Visual C++ 2019 x64)
Copyright (c) The PHP Group
Zend Engine v4.2.12, Copyright (c) Zend Technologies

staabm · 2024-03-28T08:48:53Z

as windows is no longer using a tempfile after this PR, we could now remove the whole tempfile logic which simplifies the implementation further. this could be done here in a separate commit or in a separate PR

I would suggest removing the now obsolete stuff in a separate PR, which I can open immediately after this is merged

sebastianbergmann · 2024-03-28T08:55:17Z

Thanks!

staabm force-pushed the no-shell branch 2 times, most recently from 7f2ab0b to 9c65505 Compare March 23, 2024 07:50

sebastianbergmann added type/performance Issues related to resource consumption (time and memory) feature/process-isolation Issues related to running tests in separate PHP processes labels Mar 23, 2024

staabm force-pushed the no-shell branch from 9c65505 to 75efa73 Compare March 23, 2024 12:27

sebastianbergmann changed the title ~~Do not use a shell in proc_open if not really needed~~ Do not use a shell in proc_open() if not really needed Mar 23, 2024

staabm force-pushed the no-shell branch from 6e596c3 to 1b60236 Compare March 23, 2024 15:15

crrodriguez reviewed Mar 23, 2024

View reviewed changes

src/Util/PHP/DefaultPhpProcess.php Show resolved Hide resolved

MasonM reviewed Mar 25, 2024

View reviewed changes

clxmstaab force-pushed the no-shell branch from f25151b to fb8de2d Compare March 27, 2024 07:57

staabm marked this pull request as ready for review March 27, 2024 08:04

staabm marked this pull request as draft March 27, 2024 10:13

sebastianbergmann force-pushed the no-shell branch from fb8de2d to fbf391a Compare March 28, 2024 06:31

staabm marked this pull request as ready for review March 28, 2024 08:42

staabm and others added 2 commits March 28, 2024 09:49

Do not use a shell in proc_open if not really needed

4e96332

Remove windows stdout workarounds

eeddef5

sebastianbergmann force-pushed the no-shell branch from 5c2cc5b to eeddef5 Compare March 28, 2024 08:49

sebastianbergmann merged commit 9320731 into sebastianbergmann:10.5 Mar 28, 2024
29 checks passed

staabm deleted the no-shell branch March 28, 2024 08:57

staabm mentioned this pull request Mar 28, 2024

Cleanup process handling after dropping temp-file handling #5772

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Do not use a shell in `proc_open()` if not really needed #5766

Do not use a shell in `proc_open()` if not really needed #5766

staabm commented Mar 23, 2024 •

edited

staabm commented Mar 23, 2024 •

edited

crrodriguez commented Mar 24, 2024

MasonM left a comment •

edited

staabm commented Mar 25, 2024 •

edited

crrodriguez commented Mar 25, 2024

staabm commented Mar 26, 2024 •

edited

crrodriguez commented Mar 26, 2024

staabm commented Mar 27, 2024 •

edited

codecov bot commented Mar 28, 2024 •

edited

staabm commented Mar 28, 2024 •

edited

staabm commented Mar 28, 2024

sebastianbergmann commented Mar 28, 2024

Do not use a shell in proc_open() if not really needed #5766

Do not use a shell in proc_open() if not really needed #5766

Conversation

staabm commented Mar 23, 2024 • edited

staabm commented Mar 23, 2024 • edited

crrodriguez commented Mar 24, 2024

MasonM left a comment • edited

Choose a reason for hiding this comment

staabm commented Mar 25, 2024 • edited

crrodriguez commented Mar 25, 2024

staabm commented Mar 26, 2024 • edited

crrodriguez commented Mar 26, 2024

staabm commented Mar 27, 2024 • edited

codecov bot commented Mar 28, 2024 • edited

Codecov Report

staabm commented Mar 28, 2024 • edited

staabm commented Mar 28, 2024

sebastianbergmann commented Mar 28, 2024

Do not use a shell in `proc_open()` if not really needed #5766

Do not use a shell in `proc_open()` if not really needed #5766

staabm commented Mar 23, 2024 •

edited

staabm commented Mar 23, 2024 •

edited

MasonM left a comment •

edited

staabm commented Mar 25, 2024 •

edited

staabm commented Mar 26, 2024 •

edited

staabm commented Mar 27, 2024 •

edited

codecov bot commented Mar 28, 2024 •

edited

staabm commented Mar 28, 2024 •

edited