Skip to content

Commit

Permalink
Reports: add end-to-end tests
Browse files Browse the repository at this point in the history
... for quite a few report types.

The Reports which PHPCS can generate, were until now not covered by tests and therefore had no safeguards/QA whatsoever.

This commit sets up an initial end-to-end test suite with tests for common report types.
It doesn't have an opinion on whether or not the reports as currently generated are formatted correctly.
For now, it just codifies the current functionality.

For lack of documentation about this format in the PHPUnit docs, the test docs for PHP Core are the closest available to explain the available sections and how to apply these: https://qa.php.net/phpt_details.php

With this initial setup done, it should be fairly straight-forward to expand this end-to-end test suite to cover more report types and to add tests for other command-line options as well.
Mind: these type of tests are generally slow, so unit tests/integration tests should be preferred.
  • Loading branch information
jrfnl committed Apr 18, 2023
1 parent ed8e00d commit 028bbdb
Show file tree
Hide file tree
Showing 22 changed files with 710 additions and 4 deletions.
62 changes: 62 additions & 0 deletions package.xml
Expand Up @@ -196,6 +196,32 @@ http://pear.php.net/dtd/package-2.0.xsd">
<file baseinstalldir="" name="ErrorSuppressionTest.php" role="test" />
<file baseinstalldir="" name="IsCamelCapsTest.php" role="test" />
</dir>
<dir name="EndToEnd">
<dir name="Fixtures">
<dir name="Reports">
<file baseinstalldir="" name="Dirty.php" role="test" />
<file baseinstalldir="" name="CleanClass.php" role="test" />
</dir>
</dir>
<dir name="Reports">
<file baseinstalldir="" name="checkstyle.phpt" role="test" />
<file baseinstalldir="" name="code-no-errorcodes.phpt" role="test" />
<file baseinstalldir="" name="code-with-errorcodes.phpt" role="test" />
<file baseinstalldir="" name="csv.phpt" role="test" />
<file baseinstalldir="" name="emacs-no-errorcodes.phpt" role="test" />
<file baseinstalldir="" name="emacs-with-errorcodes.phpt" role="test" />
<file baseinstalldir="" name="full-no-errorcodes.phpt" role="test" />
<file baseinstalldir="" name="full-with-errorcodes.phpt" role="test" />
<file baseinstalldir="" name="gitblame-no-errorcodes.phpt" role="test" />
<file baseinstalldir="" name="gitblame-with-errorcodes.phpt" role="test" />
<file baseinstalldir="" name="json.phpt" role="test" />
<file baseinstalldir="" name="junit.phpt" role="test" />
<file baseinstalldir="" name="source-no-errorcodes.phpt" role="test" />
<file baseinstalldir="" name="source-with-errorcodes.phpt" role="test" />
<file baseinstalldir="" name="summary.phpt" role="test" />
<file baseinstalldir="" name="xml.phpt" role="test" />
</dir>
</dir>
<dir name="Standards">
<file baseinstalldir="" name="AbstractSniffUnitTest.php" role="test" />
<file baseinstalldir="" name="AllSniffs.php" role="test" />
Expand Down Expand Up @@ -2169,6 +2195,24 @@ http://pear.php.net/dtd/package-2.0.xsd">
<install as="CodeSniffer/Core/Tokenizer/TypeIntersectionTest.inc" name="tests/Core/Tokenizer/TypeIntersectionTest.inc" />
<install as="CodeSniffer/Core/Tokenizer/UndoNamespacedNameSingleTokenTest.php" name="tests/Core/Tokenizer/UndoNamespacedNameSingleTokenTest.php" />
<install as="CodeSniffer/Core/Tokenizer/UndoNamespacedNameSingleTokenTest.inc" name="tests/Core/Tokenizer/UndoNamespacedNameSingleTokenTest.inc" />
<install as="CodeSniffer/EndToEnd/Fixtures/Reports/Dirty.php" name="tests/EndToEnd/Fixtures/Reports/Dirty.php" />
<install as="CodeSniffer/EndToEnd/Fixtures/Reports/CleanClass.php" name="tests/EndToEnd/Fixtures/Reports/CleanClass.php" />
<install as="CodeSniffer/EndToEnd/Reports/checkstyle.phpt" name="tests/EndToEnd/Reports/checkstyle.phpt" />
<install as="CodeSniffer/EndToEnd/Reports/code-no-errorcodes.phpt" name="tests/EndToEnd/Reports/code-no-errorcodes.phpt" />
<install as="CodeSniffer/EndToEnd/Reports/code-with-errorcodes.phpt" name="tests/EndToEnd/Reports/code-with-errorcodes.phpt" />
<install as="CodeSniffer/EndToEnd/Reports/csv.phpt" name="tests/EndToEnd/Reports/csv.phpt" />
<install as="CodeSniffer/EndToEnd/Reports/emacs-no-errorcodes.phpt" name="tests/EndToEnd/Reports/emacs-no-errorcodes.phpt" />
<install as="CodeSniffer/EndToEnd/Reports/emacs-with-errorcodes.phpt" name="tests/EndToEnd/Reports/emacs-with-errorcodes.phpt" />
<install as="CodeSniffer/EndToEnd/Reports/full-no-errorcodes.phpt" name="tests/EndToEnd/Reports/full-no-errorcodes.phpt" />
<install as="CodeSniffer/EndToEnd/Reports/full-with-errorcodes.phpt" name="tests/EndToEnd/Reports/full-with-errorcodes.phpt" />
<install as="CodeSniffer/EndToEnd/Reports/gitblame-no-errorcodes.phpt" name="tests/EndToEnd/Reports/gitblame-no-errorcodes.phpt" />
<install as="CodeSniffer/EndToEnd/Reports/gitblame-with-errorcodes.phpt" name="tests/EndToEnd/Reports/gitblame-with-errorcodes.phpt" />
<install as="CodeSniffer/EndToEnd/Reports/json.phpt" name="tests/EndToEnd/Reports/json.phpt" />
<install as="CodeSniffer/EndToEnd/Reports/junit.phpt" name="tests/EndToEnd/Reports/junit.phpt" />
<install as="CodeSniffer/EndToEnd/Reports/source-no-errorcodes.phpt" name="tests/EndToEnd/Reports/source-no-errorcodes.phpt" />
<install as="CodeSniffer/EndToEnd/Reports/source-with-errorcodes.phpt" name="tests/EndToEnd/Reports/source-with-errorcodes.phpt" />
<install as="CodeSniffer/EndToEnd/Reports/summary.phpt" name="tests/EndToEnd/Reports/summary.phpt" />
<install as="CodeSniffer/EndToEnd/Reports/xml.phpt" name="tests/EndToEnd/Reports/xml.phpt" />
<install as="CodeSniffer/Standards/AllSniffs.php" name="tests/Standards/AllSniffs.php" />
<install as="CodeSniffer/Standards/AbstractSniffUnitTest.php" name="tests/Standards/AbstractSniffUnitTest.php" />
</filelist>
Expand Down Expand Up @@ -2277,6 +2321,24 @@ http://pear.php.net/dtd/package-2.0.xsd">
<install as="CodeSniffer/Core/Tokenizer/TypeIntersectionTest.inc" name="tests/Core/Tokenizer/TypeIntersectionTest.inc" />
<install as="CodeSniffer/Core/Tokenizer/UndoNamespacedNameSingleTokenTest.php" name="tests/Core/Tokenizer/UndoNamespacedNameSingleTokenTest.php" />
<install as="CodeSniffer/Core/Tokenizer/UndoNamespacedNameSingleTokenTest.inc" name="tests/Core/Tokenizer/UndoNamespacedNameSingleTokenTest.inc" />
<install as="CodeSniffer/EndToEnd/Fixtures/Reports/Dirty.php" name="tests/EndToEnd/Fixtures/Reports/Dirty.php" />
<install as="CodeSniffer/EndToEnd/Fixtures/Reports/CleanClass.php" name="tests/EndToEnd/Fixtures/Reports/CleanClass.php" />
<install as="CodeSniffer/EndToEnd/Reports/checkstyle.phpt" name="tests/EndToEnd/Reports/checkstyle.phpt" />
<install as="CodeSniffer/EndToEnd/Reports/code-no-errorcodes.phpt" name="tests/EndToEnd/Reports/code-no-errorcodes.phpt" />
<install as="CodeSniffer/EndToEnd/Reports/code-with-errorcodes.phpt" name="tests/EndToEnd/Reports/code-with-errorcodes.phpt" />
<install as="CodeSniffer/EndToEnd/Reports/csv.phpt" name="tests/EndToEnd/Reports/csv.phpt" />
<install as="CodeSniffer/EndToEnd/Reports/emacs-no-errorcodes.phpt" name="tests/EndToEnd/Reports/emacs-no-errorcodes.phpt" />
<install as="CodeSniffer/EndToEnd/Reports/emacs-with-errorcodes.phpt" name="tests/EndToEnd/Reports/emacs-with-errorcodes.phpt" />
<install as="CodeSniffer/EndToEnd/Reports/full-no-errorcodes.phpt" name="tests/EndToEnd/Reports/full-no-errorcodes.phpt" />
<install as="CodeSniffer/EndToEnd/Reports/full-with-errorcodes.phpt" name="tests/EndToEnd/Reports/full-with-errorcodes.phpt" />
<install as="CodeSniffer/EndToEnd/Reports/gitblame-no-errorcodes.phpt" name="tests/EndToEnd/Reports/gitblame-no-errorcodes.phpt" />
<install as="CodeSniffer/EndToEnd/Reports/gitblame-with-errorcodes.phpt" name="tests/EndToEnd/Reports/gitblame-with-errorcodes.phpt" />
<install as="CodeSniffer/EndToEnd/Reports/json.phpt" name="tests/EndToEnd/Reports/json.phpt" />
<install as="CodeSniffer/EndToEnd/Reports/junit.phpt" name="tests/EndToEnd/Reports/junit.phpt" />
<install as="CodeSniffer/EndToEnd/Reports/source-no-errorcodes.phpt" name="tests/EndToEnd/Reports/source-no-errorcodes.phpt" />
<install as="CodeSniffer/EndToEnd/Reports/source-with-errorcodes.phpt" name="tests/EndToEnd/Reports/source-with-errorcodes.phpt" />
<install as="CodeSniffer/EndToEnd/Reports/summary.phpt" name="tests/EndToEnd/Reports/summary.phpt" />
<install as="CodeSniffer/EndToEnd/Reports/xml.phpt" name="tests/EndToEnd/Reports/xml.phpt" />
<install as="CodeSniffer/Standards/AllSniffs.php" name="tests/Standards/AllSniffs.php" />
<install as="CodeSniffer/Standards/AbstractSniffUnitTest.php" name="tests/Standards/AbstractSniffUnitTest.php" />
<ignore name="bin/phpcs.bat" />
Expand Down
1 change: 1 addition & 0 deletions phpcs.xml.dist
Expand Up @@ -10,6 +10,7 @@

<exclude-pattern>*/src/Standards/*/Tests/*\.(inc|css|js)$</exclude-pattern>
<exclude-pattern>*/tests/Core/*/*\.(inc|css|js)$</exclude-pattern>
<exclude-pattern>*/tests/EndToEnd/Fixtures/*</exclude-pattern>

<arg name="basepath" value="."/>
<arg name="colors"/>
Expand Down
4 changes: 4 additions & 0 deletions phpunit.xml.dist
Expand Up @@ -4,5 +4,9 @@
<testsuite name="PHP_CodeSniffer Test Suite">
<file>tests/AllTests.php</file>
</testsuite>
<testsuite name="End2End">
<directory suffix=".phpt">tests/EndToEnd/</directory>
<exclude>tests/EndToEnd/Fixtures</exclude>
</testsuite>
</testsuites>
</phpunit>
8 changes: 4 additions & 4 deletions scripts/ValidatePEAR/ValidatePEARPackageXML.php
Expand Up @@ -131,12 +131,12 @@ protected function checkContents()
$srcFiles = (new FileList(
$this->projectRoot.'src/',
$this->projectRoot,
'`\.(css|fixed|inc|js|php|xml)$`Di'
'`\.(css|fixed|inc|js|phpt|php|xml)$`Di'
))->getList();
$testsFiles = (new FileList(
$this->projectRoot.'tests/',
$this->projectRoot,
'`\.(css|inc|js|php|xml)$`Di'
'`\.(css|inc|js|phpt|php|xml)$`Di'
))->getList();
$files = array_merge($srcFiles, $testsFiles);

Expand Down Expand Up @@ -316,7 +316,7 @@ protected function checkPHPRelease()
}

// Check validity of the tags for files in the tests root subdirectories.
if (preg_match('`^tests/.+\.(php|inc|js|css|xml)$`', $name) === 1
if (preg_match('`^tests/.+\.(phpt|php|inc|js|css|xml)$`', $name) === 1
&& $as === str_replace('tests/', 'CodeSniffer/', $name)
) {
continue;
Expand All @@ -337,7 +337,7 @@ protected function checkPHPRelease()
* Verify that all files in the `tests` directory are listed in both `<phprelease>` tags.
*/

$testFiles = (new FileList($this->projectRoot.'tests/', $this->projectRoot, '`\.(inc|php|js|css|xml)$`Di'))->getList();
$testFiles = (new FileList($this->projectRoot.'tests/', $this->projectRoot, '`\.(inc|phpt|php|js|css|xml)$`Di'))->getList();

foreach ($testFiles as $file) {
foreach ($listedFiles as $key => $listed) {
Expand Down
12 changes: 12 additions & 0 deletions tests/EndToEnd/Fixtures/Reports/CleanClass.php
@@ -0,0 +1,12 @@
<?php
/*
* Test fixture file for End to End Report tests.
*/

namespace PHP_CodeSniffer\Tests\EndToEnd\Fixtures;

class CleanClass {
const UPPERCASE = true;

public function myMethod() {}
}
14 changes: 14 additions & 0 deletions tests/EndToEnd/Fixtures/Reports/Dirty.php
@@ -0,0 +1,14 @@
<?php
/*
* Test fixture file for End to End Report tests.
*/

class dirty_class {
const lowerCase = false;

public function My_Method() {}
}

class Second_class {}

$obj = new dirty_class();
31 changes: 31 additions & 0 deletions tests/EndToEnd/Reports/checkstyle.phpt
@@ -0,0 +1,31 @@
--TEST--
Report: Checkstyle

--SKIPIF--
<?php
if (is_file(__DIR__.'/../../../autoload.php') === false) {
print 'skip: Test cannot run from a PEAR install.';
}
?>
--ARGS--
./tests/EndToEnd/Fixtures/Reports/ -q --no-colors --report-width=80 --basepath=./tests/EndToEnd/Fixtures/Reports/ --standard=PSR1 --report=Checkstyle

--FILE--
<?php
require_once __DIR__ . '/../../../bin/phpcs';

--EXPECTF--
#!/usr/bin/env php
<?xml version="1.0" encoding="UTF-8"?>
<checkstyle version="%s">
<file name="Dirty.php">
<error line="1" column="1" severity="warning" message="A file should declare new symbols (classes, functions, constants, etc.) and cause no other side effects, or it should execute logic with side effects, but should not do both. The first symbol is defined on line 6 and the first side effect is on line 14." source="PSR1.Files.SideEffects.FoundWithSymbols"/>
<error line="6" column="1" severity="error" message="Each class must be in a namespace of at least one level (a top-level vendor name)" source="PSR1.Classes.ClassDeclaration.MissingNamespace"/>
<error line="6" column="1" severity="error" message="Class name &quot;dirty_class&quot; is not in PascalCase format" source="Squiz.Classes.ValidClassName.NotCamelCaps"/>
<error line="7" column="11" severity="error" message="Class constants must be uppercase; expected LOWERCASE but found lowerCase" source="Generic.NamingConventions.UpperCaseConstantName.ClassConstantNotUpperCase"/>
<error line="9" column="12" severity="error" message="Method name &quot;dirty_class::My_Method&quot; is not in camel caps format" source="PSR1.Methods.CamelCapsMethodName.NotCamelCaps"/>
<error line="12" column="1" severity="error" message="Each class must be in a file by itself" source="PSR1.Classes.ClassDeclaration.MultipleClasses"/>
<error line="12" column="1" severity="error" message="Each class must be in a namespace of at least one level (a top-level vendor name)" source="PSR1.Classes.ClassDeclaration.MissingNamespace"/>
<error line="12" column="1" severity="error" message="Class name &quot;Second_class&quot; is not in PascalCase format" source="Squiz.Classes.ValidClassName.NotCamelCaps"/>
</file>
</checkstyle>
73 changes: 73 additions & 0 deletions tests/EndToEnd/Reports/code-no-errorcodes.phpt
@@ -0,0 +1,73 @@
--TEST--
Report: Code, no error codes

--SKIPIF--
<?php
if (is_file(__DIR__.'/../../../autoload.php') === false) {
print 'skip: Test cannot run from a PEAR install.';
}
?>
--ARGS--
./tests/EndToEnd/Fixtures/Reports/ -q --no-colors --report-width=80 --basepath=./tests/EndToEnd/Fixtures/Reports/ --standard=PSR1 --report=Code

--FILE--
<?php
require_once __DIR__ . '/../../../bin/phpcs';

--EXPECTF--
#!/usr/bin/env php

FILE: Dirty.php
--------------------------------------------------------------------------------
FOUND 7 ERRORS AND 1 WARNING AFFECTING 5 LINES
--------------------------------------------------------------------------------
LINE 1: WARNING A file should declare new symbols (classes, functions,
constants, etc.) and cause no other side effects, or it should
execute logic with side effects, but should not do both. The
first symbol is defined on line 6 and the first side effect is
on line 14.
--------------------------------------------------------------------------------
>> 1: <?php
2: /*
3: * Test fixture file for End to End Report tests.
--------------------------------------------------------------------------------
LINE 6: ERROR Each class must be in a namespace of at least one level (a
top-level vendor name)
LINE 6: ERROR Class name "dirty_class" is not in PascalCase format
--------------------------------------------------------------------------------
4: */
5:%w
>> 6: class dirty_class {
7: const lowerCase = false;
8:
--------------------------------------------------------------------------------
LINE 7: ERROR Class constants must be uppercase; expected LOWERCASE but found
lowerCase
--------------------------------------------------------------------------------
5:%w
6: class dirty_class {
>> 7: const lowerCase = false;
8:%w
9: public function My_Method() {}
--------------------------------------------------------------------------------
LINE 9: ERROR Method name "dirty_class::My_Method" is not in camel caps
format
--------------------------------------------------------------------------------
7: const lowerCase = false;
8:%w
>> 9: public function My_Method() {}
10: }
11:
--------------------------------------------------------------------------------
LINE 12: ERROR Each class must be in a file by itself
LINE 12: ERROR Each class must be in a namespace of at least one level (a
top-level vendor name)
LINE 12: ERROR Class name "Second_class" is not in PascalCase format
--------------------------------------------------------------------------------
10: }
11:%w
>> 12: class Second_class {}
13:%w
14: $obj = new dirty_class();
--------------------------------------------------------------------------------
Time: %f secs; Memory: %dMB
79 changes: 79 additions & 0 deletions tests/EndToEnd/Reports/code-with-errorcodes.phpt
@@ -0,0 +1,79 @@
--TEST--
Report: Code, with error codes

--SKIPIF--
<?php
if (is_file(__DIR__.'/../../../autoload.php') === false) {
print 'skip: Test cannot run from a PEAR install.';
}
?>
--ARGS--
./tests/EndToEnd/Fixtures/Reports/ -qs --no-colors --report-width=80 --basepath=./tests/EndToEnd/Fixtures/Reports/ --standard=PSR1 --report=Code

--FILE--
<?php
require_once __DIR__ . '/../../../bin/phpcs';

--EXPECTF--
#!/usr/bin/env php

FILE: Dirty.php
--------------------------------------------------------------------------------
FOUND 7 ERRORS AND 1 WARNING AFFECTING 5 LINES
--------------------------------------------------------------------------------
LINE 1: WARNING A file should declare new symbols (classes, functions, constants,
etc.) and cause no other side effects, or it should execute logic with
side effects, but should not do both. The first symbol is defined on
line 6 and the first side effect is on line 14.
(PSR1.Files.SideEffects.FoundWithSymbols)
--------------------------------------------------------------------------------
>> 1: <?php
2: /*
3: * Test fixture file for End to End Report tests.
--------------------------------------------------------------------------------
LINE 6: ERROR Each class must be in a namespace of at least one level (a
top-level vendor name)
(PSR1.Classes.ClassDeclaration.MissingNamespace)
LINE 6: ERROR Class name "dirty_class" is not in PascalCase format
(Squiz.Classes.ValidClassName.NotCamelCaps)
--------------------------------------------------------------------------------
4: */
5:%w
>> 6: class dirty_class {
7: const lowerCase = false;
8:
--------------------------------------------------------------------------------
LINE 7: ERROR Class constants must be uppercase; expected LOWERCASE but found
lowerCase
(Generic.NamingConventions.UpperCaseConstantName.ClassConstantNotUpperCase)
--------------------------------------------------------------------------------
5:%w
6: class dirty_class {
>> 7: const lowerCase = false;
8:%w
9: public function My_Method() {}
--------------------------------------------------------------------------------
LINE 9: ERROR Method name "dirty_class::My_Method" is not in camel caps
format (PSR1.Methods.CamelCapsMethodName.NotCamelCaps)
--------------------------------------------------------------------------------
7: const lowerCase = false;
8:%w
>> 9: public function My_Method() {}
10: }
11:
--------------------------------------------------------------------------------
LINE 12: ERROR Each class must be in a file by itself
(PSR1.Classes.ClassDeclaration.MultipleClasses)
LINE 12: ERROR Each class must be in a namespace of at least one level (a
top-level vendor name)
(PSR1.Classes.ClassDeclaration.MissingNamespace)
LINE 12: ERROR Class name "Second_class" is not in PascalCase format
(Squiz.Classes.ValidClassName.NotCamelCaps)
--------------------------------------------------------------------------------
10: }
11:%w
>> 12: class Second_class {}
13:%w
14: $obj = new dirty_class();
--------------------------------------------------------------------------------
Time: %f secs; Memory: %dMB

0 comments on commit 028bbdb

Please sign in to comment.