Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Experiment: Use native php tests #956

Draft
wants to merge 26 commits into
base: main
Choose a base branch
from
Draft

Experiment: Use native php tests #956

wants to merge 26 commits into from

Conversation

MaartenStaa
Copy link
Collaborator

@czosel I had an idea the other day. A decent number of the bug reports on this project are due to small correctness issues. So I thought, is there a way to get ahead of the reports, and find them ourselves?

Well, the PHP project itself has many tests (.phpt files), so I set up this experiment. I pull in the php-src repository as a submodule, and made a script that generates Jest unit tests from the PHP tests, marking which should fail and which not. This branch and PR are the result.

As you can see in the commits, I've already tackled several correctness issues, but there are still plenty of failing tests. I'm jumping the gun and opening the PR to get early feedback for the idea and approach.

List of still failing tests:

  • Modifying a readonly property (test/php-src/Zend-tests-readonly_props-readonly_modification.test.js)
  • Attributes expose and verify target and repeatable data. (test/php-src/Zend-tests-attributes-020_userland_attribute_validation.test.js)
  • Test semi-reserved words as class constants (test/php-src/Zend-tests-grammar-semi_reserved_005.test.js)
  • Reserved keywords in namespace name (test/php-src/Zend-tests-namespace_name_reserved_keywords.test.js)
  • Exceptions thrown in operand cleaning must cause leak of return value (test/php-src/Zend-tests-temporary_cleaning_013.test.js)
  • Test semi-reserved words as class methods (test/php-src/Zend-tests-grammar-semi_reserved_001.test.js)
  • Enum disallows static properties (test/php-src/Zend-tests-enum-no-static-properties.test.js)
  • ZE2 An interface cannot have properties (test/php-src/tests-classes-interface_member.test.js)
  • Bug #77530: PHP crashes when parsing '(2)::class' (test/php-src/Zend-tests-bug77530.test.js)
  • Valid Unicode escape sequences: Surrogate halves (test/php-src/tests-lang-string-unicode_escape_surrogates.test.js)
  • Dynamic class name in new is not supported (test/php-src/Zend-tests-constexpr-new_dynamic_class_name.test.js)
  • errmsg: function cannot be declared private (test/php-src/Zend-tests-errmsg_002.test.js)
  • ZE2 Late Static Binding parent::/self:: forwarding while classname doesn't (test/php-src/Zend-tests-lsb_021.test.js)
  • Bug #71871: Interfaces allow final and abstract functions (test/php-src/Zend-tests-bug71871_2.test.js)
  • Constant expressions with empty dimension fetch on coalesce (test/php-src/Zend-tests-constant_expressions_coalesce_empty_dim.test.js)
  • ZE2 An interface method cannot be final (test/php-src/tests-classes-interface_method_final.test.js)
  • ZE2 An interface method cannot be private (test/php-src/tests-classes-interface_method_private.test.js)
  • using multiple access modifiers (classes) (test/php-src/Zend-tests-access_modifiers_003.test.js)
  • Bug #70183 (null pointer deref (segfault) in zend_eval_const_expr) (test/php-src/Zend-tests-bug70183.test.js)
  • foreach with list key (test/php-src/Zend-tests-foreach_list_003.test.js)
  • ZE2 An interface method must be abstract (test/php-src/tests-classes-interface_method.test.js)
  • using multiple access modifiers (abstract methods) (test/php-src/Zend-tests-access_modifiers_004.test.js)
  • abstract final methods errmsg (test/php-src/Zend-tests-access_modifiers_007.test.js)
  • Empty list() assignments are not allowed (test/php-src/Zend-tests-list_empty_error.test.js)
  • ZE2 A final method cannot be abstract (test/php-src/tests-classes-final_abstract.test.js)
  • errmsg: multiple access type modifiers are not allowed (methods) (test/php-src/Zend-tests-errmsg_010.test.js)
  • Positional arguments cannot be used after argument unpacking (test/php-src/Zend-tests-arg_unpack-positional_arg_after_unpack_error.test.js)
  • Cannot use static in trait insteadof method reference (test/php-src/Zend-tests-static_in_trait_insteadof_reference.test.js)
  • Bug #60145 (Usage of trait's use statement inside interfaces not properly checked.) (test/php-src/Zend-tests-traits-bug60145.test.js)
  • basic binary nowdoc syntax (test/php-src/Zend-tests-nowdoc_002.test.js)
  • using multiple access modifiers (methods) (test/php-src/Zend-tests-access_modifiers_001.test.js)
  • __HALT_COMPILER() basic test (test/php-src/Zend-tests-halt03.test.js)
  • ZE2 Late Static Binding ensuring extending 'static' is not allowed (test/php-src/Zend-tests-lsb_006.test.js)
  • Flexible heredoc syntax complex test 2: interpolated nested heredocs\nwith the same delimiter name (test/php-src/Zend-tests-flexible-heredoc-complex-test2.test.js)
  • Flexible heredoc syntax complex test 4: interpolated variable with\nthe same delimiter name as the heredoc (test/php-src/Zend-tests-flexible-heredoc-complex-test4.test.js)
  • basic binary heredoc syntax (test/php-src/Zend-tests-heredoc_002.test.js)
  • Bug #47516 (nowdoc can not be embed in heredoc but can be embed in double quote) (test/php-src/Zend-tests-bug47516.test.js)
  • An error should be generated when using ::class on a constant evaluated expression (test/php-src/Zend-tests-class_on_constant_evaluated_expression.test.js)
  • Backed enums type can't be union (test/php-src/Zend-tests-enum-backed-type-no-union.test.js)
  • errmsg: multiple access type modifiers are not allowed (properties) (test/php-src/Zend-tests-errmsg_009.test.js)
  • Can use readonly as a function name (test/php-src/Zend-tests-readonly_function.test.js)
  • Enum keyword can be followed by arbitrary whitespaces (test/php-src/Zend-tests-enum-keyword-whitespace.test.js)
  • Class constants can be used as a class name (test/php-src/Zend-tests-varSyntax-class_constant_static_deref.test.js)
  • Ensure a interface can not have private constants (test/php-src/tests-classes-interface_constant_inheritance_007.test.js)
  • Bug #70912 (Null ptr dereference when class property is initialised to a dereferenced value) (test/php-src/Zend-tests-bug70912.test.js)
  • Match expression with trailing comma in condition list (test/php-src/Zend-tests-match-041.test.js)
  • Foo(...) in attribute in assert (test/php-src/Zend-tests-first_class_callable_assert2.test.js)
  • foreach with list syntax (test/php-src/Zend-tests-foreach_list_001.test.js)
  • Test semi-reserved words as static class methods (test/php-src/Zend-tests-grammar-semi_reserved_002.test.js)
  • using multiple access modifiers (static methods) (test/php-src/Zend-tests-access_modifiers_006.test.js)
  • Enum disallows value property (test/php-src/Zend-tests-enum-no-value-property.test.js)
  • ::class on an expression cannot be used inside constant expressions (test/php-src/Zend-tests-class_on_expression_in_constant_expression.test.js)
  • Cannot use multiple readonly modifiers (test/php-src/Zend-tests-readonly_props-multiple_modifiers.test.js)
  • Using ::class on an object (test/php-src/Zend-tests-class_on_object.test.js)
  • using multiple access modifiers (final methods) (test/php-src/Zend-tests-access_modifiers_005.test.js)
  • Test to ensure semi reserved words allow deference (test/php-src/Zend-tests-grammar-regression_007.test.js)
  • Enum keyword can still be used in classes, namespaces, functions and constants (test/php-src/Zend-tests-enum-keyword-no-bc-break.test.js)
  • Named args after unpacking (supported) (test/php-src/Zend-tests-named_params-unpack_and_named_1.test.js)
  • Invalid constant class name in nested class constant access (test/php-src/Zend-tests-invalid_const_class_name.test.js)
  • Prevent abstract and final in the same class declaration (test/php-src/Zend-tests-access_modifiers_013.test.js)
  • Flexible heredoc syntax complex test 3: interpolated nested heredocs\nwith the same delimiter name with different levels of indentation (test/php-src/Zend-tests-flexible-heredoc-complex-test3.test.js)
  • using multiple access modifiers (attributes) (test/php-src/Zend-tests-access_modifiers_002.test.js)
  • GH-7958 (Nested CallbackFilterIterator is leaking memory) (test/php-src/Zend-tests-gh7958.test.js)
  • test for mbstring script_encoding for flex unsafe encoding (Shift_JIS) (test/php-src/Zend-tests-multibyte-multibyte_encoding_004.test.js)
  • ZE2 A final method cannot be abstract (test/php-src/tests-classes-abstract_final.test.js)
  • Flexible heredoc syntax complex test 1: interpolated nested heredocs\nwith different delimiter names (test/php-src/Zend-tests-flexible-heredoc-complex-test1.test.js)
  • encoding conversion from script encoding into internal encoding (test/php-src/Zend-tests-multibyte-multibyte_encoding_005.test.js)
  • Bug #76439: Don't always strip leading whitespace from heredoc T_ENCAPSED_AND_WHITESPACE tokens (error case) (test/php-src/Zend-tests-bug76439_2.test.js)
  • Cannot use unpacking in attribute argument list (test/php-src/Zend-tests-attributes-026_unpack_in_args.test.js)
  • First Class Callable Attribute Error (test/php-src/Zend-tests-first_class_callable_011.test.js)
  • Group use declaration list should not contain leading separator (test/php-src/Zend-tests-ns_096.test.js)
  • Interface cannot extend static, as it is reserved (test/php-src/Zend-tests-interface_extends_static.test.js)
  • Invalid use: trailing underscore (test/php-src/Zend-tests-numeric_literal_separator_002.test.js)
  • Bug #21669 ("$obj = new $this->var;" doesn't work) (test/php-src/tests-lang-bug21669.test.js)
  • ZE2 Late Static Binding ensuring implementing 'static' is not allowed (test/php-src/Zend-tests-lsb_007.test.js)
  • 081: bracketed namespace with nested unbracketed namespace (test/php-src/Zend-tests-ns_081.test.js)
  • enum_exists (test/php-src/Zend-tests-enum-enum_exists.test.js)
  • Bug #71897 (ASCII 0x7F Delete control character permitted in identifiers) (test/php-src/tests-lang-bug71897.test.js)
  • Cannot catch "static" (test/php-src/Zend-tests-catch_static.test.js)
  • Invalid consecutive numeric separators after binary literal (test/php-src/Zend-tests-bug78454_2.test.js)
  • Flexible heredoc syntax error 9: unindented variable interpolation (test/php-src/Zend-tests-flexible-heredoc-error9.test.js)
  • Zend Multibyte and ShiftJIS (test/php-src/Zend-tests-multibyte-multibyte_encoding_001.test.js)
  • Class cannot use static as a trait, as it is reserved (test/php-src/Zend-tests-class_uses_static.test.js)
  • Indirect modification of isref by-value return value not possible (test/php-src/Zend-tests-modify_isref_value_return.test.js)
  • Alternative offset syntax should emit E_COMPILE_ERROR in const expression (test/php-src/Zend-tests-alternative_offset_syntax_compile_error_in_const_expr.test.js)
  • Invalid Unicode escape sequence: Negative sign (test/php-src/tests-lang-string-unicode_escape_sign2.test.js)
  • <?= cannot be used as an identifier (test/php-src/Zend-tests-short_echo_as_identifier.test.js)
  • Flexible nowdoc syntax error 8: no ending token with explicit trailing space (test/php-src/Zend-tests-flexible-nowdoc-error8.test.js)
  • Writing to a temporary expression is not allowed (test/php-src/Zend-tests-varSyntax-writeToTempExpr.test.js)
  • Invalid Unicode escape sequence: Positive sign (test/php-src/tests-lang-string-unicode_escape_sign.test.js)
  • Cannot use "namespace" as namespace name, due to conflict with ns-relative names (test/php-src/Zend-tests-namespace_name_namespace.test.js)
  • eval() test (test/php-src/tests-lang-019.test.js)
  • Flexible heredoc syntax error 11: show erroneous line in error message (variable interpolation) (test/php-src/Zend-tests-flexible-heredoc-error11.test.js)
  • Static type is not allowed in properties (test/php-src/Zend-tests-type_declarations-static_type_property.test.js)
  • Intersection type cannot be nullable (test/php-src/Zend-tests-type_declarations-intersection_types-invalid_types-invalid_nullable_type.test.js)
  • Invalid Unicode escape sequence: Whitespace (test/php-src/tests-lang-string-unicode_escape_whitespace.test.js)
  • Invalid Unicode escape sequence: Empty (test/php-src/tests-lang-string-unicode_escape_empty.test.js)
  • Cannot use "namespace\xyz" as namespace name, due to conflict with ns-relative names (test/php-src/Zend-tests-namespace_name_namespace_start.test.js)
  • Invalid Unicode escape sequence: Incomplete (test/php-src/tests-lang-string-unicode_escape_incomplete.test.js)
  • Method cannot be readonly in trait alias (test/php-src/Zend-tests-readonly_props-readonly_method_trait.test.js)
  • Invalid consecutive numeric separators after hex literal (test/php-src/Zend-tests-bug78454_1.test.js)
  • Bug #78441 (Parse error due to heredoc identifier followed by digit) (test/php-src/Zend-tests-grammar-bug78441.test.js)
  • Test typed properties type must precede first declaration in group (test/php-src/Zend-tests-type_declarations-typed_properties_025.test.js)
  • Basic class support - attempting to create a reference to a class constant (test/php-src/tests-classes-constants_error_007.test.js)
  • Invalid octal (test/php-src/tests-lang-invalid_octal.test.js)
  • Flexible heredoc syntax error 10: unindented variable interpolation (as first value) (test/php-src/Zend-tests-flexible-heredoc-error10.test.js)

Note: it's possible that some of these are beyond the scope of this project, or are cases where the test generator is incorrectly set to expect the test to fail, or doesn't mark a test as one that should fail where it should.

Maarten Staa and others added 26 commits May 30, 2022 16:04
Add submodule with php-src.
8d82aca
@MaartenStaa
Merge pull request #941 from czosel/fix-enum-as-identifier-2
f4aab9e
@czosel
3.1.0-beta.9
fb2f446
Automatically generate Jest tests from .phpt files in php-src.
243b0ee
Handle nuances with whitespace and use statements.
ca061f4
Start differentiating between namespace_name and name.
12b4813
Handle new expression such as new ('std' . 'Class').
2cc3fa9
Handle nullsafe object operator on arbitrary expressions, like `[]?->…
5b9a744 
…foo`.
Mark some tests as throwing a parse error.
c3c8ee4
Disallow static type in argument lists.
c1636cd
Throw error on unterminated multi-line comment.
5ac08bf
Fix error on variadic function calls.
af231be 
Fixes a parse error when passing multiple argument spreads such as
`foo(...$bar, ...$baz)`. Adds extra checks for function definitions with
variadic parameters, followed by any other parameters.

Fixes #946
Fix lexing of octal literals.
424b271
Move php-src tests to their own files.
1058cad
Fix parsing of enum cases with attributes.
9821921
@czosel
Merge pull request #949 from glayzzle/fix-unpacking-multiple-arguments
9ce6b94 
Fix error on variadic function calls.
@czosel
chore: upgrade dependencies
71c2a9f
@czosel
Merge pull request #951 from czosel/upgrade-deps
6736591 
chore: upgrade dependencies
@czosel
3.1.0-beta.10
13cc2b6
Update php-src test files.
bbc70fd
Be more conservative about which .phpt files we use.
d66c5c8
Merge branch 'main' into use-native-php-tests
c722794
New snapshots after merge.
b42b8ce
Mark php-src as failing.
8640f86
Mark php-src tests as failing.
4dab16d
Mark php-src tests as not failing.
e9ac21e
@czosel
Copy link
Collaborator

czosel commented Jun 15, 2022

Hi @MaartenStaa
Wow, this sounds really exciting! I’ve been wanting to reduce the amount of boilerplate in our testing setup since a while, similar to the .phpt files from php-src - but this takes it two steps further. Unfortunately I probably won’t find time to take a closer look at this until somewhen next week. For now I just wanted to share my excitement and encourage you to keep pushing this forward 💯

@czosel
Copy link
Collaborator

czosel commented Jul 2, 2022 •
edited

@MaartenStaa In the prettier PHP plugin, we're parsing plain php files for snapshot testing using jest setupFiles, thereby avoiding the code generation step: https://github.com/prettier/plugin-php/blob/main/tests_config/run_spec.js
Would something like that work here as well? It would be nice if we could just use the .phpt files directly without the need to generate intermediate testing code.

@cseufert
Copy link
Collaborator

cseufert commented Oct 4, 2022

@MaartenStaa this is really cool, should we just create a test blacklist for failing tests, so we can work towards merging this, and them working on fixing all the cases where it fails?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@MaartenStaa @czosel @cseufert