Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

APITestCase: Validate OpenApi definition using a json schema #3828

Closed
fherbin opened this issue Nov 13, 2020 · 17 comments
Closed

APITestCase: Validate OpenApi definition using a json schema #3828

fherbin opened this issue Nov 13, 2020 · 17 comments

Comments

@fherbin
Copy link

fherbin commented Nov 13, 2020

Description
would it be possible to validate the openApi definition?
I will explain for example it will be easier:
when I want to integrate my swagger definition in my api gateway (gravitee.io), it finds me errors like in the exemple below.

it would be great to be able to get these errors directly in apiplatform, in order to consolidate our openApi definitions.

Example

**attribute paths.'/v1/edit_facture_eaus/{id}' is not of type `object`
attribute paths.'/v1/chorus_pro/factures/upload'(post).[['unknown']].name is missing
attribute paths.'/v1/chorus_pro/factures/upload'(post).responses is missing
attribute paths.'/v1/chorus_pro/factures/{refDette}/status'(get).[refDette].type is missing
attribute paths.'/v1/chorus_pro/factures/{refDette}/status'(get).responses is missing
attribute paths.'/v1/soldes/abonnement/{abonnementId}' Declared path parameter abonnementId needs to be defined as a path parameter in path or operation level
attribute paths.'/v1/soldes/client/{clientId}'. Declared path parameter clientId needs to be defined as a path parameter in path or operation level
@fherbin fherbin changed the title OpenApi / Gwagger definition validator OpenApi / Swagger definition validator Nov 13, 2020
@soyuka
Copy link
Member

soyuka commented Nov 16, 2020

We're using npx swagger-cli validate build/out/openapi/swagger_v2.json, I suggest that you use the same for now:

https://github.com/api-platform/core/blob/master/.github/workflows/ci.yml#L293

Note that you could easily add a schema validation to your test suite using:

with $this->assertMatchesJsonSchema(/* a JSON Schema as an array or as a string */);

available with our Test Client: https://api-platform.com/docs/core/testing/#the-test-httpclient

I'll add a documentation entry and this will be even easier to test on the next version of API Platform (#3407)

@fherbin
Copy link
Author

fherbin commented Nov 16, 2020

great thx !

@fherbin
Copy link
Author

fherbin commented Nov 16, 2020

@soyuka I tried to use the assertion 'assertMatchesJsonSchema' like this:

<?php

namespace App\Tests\OpenApi;

use ApiPlatform\Core\Bridge\Symfony\Bundle\Test\ApiTestCase;

class OpenApiTest extends ApiTestCase
{
    const OPEN_API_JSON_SCHEMA_V2_URL = 'https://raw.githubusercontent.com/OAI/OpenAPI-Specification/master/schemas/v2.0/schema.json';
    const OPEN_API_JSON_SCHEMA_V3_URL = 'https://raw.githubusercontent.com/OAI/OpenAPI-Specification/master/schemas/v3.0/schema.json';

    public function testOpenApiV2Validation()
    {
        static::createClient()->request(
            'GET',
            '/api/docs.json',
            [
                'headers' => ['Accept' => 'application/json'],
                'query' => ['spec_version' => 2],
            ]
        );
        $this->assertMatchesJsonSchema(file_get_contents(self::OPEN_API_JSON_SCHEMA_V2_URL));
    }

    public function testOpenApiV3Validation()
    {
        static::createClient()->request(
            'GET',
            '/api/docs.json',
            [
                'headers' => ['Accept' => 'application/json'],
                'query' => ['spec_version' => 3],
            ]
        );
        $this->assertMatchesJsonSchema(file_get_contents(self::OPEN_API_JSON_SCHEMA_V3_URL));
    }
}

but both don't work, here is the return of phpunit :

www-data@0b46516f9e2e:~/html/msfacturations$  ./bin/phpunit
PHPUnit 7.5.20 by Sebastian Bergmann and contributors.

Testing Project Test Suite
EE                                                                  2 / 2 (100%)

Time: 4.32 seconds, Memory: 46.50 MB

There were 2 errors:

1) App\Tests\OpenApi\OpenApiTest::testOpenApiV2Validation
JsonSchema\Exception\InvalidSchemaMediaTypeException: Media type application/schema+json expected

/var/www/html/msfacturations/vendor/justinrainbow/json-schema/src/JsonSchema/Uri/UriRetriever.php:92
/var/www/html/msfacturations/vendor/justinrainbow/json-schema/src/JsonSchema/Uri/UriRetriever.php:209
/var/www/html/msfacturations/vendor/justinrainbow/json-schema/src/JsonSchema/Uri/UriRetriever.php:181
/var/www/html/msfacturations/vendor/justinrainbow/json-schema/src/JsonSchema/SchemaStorage.php:52
/var/www/html/msfacturations/vendor/justinrainbow/json-schema/src/JsonSchema/SchemaStorage.php:115
/var/www/html/msfacturations/vendor/justinrainbow/json-schema/src/JsonSchema/SchemaStorage.php:138
/var/www/html/msfacturations/vendor/justinrainbow/json-schema/src/JsonSchema/SchemaStorage.php:162
/var/www/html/msfacturations/vendor/justinrainbow/json-schema/src/JsonSchema/Constraints/Constraint.php:118
/var/www/html/msfacturations/vendor/justinrainbow/json-schema/src/JsonSchema/Constraints/ObjectConstraint.php:145
/var/www/html/msfacturations/vendor/justinrainbow/json-schema/src/JsonSchema/Constraints/ObjectConstraint.php:47
/var/www/html/msfacturations/vendor/justinrainbow/json-schema/src/JsonSchema/Constraints/Constraint.php:85
/var/www/html/msfacturations/vendor/justinrainbow/json-schema/src/JsonSchema/Constraints/UndefinedConstraint.php:80
/var/www/html/msfacturations/vendor/justinrainbow/json-schema/src/JsonSchema/Constraints/UndefinedConstraint.php:51
/var/www/html/msfacturations/vendor/justinrainbow/json-schema/src/JsonSchema/Constraints/Constraint.php:118
/var/www/html/msfacturations/vendor/justinrainbow/json-schema/src/JsonSchema/Constraints/SchemaConstraint.php:92
/var/www/html/msfacturations/vendor/justinrainbow/json-schema/src/JsonSchema/Validator.php:63
/var/www/html/msfacturations/vendor/api-platform/core/src/Bridge/Symfony/Bundle/Test/Constraint/MatchesJsonSchema.php:63
/var/www/html/msfacturations/vendor/api-platform/core/src/Bridge/Symfony/Bundle/Test/ApiTestAssertionsTrait.php:109
/var/www/html/msfacturations/tests/OpenApi/OpenApiTest.php:22

2) App\Tests\OpenApi\OpenApiTest::testOpenApiV3Validation
Error: Call to a member function export() on null

/var/www/html/msfacturations/vendor/api-platform/core/src/Bridge/Symfony/Bundle/Test/ApiTestAssertionsTrait.php:109
/var/www/html/msfacturations/tests/OpenApi/OpenApiTest.php:35

ERRORS!
Tests: 2, Assertions: 2, Errors: 2.

Is there anything I didn't understand?

@soyuka
Copy link
Member

soyuka commented Nov 16, 2020

I need to dig a bit further, this should've worked indeed.

@soyuka
Copy link
Member

soyuka commented Nov 17, 2020

Hi, I tracked the bugs down, your first issue is related to: https://stackoverflow.com/questions/28302900/mediatype-should-be-application-schemajson-to-validate-schema

Basically the Swagger v2 schema uses http://swagger.io/v2/schema.json which doesn't answers with a json content-type, maybe because it's a redirection (301) and because justinrainbow/json-schema uses FileGetContents. I have a fix in mind we need to create our own UriRetrieverInterface that handles this.

The Call to a member function export() on null is way trickier still looking.

@soyuka
Copy link
Member

soyuka commented Nov 17, 2020

Okay I've found a fix!

    public function testOpenApiV3Validation()
    {
        static::createClient()->request(
            'GET',
            '/api/docs.json',
            [
                'headers' => ['Accept' => 'application/json'],
                'query' => ['spec_version' => 3],
            ]
        );
        $this->assertMatchesJsonSchema(file_get_contents(self::OPEN_API_JSON_SCHEMA_V3_URL), Constraint::CHECK_MODE_TYPE_CAST);
    }

CHECK_MODE_TYPE_CAST looks required because of the ArrayObject transformation. I'm adding a fix for the UriRetriever in the morning.

Thanks @fherbin for testing this!

@fherbin
Copy link
Author

fherbin commented Nov 17, 2020

ok, works for v3 with this commit applied :
6fce2ad

@fherbin
Copy link
Author

fherbin commented Nov 17, 2020

Hi, I tracked the bugs down, your first issue is related to: https://stackoverflow.com/questions/28302900/mediatype-should-be-application-schemajson-to-validate-schema

Basically the Swagger v2 schema uses http://swagger.io/v2/schema.json which doesn't answers with a json content-type, maybe because it's a redirection (301) and because justinrainbow/json-schema uses FileGetContents. I have a fix in mind we need to create our own UriRetrieverInterface that handles this.

The Call to a member function export() on null is way trickier still looking.

there is a property for that yes :
https://github.com/justinrainbow/json-schema/blob/fa4d2d3c1e40a222ded125b679891086c9a6c7d6/src/JsonSchema/Uri/UriRetriever.php#L38

@soyuka
Copy link
Member

soyuka commented Nov 17, 2020

See jsonrainbow/json-schema#646 for a redirection fix.

I've proposed a patch there.

I don't really have any other solution to offer as we don't want to add too much code in API Platform to cover this use case. A solution on your end would be to change the MatchesJsonSchema (copy our implementation) to something like this:

final class MyMatchesJsonSchema extends Constraint
{
    // ...

    protected function matches($other): bool
    {
        $other = $this->normalizeJson($other);

        $validator = $this->getValidator();
        $validator->validate($other, $this->schema, $this->checkMode);

        return $validator->isValid();
    }

    private function getValidator(): Validator
    {
        $uriRetriever = new UriRetriever();
        // Create your own JsonSchema\Uri\Retrievers\UriRetrieverInterface that works properly
        $uriRetriever->setUriRetriever(// insert it here);
        return new Validator(new Factory(null, $uriRetriever));
    }
}

And create your assertion like this:

    /**
     * @param object|array|string $jsonSchema
     */
    public static function assertMatchesJsonSchema($jsonSchema, ?int $checkMode = null, string $message = ''): void
    {
        $constraint = new MyMatchesJsonSchema($jsonSchema, $checkMode);

        static::assertThat(self::getHttpResponse()->toArray(false), $constraint, $message);
    }

@soyuka soyuka changed the title OpenApi / Swagger definition validator APITestCase: Validate OpenApi definition using a json schema Nov 17, 2020
@soyuka
Copy link
Member

soyuka commented Nov 26, 2020

@fherbin about the The Call to a member function export() on null issue, would you be able to update the phpunit version you're using?

@fherbin
Copy link
Author

fherbin commented Nov 26, 2020

@fherbin about the The Call to a member function export() on null issue, would you be able to update the phpunit version you're using?

PHPUnit 7.5.20 so far, yes it's possible

[...]
2) App\Tests\OpenApiTest::testOpenApiV3Validation
Error: Call to a member function export() on null

/var/www/html/msfacturations/vendor/api-platform/core/src/Bridge/Symfony/Bundle/Test/ApiTestAssertionsTrait.php:109
/var/www/html/msfacturations/tests/openApiTest.php:35

ERRORS!
Tests: 2, Assertions: 2, Errors: 2.
www-data@0b46516f9e2e:~/html/msfacturations$ bin/phpunit --version
PHPUnit 7.5.20 by Sebastian Bergmann and contributors.

@soyuka
Copy link
Member

soyuka commented Nov 26, 2020

try phpunit 8 or 9, 7 is kinda old

@fherbin
Copy link
Author

fherbin commented Nov 27, 2020

not possible, in conflict with symfony/phpunit-bridge requirements

  Problem 1
    - phpunit/phpunit 9.1.2 conflicts with symfony/phpunit-bridge[v5.1.8].
    - phpunit/phpunit 9.1.2 conflicts with symfony/phpunit-bridge[v5.1.0].
    - phpunit/phpunit 9.1.2 conflicts with symfony/phpunit-bridge[v5.1.1].
    - phpunit/phpunit 9.1.2 conflicts with symfony/phpunit-bridge[v5.1.2].
    - phpunit/phpunit 9.1.2 conflicts with symfony/phpunit-bridge[v5.1.3].
    - phpunit/phpunit 9.1.2 conflicts with symfony/phpunit-bridge[v5.1.4].
    - phpunit/phpunit 9.1.2 conflicts with symfony/phpunit-bridge[v5.1.5].
    - phpunit/phpunit 9.1.2 conflicts with symfony/phpunit-bridge[v5.1.6].
    - phpunit/phpunit 9.1.2 conflicts with symfony/phpunit-bridge[v5.1.7].
    - phpunit/phpunit 9.1.2 conflicts with symfony/phpunit-bridge[v5.1.8].
    - Installation request for phpunit/phpunit 9.1.2 -> satisfiable by phpunit/phpunit[9.1.2].
    - Installation request for symfony/phpunit-bridge ^5.1 -> satisfiable by symfony/phpunit-bridge[v5.1.0, v5.1.1, v5.1.2, v5.1.3, v5.1.4, v5.1.5, v5.1.6, v5.1.7, v5.1.8].


@soyuka
Copy link
Member

soyuka commented Nov 27, 2020

Currently I have:

symfony/phpunit-bridge                  v5.1.8             Symfony PHPUnit Bridge

And:

./vendor/bin/simple-phpunit  --version
PHPUnit 8.3.5 by Sebastian Bergmann and contributors.

I can run phpunit 9 by using:

SYMFONY_PHPUNIT_VERSION=9.4 vendor/bin/simple-phpunit 

@fherbin
Copy link
Author

fherbin commented Dec 1, 2020

ok it works by forcing a recent version of phpunit :

www-data@0b46516f9e2e:~/html/msfacturations$ SYMFONY_PHPUNIT_VERSION=9.4 vendor/bin/simple-phpunit
[...]
 matches the provided JSON Schema.
paths./api/tarification/article_fluides.get.parameters: There are no duplicates allowed in the array
paths./api/tarification/article_stocks.get.parameters: There are no duplicates allowed in the array
paths./api/chorus_pro/factures/upload.post.parameters[0].$ref: The property $ref is required
paths./api/chorus_pro/factures/upload.post.parameters[0].in: Does not have a value in the enumeration ["cookie"]
paths./api/chorus_pro/factures/upload.post.parameters[0].in: Does not have a value in the enumeration ["path"]
paths./api/chorus_pro/factures/upload.post.parameters[0].in: Does not have a value in the enumeration ["query"]
paths./api/chorus_pro/factures/upload.post.parameters[0].in: Does not have a value in the enumeration ["header"]
paths./api/chorus_pro/factures/upload.post.parameters[0]: Failed to match exactly one schema
paths./api/chorus_pro/factures/upload.post.parameters[0]: Failed to match all schemas
paths./api/chorus_pro/factures/upload.post.parameters[0].schema: String value found, but an object is required
paths./api/chorus_pro/factures/upload.post.parameters[0].schema: Failed to match exactly one schema
paths./api/chorus_pro/factures/upload.post: The property consumes is not defined and the definition does not allow additional properties
paths./api/chorus_pro/factures/upload.post: The property produces is not defined and the definition does not allow additional properties
paths./api/chorus_pro/factures/{refDette}/status.get.parameters[0].$ref: The property $ref is required
paths./api/chorus_pro/factures/{refDette}/status.get.parameters[0].content: The property content is required
paths./api/chorus_pro/factures/{refDette}/status.get.parameters[0].schema: The property schema is required
paths./api/chorus_pro/factures/{refDette}/status.get.parameters[0]: Failed to match exactly one schema
paths./api/chorus_pro/factures/{refDette}/status.get.parameters[0]: Failed to match all schemas
paths./api/chorus_pro/factures/{refDette}/status.get.parameters[0]: The property type is not defined and the definition does not allow additional properties
paths./api/chorus_pro/factures/{refDette}/status.get: The property produces is not defined and the definition does not allow additional properties

/var/www/html/msfacturations/vendor/api-platform/core/src/Bridge/Symfony/Bundle/Test/ApiTestAssertionsTrait.php:109
/var/www/html/msfacturations/tests/openApiTest.php:35

ERRORS!
Tests: 2, Assertions: 2, Errors: 1, Failures: 1.

@ihorsamusenko
Copy link

ihorsamusenko commented Dec 25, 2020

Error : Call to a member function export() on null
 /var/www/vendor/api-platform/core/src/Bridge/Symfony/Bundle/Test/ApiTestAssertionsTrait.php:110

This one will be solved when MatchesJsonSchema starts calling parent::__construct()

The issue happens in \PHPUnit\Framework\Constraint\Constraint::failureDescription()

This is reproducible on phpunit 7, for phpunit 8 work good

@soyuka
Copy link
Member

soyuka commented Dec 31, 2020

Please use phpunit 8

@soyuka soyuka closed this as completed Jan 28, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants