New Feature - Generate tests from OpenAPI / Swagger - automated contract testing #56 #225

thim81 · 2020-06-01T10:37:10Z

This feature proposal contains 2 implementations, with the aim to facilitate & automate contract testing using Postman, based on OpenApi specifications:

Provide the option to generate real basic response postman tests, generated/added based on the OpenApi spec file. (Success status check / Response time check / Content-type check / JSON schema validation (based on the OpenAPi components) / JSON body check)
Provide additional manually created Postman test scripts. The manual postman tests will be added per OpenApi operationId. This will give enough granularity to define specific tests.
Provide a option to overwrite/extend the Postman Request body, Query params, Path Variables & Headers as part of the Testsuite generation.
Provide the option to define pm.environment variables for automation, by assigning values from the response. Automatically a pm.environment variable for the POST response will be provided

An example and the properties are explained here: the https://github.com/thim81/openapi-to-postman/blob/develop/TESTGENERATION.md

The test generation for option (1) can be can be added in the test suite configuration file where all information with regards to the preferences for the different tests for option (1) can be managed. This will also allow addition of new basic tests in next versions.

The manual test postman scripts are also managed in the test suite configuration file, grouped per OpenApi OperationId.

The use-case is to facilitate API CREATE operations with dynamic data, where we provide the option to overwrite/extend the OpenAPI request body, Query params & Path Variables with more specific values or the dynamic variables provided by Postman, desired by the Testsuite.

By having the option to overwrite the Query Param, Path variables & headers, it will be easier to automate chaining / workflows as part of the automation.

By automatically creating a pm.environment variable based on the response ID property.
This ID property can be easily used for chaining the POST create operation and the DELETE operation, through the usage of this pm.environment variable.

If the test suite configuration file is not added or if certain best tests are not "enabled".
There will be no impact, since this feature implementation is fully optional.

Currently the following Postman tests are generated automatically:

Success status check
Response time check
Content-type check
JSON schema validation
JSON body check

Why I did choose not to add tests in OpenApi spec?

Currently there was no property/tag available for test (Or I havent found it in the OpenApi spec). Vendor extensions could be an option but in which format/framework would you write the tests?
OpenApi is about documenting the API while tests have a more functional aspect. Mixing these 2 concepts in 1 would risk of complicating the OpenApi?
It would not be easy to find a "generic" test description. In which format/framework would you write OpenApi tests? The Postman test format is a distinct direction and personal preference, so this would not belong in the OpenApi.

The feature proposal provide a way to add/generate Postman tests based on the OpenApi spec.
This keeps the OpenApi "clean" and allows specific Postman implementation options.

For your information, the test functionality is being used in an CI pipeline and it works great for contract testing via Postman.

The flow is defined as following:

OpenApi with examples and schemas
Filter out unwanted OpenAPI methods/Operations via openapi-filter
openapi-to-postman generates a postman collection with tests.
the postman collection ran with just setting an postman env. The examples from OpenAPI are used in the postman collection. The OpenAPI schema is used in the postman tests.
via newman we run the tests and validate the api contracts.
via the Newman report we can view what failed

This all automated in an Azure Pipeline.

Additions:

Merged Feature/cli converter options #218 for easy config file usage in the CLI tool.
Added Postman test suite settings, which allow you to define which types of basic tests to be included or not, in the postman generation.
Added Post test suite file loading (inspired by Feature/cli converter options #218) option for the CLI
Added postman-testsuite.json example

- Success status check - Response time check - Content-type check - JSON schema validation - JSON body check Merged postmanlabs#218 for easy config file usage in the CLI tool. Added Postman test suite settings, which allow you to define which types of basic tests to be included or not, in the postman generation. Added Post test suite file loading (inspired by postmanlabs#218) option for the CLI Added postman-testsuite.json example Added test suite extension option, which offers the option to include manually defined postman tests based. The test extensions are mapped based on the OpenApi operationID.

Added validation for schema.$ref

It is now possible to `overwrite` all generateTests and extend them with specific settings per OpenApiOperationId.

This helps when running newman tests with reporting via CI (like Azure devops). The reporting will contain the references.

thim81 · 2020-06-15T13:03:59Z

@VShingala Is it advised to keep PR's up-to-date with the changes on the "develop" branch?

VShingala · 2020-06-15T13:59:33Z

@thim81 yeah, it would be good to keep it up to date with develop branch.

…ostman into develop

thim81 · 2020-06-24T06:59:11Z

Testsuite configuration

Postman testsuite results

thim81 · 2020-07-01T06:37:54Z

@VShingala I want to add a test to the PR. Can you point me towards a good test that is using the CLI options and easy to understand?

kharoutaj · 2020-07-11T20:08:04Z

I took the approach of using vendor extensions for this feature for my team's project. I think it provides more granularity as you can specify tests at the individual path level in the openapi spec. I might try and clean it up if there is any interest, not sure if this implementation provides that same level of functionality. Does the operation-id field allow this?

thim81 · 2020-07-13T08:26:02Z

@kharoutaj In my PR, you can have the test suite generated based on the OpenAPI spec and still have an overwrite per path operation (each operation ID) (GET/POST/...).

So I think we have similar approach. The main reason I have kept it out of the OpenApi specification is for various reasons.
You can check the reasons in the original issue: #56 (comment)

…his will only generate tests based on the defined list of operationId's.

…ostman into develop

…ere the header is checked to present in the response

…ostman into develop

SidhNor · 2020-11-16T09:23:41Z

Hey guys, any plans to rebase / merge this?

thim81 · 2020-11-16T12:45:58Z

Hey guys, any plans to rebase / merge this?

Hi @SidhNor , I have been waiting for the CLI ability to configure the importer, which was officially added since v2.0.0 (October 15, 2020). Due to some other things, I've been a bit behind, so I'll merge this into my PR and rebase to have all the latest changes.

Pure informational, I have been using the automated contract testing over the last 4 months and it works fine for me, so if you plan to use it or have used and have feedback, please share so I can improve my PR.

…o replace an integer example

CleverCoder · 2021-03-09T15:13:22Z

Extending on my comments here: #56

The header "overwrite" isn't working for me. Seems to be due to how the "faker" stuff is designed
Wouldn't this be better implemented as a "value provider", and the current "faker" can be turned into a value provider? This way the code has a single implementation for providing a value for auth/header/body, etc.. and the provider can be the default, or this more flexible implementation
What about authorization? It seems to not address the need to change the value being provided in the auth header. the generated collection just shows . This is pretty important for automation.
Matching on operationId is too limiting. I'd suggest using RegEx against a path such as "VERB:PATH", ie: "post:/some/test/endpoint"... or ".*:/widgets".. maybe a simpler glob match..

Keep it up!!

…JSON schema properties which deletes nullable and adds "null" to type array if nullable is true

thim81 · 2021-03-11T10:57:54Z

@CleverCoder

Valid comments, I replied to them in the feature #56

… headers

…aintenance

…arams

Patch related to the stripping of "content-type" defined in OpenAPI postmanlabs#361

…content schema. This messes up the schema validation used by the testSuite. Changed the fixed stackLimit from 10 to 999 to overcome "Error: Too many levels of nesting to fake this schema" in the schema validation Fix to prevent referenced path variables being used for overwrites

Added option to match target based on the OpenApi Operation (method & path)

Added "ContentCheck" feature

…, request body or a plain value

…ResponseBody

thim81 · 2021-06-18T12:56:08Z

UPDATE: I concluded that the PR functionality does not really belong in openapi-to-postman, because this package is focussed on converting OpenAPI to Postman. Together with Nick from Apideck, we ported the PR contract testing functionality, with plenty more to the Portman package.

It supports: contract testing, overwrites, assign variables + variation testing

https://www.npmjs.com/package/@apideck/portman

thim81 · 2021-07-13T09:35:45Z

UPDATE: We just launched the 1.0 release of Portman, which now support contract, variation & integration testing!
I wrote a blog post about it on the Apideck website: https://blog.apideck.com/announcing-portman that gives more information.

SidhNor · 2021-07-13T11:27:36Z

@thim81 congrats. Nice job. Will give it a try )

ani-im · 2021-09-10T12:00:11Z

Do we have any updates regarding this?
Will this PR be merged. If so, do we have deadlines?
Thanks in advance.

thim81 · 2021-09-10T13:23:12Z

@ani-im

This PR is discontinued. Instead we ported the PR contract testing functionality, to a dedicated package >> https://www.npmjs.com/package/@apideck/portman

Under the hood it is using openapi-to-postman, to convert the OpenAPI specification to a Postman collection, with all the same conversion options, with additional options to inject: contract testing, overwrites, assign variables + variation testing

thim81 mentioned this pull request Jun 1, 2020

New Feature request - Generate tests from OpenAPI / Swagger yaml or json file #56

Open

thim81 changed the title ~~Added test generation for:~~ New Feature - Generate tests from OpenAPI / Swagger yaml or json file #56 Jun 1, 2020

thim81 mentioned this pull request Jun 1, 2020

Feature/cli converter options #218

Closed

Tim added 10 commits June 1, 2020 12:41

Minor change in the sample

c6f881f

Minor changes to the readme

2d4d9e1

Removed debug

4203d90

Added validation for schema.$ref

Added additional configuration options for the extendTests section.

1892e5a

It is now possible to `overwrite` all generateTests and extend them with specific settings per OpenApiOperationId.

Fixed code to comply to the Travis build checks

979eb80

Readme corrections

aca00be

Readme corrections

3bd9584

Improved readme

7a6f487

Added the Operation method & Path as part of the test description.

6fda601

This helps when running newman tests with reporting via CI (like Azure devops). The reporting will contain the references.

Minor optimisation of the method & path naming in the pm.tests

79e13ec

thim81 changed the title ~~New Feature - Generate tests from OpenAPI / Swagger yaml or json file #56~~ New Feature - Generate tests from OpenAPI / Swagger - automated contract testing #56 Jun 14, 2020

Merge branch 'develop' of https://github.com/postmanlabs/openapi-to-p…

d0ceea9

…ostman into develop

Tim added 6 commits July 13, 2020 13:38

Added option to define test suite limits, based on the OperationID. T…

53d2014

…his will only generate tests based on the defined list of operationId's.

Postman test generation - added support for anyOf, oneOf, allOf, $ref

84a52b7

TestExtension - added check if tests array exists

ac44b63

Merge branch 'develop' of https://github.com/postmanlabs/openapi-to-p…

19895fd

…ostman into develop

Postman test generation - added support for response header tests, wh…

c12b7ec

…ere the header is checked to present in the response

Merge branch 'develop' of https://github.com/postmanlabs/openapi-to-p…

6fb0edc

…ostman into develop

Tim added 2 commits March 9, 2021 00:04

Bug fix for path variable conversions when using a dynamic variable t…

b190b61

…o replace an integer example

Lint fix

8625e7e

Added option to convert unsupported OpenAPI(3.0) properties to valid …

af070d3

…JSON schema properties which deletes nullable and adds "null" to type array if nullable is true

Tim added 3 commits March 18, 2021 11:58

Added the option to assign pm.environment variables from the response…

a8363df

… headers

Moved all Postman Testsuite functions in a separate file for easier m…

a879d13

…aintenance

Merged version 2.5.1 from master

bbbf81a

thim81 mentioned this pull request Apr 20, 2021

Q: Is there an option to create Postman requests per response? #358

Closed

Tim added 10 commits April 20, 2021 17:28

Added support for removal of request headers, path variables, query p…

fba8363

…arams

Added the getRefObject as a temp workaround for dependency

b25fa44

Patch related to the stripping of "content-type" defined in OpenAPI postmanlabs#361

Bumped to openapi-to-postman 2.7.0 + the test suite generation

6c0d1b9

Added option to match target based on the OpenApi Operation (method & path)

Fixed lint errors

75f37ac

Removed unused variable

b841af0

Added option to disable the request query param in Postman

33a707c

Added "ContentCheck" feature

Extended the test suite assignPmVariables function by response header…

d8d10a1

…, request body or a plain value

Corrected the incorrect checkRequestBody variable definition to check…

cff6d71

…ResponseBody

Corrected the incorrect checkRequestBody variable definition to check…

b7c4877

…ResponseBody

grayside mentioned this pull request May 27, 2021

Decision: Use an OpenAPI specification for API design GoogleCloudPlatform/emblem#39

Closed

This was referenced Jul 24, 2021

How to add or convert a Collection Pre-Request Script? #158

Open

Ability to have custom functions called during conversion #26

Open

Support for import of servers defined in Open API spec #8

Open

Update README.md

14b8ff9

akshaydeo requested a review from VShingala April 3, 2023 09:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New Feature - Generate tests from OpenAPI / Swagger - automated contract testing #56 #225

New Feature - Generate tests from OpenAPI / Swagger - automated contract testing #56 #225

thim81 commented Jun 1, 2020 •

edited

thim81 commented Jun 15, 2020

VShingala commented Jun 15, 2020

thim81 commented Jun 24, 2020

thim81 commented Jul 1, 2020

kharoutaj commented Jul 11, 2020 •

edited

thim81 commented Jul 13, 2020

SidhNor commented Nov 16, 2020

thim81 commented Nov 16, 2020 •

edited

CleverCoder commented Mar 9, 2021

thim81 commented Mar 11, 2021

thim81 commented Jun 18, 2021 •

edited

thim81 commented Jul 13, 2021 •

edited

SidhNor commented Jul 13, 2021

ani-im commented Sep 10, 2021

thim81 commented Sep 10, 2021

New Feature - Generate tests from OpenAPI / Swagger - automated contract testing #56 #225

Are you sure you want to change the base?

New Feature - Generate tests from OpenAPI / Swagger - automated contract testing #56 #225

Conversation

thim81 commented Jun 1, 2020 • edited

thim81 commented Jun 15, 2020

VShingala commented Jun 15, 2020

thim81 commented Jun 24, 2020

thim81 commented Jul 1, 2020

kharoutaj commented Jul 11, 2020 • edited

thim81 commented Jul 13, 2020

SidhNor commented Nov 16, 2020

thim81 commented Nov 16, 2020 • edited

CleverCoder commented Mar 9, 2021

thim81 commented Mar 11, 2021

thim81 commented Jun 18, 2021 • edited

thim81 commented Jul 13, 2021 • edited

SidhNor commented Jul 13, 2021

ani-im commented Sep 10, 2021

thim81 commented Sep 10, 2021

thim81 commented Jun 1, 2020 •

edited

kharoutaj commented Jul 11, 2020 •

edited

thim81 commented Nov 16, 2020 •

edited

thim81 commented Jun 18, 2021 •

edited

thim81 commented Jul 13, 2021 •

edited