Validation rules: mismatch between graphql-js and spec

[IvanGoncharov]

Context #3282 by @spawnia

It looks like we have the following mismatch between spec and graphql-js.
For example, type names are checked using this rule:
https://github.com/graphql/graphql-js/blob/main/src/validation/rules/KnownTypeNamesRule.ts

But in spec, there is only a rule for checking type names on fragments and I couldn't find any rule for checking types on variable fragments.
Also, we have a significant mismatch in names and descriptions of rules.

So at some point, we need to do an audit of both spec and graphql-js to fix as many discrepancies as we can.
We need to focus on query validation first and handle SDL validation separately since it's not well documented in the spec.

[spawnia]

Renaming or changing validation rules is most likely going to be breaking. Given you just released https://github.com/graphql/graphql-js/releases/tag/v16.0.0-rc.3, perhaps now is not a good time to start working on this?

[IvanGoncharov]

@spawnia Yes, my main intention was to do an audit of discrepancies.
And also work on PRs to solve some of them on spec stuff, e.g. add stuff that we validating in JS but missing in spec like stuff that you spotted in #3282
Typically spec PR takes longer to merge so we can work on them first.

[leoloso]

I've done the mapping (as part of writing this article concerning generic error codes). @IvanGoncharov Is this what you had in mind?

The table below contains 3 colums:

• Message: The error message returned via new GraphQLError in the graphql-js source code
• Spec section: Which validation from the GraphQL spec it deals with
• Source code: File and line number where the new GraphQLError code appears

In the Spec section column, it may contain these extra strings:

• "Internal": when it relates to some internal functionality from graphql-js to build the schema, which is not documented in the schema, and does not need be documented in the schema
• "Missing in spec": validations which aren't found in the GraphQL spec, but should be there. They are all sensible validations, such as 'Cannot extend type "${typeName}" because it is not defined.', which we'd normally expect to be the case and, as such, we may take for granted (but should be defined in the spec nevertheless)
• "Optional": custom validations from graphql-js

Message	Spec section	Source code
Syntax Error: ${description}	2. Language	src/error/syntaxError.ts#L14
Must provide operation name if query contains multiple operations.	6.1. Executing Requests	src/execution/execute.ts#L299
Unknown operation named "${operationName}".	6.1. Executing Requests	src/execution/execute.ts#L318
Must provide an operation.	6.1. Executing Requests	src/execution/execute.ts#L320
Schema is not configured to execute ${operation.operation} operation.	6.2. Executing Operations	src/execution/execute.ts#L362
Expected Iterable, but did not find one for field "${info.parentType.name}.${info.fieldName}".	Internal - 3.11. List	src/execution/execute.ts#L719
Abstract type "${returnType.name}" must resolve to an Object type at runtime for field "${info.parentType.name}.${info.fieldName}". Either the "${returnType.name}" type should provide a "resolveType" function or each possible type should provide an "isTypeOf" function.	Internal - 3.6. Objects	src/execution/execute.ts#L859
Support for returning GraphQLObjectType from resolveType was removed in graphql-js@16.0.0 please return type name instead.	Internal - 3.6. Objects	src/execution/execute.ts#L868
Abstract type "${returnType.name}" must resolve to an Object type at runtime for field "${info.parentType.name}.${info.fieldName}" with	Internal - 3.6. Objects	src/execution/execute.ts#L874
Abstract type "${returnType.name}" was resolved to a type "${runtimeTypeName}" that does not exist inside the schema.	Internal - 3.6. Objects	src/execution/execute.ts#L882
Abstract type "${returnType.name}" was resolved to a non-object type "${runtimeTypeName}".	Internal - 3.6. Objects	src/execution/execute.ts#L889
Runtime Object type "${runtimeType.name}" is not a possible type for "${returnType.name}".	Internal - 3.6. Objects	src/execution/execute.ts#L896
Expected value of type "${returnType.name}" but got: ${inspect(result)}.	Internal	src/execution/execute.ts#L953
Schema is not configured to execute subscription operation.	3.3.1. Root Operation Types	src/execution/subscribe.ts#L195
The subscription field "${fieldName}" is not defined.	5.3.1. Field Selections	src/execution/subscribe.ts#L212
Too many errors processing variables, error limit reached. Execution aborted.	Internal	src/execution/values.ts#L54
Variable "$${varName}" expected value of type "${varTypeStr}" which cannot be used as an input type.	5.8.2. Variables Are Input Types	src/execution/values.ts#L87
Variable "$${varName}" of required type "${varTypeStr}" was not provided.	6.1.2. Coercing Variable Values	src/execution/values.ts#L101
Variable "$${varName}" of non-null type "${varTypeStr}" must not be null.	6.1.2. Coercing Variable Values	src/execution/values.ts#L114
Variable "$${varName}" got invalid value ${inspect(invalidValue)}	6.1.2. Coercing Variable Values	src/execution/values.ts#L132
Argument "${name}" of required type "${inspect(argType)} was not provided."	6.4.1. Coercing Field Arguments	src/execution/values.ts#L177
Argument "${name}" of required type "${inspect(argType)} was provided the variable "$${variableName}" which was not provided a runtime value."	6.4.1. Coercing Field Arguments	src/execution/values.ts#L198
Argument "${name}" of non-null type "${inspect(argType)} must not be null."	6.4.1. Coercing Field Arguments	src/execution/values.ts#L210
Argument "${name}" has invalid value ${print(valueNode)}.	6.4.1. Coercing Field Arguments	src/execution/values.ts#L223
Expected name to be a non-empty string.	B.3. Lexical Tokens	src/type/assertName.ts#L15
Names must only contain [_a-zA-Z0-9] but "${name}" does not.	B.3. Lexical Tokens	src/type/assertName.ts#L20
Names must start with [_a-zA-Z] but "${name}" does not.	B.3. Lexical Tokens	src/type/assertName.ts#L26
Enum values cannot be named: ${name}	B.4. Document Syntax	src/type/assertName.ts#L42
Enum "${this.name}" cannot represent value: ${inspect(outputValue)}	3.9. Enums	src/type/definition.ts#L1408
Enum "${this.name}" cannot represent non-string value: ${valueStr}.	3.9. Enums	src/type/definition.ts#L1418
Value "${inputValue}" does not exist in "${this.name}" enum.	3.9. Enums	src/type/definition.ts#L1426
Enum "${this.name}" cannot represent non-enum value: ${valueStr}.	3.9. Enums	src/type/definition.ts#L1441
Value "${valueStr}" does not exist in "${this.name}" enum.	3.9. Enums	src/type/definition.ts#L1451
Int cannot represent non-integer value: ${inspect(coercedValue)}	3.5.1. Int	src/type/scalars.ts#L42
Int cannot represent non 32-bit signed integer value:	3.5.1. Int	src/type/scalars.ts#L47
Int cannot represent non-integer value: ${inspect(inputValue)}	3.5.1. Int	src/type/scalars.ts#L57
Int cannot represent non 32-bit signed integer value: ${inputValue}	3.5.1. Int	src/type/scalars.ts#L62
Int cannot represent non-integer value: ${print(valueNode)}	3.5.1. Int	src/type/scalars.ts#L71
Int cannot represent non 32-bit signed integer value: ${valueNode.value}	3.5.1. Int	src/type/scalars.ts#L78
Float cannot represent non numeric value: ${inspect(coercedValue)}	3.5.2. Float	src/type/scalars.ts#L105
Float cannot represent non numeric value: ${inspect(inputValue)}	3.5.2. Float	src/type/scalars.ts#L114
Float cannot represent non numeric value: ${print(valueNode)}	3.5.2. Float	src/type/scalars.ts#L123
String cannot represent value: ${inspect(outputValue)}	3.5.3. String	src/type/scalars.ts#L151
String cannot represent a non string value: ${inspect(inputValue)}	3.5.3. String	src/type/scalars.ts#L158
String cannot represent a non string value: ${print(valueNode)}	3.5.3. String	src/type/scalars.ts#L167
Boolean cannot represent a non boolean value: ${inspect(coercedValue)}	3.5.4. Boolean	src/type/scalars.ts#L189
Boolean cannot represent a non boolean value: ${inspect(inputValue)}	3.5.4. Boolean	src/type/scalars.ts#L196
Boolean cannot represent a non boolean value: ${print(valueNode)}	3.5.4. Boolean	src/type/scalars.ts#L205
ID cannot represent value: ${inspect(outputValue)}	3.5.5. ID	src/type/scalars.ts#L228
ID cannot represent value: ${inspect(inputValue)}	3.5.5. ID	src/type/scalars.ts#L240
ID cannot represent a non-string and non-integer value: print(valueNode)	3.5.5. ID	src/type/scalars.ts#L245
Name "${name}" must not begin with "__", which is reserved by GraphQL introspection.	4. Introspection => Reserved Names	src/utilities/assertValidName.ts#L28
Expected non-nullable type "${inspect(type)}" not to be null.	3.12. Non-Null => Input Coercion	src/utilities/coerceInputValue.ts#L64
Expected type "${type.name}" to be an object.	3.10. Input Object => Input Coercion	src/utilities/coerceInputValue.ts#L93
Field "${field.name}" of required type "${typeStr}" was not provided.	3.10. Input Object => Input Coercion	src/utilities/coerceInputValue.ts#L112
Field "${fieldName}" is not defined by type "${type.name}".	3.10. Input Object => Input Coercion	src/utilities/coerceInputValue.ts#L138
Expected type "${type.name}". ${error.message}	3.5. Scalars => Input Coercion | 3.9. Enums => Input Coercion	src/utilities/coerceInputValue.ts#L163
Expected type "${type.name}".	3.5. Scalars => Input Coercion | 3.9. Enums => Input Coercion	src/utilities/coerceInputValue.ts#L179
Schema does not define the required query root type.	3.3.1. Root Operation Types	src/utilities/getOperationRootType.ts#L23
Schema is not configured for mutations.	3.3.1. Root Operation Types	src/utilities/getOperationRootType.ts#L34
Schema is not configured for subscriptions.	3.3.1. Root Operation Types	src/utilities/getOperationRootType.ts#L45
Can only have query, mutation and subscription operations.	3.3.1. Root Operation Types	src/utilities/getOperationRootType.ts#L53
Too many validation errors, error limit reached. Validation aborted.	Internal	src/validation/validate.ts#L62
The ${defName} definition is not executable.	5.1.1. Executable Definitions	src/validation/rules/ExecutableDefinitionsRule.ts#L30
Cannot query field "${fieldName}" on type "${type.name}".	5.3.1. Field Selections	src/validation/rules/FieldsOnCorrectTypeRule.ts#L58
Fragment cannot condition on non composite type "${typeStr}".	5.5.1.3. Fragments On Composite Types	src/validation/rules/FragmentsOnCompositeTypesRule.ts#L32
Fragment "${node.name.value}" cannot condition on non composite type "${typeStr}".	5.5.1.3. Fragments On Composite Types	src/validation/rules/FragmentsOnCompositeTypesRule.ts#L45
Unknown argument "${argName}" on field "${parentType.name}.${fieldDef.name}".	5.4.1. Argument Names	src/validation/rules/KnownArgumentNamesRule.ts#L39
Unknown argument "${argName}" on directive "@${directiveName}".	5.4.1. Argument Names	src/validation/rules/KnownArgumentNamesRule.ts#L87
Unknown directive "@${name}".	5.7.1. Directives Are Defined	src/validation/rules/KnownDirectivesRule.ts#L54
Directive "@${name}" may not be used on ${candidateLocation}.	5.7.2. Directives Are In Valid Locations	src/validation/rules/KnownDirectivesRule.ts#L62
Unknown fragment "${fragmentName}".	5.5.1.2. Fragment Spread Type Existence	src/validation/rules/KnownFragmentNamesRule.ts#L22
Unknown type "${typeName}".	5.5.1.2. Fragment Spread Type Existence	src/validation/rules/KnownTypeNamesRule.ts#L63
This anonymous operation must be the only defined operation.	5.2.2.1. Lone Anonymous Operation	src/validation/rules/LoneAnonymousOperationRule.ts#L29
Cannot define a new schema within a schema extension.	Missing in spec - 3.3. Schema	src/validation/rules/LoneSchemaDefinitionRule.ts#L27
Must provide only one schema definition.	Missing in spec - 3.3. Schema	src/validation/rules/LoneSchemaDefinitionRule.ts#L37
Cannot spread fragment "${spreadName}" within itself	5.5.2.2. Fragment spreads must not form cycles	src/validation/rules/NoFragmentCyclesRule.ts#L78
Variable "$${varName}" is not defined by operation "${operation.name.value}".	5.8.3. All Variable Uses Defined	src/validation/rules/NoUndefinedVariablesRule.ts#L32
Fragment "${fragName}" is never used.	5.5.1.4. Fragments Must Be Used	src/validation/rules/NoUnusedFragmentsRule.ts#L49
Variable "$${variableName}" is never used in operation "${operation.name.value}".	5.8.4. All Variables Used	src/validation/rules/NoUnusedVariablesRule.ts#L36
Fields "${responseName}" conflict because ${reasonMsg}. Use different aliases on the fields to fetch both if this was intentional.	5.3.2. Field Selection Merging	src/validation/rules/OverlappingFieldsCanBeMergedRule.ts#L83
Fragment cannot be spread here as objects of type "${parentTypeStr}" can never be of type "${fragTypeStr}".	5.5.2.3. Fragment spread is possible	src/validation/rules/PossibleFragmentSpreadsRule.ts#L38
Fragment "${fragName}" cannot be spread here as objects of type "${parentTypeStr}" can never be of type "${fragTypeStr}".	5.5.2.3. Fragment spread is possible	src/validation/rules/PossibleFragmentSpreadsRule.ts#L57
Cannot extend non-${kindStr} type "${typeName}".	Missing in spec - 3.4.3. Type Extensions	src/validation/rules/PossibleTypeExtensionsRule.ts#L68
Cannot extend type "${typeName}" because it is not defined.	Missing in spec - 3.4.3. Type Extensions	src/validation/rules/PossibleTypeExtensionsRule.ts#L82
Field "${fieldDef.name}" argument "${argDef.name}" of type "${argTypeStr}" is required, but it was not provided.	5.4.2.1. Required Arguments	src/validation/rules/ProvidedRequiredArgumentsRule.ts#L50
Directive "@${directiveName}" argument "${argName}" of type "${argType}" is required, but it was not provided.	5.4.2.1. Required Arguments	src/validation/rules/ProvidedRequiredArgumentsRule.ts#L112
Field "${fieldName}" must not have a selection since type "${typeStr}" has no subfields.	5.3.3. Leaf Field Selections	src/validation/rules/ScalarLeafsRule.ts#L29
Field "${fieldName}" of type "${typeStr}" must have a selection of subfields. Did you mean "${fieldName} { ... }"?	5.3.3. Leaf Field Selections	src/validation/rules/ScalarLeafsRule.ts#L39
Subscription "${operationName}" must not select an introspection top level field.	5.2.3.1. Single root field	src/validation/rules/SingleFieldSubscriptionsRule.ts#L56
Subscription "${operationName}" must not select an introspection top level field.	5.2.3.1. Single root field	src/validation/rules/SingleFieldSubscriptionsRule.ts#L69
Argument "${parentName}(${argName}:)" can only be defined once.	5.4.2. Argument Uniqueness	src/validation/rules/UniqueArgumentDefinitionNamesRule.ts#L69
There can be only one argument named "${argName}".	5.4.2. Argument Uniqueness	src/validation/rules/UniqueArgumentNamesRule.ts#L38
Directive "@${directiveName}" already exists in the schema. It cannot be redefined.	Missing in spec - 3.13. Directives => Validation	src/validation/rules/UniqueDirectiveNamesRule.ts#L24
There can be only one directive named "@${directiveName}".	Missing in spec - 3.13. Directives => Validation	src/validation/rules/UniqueDirectiveNamesRule.ts#L34
The directive "@${directiveName}" can only be used once at this location.	5.7.3. Directives Are Unique Per Location	src/validation/rules/UniqueDirectivesPerLocationRule.ts#L79
Enum value "${typeName}.${valueName}" already exists in the schema. It cannot also be defined in this type extension.	3.9. Enums => Type Validation	src/validation/rules/UniqueEnumValueNamesRule.ts#L50
Enum value "${typeName}.${valueName}" can only be defined once.	3.9. Enums => Type Validation	src/validation/rules/UniqueEnumValueNamesRule.ts#L56
Field "${typeName}.${fieldName}" already exists in the schema. It cannot also be defined in this type extension.	3.6. Objects => Type Validation	src/validation/rules/UniqueFieldDefinitionNamesRule.ts#L62
Field "${typeName}.${fieldName}" can only be defined once.	3.6. Objects => Type Validation	src/validation/rules/UniqueFieldDefinitionNamesRule.ts#L69
There can be only one fragment named "${fragmentName}".	5.5.1.1. Fragment Name Uniqueness	src/validation/rules/UniqueFragmentNamesRule.ts#L24
There can be only one input field named "${fieldName}".	5.6.3. Input Object Field Uniqueness	src/validation/rules/UniqueInputFieldNamesRule.ts#L41
There can be only one operation named "${operationName.value}".	5.2.1.1. Operation Name Uniqueness	src/validation/rules/UniqueOperationNamesRule.ts#L24
Type for ${operation} already defined in the schema. It cannot be redefined.	Missing in spec - 3.3.1. Root Operation Types	src/validation/rules/UniqueOperationTypesRule.ts#L47
There can be only one ${operation} type in schema.	Missing in spec - 3.3.1. Root Operation Types	src/validation/rules/UniqueOperationTypesRule.ts#L54
Type "${typeName}" already exists in the schema. It cannot also be defined in this type definition.	Missing in spec - 3.4. Types	src/validation/rules/UniqueTypeNamesRule.ts#L31
There can be only one type named "${typeName}".	Missing in spec - 3.4. Types	src/validation/rules/UniqueTypeNamesRule.ts#L41
There can be only one variable named "$${variableName}".	5.8.1. Variable Uniqueness	src/validation/rules/UniqueVariableNamesRule.ts#L31
Field "${type.name}.${fieldDef.name}" of required type "${typeStr}" was not provided.	5.6.1. Values of Correct Type	src/validation/rules/ValuesOfCorrectTypeRule.ts#L58
Field "${node.name.value}" is not defined by type "${parentType.name}".	5.6.1. Values of Correct Type	src/validation/rules/ValuesOfCorrectTypeRule.ts#L75
Expected value of type "${inspect(type)}", found ${print(node)}.	5.6.1. Values of Correct Type	src/validation/rules/ValuesOfCorrectTypeRule.ts#L87
Expected value of type "${typeStr}", found ${print(node)}.	5.6.1. Values of Correct Type	src/validation/rules/ValuesOfCorrectTypeRule.ts#L118
Expected value of type "${typeStr}", found ${print(node)}.	5.6.1. Values of Correct Type	src/validation/rules/ValuesOfCorrectTypeRule.ts#L133
Expected value of type "${typeStr}", found ${print(node)};	5.6.1. Values of Correct Type	src/validation/rules/ValuesOfCorrectTypeRule.ts#L145
Variable "$${variableName}" cannot be non-input type "${typeName}".	5.8.2. Variables Are Input Types	src/validation/rules/VariablesAreInputTypesRule.ts#L33
Variable "$${varName}" of type "${varTypeStr}" used in position expecting type "${typeStr}".	5.8.5. All Variable Usages are Allowed	src/validation/rules/VariablesInAllowedPositionRule.ts#L63
The field ${parentType.name}.${fieldDef.name} is deprecated. ${deprecationReason}	Optional - 4.2. Introspection => Deprecation	src/validation/rules/custom/NoDeprecatedCustomRule.ts#L30
Directive "@${directiveDef.name}" argument "${argDef.name}" is deprecated. ${deprecationReason}	Optional - 4.2. Introspection => Deprecation	src/validation/rules/custom/NoDeprecatedCustomRule.ts#L44
Field "${parentType.name}.${fieldDef.name}" argument "${argDef.name}" is deprecated. ${deprecationReason}	Optional - 4.2. Introspection => Deprecation	src/validation/rules/custom/NoDeprecatedCustomRule.ts#L54
The input field ${inputObjectDef.name}.${inputFieldDef.name} is deprecated. ${deprecationReason}	Optional - 4.2. Introspection => Deprecation	src/validation/rules/custom/NoDeprecatedCustomRule.ts#L69
The enum value "${enumTypeDef.name}.${enumValueDef.name}" is deprecated. ${deprecationReason}	Optional - 4.2. Introspection => Deprecation	src/validation/rules/custom/NoDeprecatedCustomRule.ts#L84
GraphQL introspection has been disabled, but the requested query contained the field "${node.name.value}".	Optional - 4. Introspection	src/validation/rules/custom/NoSchemaIntrospectionCustomRule.ts#L29

[italosantana]

I'm having this error in my code. I saw that in the source code, it will be fixed in the next release 17.0.0.

Has anyone had this error?

graphql-js/src/execution/execute.ts

Line 868 in da57238

throw new GraphQLError(

Support for returning GraphQLObjectType from resolveType was removed in graphql-js@16.0.0 please return type name instead.
[ { line: 9, column: 3 } ]
GraphQLError: Support for returning GraphQLObjectType from resolveType was removed in graphql-js@16.0.0 please return type name instead.
    at ensureValidRuntimeType (/home/italopc/workspace/safe/node_modules/graphql/execution/execute.js:829:11)
    at completeAbstractValue (/home/italopc/workspace/safe/node_modules/graphql/execution/execute.js:797:5)
    at completeValue (/home/italopc/workspace/safe/node_modules/graphql/execution/execute.js:624:12)
    at /home/italopc/workspace/safe/node_modules/graphql/execution/execute.js:486:9
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Promise.all (index 0)
    at async middleware (/home/italopc/workspace/safe/node_modules/koa-graphql/index.js:132:26)
    at async logger (/home/italopc/workspace/safe/node_modules/koa-logger/index.js:67:7)
    at async bodyParser (/home/italopc/workspace/safe/node_modules/koa-bodyparser/index.js:95:5)