Change Request: (add eslint warnings exit code)

[alSergey]

ESLint version

v8.57.0

What problem do you want to solve?

We have a fe eslint job in GitLab CI that runs eslint checks. Currently there is no way to show a warning in a job if eslint ends with a warning.

What do you think is the correct solution?

Add a new exit code, which will be triggered if eslint checks find 0 error and more than 0 warning.

fe eslint:
  script:
    - npx eslint .
   allow_failure:
      exit_codes: X

Participation

• I am willing to submit a pull request for this change.

Additional comments

No response

[alSergey]

Just in case, I’ll explain what I want:

• eslint command completed successfully (0 errors, 0 warnings) - job successful
• eslint command ended with warnings (0 errors, more than 0 warnings) - job warning
• eslint command ended with errors (more than 0 errors) - job errors

In GitLab CI this can be implemented using exit code.

[fasttime]

Thanks for the issue @alSergey. As you may already know, eslint will terminate with exit code 1 if some rule reports an error. If rules report only warnings, the exit code will be 0, but this can be changed by setting the --max-warnings option to 0, e.g.:

npx eslint . --max-warnings=0

And then any warning will cause eslint to terminate with exit code 1 like errors do.

There is no way to distinguish between only warnings and errors plus warnings just from the exit code, this can be achieved with a custom formatter.

But anyway, talking about GitLab CI, I'm not sure that you can selectively set the status of a job to warning or error from the exit code of the last command: the related issue is closed and I don't see that feature in their documentation, so I would recommend that you solve that problem first, and update the yaml code in your issue to show what you are planning to do. This would help us to understand how we can help you.

[alSergey]

@fasttime hi, I know about the --max-warnings setting, it won't help me. As I wrote above, I would like to show the job as warning if the eslint command completed with 0 errors and at least one warning.

I suggest adding a new feature flag like --max-warnings, which will control the behavior of eslint in case of termination with warnings. If we talk about GitLab CI, they have the allow_failure.exit_codes setting, which helps do what I need. How I see my .gitlab-ci.yml:

fe eslint:
  script:
    - npx eslint . --warnings-exit-code
   allow_failure:
      exit_codes: 15 # This is the exit code with which eslint will exit in case of warnings

[fasttime]

If we talk about GitLab CI, they have the allow_failure.exit_codes setting, which helps do what I need. How I see my .gitlab-ci.yml:

fe eslint:
  script:
    - npx eslint . --warnings-exit-code
   allow_failure:
      exit_codes: 15 # This is the exit code with which eslint will exit in case of warnings

So, that job will complete as successful if the exit code of the command is 0, with a warning if the exit code is 15, and with an error for any other exit code? I'm just trying to understand how things work, because the docs about allow_failure.exit_codes doesn't explain it.

Do you know if GitLab provides other ways to set the status of a job to a warning or error without checking the exit code, maybe an API? That could be simpler to implement for your case, because currently the ESLint CLI does not provide a way for plugins or formatters to control the exit code.

[alSergey]

So, that job will complete as successful if the exit code of the command is 0, with a warning if the exit code is 15, and with an error for any other exit code?

Yes, everything will be exactly as you said. I wrote a small example where you can see how it works. And this is what the pipeline looks like.

Do you know if GitLab provides other ways to set the status of a job to a warning or error without checking the exit code, maybe an API?

Unfortunately, I don’t know any other options for making a job warning, only checking for exit code. This is a built-in feature in GitLab pipelines that is widely used, so it seems like it should be used.

The allow_failure command allows you to select in which cases the job will end with an error, and in which cases with a warning. And this command has an exit_codes setting. Here is an excerpt from the documentation that describes how the command works.

When jobs are allowed to fail an orange warning indicates that a job failed. However, the pipeline is successful and the associated commit is marked as passed with no warnings.

[fasttime]

Thanks for the useful clarifications @alSergey! The reason why ESLint exits with code zero for warnings is because nonzero exit codes are typically treated as errors, and unlike errors, warnings should not enforce rule compliance or break build commands (see Rule Severities).

Now, GitLab seems to have a different idea: in order to report a warning, a command must exit with an error code - if it's true that there are no other options - and that code has to be specially annotated in their job definition. I can see that this makes it difficult to integrate GitLab's allow_failure feature with ESLint, or with any other tool that doesn't use a distinct exit code to report warnings.

Given the popularity of GitLab CI/CD, maybe it would be good to find a solution to improve this situation. I'd like to know what other members of the team think about this. @eslint/eslint-team.

[alSergey]

@fasttime Hi, I understand why the eslint command ends with exit code 0 when there are only warnings. In order not to break this behavior and to allow the job to be displayed as a warning in GitLab CI, I propose to make a feature flag.

It will work something like this. By default, the eslint command will ends with exit code 0 if there are only warnings, but if you pass this feature flag, it will ends with a different exit code. Example.

Errors and Warnings

$ npx eslint .
...
✖ 2 problems (1 error, 1 warning)

$ echo $?
1

$ npx eslint . --warnings-exit-code
...
✖ 2 problems (1 error, 1 warning)

$ echo $?
1 # Old exit code because there are errors

Only Warnings

$ npx eslint .
...
⚠ 1 problem (0 error, 1 warning)

$ echo $?
0

$ npx eslint . --warnings-exit-code
...
⚠ 1 problem (0 errors, 1 warning)

$ echo $?
3 # New exit code with feature flag !!!

All Success

$ npx eslint .
...
0 problems (0 error, 0 warning)

$ echo $?
0

$ npx eslint . --warnings-exit-code
...
0 problems (0 error, 0 warning)

$ echo $?
0 # Old exit code because there are no warnings

[mdjermanovic]

I wouldn't be opposed to adding this flag if others are in favor, though you can always pass the output from ESLint to another command or script that can then analyze it and determine the final exit code.

Here's an example I found with eslint warnings and allow_failure:

https://blog.vasi.li/soft-failing-the-build-in-gitlab-ci-on-eslint-warnings

[alSergey]

Yes, you are right. It really can be done as you wrote. But this introduces additional difficulties. First you need to parse the output and understand whether there is only a warning, or there are also errors. Then you need to print the original output so that it appears in the console. At the very end, you need to enter the exit code yourself.

Given the popularity of GitLab CI, I would like to have a more native solution to this problem.

[aladdin-add]

https://eslint.org/docs/latest/use/command-line-interface#exit-codes

1: Linting was successful and there is at least one linting error, or there are more linting warnings than allowed by the --max-warnings option.

I'm👍 to distinction between the two cases in the exitcode (e.g. 3 for --max-warnings)(and possible parsing errors?). better to put an RFC to explore: https://github.com/eslint/rfcs

[nzakas]

I'm not opposed to adding a new flag, but we'd definitely need an RFC for that.

[alSergey]

@nzakas hi, is there anything you need from me?

[fasttime]

@alSergey I think there is agreement that we can add a CLI flag to change the exit code as you suggested, but some reviewers consider this change sufficiently complex that it will require a RFC (request for comments) to discuss the details and formalize the process of finding the best design. To start a RFC, just follow the instructions in this readme file and fill in the design template with the requested information.