Improve the missing backends exception

[mpkorstanje]

🤔 What's the problem you're trying to solve?

When users encounter this exception:

io.cucumber.core.exception.CucumberException: No backends were found. Please make sure you have a backend module on your CLASSPATH.

	at io.cucumber.core.runtime.BackendServiceLoader.get(BackendServiceLoader.java:39)
	at io.cucumber.core.runtime.BackendServiceLoader.get(BackendServiceLoader.java:33)
	at io.cucumber.core.runtime.SingletonRunnerSupplier.creat

It is not clear at all what they've done wrong.

✨ What's your proposed solution?

Improve the exception. Explain that Cucumber consists of

• something to run Cucumber (cucumber-core)
• something to declare step defintions (the backend)
• optionally a DI container.
• optionally integration with JUnit4/5/TestNG/ect

[relentless-pursuit]

When you talk about improving the exception message using step definitions, are you asking to verify the step definitions if they are a part of classpath?

I have added a sample code for your reference.

// Verify that step definitions have been declared and are available on the classpath
try (ScanResult scanResult = new ClassGraph().enableClassInfo().scan()) {
ClassInfoList stepDefinitionClasses = scanResult.getClassesWithAnnotation("io.cucumber.java.*");
if (stepDefinitionClasses.isEmpty()) {
String errorMessage = "No step definitions were found. Please make sure you have declared your step definitions on the classpath.";
LOGGER.error(errorMessage);
throw new CucumberException(errorMessage);
}
}

I am using the ClassGraph library to scan the classpath for classes that have been annotated with @given, @when, or @then annotations. If no step definition classes are found, we throw a CucumberException with an appropriate error message.

Does this improve the exception message?

[mpkorstanje]

The exception I originally mentioned typically happens when a user tries to use cucumber without either cucumber-java or cucumber-java8 as a dependency. These dependencies contain the JavaBackend and Java8Backend which are Backend implementations. The exception you mention would happen only if/when either backend is present but doesn't discover any step definitions.

[relentless-pursuit]

The exception I originally mentioned typically happens when a user tries to use cucumber without either cucumber-java or cucumber-java8 as a dependency. These dependencies contain the JavaBackend and Java8Backend which are Backend implementations. The exception you mention would happen only if/when either backend is present but doesn't discover any step definitions.

Thank you for your response. So, I assume, we just need to add few details to the error message. Can I improve the exception with the following message:

"No backends were found. Cucumber consists of two main components: cucumber-core, which runs Cucumber, and a backend, which declares step definitions. Please make sure you have a backend module on your CLASSPATH, such as cucumber-java or cucumber-java8, which contain the JavaBackend and Java8Backend implementations. If you already have a backend module on your CLASSPATH, this error may occur if there are no step definitions present in your project."

This message provides more information about the cause of the exception, explains the role of the backend in Cucumber, and provides suggestions for resolving the issue. Additionally, it mentions the possibility of the error occurring when there are no step definitions present in the project, which could be a common cause of the error.

[mpkorstanje]

Aside from being factually incorrect, the message you propose is rather woolly and mentions (to the user) irrelevant implementation details.

Out of curiosity, did you just ask ChatGTP to write an error message based on my input?

[relentless-pursuit]

Aside from being factually incorrect, the message you propose is rather woolly and mentions (to the user) irrelevant implementation details.

Out of curiosity, did you just ask ChatGTP to write an error message based on my input?

Pardon me. I am new to open source contribution. Though, I had exposure to cucumber testing framework during my spring-boot-uplift at work. I have been taking chatGPT's help lately. However, I do intend to act responsibly and make sure have my doubts cleared before I initiate a PR. It can become little difficult to comprehend the issues, therefore ChatGPT. However, your guidance on the same would be much appreciated.

I understood your above comment. Is it possible for you to showcase what an exception message would look like, an example maybe? I have understood the DI container, and how can it be used in exception message. But, I am still confused about point 1 and 2 in your propsoed solution.

[mpkorstanje]

That's cool.

Unfortunately ChatGTP is only a language model. This means that it will generate text that sounds plausibele and convincing but also has no basis in factual reality. If you do not have the knowledge of the underlying system you will not be able to tell ofcourse. So you are setting yourself up to be disappointed.

Additionally, your real writing voice has certain mannerisms (I would guess you are from India) while ChatGTP has other distinct mannerisms. The switch between the two is rather obvious.

I would prefer it if going forward you didn't use ChatGTP at all. I would rather talk to a person in what I assume is a second language for both of us. I don't mind if it's not perfect. I would rather see that you learn from it.

Now I would like to show you want the exception should look like but writing it will take a little time. So I made this issue as a reminder for the future.

Instead I think it is better if you spend some trying to understand the problem. Perhaps you can start by reproducing it. You can use the skeleton project to get started quickly. Removing the cucumber-java dependency should do the trick.

Once you have the exception. Look at the code that causes it. If you can explain to me what is going on there we can move on to the next step.

https://github.com/cucumber/cucumber-java-skeleton

[relentless-pursuit]

Yes, I am from India. Thank you for such warm gesture. I did as mentioned starting with Java-Skeleton repository. So, without removing the cucumber-java, it already had some issues with steps. Whenever I try to run my scenario from feature file, it gave me following error:

Scenario: a few cukes               # io/cucumber/skeleton/belly.feature:3
  Given I have 42 cukes in my belly # io.cucumber.skeleton.StepDefinitions.I_have_cukes_in_my_belly(int)
  When I wait 1 hour
  Then my belly should growl
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.088 s <<< FAILURE! - in io.cucumber.skeleton.RunCucumberTest
[ERROR] Belly.Belly - a few cukes  Time elapsed: 0.051 s  <<< ERROR!
io.cucumber.junit.platform.engine.UndefinedStepException: 
The step 'I wait 1 hour' and 1 other step(s) are undefined.
You can implement these steps using the snippet(s) below:

@When("I wait {int} hour")
public void i_wait_hour(Integer int1) {
    // Write code here that turns the phrase above into concrete actions
    throw new io.cucumber.java.PendingException();
}
@Then("my belly should growl")
public void my_belly_should_growl() {
    // Write code here that turns the phrase above into concrete actions
    throw new io.cucumber.java.PendingException();
}

However, it had no impact even if i had commented out cucumber-java dependency from pom.xml. here:

Do you want me to explain UndefinedStepException?
As I understand, the feature file's When and Then are not being implemented in StepDefinition Class. And, for that reason, it is showing UndefinedStepException. And, as suggestion, It is asking them to be implemented in StepDefinition class as mentioned above.

But, I also corrected them and did pass my scenarios. Now, when everything was working fine after having implemented in my StepDefinition class, I did comment out cucumber-java dependency to see the result. And, that threw me compilation errors.

I hope I didn't confuse you.

[mpkorstanje]

Sounds like you're getting there!

If you want to use the project after removing the cucumber-java dependency you'll also have to remove the classes that use imports from this dependency. This might make the project look non-functional but should still get you the error we're trying to reproduce.

One of the real world situations this may happen in is when people only upgrade cucumber-core and not cucumber-java. For example this Stackoverflow question. Or leave out cucumber-java altogether.

[relentless-pursuit]

Before All/After All failed
io.cucumber.core.exception.CompositeCucumberException: There were 2 exceptions. The details are in the stacktrace below.
	at io.cucumber.core.runtime.RethrowingThrowableCollector.getThrowable(RethrowingThrowableCollector.java:57)
	at io.cucumber.core.runtime.CucumberExecutionContext.getThrowable(CucumberExecutionContext.java:102)
	at io.cucumber.core.runtime.CucumberExecutionContext.finishTestRun(CucumberExecutionContext.java:97)
	at io.cucumber.core.runtime.Runtime.run(Runtime.java:89)
	at io.cucumber.core.cli.Main.run(Main.java:87)
	at io.cucumber.core.cli.Main.main(Main.java:30)
	Suppressed: io.cucumber.core.exception.CucumberException: No backends were found. Please make sure you have a backend module on your CLASSPATH.
		at io.cucumber.core.runtime.BackendServiceLoader.get(BackendServiceLoader.java:39)
		at io.cucumber.core.runtime.BackendServiceLoader.get(BackendServiceLoader.java:33)
		at io.cucumber.core.runtime.SingletonRunnerSupplier.createRunner(SingletonRunnerSupplier.java:43)
		at io.cucumber.core.runtime.SingletonRunnerSupplier.get(SingletonRunnerSupplier.java:35)
		at io.cucumber.core.runtime.RethrowingThrowableCollector.executeAndThrow(RethrowingThrowableCollector.java:35)
		at io.cucumber.core.runtime.CucumberExecutionContext.getRunner(CucumberExecutionContext.java:134)
		at io.cucumber.core.runtime.CucumberExecutionContext.runBeforeAllHooks(CucumberExecutionContext.java:86)
		at io.cucumber.core.runtime.Runtime.lambda$run$0(Runtime.java:83)
		at io.cucumber.core.runtime.Runtime.execute(Runtime.java:99)
		at io.cucumber.core.runtime.Runtime.run(Runtime.java:82)
		... 2 more
	Suppressed: io.cucumber.core.exception.CucumberException: No backends were found. Please make sure you have a backend module on your CLASSPATH.
		at io.cucumber.core.runtime.BackendServiceLoader.get(BackendServiceLoader.java:39)
		at io.cucumber.core.runtime.BackendServiceLoader.get(BackendServiceLoader.java:33)
		at io.cucumber.core.runtime.SingletonRunnerSupplier.createRunner(SingletonRunnerSupplier.java:43)
		at io.cucumber.core.runtime.SingletonRunnerSupplier.get(SingletonRunnerSupplier.java:35)
		at io.cucumber.core.runtime.RethrowingThrowableCollector.executeAndThrow(RethrowingThrowableCollector.java:35)
		at io.cucumber.core.runtime.CucumberExecutionContext.getRunner(CucumberExecutionContext.java:134)
		at io.cucumber.core.runtime.CucumberExecutionContext.runAfterAllHooks(CucumberExecutionContext.java:91)
		at io.cucumber.core.runtime.Runtime.execute(Runtime.java:99)
		at io.cucumber.core.runtime.Runtime.run(Runtime.java:87)
		... 2 more

Now, I am able to reproduce the exception. Thanks a ton. If I could try to explain the exception, cucumber-jvm repository has a Class named BackendServiceLoader. And, it throws the above exception when it fails to create Backend instance. And, the reasons for such an exception can be as mentioned above i.e. If the user has left cucumber-java dependency in their pom.xml or upgrading cucumber-core and not cucumber-java. There could be other possible scenarios too, which I am not aware.

How to proceed from here?

[mpkorstanje]

So imagine you're using Cucumber. You're also new to software development, testing and Cucumber. Then you get this error message. It's not very helpful.

Can you change it in such a way that becomes helpful?

I imagine that this explanation might have a few parts. One to explain that cucumber has multiple modules, one to explain what those modules are, one to explain the problem and finally how to solve it.

edit: It's night here. You can think about it for a while.

[relentless-pursuit]

No backends were found. Cucumber relies on backend module to run steps. It appears to be missing from your CLASSPATH. Please make sure you have a backend module in your project's dependencies or explicitly added to your CLASSPATH. For example, you can add the cucumber-jvm backend module to your project. If you already have cucumber-jvm in your CLASSPATH, please make sure the version of cucumber-core and cucumber-jvm are in sync.

What improvement/correction could I bring in to the above exception message?

[mpkorstanje]

Cucumber relies on backend module to run steps.

Sounds about right. The module both discovers and run steps.

Please make sure you have a backend module in your project's dependencies or explicitly added to your CLASSPATH.

Yes!

For example, you can add the cucumber-jvm backend module to your project.

I don't think this is quite right. Did you try this as a solution in tbe the broken project we are using to reproduce the problem?

If you already have cucumber-jvm in your CLASSPATH, please make sure the version of cucumber-core and cucumber-jvm are in sync.

Very nice! I didn't think to include that.

[relentless-pursuit]

Cucumber relies on backend module to run steps.

Sounds about right. The module both discovers and run steps.

Please make sure you have a backend module in your project's dependencies or explicitly added to your CLASSPATH.

Yes!

For example, you can add the cucumber-jvm backend module to your project.

I don't think this is quite right. Did you try this as a solution in tbe the broken project we are using to reproduce the problem?

If you already have cucumber-jvm in your CLASSPATH, please make sure the version of cucumber-core and cucumber-jvm are in sync.

Very nice! I didn't think to include that.

I am a bit confused. When I say this, 'For example, you can add the cucumber-jvm backend module to your project.', Doesn't it guide the customer/user to add a backend module? And, isn't cucumber-jvm that backend module?

edit: Should it be cucumber-java instead of cucumber-jvm for the statement 'For example, you can add the cucumber-jvm backend module to your project.'?

[mpkorstanje]

The best way to know if your instructions fix the problem is to try it out. You've made a reproducer so you can use it to test what happens when you use cucumber-jvm or cucumber-java.

Once you've tried it and know which one works, we can look at why the other one didn't.

[relentless-pursuit]

So, as mentioned above, I just uncommented cucumber-java and the corresponding step class, and i no longer could see the error message i.e.

[ERROR] Errors: 
[ERROR]   No backends were found. Please make sure you have a backend module on your CLASSPATH.

And, when I visit the documentation web page of Cucumber, I see the dependency for cucumber-jvm mentioned as

<dependency>
    <groupId>io.cucumber</groupId>
    <artifactId>cucumber-java</artifactId>
    <version>7.11.1</version>
    <scope>test</scope>
</dependency>

Does this mean I can add this to the exception message?
'For example, you can add the cucumber-jvm backend module to your project.'

Reference:https://cucumber.io/docs/installation/java/ #

[mpkorstanje]

I think you understand what to do to trigger the problem and to make it go away. But I don't think you've quite connected that yet to helping someone fix the problem.

I think if you told someone to add the cucumber-jvm as a backend, they'd would literally try that. Try doing that and see what happens.

[relentless-pursuit]

I think you understand what to do to trigger the problem and to make it go away. But I don't think you've quite connected that yet to helping someone fix the problem.

I think if you told someone to add the cucumber-jvm as a backend, they'd would literally try that. Try doing that and see what happens.

Got it. I deleted my earlier response. I would have a more refined exception message as below:

No backends were found. Cucumber relies on backend module to discover and run steps. It appears to be missing from your CLASSPATH. Please make sure you have a backend module in your project's dependencies or explicitly added to your CLASSPATH. If you already have cucumber-jvm in your CLASSPATH, please make sure the version of cucumber-core and cucumber-jvm are in sync.

Edit: How about a slight change in the last line? Does it make sense?
If you already have backend module in your CLASSPATH, please make sure Cucumber version is the same for all Cucumber dependencies.

[mpkorstanje]

So, I made a project to reproduce this problem and I keep following your advice of adding cucumber-jvm as a dependency. It doesn't work, the error doesn't go away!

Can you tell me what I'm doing wrong?

And perhaps that sounds a bit silly. We both know the solution. But we are trying to help people who don't know. So we must act like people who don't know if we want to test the advice we give them.

If it helps, completely remove the code you commented from your project. It will make it look more like the work of someone who made the mistake and less like a tool to reproduce the problem.

Also, I'm not telling you exactly what is wrong with your message. The point of this exercise is that you learn how to verify if your own instructions are correct. By following is them.

[relentless-pursuit]

Makes sense.