Call tests via flutter test instead of flutter drive possible?

[golane-august]

Can you use integration_test package instead of driver package to test the app?

Use:

flutter test integration_test/gherkin_suite_test.dart

Instead of:

flutter drive --driver=test_driver/integration_test_driver.dart --target=integration_test/gherkin_suite_test.dart

But I get the error with flutter test:

flutter test integration_test/gherkin_suite_test.dart
00:00 +0: loading /Users/august/Coding/git/jonsamwell/flutter_gherkin/example_with_integration_test/integration_test/gherkin_suite_test.dart                                                                                                                                                                           R00:11 +0: loading /Users/august/Coding/git/jonsamwell/flutter_gherkin/example_with_integration_test/integration_test/gherkin_suite_test.dart                                                                                                                                                                      11.7s
✓  Built build/app/outputs/flutter-apk/app-debug.apk.
00:12 +0: loading /Users/august/Coding/git/jonsamwell/flutter_gherkin/example_with_integration_test/integration_test/gherkin_suite_test.dart                                                                                                                                                                           I00:13 +0: loading /Users/august/Coding/git/jonsamwell/flutter_gherkin/example_with_integration_test/integration_test/gherkin_suite_test.dart                                                                                                                                                                      656ms
00:15 +0 -1: loading /Users/august/Coding/git/jonsamwell/flutter_gherkin/example_with_integration_test/integration_test/gherkin_suite_test.dart [E]                                                                                                                                                                    
  Failed to load "/Users/august/Coding/git/jonsamwell/flutter_gherkin/example_with_integration_test/integration_test/gherkin_suite_test.dart": Bad state: Can't call group() once tests have begun running.
  package:test_api                                                                                                                  Declarer.group
  package:flutter_test/src/test_compat.dart 189:13                                                                                  group
  package:flutter_gherkin/src/flutter/runners/gherkin_integration_test_runner.dart 117:5                                            GherkinIntegrationTestRunner.runFeature
  Users/august/Coding/git/jonsamwell/flutter_gherkin/example_with_integration_test/integration_test/gherkin_suite_test.g.dart 34:5  _CustomGherkinIntegrationTestRunner.testFeature0
  Users/august/Coding/git/jonsamwell/flutter_gherkin/example_with_integration_test/integration_test/gherkin_suite_test.g.dart 26:5  _CustomGherkinIntegrationTestRunner.onRun
  package:flutter_gherkin/src/flutter/runners/gherkin_integration_test_runner.dart 94:5                                             GherkinIntegrationTestRunner.run
  
00:16 +0 -1: (tearDownAll) - did not complete [E]                                                                                                                                                                                                                                                                      
00:16 +0 -1: Some tests failed.                         

I used the example in integration_test__package_support: https://github.com/jonsamwell/flutter_gherkin/tree/integration_test__package_support/example_with_integration_test

I also tested flutter run with tagExpression @failure-expected, but then the result doesn't match the failure:

flutter run integration_test/gherkin_suite_test.dart
...
I/flutter (16379): FAILED: Scenario Failed expect() should be added to json report # ./integration_test/features/failure.feature:0
...
I/flutter (16379): 00:03 +3: All tests passed!

[golane-august]

Also calling this with tagExpression: '@failure-expected':

 flutter drive --driver=test_driver/integration_test_driver.dart --target=integration_test/gherkin_suite_test.dart -d chrome

results in:

FAILED: Scenario Failed expect() should be added to json report # ./integration_test/features/failure.feature:0
00:03 +2: (tearDownAll)
2 scenarios (2 failed)
4 steps (2 passed, 2 failed)
0:00:03.088000
00:03 +3: All tests passed!
Application finished.

Note the "All tests passed!", which sould not occur.

[luvetter]

The problem seems to be that the generated executeTestSuite method is synchronous, but the run method is asynchronous:

void executeTestSuite({
  required FlutterTestConfiguration configuration,
  required StartAppFn appMainFunction,
  Timeout scenarioExecutionTimeout = const Timeout(const Duration(minutes: 10)),
  AppLifecyclePumpHandlerFn? appLifecyclePumpHandler,
  LiveTestWidgetsFlutterBindingFramePolicy? framePolicy,
}) {
  _CustomGherkinIntegrationTestRunner(
    configuration: configuration,
    appMainFunction: appMainFunction,
    appLifecyclePumpHandler: appLifecyclePumpHandler,
    scenarioExecutionTimeout: scenarioExecutionTimeout,
    framePolicy: framePolicy,
  ).run();
}

Also the main method is done before the test setup is complete:

@GherkinTestSuite(
  useAbsolutePaths: false,
)
void main() {
  executeTestSuite(
    appMainFunction: appInitializationFn,
    configuration: gherkinTestConfiguration,
  );
}

But Flutter assumes that with the end of the main method the test declaration is finished and does not allow any more calls on the test api (group, test, testWidgets, ...)

You could probably make the executeTestSuite method asynchronous and await it.

Future executeTestSuite({
  required FlutterTestConfiguration configuration,
  required StartAppFn appMainFunction,
  Timeout scenarioExecutionTimeout = const Timeout(const Duration(minutes: 10)),
  AppLifecyclePumpHandlerFn? appLifecyclePumpHandler,
  LiveTestWidgetsFlutterBindingFramePolicy? framePolicy,
}) async {
  return _CustomGherkinIntegrationTestRunner(
    configuration: configuration,
    appMainFunction: appMainFunction,
    appLifecyclePumpHandler: appLifecyclePumpHandler,
    scenarioExecutionTimeout: scenarioExecutionTimeout,
    framePolicy: framePolicy,
  ).run();
}

@GherkinTestSuite(
  useAbsolutePaths: false,
)
void main() async {
  await executeTestSuite(
    appMainFunction: appInitializationFn,
    configuration: gherkinTestConfiguration,
  );
}

What do you think @jonsamwell ?

[AFASbart]

@luvetter I edited you comment to translate the german sentence 😉