Test completed json doesn't include location

[razzeee]

It would be nice to get the location of the failure. Am I missing something here?

The command I ran was: elm-test "tests\\RoutingTests.elm" --report json

The current return looks like this:

{
  "event": "testCompleted",
  "status": "fail",
  "labels": [
    "RoutingTests",
    "Route.fromUrl",
    "Parsing hash: \"#login\""
  ],
  "failures": [
    {
      "given": null,
      "message": "Expect.equal",
      "reason": {
        "type": "Equality",
        "data": {
          "expected": "Just Login",
          "actual": "Nothing",
          "comparison": "Expect.equal"
        }
      }
    }
  ],
  "duration": "3"
}

[razzeee]

This would also be a priority for us, but we would like to get this on non failure items too. So that we can make a test selectable and thus go to it.

[lydell]

Do you mean the source code line and column number, or something else?

[razzeee]

I guess the line would be enough, but column would be even better. I also thought we might need the file path, but judging from https://github.com/frawa/vscode-elm-test-runner that might not be needed.

[harrysarson]

This function

node-test-runner/lib/Parser.js

Line 27 in c493674

function extractExposedPossiblyTests(

returns a promise resolving to an array of strings which are the names of elm values that might be tests.

To solve this I think that the function would have to resolve to an array of some object

{ name: string
, line: number
, column: number
}

and then here
https://github.com/rtfeldman/node-test-runner/blob/master/elm/src/Test/Runner/Node.elm#L367
instead of run taking a List ( String, List (Maybe Test) ) (the string is the module name) it would take a List ( String, List (PossibleTest) ) which would be

type alias PossibleTest=
  { test : Maybe Test
  , line : Int
  , column: Int
  }

[lydell]

Are we talking about the location of

1. the Expect.equal (or whatever) that failed
2. or the “whole” test

Either way, I don’t know how to find locations in this example (I wrote this here on GitHub so there might be mistakes that don’t compile):

module Tests exposing (tests)

import Test
import Expect

tests =
    Test.describe "Cool stuff"
		[ foo
		]

foo = bar

bar =
	baz title (body 1)

baz = Test.test

title = "My test title"

body n _ =
	check 0 n

check = Tuple.first (Expect.equal, ())

On the other hand, nobody writes tests like that, but in theory that’s what we have to work with.

Random note: Debug.todo compiles to code that has the line and column number in it (since that’s part of the message). Maybe that can be abused somehow.

[razzeee]

Current implementation of the test runner seems to go to the describe and match the name, so it's rather expensive to do and dumb, as it will break if your test name is generated somehow

[harrysarson]

Yeah, of course. My idea in #368 (comment) would give only the location of the top level "test" (in practice it would be a describe block) which isn't that helpful.

@lydell is right on the money: we can never come up with a scheme that can locate every test. You can generate tests in all sorts of ways that subvert attempts to locate tests.

Even if elm-explorations/test attached location info into Expect calls you could (and libraries may well) wrap Expect calls in helper functions that would then give unhelpful locations (i.e. in external libraries).

Maybe we need to think about alternatives ways of structuring tests, rust uses #[test] annotations which then gives a concrete thing to refer to in the test results maybe that is something worth thinking about in elm.

[lydell]

Maybe we need to think about alternatives ways of structuring tests, rust uses #[test] annotations which then gives a concrete thing to refer to in the test results maybe that is something worth thinking about in elm.

I’ve been thinking along those lines too. In the C#/F# world, tests are commonly marked too (with [<Test>] or [<Fact>]). But not always. There is one test framework called Expecto that looks more like Elm tests, and they have the same problem with editor integration basically: haf/expecto#326

One way could be to ditch describe completely. The only thing users would be able to do is to expose single tests. Then we could parse out the location of every test like Harry initially suggested. People would have to use new files instead of new describes to group tests (which maybe isn’t so bad? 🤔)

[gampleman]

One way could be to ditch describe completely. The only thing users would be able to do is to expose single tests. Then we could parse out the location of every test like Harry initially suggested. People would have to use new files instead of new describes to group tests (which maybe isn’t so bad? 🤔)

The problem with that is that it makes generating many tests impossible. I often have something like:

suite =  
  [ ("foo", 34, True)
  , ("bar", 1, False)
  -- ...
  ] 
  |> List.map (\(name, val, res) ->
     Test.test name <|
        \() ->
             functionUnderTest val 
                 |> Expect.equal res
       )
  |> Test.desribe "functionUnderTest"

(except of course the input list might be generated in some way more complex manner).

One solution would be if the runner could execute values that are Test or List Test.

[lydell]

Good point. We’d need some other way of generating tests. List Test would bring us back to the initial problem again, I think though. So maybe something like Expect.each : (a -> Expectation) -> List ( String, a ) -> Expectation

[lydell]

Here’s another idea for lists of tests: each : String -> List ( String, a ) -> (a -> Expectation) -> Expectation

from Imgflip Meme Generator

-- Unit test
absTest : Test
absTest =
    test "abs" <|
        \_ ->
            abs -1
                |> Expect.equal 1

-- Parameterized test
absTest2 : Test
absTest2 =
    each "abs"
        [ ( "negative number", ( -1, 1 ) )
        , ( "zero", ( 0, 0 ) )
        , ( "positive number", ( 1, 1 ) )
        ]
    <|
        \( input, expected ) ->
            abs input
                |> Expect.equal expected

-- Property test
absTest3 : Test
absTest3 =
    fuzz Fuzz.int "abs" <|
        \input ->
            abs input
                |> Expect.atLeast 0