Add automated testing

[nicksellen]

We discussed the idea to add automated testing in slack already (last discussion is https://freegle.slack.com/archives/C6LCTJ280/p1576346351010700). There is broad agreement about the need to add some, but we need to work out the exact approach.

I recently evaluated frontend testing approaches for https://trustroots.org and you can see my evaluation in this comment. That concerns using react (whereas we use vue), but the philosophy is the same, and even the library we settled on there has a vue variant.

The key bit from my evaluation I think is:

Philosophy generally summed up as:

testing using user facing features instead of implementation details

Have a look at these articles:

• Write tests. Not too many. Mostly integration.
• Why I Never Use Shallow Rendering

You can see an example from trustroots how a test file might look, key points:

• focus on what the user is doing, not the implementation
• uses a lot of aria roles kind of stuff to find the correct elements on the page (so can trigger actually adding better accessibility aria stuff in the process)
• assertions are based on what the user will see/do and the API calls made
• only the API calls are mocked
• no random waiting for elements to appear (or nextTick stuff)

Those tests are for quite focused individual components, whereas in iznik most of the code is in page level components that do quite a lot of different things, and interact with the store a lot, so it will be harder, we may need to split things out into more separate components (which would be good idea in general imo). But also would like to be test those full page-level components.

I welcome any discussion on these approaches! I think we should throw around ideas until we are happy with an approach, before starting work on this.

[edwh]

Is your suggested choice something we can add on travis? Almost certainly yes, but that's what we use for the backend testing, and it's free for open source, so I'd like to do that.

I've not really read this yet. More later.

[nicksellen]

For sure, any of these things can run in travis.

[edwh]

Have you looked into automated visual testing at all?

[nicksellen]

Not recently, a quick search brought up this guide https://markus.oberlehner.net/blog/visual-regression-tests-for-vue-applications-with-jest-and-puppeteer/. I never felt particularly convinced that it's useful and not flakey.

[edwh]

The problem I always have (which may just be me) with jest style scripts is that the test scripts look nothing like the actual function you're testing.

In (say) phpunit, the test code looks quite like the code that actually uses the thing you're testing. So the test scripts look like the function.

That's true if you're using jest to test a JS library that does some functional stuff. But jest test scripts look nothing like a web UI does on a screen; at best they're testing the DOM.

They're not really testing user-facing features, because the user-facing features are fundamentally a visual experience. A visual test script wouldn't be dealing with aria roles - it would be more like "click the green button near the top right and check a modal pops up". That kind of test script would be much closer to the user experience.

There are people who claim that they can use more modern techniques to avoid flakiness, e.g. https://applitools.com/. No idea whether that's true or not, but it's the kind of area that might have moved on a bit. Could you spend a bit of time looking around that?

[nicksellen]

which may just be me

I think it's just you. It feels like I'd have to write a book to justify adding some tests, but I'm not going to do that of course.

I would consider adding visual regression testing later, but not as the basic layer of testing. Those kind of tests compare a baseline image with the current version, and checks for differences in the rendered images, they are not concerned with stepping through and clicking buttons.

[edwh]

Ok. You're probably right. I will do a little googling to see if there are any test tools of the kind I think I would like. I won't find any, and that will be confirmation that I am just wrong about what I think I want, and I'll just have to get over that :-).

[edwh]

You're right - that facet is just me. I'll forget about it.

Recent experience since we went live shows me that it's pretty easy to break the code on one specific device, but not on others. Or to fix it for one device and break others. Or to not notice that it's broken on some devices when it's first written. We have had too many "it just doesn't work on device X" problems.

This makes me feel that what we need most is end to end testing using real devices. That article says you get more benefit as you move from UT to E2E, which I agree with strongly, but it then focuses on integration testing. I think that's coming from a mindset of E2E tests being lovely but being really expensive to write.

However although the tests themselves are quite expensive to write, I think what that isn't paying enough attention to is the ability to run them in parallel on real devices. If you can run a single E2E test on tens of devices, the benefit of that test leaps up massively. If we had tests that covered posting an offer, searching for a post, and replying with sign up then we would have tested the key function of the site for most users.

So I wonder if we should try some automated E2E tests. We get Browserstack for free, and they have automation. Saucelabs does too; I've just applied for an open source free account from them.

[JayyajGH]

My main testing experience so far has been to use Ava with vue-dev-tools. One thing I found when added it retrospectively is that you usually need to do a pile of refactoring to make your code testable. I see that as a good thing though. I found this a great setup which easily works with Nuxt and it useful for unit testing your JS and any conditional elements logic etc.

I've also done visual regression testing using Backstop/Casper/Phantom. This was really useful as the project was a large CMS driven brochureware site with hundreds of pages that we would never had the time to test manually. This setup now can be done just using Backstop and takes seconds to setup. I've not used the new version so not sure what improvements have been made. It would probably be slightly harder for stuff behind a login but I imagine it would be possible. This will just test that the page looked the same now as it did before a change in multiple breakpoints; not devices.

The visual regression part of this link shows how easy it is:
https://www.javascriptjanuary.com/blog/getting-started-with-front-end-testing

On other non-Vue projects I've used Selenium for E2E testing. I think this can hook up to browserstack too. My experience of this was that it was great but took a lot of effort to initially set up and also maintain. So I believe you get a big benefit but you need to put in the effort to get it.

Is the "it just doesn't work on device X" actually "it just doesn't work on screensize 768" or "it just doesn't work on Google Pixel 2"?

[edwh]

Ava would be an alternative to Jest? I'd not heard of it. What is it that makes it work especially well with Nuxt?

I've used Phantom in the past, but I think for our kind of stuff you need something that's hooked into events so that you know when everything has finished rendering. I've not yet seen anything like that which is Vue/Nuxt-aware in that way, but not looked for long enough to believe nothing exists.

It's definitely not just screen size - that might mean it looks rubbish but it will generally work. It's that there are a whole load of browser-specific gotchas on old-but-not-that-old devices. Older Safari, older IE, older Firefox, older Android stock browser.

So the risk is that we put a bunch of effort into testing and end up with something that...still only works really well on Chrome. Some of the minor functional issues that can creep in through lack of decent automated tests are very much less important than the "isn't usable on IE11" type issues.

[JayyajGH]

Ava is the tool that is referenced in the Nuxt docs...

https://nuxtjs.org/guide/development-tools/

Ava is a test runner. So yeah, like Jest or Mocha. I'd only ever had a quick play with Jest before so Ava was the first real Javascript test runner I'd used before so not much to compare it too. I just found it easy to set up with Nuxt. There were a few docs around on the internet that explained how to do it and I couldn't find that for any other framework. There might be loads now though. That was a couple of years back.

Most of the visual testing tools will have some kind of wait process to sync up with a finished render. I've used the one in selenium before. I think that must poll until the element is visible. I think backstop just had a period of seconds it will wait before checking. It probably needs to be "javascript aware" rather than specifically "Vue/Nuxt" aware. i.e. the element is dynamically created and might not be available instantly.

OK so it is definitely devices you need to check then. Sounds like hooking into browserstack might be the best option then. Especially as its free. So many tools!

My personal opinion is that I really like unit tests. They make you refactor your code in a good way and they are quick and easy to write alongside the methods. They also make you think about error handling and help you write more robust methods. They also give you confidence to make code changes too.

In the case of Freegle it sounds like some kind of visual regression testing would be really useful mainly due to the type of users we have. i.e. ones with older devices. Having a few journeys that are tested on a bunch of devices would probably be useful. i.e. add an item, search for an item etc. But what are you testing? That the screen looks the same as it did before or that the buttons are visible and fields are interactive etc. Or both? e.g. the continue button could be displayed off screen on a particular device but javascript could still find it and click it in an E2E test. But a user couldn't.

[JayyajGH]

I think my suggestion would be to choose a javascript runner like Ava or Jest. Whatever seems to be the best one at the moment. When adding new methods then add some unit tests. If you are bored then go back and add them for other methods. If a bug is found in a method then add some tests to that and just expand them as time goes on.

Then as a separate step choose an E2E tool which hooks into a "real device" like browserstack.

I see those two types of tests as very separate things and it would allow you to choose the best tool for the job.

[edwh]

I agree UT and E2E are separate things, primarily because I want E2E to use real devices and therefore it has to use some other infrastructure like browserstack.

There is a good argument that IT is generally better bang-for-buck than UT - see Nick's link at the top of the thread. I agree with this - the server testing is about 50/50 UT/IT.

For UT/IT, ava is the Nuxt recommendation (thanks, hadn't noticed); Jest is more common. We need to pick one. @nicksellen may have a view, so we should wait a bit for him to chip in (he's away at the moment).

If we settled on something then people could write some UTs, and I'd then like to get a coverage report out of that using coveralls.io, which is what we currently use for the server testing.

I would personally prioritise E2E at the moment, because I'm seeing the problems caused by weird browsers. But that's probably just an argument for me doing the E2E testing. So I'll have a play with Browserstack's stuff when I get a chance and report back on whether it's nice.

[JayyajGH]

Yeah I think i'd have to agree with the ideas in Nick's link. And I'd definitely agree that this project seems like it needs E2E testing the most. At least at the moment.

I don't have many opinions on which test runner we use for UT/IT. Ava was easy to set up so I used it. They all looks pretty much the same when you're writing tests!

I've used NYC for code coverage which is basically Istanbul. But once again, I'm sure they are all pretty much the same. As long as it's easy to set up and it works.

[edwh]

I came across this discussion of Cypress vs Selenium, which I found useful. It rules out Cypress for us since cross-browser testing is a goal.

https://medium.com/@arnonaxelrod/my-experience-moving-from-selenium-to-cypress-2a23072993e5