-
Notifications
You must be signed in to change notification settings - Fork 499
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(test): POC e2e tests #3998
Changes from 1 commit
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,51 @@ | ||
const expect = require('chai').expect; | ||
|
||
describe('InstantSearch e-commerce demo', () => { | ||
before(() => { | ||
browser.maximizeWindow(); | ||
}); | ||
|
||
it('must work', async () => { | ||
await browser.url( | ||
'https://deploy-preview-3967--instantsearchjs.netlify.com/examples/e-commerce/' | ||
); | ||
|
||
// Select "Apple" brand in list | ||
const brand = await browser.$('span=Apple'); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. is there a reason this isn't There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We are not in the browser context here so we have to use the selector from WebdriverIO. It is a bit different than There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can we rely on something else than There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The But we can also use CSS selectors if needed. https://webdriver.io/docs/selectors.html Targeting the right elements is tricky in E2E test as the UI and/or CSS classes can change frequently. IMHO in our case we should rely on CSS classes, since they are pretty stable in InstantSearch (any change would be a breaking change). There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I wish to be able to use something like ByLabelText here :( If There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Since we use them for E2E tests we can imagine the usage for There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, that's also a possibility. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. test- attributes could be nice by I don't see how we can do it without altering every the DOM of every widget. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe this can help: https://github.com/testing-library/testcafe-testing-library. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sadly this is not compatible with WebdriverIO 😞 |
||
await brand.click(); | ||
|
||
// Fill search input with "macbook" | ||
const searchInput = await browser.$('[type=search]'); | ||
await searchInput.setValue('macbook'); | ||
|
||
// Wait for the results list to be updated (wait for the "macbook" word to be highlighted) | ||
await browser.waitUntil( | ||
async () => (await browser.$$('mark=MacBook')).length > 0, | ||
10000 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. can we not rely on the search callback instead of a magic number? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What do you mean by search callback? You can find the API documentation here https://webdriver.io/docs/api.html There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We can wait for the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Since it is an E2E test we're not supposed to interact with InstantSearch differently from a user. We should only wait for a visual feedback that the search is done, like a real user. |
||
); | ||
|
||
// Get title for each result | ||
const hits = await browser.$$('.hit h1'); | ||
const hitsText = await Promise.all(hits.map(hit => hit.getText())); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why do we extract only the text and not the HTML for also the highlighted parts? This seems to cover better the actual usage. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Here I'm only testing that I got the right list of results, I do not test that the text is correctly highlighted. |
||
|
||
// Compare them to expected titles | ||
expect(hitsText).to.deep.equal([ | ||
'Apple - MacBook Air® (Latest Model) - 13.3" Display - Intel Core i5 - 8GB Memory - 128GB Flash Storage - Silver', | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Here I think what we expect is to have Apple in all the hits text so maybe we should just check that? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Mmm, I prefer to test that we get exactly the results we expect, and checking for only "Apple" is too vague IMO. I find this as a good thing to have the test to fail if the ranking or a hit change 😇 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is something we can do with a dedicated demo: we can show the value of the attribute that will be filtered in the UI so we can test it There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That would be equivalent to exposing some internals for testing, not a good practice and it would ruin the purpose of having end-2-end tests. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not really true, the attribute is what proves that it's filtered, since the filter isn't related to the search query per se, and can be in multiple attributes There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Let's not forget that we are supposed to test from a end user POV, so what matter at the end is that the result list is correct. |
||
'Apple - MacBook Air® (Latest Model) - 13.3" Display - Intel Core i5 - 8GB Memory - 256GB Flash Storage - Silver', | ||
'Apple - Macbook® (Latest Model) - 12" Display - Intel Core M3 - 8GB Memory - 256GB Flash Storage - Space Gray', | ||
'Apple - Macbook® (Latest Model) - 12" Display - Intel Core M3 - 8GB Memory - 256GB Flash Storage - Gold', | ||
'Apple - Macbook® (Latest Model) - 12" Display - Intel Core M3 - 8GB Memory - 256GB Flash Storage - Rose Gold', | ||
'Apple - Macbook® (Latest Model) - 12" Display - Intel Core M3 - 8GB Memory - 256GB Flash Storage - Silver', | ||
'Apple - Macbook® (Latest Model) - 12" Display - Intel Core M5 - 8GB Memory - 512GB Flash Storage - Space Gray', | ||
'Apple - Macbook® (Latest Model) - 12" Display - Intel Core M5 - 8GB Memory - 512GB Flash Storage - Rose Gold', | ||
'Apple - Macbook® (Latest Model) - 12" Display - Intel Core M5 - 8GB Memory - 512GB Flash Storage - Gold', | ||
'Apple - Macbook® (Latest Model) - 12" Display - Intel Core M5 - 8GB Memory - 512GB Flash Storage - Silver', | ||
'Apple - MacBook Pro with Retina display - 13.3" Display - 8GB Memory - 128GB Flash Storage - Silver', | ||
'Apple - MacBook Pro® - 13" Display - Intel Core i5 - 8 GB Memory - 256GB Flash Storage (latest model) - Space Gray', | ||
'Apple - MacBook® Pro - 15.4" Display - Intel Core i7 - 16GB Memory - 256GB Flash Storage - Silver', | ||
'Apple - MacBook Pro® - 13" Display - Intel Core i5 - 8 GB Memory - 256GB Flash Storage (latest model) - Silver', | ||
'Apple - MacBook® Pro - Intel Core i5 - 13.3" Display - 4GB Memory - 500GB Hard Drive - Silver', | ||
'Apple - MacBook Pro 13.3" Refurbished Laptop - Intel Core i5 - 4GB Memory - 320GB - Silver', | ||
]); | ||
}); | ||
}); |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,62 @@ | ||
const config = { | ||
hostname: 'hub.browserstack.com', | ||
port: 443, | ||
baseUrl: '', | ||
user: process.env.BROWSERSTACK_USER, | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not sure yet if these E2E tests are also meant to be run on our local machines. If so, we can consider adding There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. When the e2e tests fail on CI server, we'll probably want to try it again on our local machines. So let's consider adding it :) |
||
key: process.env.BROWSERSTACK_KEY, | ||
|
||
specs: ['./e2e/test.js'], | ||
maxInstances: 10, | ||
|
||
// Capabilities Generator | ||
// https://www.browserstack.com/automate/capabilities?tag=selenium-4 | ||
commonCapabilities: { | ||
'browserstack.debug': true, | ||
'browserstack.video': true, | ||
'browserstack.console': 'verbose', | ||
'browserstack.networkLogs': true, | ||
}, | ||
|
||
capabilities: [ | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What is the time impact of adding multiple other combinations? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It depends of the offer on the test service. With an offer like Automate Pro on BrowserStack we can run 5 tests in parallel, so 5 combinations in total without impacting the test duration. |
||
{ | ||
os: 'Windows', | ||
osVersion: '10', | ||
resolution: '1680x1050', | ||
browserName: 'Chrome', | ||
browserVersion: '75.0', | ||
}, | ||
{ | ||
os: 'OS X', | ||
osVersion: 'Mojave', | ||
resolution: '1600x1200', | ||
browserName: 'Firefox', | ||
browserVersion: '68.0', | ||
}, | ||
{ | ||
osVersion: '12', | ||
deviceName: 'iPhone XS', | ||
realMobile: 'true', | ||
}, | ||
], | ||
|
||
logLevel: 'trace', | ||
coloredLogs: true, | ||
waitforTimeout: 10000, | ||
connectionRetryTimeout: 90000, | ||
connectionRetryCount: 0, | ||
|
||
framework: 'mocha', | ||
reporters: ['spec'], | ||
mochaOpts: { | ||
ui: 'bdd', | ||
timeout: 60000, | ||
}, | ||
}; | ||
|
||
// Map common capabilities to capabilities | ||
config.capabilities = config.capabilities.map(capability => ({ | ||
...config.commonCapabilities, | ||
...capability, | ||
})); | ||
|
||
exports.config = config; |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,60 @@ | ||
const config = { | ||
hostname: 'hub.crossbrowsertesting.com', | ||
port: 80, | ||
baseUrl: '', | ||
user: process.env.CBT_USER, | ||
key: process.env.CBT_KEY, | ||
services: ['crossbrowsertesting'], | ||
|
||
specs: ['./e2e/test.js'], | ||
maxInstances: 10, | ||
|
||
// Capabilities Generator | ||
// https://app.crossbrowsertesting.com/selenium/run | ||
commonCapabilities: { | ||
record_video: 'true', | ||
record_network: 'true', | ||
}, | ||
|
||
capabilities: [ | ||
{ | ||
browserName: 'Chrome', | ||
version: '75', | ||
platform: 'Windows 10', | ||
screenResolution: '1680x1050', | ||
}, | ||
{ | ||
browserName: 'Firefox', | ||
platform: 'Mac OSX 10.14', | ||
screenResolution: '1440x1050', | ||
}, | ||
{ | ||
browserName: 'Safari', | ||
deviceName: 'iPhone XR Simulator', | ||
platformVersion: '12.0', | ||
platformName: 'iOS', | ||
deviceOrientation: 'portrait', | ||
}, | ||
], | ||
|
||
logLevel: 'trace', | ||
coloredLogs: true, | ||
waitforTimeout: 10000, | ||
connectionRetryTimeout: 90000, | ||
connectionRetryCount: 0, | ||
|
||
framework: 'mocha', | ||
reporters: ['spec'], | ||
mochaOpts: { | ||
ui: 'bdd', | ||
timeout: 60000, | ||
}, | ||
}; | ||
|
||
// Map common capabilities to capabilities | ||
config.capabilities = config.capabilities.map(capability => ({ | ||
...config.commonCapabilities, | ||
...capability, | ||
})); | ||
|
||
exports.config = config; |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,55 @@ | ||
const config = { | ||
hostname: 'cloud.seetest.io', | ||
protocol: 'https', | ||
port: 443, | ||
|
||
specs: ['./e2e/test.js'], | ||
maxInstances: 10, | ||
|
||
// Capabilities Generator | ||
// https://cloud.seetest.io/index.html#/quickstart | ||
commonCapabilities: {}, | ||
|
||
capabilities: [ | ||
{ | ||
browserName: 'chrome', | ||
browserVersion: '75.0.3770.142', | ||
platform: 'Mac OS X High Sierra', | ||
accessKey: process.env.EXPERITEST_ACCESS_KEY, | ||
}, | ||
{ | ||
browserName: 'firefox', | ||
browserVersion: '68.0.1', | ||
platform: 'Windows 10', | ||
|
||
accessKey: process.env.EXPERITEST_ACCESS_KEY, | ||
}, | ||
{ | ||
platformName: 'ios', | ||
deviceQuery: "@os='ios' and @version='12.0.1' and @category='PHONE'", | ||
|
||
accessKey: process.env.EXPERITEST_ACCESS_KEY, | ||
}, | ||
], | ||
|
||
logLevel: 'trace', | ||
coloredLogs: true, | ||
waitforTimeout: 10000, | ||
connectionRetryTimeout: 90000, | ||
connectionRetryCount: 0, | ||
|
||
framework: 'mocha', | ||
reporters: ['spec'], | ||
mochaOpts: { | ||
ui: 'bdd', | ||
timeout: 60000, | ||
}, | ||
}; | ||
|
||
// Map common capabilities to capabilities | ||
config.capabilities = config.capabilities.map(capability => ({ | ||
...config.commonCapabilities, | ||
...capability, | ||
})); | ||
|
||
exports.config = config; |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,56 @@ | ||
const config = { | ||
hostname: 'hub.lambdatest.com', | ||
port: 80, | ||
baseUrl: '', | ||
user: process.env.LT_USER, | ||
key: process.env.LT_KEY, | ||
|
||
specs: ['./e2e/test.js'], | ||
maxInstances: 10, | ||
|
||
// Capabilities Generator | ||
// https://www.lambdatest.com/capabilities-generator/ | ||
commonCapabilities: { | ||
visual: true, | ||
video: true, | ||
console: true, | ||
network: true, | ||
}, | ||
|
||
capabilities: [ | ||
{ | ||
platform: 'Windows 10', | ||
browserName: 'Chrome', | ||
version: '76.0', | ||
resolution: '1680x1050', | ||
}, | ||
{ | ||
platform: 'macOS Mojave', | ||
browserName: 'Firefox', | ||
version: '67.0', | ||
resolution: '1600x900', | ||
}, | ||
// No mobile test :( | ||
], | ||
|
||
logLevel: 'trace', | ||
coloredLogs: true, | ||
waitforTimeout: 10000, | ||
connectionRetryTimeout: 90000, | ||
connectionRetryCount: 0, | ||
|
||
framework: 'mocha', | ||
reporters: ['spec'], | ||
mochaOpts: { | ||
ui: 'bdd', | ||
timeout: 60000, | ||
}, | ||
}; | ||
|
||
// Map common capabilities to capabilities | ||
config.capabilities = config.capabilities.map(capability => ({ | ||
...config.commonCapabilities, | ||
...capability, | ||
})); | ||
|
||
exports.config = config; |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,57 @@ | ||
const config = { | ||
hostname: 'ondemand.eu-central-1.saucelabs.com', | ||
port: 443, | ||
user: process.env.SAUCELABS_USER, | ||
key: process.env.SAUCELABS_KEY, | ||
|
||
specs: ['./e2e/test.js'], | ||
maxInstances: 10, | ||
|
||
// Capabilities Generator | ||
// https://wiki.saucelabs.com/display/DOCS/Platform+Configurator#/ | ||
commonCapabilities: {}, | ||
|
||
capabilities: [ | ||
{ | ||
browserName: 'chrome', | ||
platform: 'Windows 10', | ||
version: '75.0', | ||
screenResolution: '1680x1050', | ||
}, | ||
{ | ||
browserName: 'firefox', | ||
// https://github.com/webdriverio/webdriverio/issues/3443 | ||
'sauce:options': { | ||
seleniumVersion: '3.11.0', | ||
}, | ||
}, | ||
{ | ||
browserName: 'Safari', | ||
deviceName: 'iPhone XS Simulator', | ||
deviceOrientation: 'portrait', | ||
platformVersion: '12.2', | ||
platformName: 'iOS', | ||
}, | ||
], | ||
|
||
logLevel: 'trace', | ||
coloredLogs: true, | ||
waitforTimeout: 10000, | ||
connectionRetryTimeout: 90000, | ||
connectionRetryCount: 0, | ||
|
||
framework: 'mocha', | ||
reporters: ['spec'], | ||
mochaOpts: { | ||
ui: 'bdd', | ||
timeout: 60000, | ||
}, | ||
}; | ||
|
||
// Map common capabilities to capabilities | ||
config.capabilities = config.capabilities.map(capability => ({ | ||
...config.commonCapabilities, | ||
...capability, | ||
})); | ||
|
||
exports.config = config; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For end to end test, I'd tend to use
it
in a slightly different way that in a unit test, for readability of the report and to clarify the intent of the commands that failed.Example in this case:
I'm just suggesting. This test is small enough that it's doesn't beg for it 😄
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I do not have any opinion on this, it makes the test a bit longer but one advantage of this is that it annotates the test, for example on Sauce Labs you'll get:
(this is similar to the use of
sauce:context
)