testing-starter-kit

QA Engineers and Developers now have a wide variety of free open source tools for testing websites and portals, but it can be a challenging task to find them all and make informed decision on what and how to use.

This is a collection of free open source software testing tools with some links on how to use them.

The scope of this particular list is limited to things that QA Automation Engineers, Software Development Engineers in Test (SDET), Web Developers, etc. are likely to find useful.

You can help by sending Pull Requests to add more tools and links to "how to use" instructions.

We love all the free things out there, but it would be good to keep it on topic. It's a bit of a grey line sometimes, so this is a bit opinionated.

Table of Contents

• Code Editors and IDE:
  • VS Code
  • Sublime Text
  • IntelliJ IDEA
  • CodePen
• Automated Browser Testing (end-to-end):
  • Selenium
  • RSpec
  • Protractor
  • Nightwatch
  • WebdriverIO
  • Puppeteer
  • TestCafe
  • Cypress
  • Playwright
• API Testing:
  • cURL
  • Dredd
  • Postman
  • Pactum
• Unit Testing:
  • Mocha
  • Chai
  • Jasmine
  • Karma
  • Jest
  • AVA
• Load Testing:
  • JMeter
  • Tsung
  • Gatling
  • Artillery
  • k6
• Security Testing
• Linting and Code Quality:
  • ESLint
• Version Control Systems (VCS):
  • Git
• Continuous Integration (CI / CD):
  • Jenkins
  • Travis
  • GitLab
  • GitHub Actions
• Containerization Tools:
  • Docker
  • Kubernetes
• Command Line Tools (CLI):
  • Grep
• Bug Tracking and Task Management:
  • Redmine
  • Mantis
  • Jira
• Node.js
• Leadership in QA
• Job Search

Code Editors and IDE

• VS Code: initial release date: April 29, 2015, written in: TypeScript, JavaScript, Cascading Style Sheets (CSS). Visual Studio Code is a source code editor developed by Microsoft for Windows, Linux and macOS. It includes support for debugging, embedded Git control, syntax highlighting, intelligent code completion, snippets and code refactoring. Plugins and extensions for it can be written in JavaScript (a lot of them are built and maintained by community).

• Sublime Text: initial release date: January 18, 2008, written in: C++, Python. Sublime Text is a cross-platform source code editor. It natively supports many programming and markup languages. Plugins and extensions for it can be written in Python (tons of them can be installed and used).

• IntelliJ IDEA: initial release date: January 2001, written in: Java. IntelliJ IDEA is a Java integrated development environment for developing computer software. It is developed by JetBrains and is available as an Apache 2 Licensed community edition and in a proprietary commercial edition. Both can be used for commercial development.

• CodePen: initial release date: 2012. CodePen is an online community for testing and showcasing user-created HTML, CSS and JavaScript code snippets. It functions as an online code editor and open-source learning environment, where developers can create code snippets, creatively named "pens", and test them.

Automated Browser Testing (end-to-end)

• Selenium: initial release date: 2004, written in: Java. Selenium automates browsers. Primarily, it is for automating web applications for testing purposes, but is certainly not limited to just that. Selenium has the support of some of the largest browser vendors who have taken (or are taking) steps to make Selenium a native part of their browser. It is also the core technology in countless other browser automation tools, APIs and frameworks. Selenium has 2 incarnations:

  • Selenium IDE: is more simple and fast way to create automated testing scripts by clicking and recording your actions with record-and-playback Firefox add-on. It is mostly used to create simple scripts to aid in automation-aided exploratory testing. It requires some knowledge of HTML.
  • Selenium WebDriver: is more sophisticated and powerful tool to create robust, browser-based regression automation suites and tests. It requires some knowledge of programming languages - can be used one of the following: Java, JavaScript, Python, Ruby, C#.

• RSpec: initial release date: August 2005, written in: Ruby. RSpec is a 'Domain Specific Language' testing tool written in Ruby to test Ruby code. It is a behavior-driven development framework which is extensively used in the production applications. It utilizes easy-to-read syntax (describe, it). Here is a free, once-weekly e-mail on how to use Selenium with RSpec (also see the full archive of useful tips).

• Protractor: initial release date: Jun 12, 2013, written in: JavaScript. Protractor is an end-to-end test framework for Angular and AngularJS applications (can be used for non-Angular websites too). Protractor runs tests against your application running in a real browser, interacting with it as a user would.

  Note: the Angular team announced that Protractor was officially sunsetted at the end of 2022.

• Nightwatch: initial release date: January, 2014, written in: JavaScript. Nightwatch was created as a tool to enable writing of automated UI tests in a straightforward way with as little configuration and additional libraries as possible.

• WebdriverIO: initial release date: May 17, 2012, written in: JavaScript. WebdriverIO is a test automation framework that allows you to run tests based on the Webdriver protocol and Appium automation technology. It provides support for your favorite BDD/TDD test framework and will run your tests locally or in the cloud using Sauce Labs, BrowserStack or TestingBot.

  Useful links:

  • Building Scalable And Stable e2e Test Suites With WebdriverIO.
  • A Tour Through the Newly Architected WebdriverIO Home.
  • Front-end Testing with Kevin - a collection of WebdriverIO tutorial videos.
  • A look at the browser.execute command.

• Puppeteer: initial release date: August 16, 2017, written in: JavaScript. Puppeteer is a Node.js library maintained by Chrome's development team from Google. Puppeteer provides a high-level API to control Chrome or Chromium (by default headless, but can be configured to run full non-headless Chrome or Chromium) or interact with the DevTools protocol.

  Useful links:

  • Web Scraping with a Headless Browser: A Puppeteer Tutorial.
  • Getting to Know Puppeteer Using Practical Examples.
  • Puppeteer keyboard keys to use in keyboard.type.
  • Puppeteer Snippets extension for VS Code.

• TestCafe: initial release date: October 17, 2016, written in: JavaScript. TestCafe is Node.js tool to automate end-to-end web testing. TestCafe runs on Windows, MacOS, Linux and supports desktop, mobile, remote and cloud browsers (UI or headless).

  Useful links:

  • Why TestCafe is better than Selenium.
  • Injecting cookies in your TestCafe automation.
  • Integrating TestCafe With TestRail.

• Cypress: initial release date: September 10, 2017, written in: JavaScript. Cypress is a next generation front-end testing tool built for the modern web. It is most often compared to Selenium, however Cypress is both fundamentally and architecturally different. Cypress is not constrained by the same restrictions as Selenium. This enables you to write faster, easier and more reliable tests. At the same time Cypress has a set of limitations/trade-offs.

• Playwright: initial release date: 31 January, 2020, written in: JavaScript. Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API. Playwright is built to enable cross-browser web automation that is ever-green, capable, reliable and fast.

  Useful links:

  • The complete Playwright cheatsheet.
  • Playwright tips that will make your life easier.
  • Mastering the art of detecting and handling flaky tests in Playwright.

Useful links:

• The Big List of Naughty Strings which have a high probability of causing issues when used as user-input data.

API Testing

• cURL: initial release date: 1997, written in: C. The name stands for "Client URL". cURL is providing a library and command-line toolfor transferring data using various protocols. Installing and using cURL on Mac OS:

  • /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
  • brew install curl
  • curl http://localhost:3000 # Server will receive a GET HTTP request
  • curl -X POST http://localhost:3000 # Server will receive a POST HTTP request
  • curl -X PUT http://localhost:3000 # Server will receive a PUT HTTP request
  • curl -X DELETE http://localhost:3000 # Server will receive a DELETE HTTP request

• Postman: initial release date: February 19, 2016, written in: JavaScript. Postman is a powerful API testing suite which has become a must-have tool for many developers. It has the ability to make various types of HTTP requests, i.e. GET, POST, PUT, PATCH. It is available for Windows, macOS, and Linux. To query an API endpoint, you’ll need to do the following steps:

  1. Enter the URL that you want to query in the URL bar in the top section.
  2. Select the HTTP method on the left of the URL bar to send the request.
  3. Click on the "Send" button - Postman will then send the request to the application, retrieve any responses and display it in the lower window.

  Also Postman has a CLI collection runner - Newman. It allows you to effortlessly run and test a Postman collection directly from the command line.

  Useful links:

  • The Basics of Using Postman for API Testing.
  • Writing tests in Postman.
  • 16 Postman assertions.
  • Part 1: How to re-use code.
  • Part 2: Customizing Execution Order.
  • Part 3: Single collection pattern.
  • Part 4: Easy assertions.

• Dredd: initial release date: October 25, 2013, written in: JavaScript. Dredd is a language-agnostic command-line tool for validating API description document against backend implementation of the API. Dredd reads your API description and step by step validates whether your API implementation replies with responses as they are described in the documentation.

  Useful links:

  • Testing your API with Dredd

Unit Testing

• Mocha: initial release date: 22 November 2011, written in: JavaScript. Mocha is a feature-rich JavaScript test framework, making asynchronous testing simple and fun. Mocha tests run serially, allowing for flexible and accurate reporting, while mapping uncaught exceptions to the correct test cases.

• Chai: initial release date: January, 2014, written in: JavaScript. Chai is a BDD / TDD assertion library that can be delightfully paired with any JavaScript testing framework. Chai has several interfaces that allow the developer to choose the most comfortable: the chain-capable BDD styles provide an expressive language & readable style, while the TDD assert style provides a more classical feel.

• Jasmine: initial release date: September 14, 2010, written in: JavaScript. Jasmine is a behavior-driven development framework for testing JavaScript code. It does not depend on any other JavaScript frameworks. It does not require a DOM. And it has a clean, obvious syntax so that you can easily write tests with easy-to-read syntax (describe, it). It has built in assertion library and can be used as a testing framework both for unit tests and end-to-end tests.

• Karma: initial release date: March 11, 2012, written in: JavaScript. A simple tool that allows you to execute JavaScript code in multiple real browsers. Karma is not a testing framework, nor an assertion library. Karma just launches an HTTP server, and generates the test runner HTML file you probably already know from your favourite testing framework. So for testing purposes you can use pretty much anything you like. There are already plugins for most of the common testing frameworks: Jasmine, Mocha and many others.

• Jest: initial release date: April, 2016, written in: JavaScript. Jest is a delightful JavaScript Testing Framework with a focus on simplicity. It was developed by Facebook.

  Useful links:

  • The only 3 steps you need to mock an API call in Jest.

• AVA: initial release date: April 14, 2018, written in: JavaScript. AVA is a test runner for Node.js with a concise API, detailed error output, embrace of new language features and process isolation that let you write tests more effectively.

Useful links:

• Node.js & JavaScript Unit Testing Best Practices.
• How to test software: mocking, stubbing, and contract testing.
• Unit testing Node.js applications using Mocha, Chai, and Sinon.
• Node Testing Essentials (A node developer's guide to testing).

Load Testing

• JMeter: initial release date: December 15, 1998, written in: Java. Apache JMeter is a load testing tool for analyzing and measuring the performance of a variety of services, with a focus on web applications.

  Useful links:

  • JMeter Tutorials - set of short YouTube videos.
  • Load Testing with JMeter: Part 1 - Getting Started, Part 2 - Headless Testing and Jenkins Integration, Part 3 - Replaying Apache Logs.
  • Introduction to Apache JMeter Performance Testing.
  • How to do multiple user logins with different inputs in JMeter.

• Tsung: initial release date: May 29, 2006, written in: Erlang. Tsung is an open-source multi-protocol distributed load testing tool. It can currently stress test HTTP, WebDAV, LDAP, MySQL, PostgreSQL, SOAP and XMPP servers. Tsung can simulate hundreds of simultaneous users on a single system. It can also function in a clustered environment.

  Useful links:

  • Building a Load Test with Tsung for a Login and Post Session with dynamic url-encoded variables.

• Gatling: initial release date: January 13, 2012, written in: Scala. Gatling is a highly capable load testing tool designed for ease of use, maintainability and high performance.

• Artillery: initial release date: November 17, 2015, written in: JavaScript. Artillery is a modern, powerful, easy-to-use load-testing toolkit. Artillery has a strong focus on developer happiness & ease of use, and a batteries-included philosophy. Our goal is to help developers build faster, more resilient and more scalable applications.

• k6: initial release date: January 18, 2017, written in: Go, but scripts are written in JavaScript (see the running k6 guide). k6 is an open-source load testing tool that makes performance testing easy and productive for engineering teams. k6 is free, developer-centric, and extensible. Using k6, you can test the reliability and performance of your systems and catch performance regressions and problems earlier. k6 will help you to build resilient and performant applications that scale.

Security Testing

Useful links:

• Node.js Security Checklist.
• Node Security: Fixing OWASP Top 10.

Linting and Code Quality

• ESLint: initial release date: June 30, 2013, written in: JavaScript. ESLint a pluggable and configurable linter tool for identifying and reporting on patterns in JavaScript.

  Useful links:

  • JavaScript Naming Conventions.
  • Why npm lockfiles can be a security blindspot for injecting malicious modules.

Version Control Systems (VCS)

• Git: initial release date: April 7, 2005, written in: C, Perl, Tcl, Python. Git is a version-control system for tracking changes in computer files and coordinating work on those files among multiple people. It is primarily used for source-code management in software development, but it can be used to keep track of changes in any set of files.

  Useful links:

  • Git workflow guidelines with easy and clear instructions on "What to do" and explanations on "Why to do it this way".
  • Removing sensitive data from a repository.
  • Markdown is a lightweight markup language used all over GitHub for documentation and instructions (for example README.md files). Here is a good Markdown Cheatsheet.

Continuous Integration (CI / CD)

• Jenkins: initial release date: February 2, 2011, written in: Java. Jenkins is commonly used for: building projects, running tests to detect bugs and other issues as soon as they are introduced, static code analysis, deployment, execute repetitive tasks, save time, and optimize your development process. It usually has to be installed on your servers / machines.

• Travis: initial release date: November, 2010, written in: Ruby. Travis is famous for: quick setup (just add .travis.yml configuration file, login with Github, tell Travis CI to test a project and then push to Github), live build views, pull request support, pre-installed database services, auto deployments on passing builds, clean VMs for every build, etc. With Travis CI testing your open source projects will always be free! It does not need to be installed on your servers / machines, it just needs to be enabled for your GitHub repository.

• GitLab: initial release date: October, 2011, written in: Ruby, Go, JavaScript. GitLab is a web-based Git-repository manager providing wiki, issue-tracking and CI/CD pipeline features. It usually has to be installed on your servers / machines.

• GitHub Actions: initial release date: October, 2018, written in: Ruby, Java, JavaScript. GitHub Actions are used to automate, customize, and execute your software development workflows right in your repository. You can create actions to perform any job you'd like, including CI/CD, and combine actions in a completely customized workflow.

  Useful links:

  • A first look at GitHub Actions.

Containerization Tools

• Docker: initial release date: December 21, 2013, written in: Go. Docker is essentially, an isolated environment for your code to run in.

  Useful links:

  • A Practical Introduction to Container Terminology.
  • A Better Way to Develop Node.js with Docker.
  • Docker 101 workshop - introduction to Docker and basic concepts.
  • Top 4 Tactics To Keep Node.js Rockin’ in Docker.

• Kubernetes: initial release date: 7 June, 2014, written in: Go. Kubernetes (k8s) is an open-source system for automating deployment, scaling, and management of containerized applications. A container orchestrator like Kubernetes usually does two things:

  • Dynamically schedules container workloads within a cluster of computers. This is often referred to as distributed computing.
  • Provides a standardized application definition file (kube yaml, docker compose, etc.)

  Useful links:

  • Kubernetes Production Patterns and Anti-Patterns

Command Line Tools (CLI)

• Grep: initial release date: November, 1974, written in: PDP-11 assembly. Grep is a command-line utility for searching plain-text data sets for lines that match a regular expression. Its name comes from g/re/p (globally search a regular expression and print), which has the same effect: doing a global search with the regular expression and printing all matching lines.

  Useful links:

  • Manipulating text at the command line with grep.

• AWK: initial release date: 1977, written in: C. AWK is a scripting language used for manipulating data and generating reports.The awk command programming language requires no compiling, and allows the user to use variables, numeric functions, string functions, and logical operators.

Bug Tracking and Task Management

• Redmine: initial release date: June 25, 2006, written in: Ruby. Redmine is a free and open source, web-based project management and issue tracking tool. It allows users to manage multiple projects and associated subprojects. It features per project wikis and forums, time tracking, and flexible, role-based access control.

• Mantis: initial release date: Nov 6, 2008, written in: PHP. Mantis Bug Tracker is a free and open source, web-based bug tracking system. The most common use of MantisBT is to track software defects. However, MantisBT is often configured by users to serve as a more generic issue tracking system and project management tool.

• Jira: initial release date: 2002, written in: Java. Jira used to be an open source tool, but then became a commercial product developed by Atlassian for issue tracking and agile project management. The name Jira comes from Godjira or Japanese for Godzilla. Jira is used by a large number of companies globally for project, time, requirements, task, bug, change, code, test, release and sprint management.

Useful links:

• Scrum, Kanban, and ScrumBan: What’s the Difference.

Node.js

Useful links:

• Node.js & JavaScript Testing Best Practices.
• Node.js Crash Course - nice 90 minute video on starting with Node.js.
• Cheatsheet on main Node.js functions and processes.
• Practical Node.js - open source book.
• How to use Docker for Node.js development.
• Differences between iterating over an array with the 4 primary looping constructs in JavaScript.
• The Pitfalls of Async/Await in Array Loops.
• Guide to JSON.stringify in JavaScript.
• Very expressive videos about Functional Programming in JavaScript by Mattias Petter Johansson Twitter | Youtube:
  • Part 1: Higher-order functions;
  • Part 2: Map;
  • Part 3: Reduce Basics;
  • Part 4: Reduce Advanced;
  • Part 5: Closures;
  • Part 6: Currying;
  • Part 7: Recursion;
  • Part 8: Promises;
  • Composition over Inheritance;
  • Factory Functions in JavaScript.
• When Not to Use Lock Files with Node.js.
• 7 Methods for Working with Directories in Node.js.
• How to build a CLI with Node.js.
• How to access Google Spreadsheets with Node.js: article 1, article 2.
• A Guide to Node.js logging.
• Understanding npm link.
• Monorepos and npm.
• JavaScript modules.
• Types of test unreliability - informative 30 min video with code examples by Rich Trott.
• Running Python scripts from Node.js.
• JavaScript Visualized: the JavaScript Engine.
• Object.assign vs Object Spread in Node.js.
• How To Write Shell Scripts in Node with Google’s zx Library.
• The Death of a Node.js Process.

Leadership in QA

Useful links:

• Leading quality in software organizations.
• The impact of your conversation style on your team.
• One key metric should drive quality engineering.
• What you should really measure in your annual performance reviews (and why).
• How to Make Good Code Reviews Better.
• Emotional intelligence: 7 essential tips for leaders.
• IT burnout.
• Job interviews: how to spot toxic management.
• The Joys of Managing Engineering Teams by Maryam Umar.
• Being successfull as a team glue.
• An Introvert's Guide to being a Mentor - with a plan.
• How to Mentor Someone Who Doesn’t Know What Their Career Goals Should Be.
• How to be a sponsor when you're a developer.
• Getting mentors.
• Difference between mentorship and sponsorship.
• Maybe getting rid of your QA team was bad, actually.

Job Search

• Tips on writing good resumes.
• How to advance your career by contributing to open source projects.
• A curated list of established remote/distributed/virtual tech companies.

Thanks

If this list of free open source software testing tools and links was helpful to you, please give it a ★ Star on GitHub.