-
-
Notifications
You must be signed in to change notification settings - Fork 149
Testing LedgerSMB
LedgerSMB includes tests which allow confirmation of a successful installation and enable bugs to be caught during development. All code contributions submitted via Github are automatically tested using Travis CI before being merged.
During development - of code, or new tests - it can be useful to run the test suite locally. LedgerSMB provides a Docker Compose based test and development infrastructure to get up-and-running quickly.
Development tests (found in xt/) should not be run against a production database server as these tests mess with roles and databases which could affect existing data. General tests (found in t/) are safe to run against a production system and can be used to confirm correct installation.
A minimum 4GB RAM is required to run tests using the Docker Compose containers (based on empirical testing). Insufficient memory will cause the tests to fail in non-obvious and inconsistent ways, for example the prove command may stop without completing all tests, yet display no error or status message.
The following steps will install LedgerSMB test infrastructure on a fresh Debian 9 (Stretch) system.
## Installation on new Debian 9 Stretch machine
## Tested 24 December 2017 by Nick Prater
# Install base packages
apt update
apt upgrade
apt dist-upgrade
apt install git curl
# Install docker-compose
# Needs more recent docker-compose than provided by distribution
# Check for latest version number before running
# see: https://docs.docker.com/compose/install/#install-compose
curl -L \
https://github.com/docker/compose/releases/download/1.18.0/docker-compose-`uname -s`-`uname -m` \
-o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
# Install docker
# Needs more recent docker than shipped by distribution
# see: https://docs.docker.com/engine/installation/linux/docker-ce/debian/#set-up-the-repository
apt install \
apt-transport-https \
ca-certificates \
curl \
gnupg2 \
software-properties-common
curl -fsSL https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")/gpg | sudo apt-key add -
add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") \
$(lsb_release -cs) \
stable"
apt update
apt install docker-ce
# Add a user `lsmb-dev` for development
adduser lsmb-dev # answer prompts
adduser lsmb-dev docker
# Install lsmb repositories
# As development user
su - lsmb-dev
git clone https://github.com/ledgersmb/LedgerSMB.git ledgersmb
git clone https://github.com/ledgersmb/ledgersmb-dev-docker.git
# Install dojo
cd ledgersmb
git submodule init
git submodule update
# Use config for 'unbuilt' dojo
cp ~/ledgersmb/doc/conf/ledgersmb.conf.unbuilt-dojo ~/ledgersmb/ledgersmb.conf
# Create a directory for screenshots
mkdir ~/ledgersmb/screens
# Adjust permissions so that tests run
chmod ugo+rwx ~/ledgersmb/t/data # so that test t/12-sysconfig.t runs
chmod ugo+rwx ~/ledgersmb/screens # so we can write screenshots
# Start Ledgersmb containers
cd ~/ledgersmb
docker-compose -f ~/ledgersmb-dev-docker/docker-compose.yml up -d # start
The following command will run the full test suite on master, skipping those tests tagged with @wip which are still being developed and expected to fail.
docker exec -ti ledgersmb-dev-docker_lsmb_1 \
prove \
--pgtap-option dbname=lsmbtestdb \
--feature-option tags=~@wip \
-r t/ xt/
If tests fail after making significant changes, it can be useful to normalise the test environment, using the following commands, which provide a consistent stating point for the tests:
# Restart the lsmb container to ensure Starman picks up
# any code changes.
docker restart ledgersmb-dev-docker_lsmb_1
# Ensure necessary perl modules are installed
# In case recent commits have introduced new dependencies
docker exec -ti --user=root ledgersmb-dev-docker_lsmb_1 \
cpanm \
--quiet --notest \
--with-develop \
--with-feature=starman \
--with-feature=latex-pdf-ps \
--with-feature=openoffice \
--with-feature=xls \
--with-feature=edi \
--installdeps .
# Drop any test databases left over from previous failed or
# aborted test runs. After a complete test run, test
# xt/89-dropdb.t normally does this cleanup
docker exec -ti --user=postgres ledgersmb-dev-docker_db_1 psql -c 'DROP DATABASE IF EXISTS lsmbtest_database'
docker exec -ti --user=postgres ledgersmb-dev-docker_db_1 psql -c 'DROP DATABASE IF EXISTS lsmbtestdb'
Testing ledgersmb 1.5 requires slightly different commands:
# Checkout 1.5 source
cd ~/ledgersmb
git checkout 1.5
# Start containers
docker-compose -f ~/ledgersmb-dev-docker/docker-compose.yml up -d
# Set permissions so that tests can run
chmod ugo+rwx t/var
chmod ugo+rwx t/data
chmod ugo+rw sql/modules/BLACKLIST
# Run tests
docker exec -ti ledgersmb-dev-docker_lsmb_1 prove t xt
All being well, after running the tests you should see a message All tests successful. and Result: PASS. If not, examine the test output to see what went wrong. Adding a -v option to the prove command will display each step within each test.
The tests deliberately exercise various failure conditions to ensure they are handled correctly (for example missing directories and faulty sql modules) so it is normal to see some error messages displayed on the console.
If no summary or error message is displayed by prove, ensure that you have sufficient memory for running the tests.
Test failures and errors will result if there is insufficient memory or free disk space when the tests are run, particularly with the BDD tests in xt/66-cucumber.
Occasionally the BDD tests produce spurious errors. This is understood to be caused by the underlying DOM within the test browser being altered once the test is running. This is a timing issue and as such is difficult to replicate. If you encounter this issue, running the tests again will usually be successful. A typical error message is:
# # Failed test 'Test compiled'
# # at /usr/local/share/perl/5.24.1/Test/BDD/Cucumber/Executor.pm line 447.
# # Error while executing command: getElementTagName: An element command failed because the referenced element is no longer attached to the DOM.: {"errorMessage":"Element is no longer attached to the DOM","request":{"headers":{"Accept":"application/json","Connection":"TE, close","Content-Length":"2","Content-Type":"application/json; charset=utf-8","Host":"selenium:4422","TE":"deflate,gzip;q=0.3","User-Agent":"libwww-perl/6.15"},"httpVersion":"1.1","method":"GET","url":"/name","urlParsed":{"anchor":"","query":"","file":"name","directory":"/","path":"/name","relative":"/name","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/name","queryKey":{},"chunks":["name"]},"urlOriginal":"/session/54361530-e8b3-11e7-8f5e-4b60b5851240/element/:wdc:1514124257863/name"}} at /usr/local/share/perl/5.24.1/Selenium/Remote/Driver.pm line 327.
# # at /usr/local/share/perl/5.24.1/Selenium/Remote/Driver.pm line 327.
For Starman:
docker logs -f ledgersmb-dev-docker_lsmb_1
For the database:
docker logs -f ledgersmb-dev-docker_db_1
When debugging a failing browser-based test, such as those in xt/66-cucumber it can be useful to capture "screenshots" from the virtual browser at each step of the test. This can be enabled be editing
t/.pherkin.yaml and setting:
pre-step: 1
post-step: 1
pre-scenario: 1
post-scenario: 1
Using the default configuration, screenshots are written to the screens/ directory, but this can be altered by changing the screenshots_dir directive in t/.pherkin.yaml.
-
t/.proverc: configures how to run the tests for each file extension (.t, .pg, .feature) -
t/.pherkin.yaml: configures browser testing, including screenshots -
t/testrules.yml: configures the order in which tests are run, and which can be run in parallel -
~/ledgersmb-dev-docker/docker-compose.yml: configuration, including environment variables, used to start the docker containers
In general, it is possible to run individual tests by specifying them to the prove command. For example:
docker exec -ti ledgersmb-dev-docker_lsmb_1 prove --pgtap-option dbname=lsmbtestdb xt/07.1-pod.t
However, note that some tests require special database setup and must be bookended by xt/40-dbsetup.t and xt/89-dropdb.t. These sequencing rules are defined in t/testrules.yml.
docker ps
docker stats
docker-compose -f ~/ledgersmb-dev-docker/docker-compose.yml down
docker-compose -f ~/ledgersmb-dev-docker/docker-compose.yml pull
See documentation online at https://ledgersmb.org/content/writing-tests and in the following README files: