forked from django/django
/
unit-tests.txt
616 lines (437 loc) · 21.4 KB
/
unit-tests.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
==========
Unit tests
==========
Django comes with a test suite of its own, in the ``tests`` directory of the
code base. It's our policy to make sure all tests pass at all times.
We appreciate any and all contributions to the test suite!
The Django tests all use the testing infrastructure that ships with Django for
testing applications. See :doc:`/topics/testing/overview` for an explanation of
how to write new tests.
.. _running-unit-tests:
Running the unit tests
======================
Quickstart
----------
First, `fork Django on GitHub <https://github.com/django/django/fork>`__.
Second, create and activate a virtual environment. If you're not familiar with
how to do that, read our :doc:`contributing tutorial </intro/contributing>`.
Next, clone your fork, install some requirements, and run the tests:
.. console::
$ git clone https://github.com/YourGitHubName/django.git django-repo
$ cd django-repo/tests
$ python -m pip install -e ..
$ python -m pip install -r requirements/py3.txt
$ ./runtests.py
Installing the requirements will likely require some operating system packages
that your computer doesn't have installed. You can usually figure out which
package to install by doing a Web search for the last line or so of the error
message. Try adding your operating system to the search query if needed.
If you have trouble installing the requirements, you can skip that step. See
:ref:`running-unit-tests-dependencies` for details on installing the optional
test dependencies. If you don't have an optional dependency installed, the
tests that require it will be skipped.
Running the tests requires a Django settings module that defines the databases
to use. To help you get started, Django provides and uses a sample settings
module that uses the SQLite database. See :ref:`running-unit-tests-settings` to
learn how to use a different settings module to run the tests with a different
database.
Having problems? See :ref:`troubleshooting-unit-tests` for some common issues.
Running tests using ``tox``
---------------------------
`Tox <https://tox.readthedocs.io/>`_ is a tool for running tests in different
virtual environments. Django includes a basic ``tox.ini`` that automates some
checks that our build server performs on pull requests. To run the unit tests
and other checks (such as :ref:`import sorting <coding-style-imports>`, the
:ref:`documentation spelling checker <documentation-spelling-check>`, and
:ref:`code formatting <coding-style-python>`), install and run the ``tox``
command from any place in the Django source tree:
.. console::
$ python -m pip install tox
$ tox
By default, ``tox`` runs the test suite with the bundled test settings file for
SQLite, ``flake8``, ``isort``, and the documentation spelling checker. In
addition to the system dependencies noted elsewhere in this documentation,
the command ``python3`` must be on your path and linked to the appropriate
version of Python. A list of default environments can be seen as follows:
.. console::
$ tox -l
py3
flake8
docs
isort>=5.1.0
Testing other Python versions and database backends
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In addition to the default environments, ``tox`` supports running unit tests
for other versions of Python and other database backends. Since Django's test
suite doesn't bundle a settings file for database backends other than SQLite,
however, you must :ref:`create and provide your own test settings
<running-unit-tests-settings>`. For example, to run the tests on Python 3.9
using PostgreSQL:
.. console::
$ tox -e py39-postgres -- --settings=my_postgres_settings
This command sets up a Python 3.9 virtual environment, installs Django's
test suite dependencies (including those for PostgreSQL), and calls
``runtests.py`` with the supplied arguments (in this case,
``--settings=my_postgres_settings``).
The remainder of this documentation shows commands for running tests without
``tox``, however, any option passed to ``runtests.py`` can also be passed to
``tox`` by prefixing the argument list with ``--``, as above.
Tox also respects the :envvar:`DJANGO_SETTINGS_MODULE` environment variable, if
set. For example, the following is equivalent to the command above:
.. code-block:: console
$ DJANGO_SETTINGS_MODULE=my_postgres_settings tox -e py39-postgres
Windows users should use:
.. code-block:: doscon
...\> set DJANGO_SETTINGS_MODULE=my_postgres_settings
...\> tox -e py39-postgres
Running the JavaScript tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Django includes a set of :ref:`JavaScript unit tests <javascript-tests>` for
functions in certain contrib apps. The JavaScript tests aren't run by default
using ``tox`` because they require ``Node.js`` to be installed and aren't
necessary for the majority of patches. To run the JavaScript tests using
``tox``:
.. console::
$ tox -e javascript
This command runs ``npm install`` to ensure test requirements are up to
date and then runs ``npm test``.
Running tests using ``django-docker-box``
-----------------------------------------
`django-docker-box`_ allows you to run the Django's test suite across all
supported databases and python versions. See the `django-docker-box`_ project
page for installation and usage instructions.
.. _django-docker-box: https://github.com/django/django-docker-box/
.. _running-unit-tests-settings:
Using another ``settings`` module
---------------------------------
The included settings module (``tests/test_sqlite.py``) allows you to run the
test suite using SQLite. If you want to run the tests using a different
database, you'll need to define your own settings file. Some tests, such as
those for ``contrib.postgres``, are specific to a particular database backend
and will be skipped if run with a different backend. Some tests are skipped or
expected failures on a particular database backend (see
``DatabaseFeatures.django_test_skips`` and
``DatabaseFeatures.django_test_expected_failures`` on each backend).
To run the tests with different settings, ensure that the module is on your
:envvar:`PYTHONPATH` and pass the module with ``--settings``.
The :setting:`DATABASES` setting in any test settings module needs to define
two databases:
* A ``default`` database. This database should use the backend that
you want to use for primary testing.
* A database with the alias ``other``. The ``other`` database is used to test
that queries can be directed to different databases. This database should use
the same backend as the ``default``, and it must have a different name.
If you're using a backend that isn't SQLite, you will need to provide other
details for each database:
* The :setting:`USER` option needs to specify an existing user account
for the database. That user needs permission to execute ``CREATE DATABASE``
so that the test database can be created.
* The :setting:`PASSWORD` option needs to provide the password for
the :setting:`USER` that has been specified.
Test databases get their names by prepending ``test_`` to the value of the
:setting:`NAME` settings for the databases defined in :setting:`DATABASES`.
These test databases are deleted when the tests are finished.
You will also need to ensure that your database uses UTF-8 as the default
character set. If your database server doesn't use UTF-8 as a default charset,
you will need to include a value for :setting:`CHARSET <TEST_CHARSET>` in the
test settings dictionary for the applicable database.
.. _runtests-specifying-labels:
Running only some of the tests
------------------------------
Django's entire test suite takes a while to run, and running every single test
could be redundant if, say, you just added a test to Django that you want to
run quickly without running everything else. You can run a subset of the unit
tests by appending the names of the test modules to ``runtests.py`` on the
command line.
For example, if you'd like to run tests only for generic relations and
internationalization, type:
.. console::
$ ./runtests.py --settings=path.to.settings generic_relations i18n
How do you find out the names of individual tests? Look in ``tests/`` — each
directory name there is the name of a test.
If you want to run only a particular class of tests, you can specify a list of
paths to individual test classes. For example, to run the ``TranslationTests``
of the ``i18n`` module, type:
.. console::
$ ./runtests.py --settings=path.to.settings i18n.tests.TranslationTests
Going beyond that, you can specify an individual test method like this:
.. console::
$ ./runtests.py --settings=path.to.settings i18n.tests.TranslationTests.test_lazy_objects
You can run tests starting at a specified top-level module with ``--start-at``
option. For example:
.. console::
$ ./runtests.py --start-at=wsgi
You can also run tests starting after a specified top-level module with
``--start-after`` option. For example:
.. console::
$ ./runtests.py --start-after=wsgi
Note that the ``--reverse`` option doesn't impact on ``--start-at`` or
``--start-after`` options. Moreover these options cannot be used with test
labels.
Running the Selenium tests
--------------------------
Some tests require Selenium and a Web browser. To run these tests, you must
install the selenium_ package and run the tests with the
``--selenium=<BROWSERS>`` option. For example, if you have Firefox and Google
Chrome installed:
.. console::
$ ./runtests.py --selenium=firefox,chrome
See the `selenium.webdriver`_ package for the list of available browsers.
Specifying ``--selenium`` automatically sets ``--tags=selenium`` to run only
the tests that require selenium.
Some browsers (e.g. Chrome or Firefox) support headless testing, which can be
faster and more stable. Add the ``--headless`` option to enable this mode.
.. _selenium.webdriver: https://github.com/SeleniumHQ/selenium/tree/trunk/py/selenium/webdriver
.. _running-unit-tests-dependencies:
Running all the tests
---------------------
If you want to run the full suite of tests, you'll need to install a number of
dependencies:
* argon2-cffi_ 19.1.0+
* asgiref_ 3.3.2+ (required)
* bcrypt_
* colorama_
* docutils_
* geoip2_
* jinja2_ 2.7+
* numpy_
* Pillow_ 6.2.0+
* PyYAML_
* pytz_ (required)
* pywatchman_
* setuptools_
* memcached_, plus a :ref:`supported Python binding <memcached>`
* gettext_ (:ref:`gettext_on_windows`)
* selenium_
* sqlparse_ 0.2.2+ (required)
* tblib_ 1.5.0+
You can find these dependencies in `pip requirements files`_ inside the
``tests/requirements`` directory of the Django source tree and install them
like so:
.. console::
$ python -m pip install -r tests/requirements/py3.txt
If you encounter an error during the installation, your system might be missing
a dependency for one or more of the Python packages. Consult the failing
package's documentation or search the Web with the error message that you
encounter.
You can also install the database adapter(s) of your choice using
``oracle.txt``, ``mysql.txt``, or ``postgres.txt``.
If you want to test the memcached cache backend, you'll also need to define
a :setting:`CACHES` setting that points at your memcached instance.
To run the GeoDjango tests, you will need to :doc:`setup a spatial database
and install the Geospatial libraries</ref/contrib/gis/install/index>`.
Each of these dependencies is optional. If you're missing any of them, the
associated tests will be skipped.
To run some of the autoreload tests, you'll need to install the Watchman_
service.
.. _argon2-cffi: https://pypi.org/project/argon2-cffi/
.. _asgiref: https://pypi.org/project/asgiref/
.. _bcrypt: https://pypi.org/project/bcrypt/
.. _colorama: https://pypi.org/project/colorama/
.. _docutils: https://pypi.org/project/docutils/
.. _geoip2: https://pypi.org/project/geoip2/
.. _jinja2: https://pypi.org/project/Jinja2/
.. _numpy: https://pypi.org/project/numpy/
.. _Pillow: https://pypi.org/project/Pillow/
.. _PyYAML: https://pyyaml.org/wiki/PyYAML
.. _pytz: https://pypi.org/project/pytz/
.. _pywatchman: https://pypi.org/project/pywatchman/
.. _setuptools: https://pypi.org/project/setuptools/
.. _memcached: https://memcached.org/
.. _gettext: https://www.gnu.org/software/gettext/manual/gettext.html
.. _selenium: https://pypi.org/project/selenium/
.. _sqlparse: https://pypi.org/project/sqlparse/
.. _pip requirements files: https://pip.pypa.io/en/latest/user_guide/#requirements-files
.. _tblib: https://pypi.org/project/tblib/
.. _Watchman: https://facebook.github.io/watchman/
Code coverage
-------------
Contributors are encouraged to run coverage on the test suite to identify areas
that need additional tests. The coverage tool installation and use is described
in :ref:`testing code coverage<topics-testing-code-coverage>`.
Coverage should be run in a single process to obtain accurate statistics. To
run coverage on the Django test suite using the standard test settings:
.. console::
$ coverage run ./runtests.py --settings=test_sqlite --parallel=1
After running coverage, generate the html report by running:
.. console::
$ coverage html
When running coverage for the Django tests, the included ``.coveragerc``
settings file defines ``coverage_html`` as the output directory for the report
and also excludes several directories not relevant to the results
(test code or external code included in Django).
.. _contrib-apps:
Contrib apps
============
Tests for contrib apps can be found in the ``tests/`` directory, typically
under ``<app_name>_tests``. For example, tests for ``contrib.auth`` are located
in ``tests/auth_tests``.
.. _troubleshooting-unit-tests:
Troubleshooting
===============
Test suite hangs or shows failures on ``main`` branch
-----------------------------------------------------
Ensure you have the latest point release of a :ref:`supported Python version
<faq-python-version-support>`, since there are often bugs in earlier versions
that may cause the test suite to fail or hang.
On **macOS** (High Sierra and newer versions), you might see this message
logged, after which the tests hang::
objc[42074]: +[__NSPlaceholderDate initialize] may have been in progress in
another thread when fork() was called.
To avoid this set a ``OBJC_DISABLE_INITIALIZE_FORK_SAFETY`` environment
variable, for example::
$ OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES ./runtests.py
Or add ``export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES`` to your shell's
startup file (e.g. ``~/.profile``).
Many test failures with ``UnicodeEncodeError``
----------------------------------------------
If the ``locales`` package is not installed, some tests will fail with a
``UnicodeEncodeError``.
You can resolve this on Debian-based systems, for example, by running:
.. code-block:: console
$ apt-get install locales
$ dpkg-reconfigure locales
You can resolve this for macOS systems by configuring your shell's locale:
.. code-block:: console
$ export LANG="en_US.UTF-8"
$ export LC_ALL="en_US.UTF-8"
Run the ``locale`` command to confirm the change. Optionally, add those export
commands to your shell's startup file (e.g. ``~/.bashrc`` for Bash) to avoid
having to retype them.
Tests that only fail in combination
-----------------------------------
In case a test passes when run in isolation but fails within the whole suite,
we have some tools to help analyze the problem.
The ``--bisect`` option of ``runtests.py`` will run the failing test while
halving the test set it is run together with on each iteration, often making
it possible to identify a small number of tests that may be related to the
failure.
For example, suppose that the failing test that works on its own is
``ModelTest.test_eq``, then using:
.. console::
$ ./runtests.py --bisect basic.tests.ModelTest.test_eq
will try to determine a test that interferes with the given one. First, the
test is run with the first half of the test suite. If a failure occurs, the
first half of the test suite is split in two groups and each group is then run
with the specified test. If there is no failure with the first half of the test
suite, the second half of the test suite is run with the specified test and
split appropriately as described earlier. The process repeats until the set of
failing tests is minimized.
The ``--pair`` option runs the given test alongside every other test from the
suite, letting you check if another test has side-effects that cause the
failure. So:
.. console::
$ ./runtests.py --pair basic.tests.ModelTest.test_eq
will pair ``test_eq`` with every test label.
With both ``--bisect`` and ``--pair``, if you already suspect which cases
might be responsible for the failure, you may limit tests to be cross-analyzed
by :ref:`specifying further test labels <runtests-specifying-labels>` after
the first one:
.. console::
$ ./runtests.py --pair basic.tests.ModelTest.test_eq queries transactions
You can also try running any set of tests in reverse using the ``--reverse``
option in order to verify that executing tests in a different order does not
cause any trouble:
.. console::
$ ./runtests.py basic --reverse
Seeing the SQL queries run during a test
----------------------------------------
If you wish to examine the SQL being run in failing tests, you can turn on
:ref:`SQL logging <django-db-logger>` using the ``--debug-sql`` option. If you
combine this with ``--verbosity=2``, all SQL queries will be output:
.. console::
$ ./runtests.py basic --debug-sql
Seeing the full traceback of a test failure
-------------------------------------------
By default tests are run in parallel with one process per core. When the tests
are run in parallel, however, you'll only see a truncated traceback for any
test failures. You can adjust this behavior with the ``--parallel`` option:
.. console::
$ ./runtests.py basic --parallel=1
You can also use the :envvar:`DJANGO_TEST_PROCESSES` environment variable for
this purpose.
Tips for writing tests
======================
.. highlight:: python
Isolating model registration
----------------------------
To avoid polluting the global :attr:`~django.apps.apps` registry and prevent
unnecessary table creation, models defined in a test method should be bound to
a temporary ``Apps`` instance::
from django.apps.registry import Apps
from django.db import models
from django.test import SimpleTestCase
class TestModelDefinition(SimpleTestCase):
def test_model_definition(self):
test_apps = Apps(['app_label'])
class TestModel(models.Model):
class Meta:
apps = test_apps
...
.. function:: django.test.utils.isolate_apps(*app_labels, attr_name=None, kwarg_name=None)
Since this pattern involves a lot of boilerplate, Django provides the
:func:`~django.test.utils.isolate_apps` decorator. It's used like this::
from django.db import models
from django.test import SimpleTestCase
from django.test.utils import isolate_apps
class TestModelDefinition(SimpleTestCase):
@isolate_apps('app_label')
def test_model_definition(self):
class TestModel(models.Model):
pass
...
.. admonition:: Setting ``app_label``
Models defined in a test method with no explicit
:attr:`~django.db.models.Options.app_label` are automatically assigned the
label of the app in which their test class is located.
In order to make sure the models defined within the context of
:func:`~django.test.utils.isolate_apps` instances are correctly
installed, you should pass the set of targeted ``app_label`` as arguments:
.. code-block:: python
:caption: tests/app_label/tests.py
from django.db import models
from django.test import SimpleTestCase
from django.test.utils import isolate_apps
class TestModelDefinition(SimpleTestCase):
@isolate_apps('app_label', 'other_app_label')
def test_model_definition(self):
# This model automatically receives app_label='app_label'
class TestModel(models.Model):
pass
class OtherAppModel(models.Model):
class Meta:
app_label = 'other_app_label'
...
The decorator can also be applied to classes::
from django.db import models
from django.test import SimpleTestCase
from django.test.utils import isolate_apps
@isolate_apps('app_label')
class TestModelDefinition(SimpleTestCase):
def test_model_definition(self):
class TestModel(models.Model):
pass
...
The temporary ``Apps`` instance used to isolate model registration can be
retrieved as an attribute when used as a class decorator by using the
``attr_name`` parameter::
from django.db import models
from django.test import SimpleTestCase
from django.test.utils import isolate_apps
@isolate_apps('app_label', attr_name='apps')
class TestModelDefinition(SimpleTestCase):
def test_model_definition(self):
class TestModel(models.Model):
pass
self.assertIs(self.apps.get_model('app_label', 'TestModel'), TestModel)
Or as an argument on the test method when used as a method decorator by using
the ``kwarg_name`` parameter::
from django.db import models
from django.test import SimpleTestCase
from django.test.utils import isolate_apps
class TestModelDefinition(SimpleTestCase):
@isolate_apps('app_label', kwarg_name='apps')
def test_model_definition(self, apps):
class TestModel(models.Model):
pass
self.assertIs(apps.get_model('app_label', 'TestModel'), TestModel)