Tests test_iter3 and test_iter2 fails on big endian

[mcepl]

While building packages for openSUSE, test suite fails on all big endian architectures (ppc64, s390x, and armv7l):

[   42s] ======================================================================
[   42s] FAIL: test_iter3 (unittests.TestAutomatonIterSearch)
[   42s] ----------------------------------------------------------------------
[   42s] Traceback (most recent call last):
[   42s]   File "/home/abuild/rpmbuild/BUILD/pyahocorasick-1.4.1/unittests.py", line 775, i
n test_iter3
[   42s]     self.assertEqual(L, C)
[   42s] AssertionError: Lists differ: [] != [(6, 'he'), (7, 'her'), (8, 'hers')]
[   42s]
[   42s] Second list contains 3 additional elements.
[   42s] First extra element 0:
[   42s] (6, 'he')
[   42s]
[   42s] - []
[   42s] + [(6, 'he'), (7, 'her'), (8, 'hers')]
[   42s]
[   42s] ======================================================================
[   42s] FAIL: test_iter2 (unittests.TestAutomatonIterSearchWithIgnoreWhiteSpace)
[   42s] ----------------------------------------------------------------------
[   42s] Traceback (most recent call last):
[   42s]   File "/home/abuild/rpmbuild/BUILD/pyahocorasick-1.4.1/unittests.py", line 890, i
n test_iter2
[   42s]     self.assertEqual(L, self.correct_positons_start_12)
[   42s] AssertionError: Lists differ: [(4, 'she'), (4, 'he'), (6, 'her'), (8, 'he[49 chars
]he')] != [(13, 'he')]
[   42s]
[   42s] First differing element 0:
[   42s] (4, 'she')
[   42s] (13, 'he')
[   42s]
[   42s] First list contains 7 additional elements.
[   42s] First extra element 1:
[   42s] (4, 'he')
[   42s]
[   42s] - [(4, 'she'),
[   42s] -  (4, 'he'),
[   42s] -  (6, 'her'),
[   42s] -  (8, 'he'),
[   42s] -  (9, 'her'),
[   42s] -  (11, 'hers'),
[   42s] -  (13, 'she'),
[   42s] -  (13, 'he')]
[   42s] ? ^
[   42s]
[   42s] + [(13, 'he')]
[   42s] ? ^

Complete build on PPC64 with all details

[WojciechMula]

Thank you for reporting this problem. Can anybody help to debug?

[pombredanne]

@mcepl How can we debug this?

[mcepl]

@mcepl How can we debug this?

Tell me which logs and other information I should collect?

[pombredanne]

@mcepl re logs.
Where are you building for openSUSE, on all big endian architectures (ppc64, s390x, and armv7l)?
Could you help us by having a way to easily run the tests on demand or on commit on all these architectures?
That would be awesome!

Last I checked in my cave, I could not find any leftover mainframe in a corner to use to run tests ;) 😇

[mcepl]

@mcepl re logs. Where are you building for openSUSE, on all big endian architectures (ppc64, s390x, and armv7l)? Could you help us by having a way to easily run the tests on demand or on commit on all these architectures? That would be awesome!

See https://build.opensuse.org/package/show/devel:languages:python/python-pyahocorasick ... if I can just run the test suite, I can run it on all these (and probably some other) platforms.

[pombredanne]

@mcepl thanks... the basic test suite is with make test (for py2 and py3) or just make test3 for Python 3 only. that's for the latest master code and the latest 1.4.4 release. The set of extended tests are listed in https://github.com/WojciechMula/pyahocorasick/blob/master/release_checklist.txt

[pombredanne]

I see your run them with pytest per:

pytest-3.9 --ignore=_build.python39 --ignore=_build.python310 --ignore=_build.python38 -v -k '' unittests.py
... this works too.
@WojciechMula FYI, I want to reorg the test suite to be also all under a test/ directory (a pet pieve of mine ;) )

[mcepl]

I see your run them with pytest per:

pytest-3.9 --ignore=_build.python39 --ignore=_build.python310 --ignore=_build.python38 -v -k '' unittests.py
... this works too.
@WojciechMula FYI, I want to reorg the test suite to be also all under a test/ directory (a pet pieve of mine ;) )

I have no particular problem with unittest v. pytest, but I am not big fan of other testing packages (nose2, or some other ones, and nose1 is dead and being removed from openSUSE this week).

[eclipseo]

Still having issue on s390x (Python 3.12, pyahocorasick-2.0.0)

=================================== FAILURES ===================================
______________________________ test_basic_unicode ______________________________
    @pytest.mark.skipif(not ahocorasick.unicode, reason="Run only with unicode build")
    @pytest.mark.xfail(on_windows, reason="Fails on Windows")
    def test_basic_unicode():
        automaton = ahocorasick.Automaton()
        words = "he e hers his she hi him man he".split()
        #        0  1 2    3   4   5  6   7   8
        for i, w in enumerate(words):
            automaton.add_word(w, (i, w))
        query = "he rshershidamanza "
        #        01234567890123
        automaton.make_automaton()
    
        assert query[2:8] == ' rsher'
        results = list(automaton.iter(string=query, start=2, end=8))
>       assert results == [(6, (4, 'she')), (6, (8, 'he')), (6, (1, 'e'))]
E       AssertionError: assert [] == [(6, (4, 'she...(6, (1, 'e'))]
E         Right contains 3 more items, first extra item: (6, (4, 'she'))
E         Use -v to get more diff
tests/test_basic.py:115: AssertionError
_____________________________ test_issue_10_case_1 _____________________________
    def test_issue_10_case_1():
        ac = ahocorasick.Automaton()
        ac.add_word(conv('S'), 1)
        ac.make_automaton()
        buffer = conv('SSS')
        count = 0
        for _item in ac.iter(buffer, 0, 3):  # this causes an error
            count += 1
    
>       assert count == 3
E       assert 0 == 3
tests/test_issue_10.py:23: AssertionError
_____________________________ test_issue_10_case_2 _____________________________
    def test_issue_10_case_2():
        ac = ahocorasick.Automaton()
        ac.add_word(conv('S'), 1)
        ac.make_automaton()
        buffer = conv('SSS')
        count = 0
        for _item in ac.iter(buffer, 0, 2):  # no error, but it misses the last 'S' in the buffer
            count += 1
    
>       assert count == 2
E       assert 0 == 2
tests/test_issue_10.py:35: AssertionError
______________________ TestAutomatonIterSearch.test_iter3 ______________________
self = <test_unit.TestAutomatonIterSearch testMethod=test_iter3>
    def test_iter3(self):
        A = self.add_words_and_make_automaton()
    
        start = 4
        end = 9
    
        C = []
        for index, word in A.iter(conv(self.string[start:end])):
            C.append((index + start, word))
    
        L = []
        for index, word in A.iter(conv(self.string), start, end):
            L.append((index, word))
    
>       self.assertEqual(L, C)
E       AssertionError: Lists differ: [] != [(6, 'he'), (7, 'her'), (8, 'hers')]
E       
E       Second list contains 3 additional elements.
E       First extra element 0:
E       (6, 'he')
E       
E       - []
E       + [(6, 'he'), (7, 'her'), (8, 'hers')]
tests/test_unit.py:737: AssertionError
____________ TestAutomatonIterSearchWithIgnoreWhiteSpace.test_iter2 ____________
self = <test_unit.TestAutomatonIterSearchWithIgnoreWhiteSpace testMethod=test_iter2>
    def test_iter2(self):
        self.add_words_and_make_automaton()
        A = self.A
        self.assertEqual(A.kind, ahocorasick.AHOCORASICK)
    
        L = []
        for index, word in A.iter(conv(self.string), ignore_white_space=True, start=12):
            L.append((index, word))
>       self.assertEqual(L, self.correct_positons_start_12)
E       AssertionError: Lists differ: [(4, 'she'), (4, 'he'), (6, 'her'), (8, 'he[49 chars]he')] != [(13, 'he')]
E       
E       First differing element 0:
E       (4, 'she')
E       (13, 'he')
E       
E       First list contains 7 additional elements.
E       First extra element 1:
E       (4, 'he')
E       
E       - [(4, 'she'),
E       -  (4, 'he'),
E       -  (6, 'her'),
E       -  (8, 'he'),
E       -  (9, 'her'),
E       -  (11, 'hers'),
E       -  (13, 'she'),
E       -  (13, 'he')]
E       ? ^
E       
E       + [(13, 'he')]
E       ? ^
tests/test_unit.py:849: AssertionError
=========================== short test summary info ============================
FAILED tests/test_basic.py::test_basic_unicode - AssertionError: assert [] ==...
FAILED tests/test_issue_10.py::test_issue_10_case_1 - assert 0 == 3
FAILED tests/test_issue_10.py::test_issue_10_case_2 - assert 0 == 2
FAILED tests/test_unit.py::TestAutomatonIterSearch::test_iter3 - AssertionErr...
FAILED tests/test_unit.py::TestAutomatonIterSearchWithIgnoreWhiteSpace::test_iter2
=================== 5 failed, 158 passed, 7 skipped in 2.14s ===================