shuffling the elements of a list after converting to binary

[dimitriusssBR]

A strange problem after compiling with pyinstaller(auto-py-to-exe)
When Im running a code on Pycharm it finds the items in the stipulated order, but after compiling in .exe the order of the objects simply changes. Code follows:

Objects are correctly identified in pycharm, but after compiling in .exe the order changes completely.

plays =0
elements = self.find_all(label="arena_150", matching=0.7, waiting_time=3000, grayscale=True)
elements_list = list(elements)
number_of_elements = len(elements_list)
print('Energy available: ' + str(number_of_elements))
for _ in range(int(number_of_elements)):
x = elements_list[plays].left
y = elements_list[plays].top
self.mouse_move(x, y)
pyautogui.click()
plays +=1
time.sleep(3)

[rokm]

Can you provide full minimal example that reproduces the problem?

[dimitriusssBR]

Sure,
I work with computer vision, my code finds the 5 identical images from left to right and top to bottom and creates a list in that order. After creating a binary with pyinstaller, the order of objects changes.

When I run my code in pycharm:

[Box(left=1413, top=654, width=66, height=38)
Box(left=981, top=753, width=66, height=38)
Box(left=1154, top=753, width=66, height=38)
Box(left=1326, top=753, width=66, height=38)
Box(left=1499, top=753, width=66, height=38)]

after .exe:

[Box(left=1154, top=753, width=66, height=38)
Box(left=1499, top=753, width=66, height=38)
Box(left=981, top=753, width=66, height=38)
Box(left=1413, top=654, width=66, height=38)
Box(left=1326, top=753, width=66, height=38)]

I'm using the same code and the same image in this test

[bwoodsend]

That's not a reproducible example. We can't run it nor can we audit the source code for causes of indeterminism.

[rokm]

If you cannot share a reproducible example, you'll need to audit the code that generates this list yourself, and determine why it is generated differently.

[dimitriusssBR]

Ok, full code:

from botcity.core import DesktopBot
import time

class Bot(DesktopBot):

def action(self, execution=None):
    plays = 0
    while self.find("arena_150", matching=0.8, waiting_time=3000, grayscale=False):
        print('Above not found - Arena Attack 150')
        elements = self.find_all(label="arena_150", matching=0.8, waiting_time=1000, grayscale=False)
        elements_list = list(elements)
        print(elements_list)
        time.sleep(3)

Open arena.png (first image) in scale 100% and leave this image in the foreground

Create a "sources" folder an put "arena_150.png" (2nd image) inside

Run code - it will produce a list in the order in which the elements are found on the screen

after the .exe the list becomes random

[rokm]

Hmm, I cannot reproduce this. On my test Windows system, the unfrozen and frozen variant return:

>python program.py
Above not found - Arena Attack 150
[Box(left=1536, top=739, width=52, height=29), Box(left=1191, top=739, width=52, height=29), Box(left=1709, top=739, width=52, height=29), Box(left=1364, top=739, width=52, height=29), Box(left=1363, top=739, width=52, height=29), Box(left=1622, top=640, width=52, height=29)]

and

dist>program\program.exe
Above not found - Arena Attack 150
[Box(left=1536, top=739, width=52, height=29), Box(left=1191, top=739, width=52, height=29), Box(left=1709, top=739, width=52, height=29), Box(left=1364, top=739, width=52, height=29), Box(left=1363, top=739, width=52, height=29), Box(left=1622, top=640, width=52, height=29)]

respectively.

---

The sort order seems to be determined by the template-match score:
https://github.com/botcity-dev/botcity-framework-core-python/blob/86c9cc994d3b4304191d7396cacede466eea28fc/botcity/core/cv2find.py#L141

Can you try inserting the following print in your local copy of botcity/core/cv2find.py:

[...]
    # Order results before sending back
    ordered = sorted(zip(matchx, matchy), key=lambda p: result[p[1]][p[0]], reverse=True)
    for x, y in ordered:
        print(f"Box: x={x}, y={y}, w={needle_width}, h={needle_height}, score={result[y][x]}")  # Added
        yield Box(x, y, needle_width, needle_height)

Then rebuild the frozen application and run both unfrozen script and frozen application again.

This should print the box info together with their scores; can you check if the scores are identical in the frozen and unfrozen version? In my case, they are:

>python program.py
Box: x=1536, y=739, w=52, h=29, score=0.9999576210975647
Above not found - Arena Attack 150
Box: x=1536, y=739, w=52, h=29, score=0.9999576210975647
Box: x=1191, y=739, w=52, h=29, score=0.967707633972168
Box: x=1709, y=739, w=52, h=29, score=0.9400887489318848
Box: x=1364, y=739, w=52, h=29, score=0.8545419573783875
Box: x=1363, y=739, w=52, h=29, score=0.8460010886192322
Box: x=1622, y=640, w=52, h=29, score=0.82472825050354
[Box(left=1536, top=739, width=52, height=29), Box(left=1191, top=739, width=52, height=29), Box(left=1709, top=739, width=52, height=29), Box(left=1364, top=739, width=52, height=29), Box(left=1363, top=739, width=52, height=29), Box(left=1622, top=640, width=52, height=29)]

dist>program\program.exe
Box: x=1536, y=739, w=52, h=29, score=0.9999576210975647
Above not found - Arena Attack 150
Box: x=1536, y=739, w=52, h=29, score=0.9999576210975647
Box: x=1191, y=739, w=52, h=29, score=0.967707633972168
Box: x=1709, y=739, w=52, h=29, score=0.9400887489318848
Box: x=1364, y=739, w=52, h=29, score=0.8545419573783875
Box: x=1363, y=739, w=52, h=29, score=0.8460010886192322
Box: x=1622, y=640, w=52, h=29, score=0.82472825050354
[Box(left=1536, top=739, width=52, height=29), Box(left=1191, top=739, width=52, height=29), Box(left=1709, top=739, width=52, height=29), Box(left=1364, top=739, width=52, height=29), Box(left=1363, top=739, width=52, height=29), Box(left=1622, top=640, width=52, height=29)]

[dimitriusssBR]

Hmm, I cannot reproduce this. On my test Windows system, the unfrozen and frozen variant return:

>python program.py
Above not found - Arena Attack 150
[Box(left=1536, top=739, width=52, height=29), Box(left=1191, top=739, width=52, height=29), Box(left=1709, top=739, width=52, height=29), Box(left=1364, top=739, width=52, height=29), Box(left=1363, top=739, width=52, height=29), Box(left=1622, top=640, width=52, height=29)]

and

dist>program\program.exe
Above not found - Arena Attack 150
[Box(left=1536, top=739, width=52, height=29), Box(left=1191, top=739, width=52, height=29), Box(left=1709, top=739, width=52, height=29), Box(left=1364, top=739, width=52, height=29), Box(left=1363, top=739, width=52, height=29), Box(left=1622, top=640, width=52, height=29)]

respectively.

The sort order seems to be determined by the template-match score: https://github.com/botcity-dev/botcity-framework-core-python/blob/86c9cc994d3b4304191d7396cacede466eea28fc/botcity/core/cv2find.py#L141

Can you try inserting the following print in your local copy of botcity/core/cv2find.py:

[...]
    # Order results before sending back
    ordered = sorted(zip(matchx, matchy), key=lambda p: result[p[1]][p[0]], reverse=True)
    for x, y in ordered:
        print(f"Box: x={x}, y={y}, w={needle_width}, h={needle_height}, score={result[y][x]}")  # Added
        yield Box(x, y, needle_width, needle_height)

Then rebuild the frozen application and run both unfrozen script and frozen application again.

This should print the box info together with their scores; can you check if the scores are identical in the frozen and unfrozen version? In my case, they are:

>python program.py
Box: x=1536, y=739, w=52, h=29, score=0.9999576210975647
Above not found - Arena Attack 150
Box: x=1536, y=739, w=52, h=29, score=0.9999576210975647
Box: x=1191, y=739, w=52, h=29, score=0.967707633972168
Box: x=1709, y=739, w=52, h=29, score=0.9400887489318848
Box: x=1364, y=739, w=52, h=29, score=0.8545419573783875
Box: x=1363, y=739, w=52, h=29, score=0.8460010886192322
Box: x=1622, y=640, w=52, h=29, score=0.82472825050354
[Box(left=1536, top=739, width=52, height=29), Box(left=1191, top=739, width=52, height=29), Box(left=1709, top=739, width=52, height=29), Box(left=1364, top=739, width=52, height=29), Box(left=1363, top=739, width=52, height=29), Box(left=1622, top=640, width=52, height=29)]

dist>program\program.exe
Box: x=1536, y=739, w=52, h=29, score=0.9999576210975647
Above not found - Arena Attack 150
Box: x=1536, y=739, w=52, h=29, score=0.9999576210975647
Box: x=1191, y=739, w=52, h=29, score=0.967707633972168
Box: x=1709, y=739, w=52, h=29, score=0.9400887489318848
Box: x=1364, y=739, w=52, h=29, score=0.8545419573783875
Box: x=1363, y=739, w=52, h=29, score=0.8460010886192322
Box: x=1622, y=640, w=52, h=29, score=0.82472825050354
[Box(left=1536, top=739, width=52, height=29), Box(left=1191, top=739, width=52, height=29), Box(left=1709, top=739, width=52, height=29), Box(left=1364, top=739, width=52, height=29), Box(left=1363, top=739, width=52, height=29), Box(left=1622, top=640, width=52, height=29)]

Tkx for u assistante:

It founds 6 objects (there are only 5, its "double-checking" one of them "+150 in arena.png")

this "duplicate" object could be the cause of the order shuffle

.py version:

Box: x=1370, y=766, w=52, h=29, score=0.9999566078186035
Above not found - Arena Attack 150
Box: x=1370, y=766, w=52, h=29, score=0.9999566078186035
Box: x=1025, y=766, w=52, h=29, score=0.967707633972168
Box: x=1543, y=766, w=52, h=29, score=0.9400887489318848
Box: x=1198, y=766, w=52, h=29, score=0.8545430302619934
Box: x=1197, y=766, w=52, h=29, score=0.846000075340271
Box: x=1456, y=667, w=52, h=29, score=0.8247270584106445
[Box(left=1370, top=766, width=52, height=29), Box(left=1025, top=766, width=52, height=29), Box(left=1543, top=766, width=52, height=29), Box(left=1198, top=766, width=52, height=29), Box(left=1197, top=766, width=52, height=29), Box(left=1456, top=667, width=52, height=29)]

[rokm]

Yeah, I can see the same duplicate in my output, but it happens in both .py and .exe version.

[dimitriusssBR]

On my .exe the changes made to botcity/core/cv2 find.py were not printed

[dimitriusssBR]

Yeah, I can see the same duplicate in my output, but it happens in both .py and .exe version.

you got it right, in .py objects are shown in the order they appear on the screen, after .exe they are shown by best score

[rokm]

On my .exe the changes made to botcity/core/cv2 find.py were not printed

Have you rebuilt the program (maybe with added --clean option to force full rebuild)?

Yeah, I can see the same duplicate in my output, but it happens in both .py and .exe version.
you got it right, in .py objects are shown in the order they appear on the screen, after .exe they are shown by best score

Hmm, but in your ".py version" output above, they were sorted by score, were they not?

[dimitriusssBR]

On my .exe the changes made to botcity/core/cv2 find.py were not printed

Have you rebuilt the program (maybe with added --clean option to force full rebuild)?

Yeah, I can see the same duplicate in my output, but it happens in both .py and .exe version.
you got it right, in .py objects are shown in the order they appear on the screen, after .exe they are shown by best score

Hmm, but in your ".py version" output above, they were sorted by score, were they not?

Yeh, u right. It was changed on botcity-core v0.4.0. Isnt a pyinstaller problem.

I will search for a solution to list the objets in the order they appear, not by score

[dimitriusssBR]

On my .exe the changes made to botcity/core/cv2 find.py were not printed

Have you rebuilt the program (maybe with added --clean option to force full rebuild)?

Yeah, I can see the same duplicate in my output, but it happens in both .py and .exe version.
you got it right, in .py objects are shown in the order they appear on the screen, after .exe they are shown by best score

Hmm, but in your ".py version" output above, they were sorted by score, were they not?

the error is the divergence of versions between the .py and the generated .exe. When running the .py it uses version 0.3.0 of botcity core which lists the objects in the order they appear on the screen, when I generate the .exe it is using version 0.4.0 which lists the objects by score.

That's why the changes I make in cv2 don't appear in the .exe (print(f"Box: x={x}, y={y}, w={needle_width}, h={needle_height}, score={result[y][x]}") # Added)

[rokm]

So in other words, you are using a different python environment when running PyInstaller compared to when you are running/developing your .py program, and those two environments have different versions of botcity-core installed.

[dimitriusssBR]

So in other words, you are using a different python environment when running PyInstaller compared to when you are running/developing your .py program, and those two environments have different versions of botcity-core installed.

I'm going to study the environments to try to correct this divergence between .py and .exe