Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some lively.morphic/text tests regularly fail in CI #1524

Open
linusha opened this issue May 16, 2024 · 1 comment
Open

Some lively.morphic/text tests regularly fail in CI #1524

linusha opened this issue May 16, 2024 · 1 comment
Assignees
Labels
🔍 cause unknown A problem which needs more investigation/is not yet reproducible 🐞 bug Something isn't working ⏱️ postponed A problem/idea that is known/documented but is not urgent/has previous pathdependencies.

Comments

@linusha
Copy link
Contributor

linusha commented May 16, 2024

Ever since merging #1481 our CI pipelines failed more or less reliably.

We often had trouble reproducing this locally. Although it was not impossible to reproduce it locally, it did work more often than not and inside of a running lively environment the tests never failed.

We found out that disabling the caching inside of the actions, thus enforcing a clean reinstallation of lively reliably seems to solve the issue.
We then tried to move the lively.morphic tests, which are the failing tests, to the start of the pipeline for easier testing. This lead to the tests failing again.

We briefly experimented with setting headless: 'shell' instead of headless: 'new'. However, this would need to be investigated further.

Overall the situation is highly unsatisfactory and we have no real indicator what the exact cause for the problems is. For now, we just hope that the problem does not persist, but we should investigate how we can harden the pipeline. Enabling the caching again would also be quite nice as the runtime of the pipelines skyrockets drastically, which is especially annoying for the PR checks.
In the worst case we could try to force the installation only for the daily checks and for lively.morpic.

@linusha linusha added 🐞 bug Something isn't working ⏱️ postponed A problem/idea that is known/documented but is not urgent/has previous pathdependencies. labels May 16, 2024
@linusha linusha self-assigned this May 16, 2024
@linusha linusha added the 🔍 cause unknown A problem which needs more investigation/is not yet reproducible label May 16, 2024
@linusha linusha changed the title CI Pipeline requires new installation for some lively.morphic/text tests to be green Some lively.morphic/text tests regularly fail in CI May 22, 2024
@linusha
Copy link
Contributor Author

linusha commented May 24, 2024

Upon further investigation, all of the above information seems to be incorrect. We found out that retrying the layout tests reliably fixes the problem. This is highly curious, as we tried to capture the console output of the failing pipeline runs which threw Error rendering morphs, which is usually unrecoverable and all subsequent tests run in the same lively session.

As of now, we will try to reactivate the caches and we think that the culprit is inside of the text layout tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🔍 cause unknown A problem which needs more investigation/is not yet reproducible 🐞 bug Something isn't working ⏱️ postponed A problem/idea that is known/documented but is not urgent/has previous pathdependencies.
Projects
None yet
Development

No branches or pull requests

1 participant