You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The expensive operation before starting the viz server is loading the data from the Kedro session (possibly related to Lazy Loading of Catalog Items kedro#2829 ?)
Most of the time taken to load the data is from catalog and pipelines_dict resolution, which worsens as the pipeline count increases
Originally posted by AhdraMeraliQB January 6, 2024
Description
There are several features across the Kedro organisation that could benefit from manual testing on large projects to evaluate performance. The proposal is to create several kedro projects of varying size that can be used to test and experiment with.
The Kedro-Viz team carried out a performance analysis using an internal QB pipeline, with preliminary results shown here kedro-org/kedro-viz#1064
(from kedro-org/kedro-viz#1064 (comment), summary of internal report).
There is preliminary evidence that the Kedro Framework CLI is a bottleneck for Kedro Viz.
This is on top of the already existing evidence that Kedro takes a lot of time to load even for trivial commands or almost empty projects #1476
We noted that there are several factors that make a pipeline "complex":
In I expanded on @AhdraMeraliQB's original proposal and suggested that we create a family of pipelines, comprising
Comes from #3790
Originally posted by AhdraMeraliQB January 6, 2024
Description
There are several features across the Kedro organisation that could benefit from manual testing on large projects to evaluate performance. The proposal is to create several kedro projects of varying size that can be used to test and experiment with.
This could be particularly useful for testing Viz features, CC @rashidakanchwala, @NeroOkwa
...
The text was updated successfully, but these errors were encountered: