Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[QUESTION] Why does asof_join line take long to spin up? #11

Open
dxtrous opened this issue Mar 14, 2024 · 0 comments
Open

[QUESTION] Why does asof_join line take long to spin up? #11

dxtrous opened this issue Mar 14, 2024 · 0 comments
Assignees
Labels
question Further information is requested

Comments

@dxtrous
Copy link
Member

dxtrous commented Mar 14, 2024

I noticed that when running in Colab building up a compute graph for an asof_join takes several seconds, regardless of table size.

In the example below, taken from API documentation, it takes 4s, then if you duplicate the line in the cell, it takes 8s, etc.

Why is this the case? Does this only happen in interactive mode?

image

# -*- coding: utf-8 -*-
"""Colab_test_asof_join.ipynb

Automatically generated by Colaboratory.

Original file is located at
    https://colab.research.google.com/drive/1pWox7vvDoSuohRZ1EpWR2QDGSRSOmgsV
"""

!pip install pathway

import pathway as pw

t1 = pw.debug.table_from_markdown(
    '''
    value | event_time | __time__
      2   |      2     |     4
      3   |      5     |     6
      4   |      1     |     8
      5   |      7     |    14
'''
)
t2 = pw.debug.table_from_markdown(
    '''
    value | event_time | __time__
      42  |      1     |     2
       8  |      4     |    10
'''
)

result_join = t1.join(t2, t1.event_time ==t2.event_time, how=pw.JoinMode.LEFT).select(event_time = t1.event_time)

result_asof_join = t1.asof_join(
    t2, t1.event_time, t2.event_time, how=pw.JoinMode.LEFT
).select(
    left_value=t1.value,
    right_value=t2.value,
    left_time=t1.event_time,
    right_time=t2.event_time,
)```
@dxtrous dxtrous added the question Further information is requested label Mar 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants