How can I find similar pattern between latest 30 data points of one time series and all the historical data of another time series?

[rkisdp]

Hi @seanlaw,

I have two different time series.

Let's say:

first_data = np.random.rand(100)
second_data = np.random.rand(100)

I want to find similar patterns between the latest 30 data points of first_data which is like(first_data.tail(30)) and within all the datapoint points of the second_data time series.

in other words, I want the best contiguous 30 data points in the second_data time-series which have a similar pattern to the latest 30 data points of the first_data time series and I also want to know which 30 data points are showing a similar pattern to the latest 30 data points of the first_data time series in the graph.

Can you please help me? I am unable to get my desired results so far by just reading the documentation.

[seanlaw]

@rkisdp Thank you for your question and welcome to the STUMPY community.

I am unable to get my desired results so far by just reading the documentation.

In the future, it would be beneficial for you to share the code that you've tried as you might be very close. In the meantime, if you look at the fast pattern matching tutorial, you'll see that finding closest matches can be accomplished by using the mass function:

import stumpy
import numpy as np

first_data = np.random.rand(100)
second_data = np.random.rand(100)

m = 30
Q = first_data[-m:]  # Note that NumPy arrays do not have a `tail` function associated with it. You may be thinking of Pandas
T = second_data

distance_profile = stumpy.mass(Q, T) 
idx = np.argmin(distance_profile)  # Find the index location (within `T`) that best matches your query subsequence `Q`
print(idx, T[idx : idx + m])  # This will print the index location AND the actual best match subsequence

I hope that helps.

[rkisdp]

Thank you so so so much for your reply @seanlaw,

Only you in this world can save my life.

I just tried your code, but not getting the desired result.

In this image, the similarity of the pattern is shown at the tail of the second graph, but I think it should be somewhere
in between the older data of Graph 2.

Can you please please please help me?

[seanlaw]

@rkisdp Are you able to provide the two datasets here (drag and drop them into the comment text box) and also provide the code that you used to analyze and generate the plots? Otherwise, it is impossible for us to assist you without more information.

[rkisdp]

Thank you @seanlaw,

You can find the code along with the dataset from this pastebin link.

https://pastebin.com/5Bup2auH

[NimaSarajpoor]

@seanlaw Thanks for the input!That was helpful 👍

@rkisdp
I might be wrong but the y value in the two time series are completely in different orders of magnitude. And when you do not normalize each subsequence, your distance between subsequences become very large, and ... (I think you should see where I am going with this...)

If you do not want to normalize each subsequence, then you may want to normalize the whole time series. Normalize the whole time series for both data and that might help (you as the user should decide whether that makes sense to normalize the whole time series or not. If it does not, then ignore my note)

Suggestion:
You may want to think what exactly you want to accomplish and then see your data and try to understand what a method tries to do for you, and THEN use that method.

[rkisdp]

I got your point this time.

I normalized my dataset, but still got the same first result.

https://pastebin.com/6t77aBjE

[seanlaw]

The first thing I noticed is that your code did not select the last 30 datapoints (but you may have executed with it):

distance_profile = stumpy.mass(Q[-m:], T, normalize=False) 

I think it should be somewhere in between the older data of Graph 2.

What evidence do you have for this? At the end of the day, stumpy.mass (with normalize=False) simply computes the Euclidean distance between your query subsequence in first_data[-m:] and every subsequence in second_data. There is no magic here. If you have concrete evidence that a particular subsequence in second_data is a close match to your query then you can compute the Euclidean distance by hand and convince yourself that it is NOT smaller than the subsequence located at index 71 (i.e., second_data[71 : 71 + m]), which has a distance of 633888790358.7148 from your query. Note that this value is large because you have chosen to set normalize=False. In other words, you care about the amplitudes of your subsequences whereas, with normalize=True, you only care about the shapes of your subsequences.

To be clear, as much as you "want" for there to be a match in a specific region, using STUMPY will not give you that (unless you limit the search to that region). stumpy.mass gives you exactly what you requested: "The Euclidean distance between the query and an input time series. Nothing more, nothing less."

[rkisdp]

@seanlaw @NimaSarajpoor Thank you so so much, you helped me a lot.

I just found it.

You can see here even after normalization and other things it is still finding the similarity of pattern in the wrong area of the second graph.

[NimaSarajpoor]

While sharing an image can be helpful in some cases, it does not give us information about what you are doing on your end. You are trying to see why the code produces wrong result. So, it makes sense that the code should be provided here.

So, if I were you, I would:

(1) write data into csv files and do drag&drop here

(2) provide us with MINIMUM reproducible example (you may want to Google it to understand it and try your best to do it). Then copy your code here. You can enclose your code with three backticks when you want to write it here.

[seanlaw]

You can see here even after normalization and other things it is still finding the similarity of pattern in the wrong area of the second graph.

@rkisdp You continue to say this but have yet to provide any direct evidence to support this claim. Please answer these questions:

1. What index provides the "correct" and most similar pattern to your query subsequence?
2. What is the Euclidean distance of this matching subsequence to your query subsequence?
3. Is that distance larger or smaller than the closest match returned by stumpy.mass

Please show all of the code that you've written to answer these questions.