Stumpi pre-calculated mp

[tomweb153]

Hi,

I have a question regarding the use of a pre-calculated mp for the stumpi function. For me, it isn't clear how to use the "mp" parameter.

My use case:

• Calculate mp with stumpy.stump for data (1 week of data) -> initially
• Save all information from mp in database table
• Collect a days worth of data
• Load existing mp from database -> every day
• Update existing (1 week long) mp with stumpi -> egress=True
• Search if anomalies have occured in new data
• Save updated mp (still 1 week long) in database table

My restrictions:

• Tools for fully online mp calculation are not available
• Have to rely on database (as data only arrives there)
• Data can only be retrieved day by day (not as it arrives)

Is it possible to update an existing mp that contains information about 1 week of data with 1 day of new data (trying to prevent having to calculate the whole mp every time new data arrives)?
Your documentation says that the pre-calculated mp has to be the same length as the data one wants to calculate anything for. So what does the mp parameter exactly do, when passing a pre-calculated mp from day 1 and new data from day 2 (seems like it is not possible to update something of different lengths).

Here is a very short implementation demonstrating what I'm trying to do. The vertical line in the first plot marks where updatedata starts. I was expecting the mp to have changed after the stumpi usage.

Thank you for your help.
Best Regards
Tom

[seanlaw]

@tomweb153 Thank you for your question and welcome to the STUMPY community. When you say "Collect a days worth of data", can you please clarify if that is a single data point (representing the whole day) or, say, 24 data points (one data point for each hour of that day)?

[tomweb153]

Hello @seanlaw
Thanks for the warm welcome.
I have a dynamic sensor that collects about 2880 readings per day (2 readings per minute, if there is uninterrupted communication). So when I talk about a day of data I mean 2880 readings.

To clarify my use case further:

• 1 week of data is 20160 data points
• 1 day of data is 2880 data Points
• I want to incrementally update my mp
• I only want to keep 1 week of calculated mps

[seanlaw]

Thank you for the extra information @tomweb153

Following what you showed above, let's say you already have one weeks worth of data, you pre-compute the matrix profile for it:

import numpy as np
import stumpy

past_week_data = np.random.rand(20160)
m = 30
mp = stumpy.stump(past_week_data, m, normalize=False)

Hopefully, there is nothing unusual here but please feel free to ask any questions. Now, let's set up our matrix profile stream:

stream = stumpy.stumpi(past_week_data, m, normalize=False, egress=True)

Our stream has been initialized and now we can pass single/individual datapoints to it to update the matrix profile. Let's say we have some new data:

day_one_data = np.random.rand(2880)  # or retrieve from database

The stream has no concept of 2880 data points at this time. It only understands a SINGLE data point so let's update our matrix profile one data point a time:

for i in range(len(day_one_data):
    stream.update(day_one_data[i])

And (I think) that's it! For the second day, you'd do:

day_two_data = np.random.rand(2880)  # or retrieve from database
for i in range(len(day_two_data):
    stream.update(day_two_data[i])

Certainly, you don't need to call the new data day_one_data and day_two_data since, again, the stream only cares about the next single data point that you feed into it in order to update the matrix profile. Please let me know if that makes sense or if you have any further questions.

In case you missed it , I also recommend looking at this tutorial.

[tomweb153]

@seanlaw
Thank you for your response. I have been following your tutorials on the stumpy webpage for the past few months. They have been incredibly helpful, and I want to express my gratitude once again!

I have already considered a solution similar to what you suggested.
However, I believe I would need to initialize a streaming object with data from one week prior to updating my mp, right? If I understand this correctly, it would involve repetitive calculations because I cannot initiate the streaming object, update it, and then wait for new data, all while keeping the streaming object ready.

The challenge here is that I receive daily updates at a specific time from the database where my sensor readings are stored. Additionally, I have multiple sensors for which I need to perform anomaly detection (although I didn't mention this earlier, I'm dealing with approximately 10,000 sensors feeding data into the database). So I am currently searching for a effective solution to do anomaly detection on my sensor readings.
I was hoping that the option to provide the stumpi function with a mp, would solve my problem.

My ideas are mainly based on the following paper, published in 2023 by Eamonn Keogh and his colleagues: "Matrix Profile XXVIII: Discovering Multi-Dimensional Time Series Anomalies with K of N Anomaly Detection". In this context 5..7 sensors can be seen as a single system.

I had hoped to find an efficient way to calculate the mp for each sensor (or system) over the course of one weeks data, save these calculated mp in my database, and simply update them as new data becomes available. It turns out that one week provides the optimal amount of data required for secure anomaly detection within the given systems.

If you are available over the next few days, would you be open to have a video call to further discuss this? Please let me know if you are available, and I can send you an invitation via Teams or Zoom. I'll ensure the timing aligns with my location in Leipzig, Germany and your location, to minimize any inconveniences.

[seanlaw]

Thank you for your response. I have been following your tutorials on the stumpy webpage for the past few months. They have been incredibly helpful, and I want to express my gratitude once again!

Thank you for your kind words. It means a lot!

However, I believe I would need to initialize a streaming object with data from one week prior to updating my mp, right? If I understand this correctly, it would involve repetitive calculations because I cannot initiate the streaming object, update it, and then wait for new data, all while keeping the streaming object ready.

@tomweb153 I'm not sure I fully understand what problem you are facing so please elaborate. There is nothing stopping you from initiating the stream with np.nan data if the data doesn't already exist:

import numpy as np
import stumpy

past_week_data = np.full(20160, np.nan)
m = 30
mp = stumpy.stump(past_week_data, m, normalize=False)

Now, some folks have mentioned using cloudpickle to pickle this stream object and then storing it either locally (on disk) or into some database (I have not tried this myself and so I cannot comment). And then, when the new data arrives, retrieve the last stream object, load it into Python, retrieve the NEW data, and then update the stream accordingly. Note that since the initial "fake" data is np.nan, those will never produce a valid distance for consideration into the matrix profile.

So, Once the first m=30 points have been used to update the stream, then you will start having actual distances in your matrix profile after the 31st data point. Certainly, you'll need to wait for 20160 data points before you are guaranteed to have had a full weeks of data in your matrix profile.

Is this what you mean? If not, please clarify.

Note that at some point, we are moving beyond STUMPY and matrix profiles and going into architecture and production scalability of your problem, which is (sadly) beyond the scope of the package. I imagine that there may be something you can pull together by putting your 10K streaming objects into some sort of cloud storage (e.g. S3) and then create a cloud function (e.g., Lambda) that accepts a new day's data and location of the corresponding streaming object and updates the stream accordingly. I have little to no experience here :(

If you are available over the next few days, would you be open to have a video call to further discuss this? Please let me know if you are available, and I can send you an invitation via Teams or Zoom. I'll ensure the timing aligns with my location in Leipzig, Germany and your location, to minimize any inconveniences.

Unfortunately, due to my full time job and family, I am unable to offer personal consultations. However, you may continue posting your questions here in case others may have some suggestions too.

[tomweb153]

@seanlaw

Unfortunately, due to my full time job and family, I am unable to offer personal consultations. However, you may continue posting your questions here in case others may have some suggestions too.

I fully understand that!

Thank you for the suggestion to use cloudpickle. I think storing the streaming object may be the key for solving my problem.

I'm not sure I fully understand what problem you are facing so please elaborate.

Sorry, English is not my main language, but I will try to elaborate my problem:

• I have a lot of sensors (5..7 combined are ONE system) that provide me with 2 readings/minute
• I want to check whether some of the systems are experiencing anormal states
• I am very limited on how I can implement an anomaly detection for the sensors
  • no infrastructure for continuous calculations using stumpi (so that I could access the readings as they arrive)
• I thought of using data of one week (e.g. 11. Oct to 18. Oct) for an initial calculation of all the neccessary mps
  • do anomaly detection on the mps (e.g. use the left mp as stated in your tutorial)
  • save the mps into a database table (seems I have to save the streaming object!)
  • wait 1 day, so I can access the data that was collected for the last day (e.g. 19. Oct)
  • get the streaming object out of the database
  • update my streaming object with the newly available data (streaming object with egress=True )
  • check the updated mps for anomalies
  • save updated streaming object back to database
  • -> repeat this process every day (I'd call it "pseudo-streaming-anomaly-detection")

I hope, I could clarify what I'm trying to do.

[seanlaw]

Frankly, at some point, you need to ask yourself if it takes MORE total time to:

1. pickle the stream object
2. upload/download the stream object
3. upload/download the new data
4. load the stream object into memory
5. update the stream and potentially incur a numba warm up cost

than it would take to just re-compute the matrix profile each time.