wrong period starts for 45m candles in PAPERTRADE mode

[movy]

Saleh, first of all, I hope you're well during these trying times.
I found some weird behaviour, upon investigation it looks like a bug in Jesse's candle generation.

Describe the bug
The assumption with any trading system is that candles[-1] is either closed previous or the current candle. With Jesse however, this is not guaranteed, as during papertrade mode quite often the most recent 45m candle(s) contain erratic open time and OCHL values which leads to wrong orders. This is never a problem in backtesting mode as candles[-1] is always previous candle there. All this leads to wrong assumptions how backtested strategy will work in papertrade/live mode and to wrong trades being issued.

I log previous candle, as my indicator depends on its values. Sometime ago I noticed that the most recent candle can contain open timestamp equal to current time, i.e. 0 second candle basically. This may happen when Jesse executes the strategy after a new candle start, i.e. 1 second later for example. As a quick crutch, I added a check for such candles in my indicator to chomp them:

    if time - candles[-1, 0] < 60 * 1000:
        candles = candles[:-1]

Recently I noticed that some trades still open at wrong times (which usually leads to a loss) and these trades are not present in forward-test for the same time period.

I added current_time - candle_open_time difference calculation to my candle logs and discovered that quite a few candles still have 0 second length or length less than 45m even after chomping:

2023-10-12T18:00:00 ETH-USDT  prev_candle: [1.697131e+12 1.526480e+03 1.529580e+03 1.530760e+03 1.522040e+03 1.106883e+05], open-time diff: 2700000.0
2023-10-12T18:45:01 ETH-USDT  prev_candle: [1.697136e+12 1.533550e+03 1.533450e+03 1.533550e+03 1.533450e+03 1.447000e+00], open-time diff: 0.0
2023-10-12T19:30:01 ETH-USDT  prev_candle: [1.697139e+12 1.529920e+03 1.529910e+03 1.529920e+03 1.529910e+03 1.900000e-01], open-time diff: 0.0
2023-10-12T20:15:01 ETH-USDT  prev_candle: [1.697139e+12 1.529920e+03 1.532560e+03 1.534160e+03 1.529540e+03 4.164413e+04], open-time diff: 2700000.0
2023-10-12T21:00:01 ETH-USDT  prev_candle: [1.697142e+12 1.532570e+03 1.535870e+03 1.535880e+03 1.530600e+03 3.195940e+04], open-time diff: 2700000.0
2023-10-12T21:45:00 ETH-USDT  prev_candle: [1.697144e+12 1.535880e+03 1.535930e+03 1.537900e+03 1.533000e+03 2.991551e+04], open-time diff: 2700000.0
2023-10-12T22:30:00 ETH-USDT  prev_candle: [1.697147e+12 1.535920e+03 1.537560e+03 1.539770e+03 1.534520e+03 3.097970e+04], open-time diff: 2700000.0
2023-10-12T23:15:01 ETH-USDT  prev_candle: [1.697150e+12 1.537560e+03 1.537130e+03 1.539430e+03 1.536080e+03 2.388679e+04], open-time diff: 2700000.0
2023-10-13T00:00:01 ETH-USDT  prev_candle: [1.697152e+12 1.537140e+03 1.538810e+03 1.539490e+03 1.534480e+03 2.317264e+04], open-time diff: 2700000.0
2023-10-13T00:45:01 ETH-USDT  prev_candle: [1.697155e+12 1.538820e+03 1.542990e+03 1.545800e+03 1.537460e+03 8.129657e+04], open-time diff: 2700000.0
2023-10-13T01:30:00 ETH-USDT  prev_candle: [1.697158e+12 1.543000e+03 1.541700e+03 1.546960e+03 1.541140e+03 4.012142e+04], open-time diff: 2700000.0
2023-10-13T02:15:00 ETH-USDT  prev_candle: [1.697161e+12 1.541690e+03 1.542330e+03 1.544060e+03 1.541260e+03 2.049869e+04], open-time diff: 2700000.0
2023-10-13T03:00:01 ETH-USDT  prev_candle: [1.697166e+12 1.541930e+03 1.542120e+03 1.542120e+03 1.541930e+03 1.775000e+01], open-time diff: 0.0
2023-10-13T03:45:01 ETH-USDT  prev_candle: [1.697169e+12 1.538340e+03 1.538210e+03 1.538340e+03 1.538210e+03 3.851800e+01], open-time diff: 0.0
2023-10-13T04:30:00 ETH-USDT  prev_candle: [1.697169e+12 1.538340e+03 1.539730e+03 1.541140e+03 1.537270e+03 2.745193e+04], open-time diff: 2700000.0

(note open-time diff values above)

Another example with prev candle open time = 7 minute delta from current:

2023-10-13T12:15:00 LRC-USDT prev_candle: [1.697174e+12 1.669000e-01 1.670000e-01 1.671000e-01 1.669000e-01 1.428210e+05], open-time diff: 420000.0

Upon further investigation it turned out that in Postgres 45m candles are sometimes formed at random periods:

select to_timestamp(timestamp / 1000), symbol, timeframe, open, close, high, low, volume
from candle
where symbol = 'LRC-USDT'
order by timestamp desc
limit 1000

Output:

Please note how around 00:53 candles began to start their period at wrong times. For the record, this Jesse instance was restarted at 00:35, however previous restart was at 23:53, not sure whether it bears any significance or just a coincidence.

So, I'm at loss here. Looks like there is a bug with the 45m period start calculation, and nothing can be done on client side to fix it. I will look into Jesse's code later today, but maybe you have an idea where to look for this error?

Enviroment (please complete the following information):

• Ubuntu 22.04, python 3.9, Jesse 0.42

[movy]

I have updated the issue with some extra information (in case you come here from email notifications)

[movy]

Just double checked all my instances, and the problem is confirmed to be only with 45m candles.

[saleh-mir]

I appreciate the detailed explanation that you provided.

If the timestamp of the candles is as you see in the pictures, then yes there is a bug with 45m candles.

The assumption with any trading system is that candles[-1] is either closed previous or the current candle. With Jesse however, this is not guaranteed, as during papertrade mode quite often the most recent 45m candle(s) contain erratic open time and OCHL values which leads to wrong orders.

I'd like to clarify that in Jesse my assumption is that we calculate candles the same way it is on trading view. Which is at the closing time yes that is how you expected. But if you try to make a trade or some calculation during the candle's timestamp, then [-1] returns and open candle. And yes this might make it a little different between the results that you get in a backtest or Live trade. But so far that hasn't been a problem for me. Because let's say once again is past the coast time. Still the closing price of the candle you are getting must be very similar to the previous one. Because it's been just one second how much can the price change in that one second?

[movy]

The current/closing price won't be different of course, but in case I need to see the last candle's color for example, or whether it was a pump/dump candle, my strategy will have a wrong impression if candles[-1] is only 0 second long.

I checked the code, and I believe the 45m bug is inside the binary live trading plugin, so I cannot reach it, but this 0-second candles stuff I should be able to patch client-side. Strange thing is that I trim candles[-1] in my indicator code, but seems like the last 0-second candle is still there. Running more experiments today.

[movy]

So, I managed to get rid of all 0-second candles inside my indicator using

    candles = candles[time - candles[:, 0] > 60 * 1000]

Apparently, at times there're more than one 0-sec candle in the array, as in logs I can see sometimes 2 or even 3 candles are dropped. Not sure whether it's related to the main issue, but worth documenting or creating a helper function I think.

[saleh-mir]

This is useful. Thank you for explaining the use cases. Can you please submit a PR to add this as a utility function?

By the way, in case you are curious the code for handling candles is inside the main framework:
https://github.com/jesse-ai/jesse/blob/master/jesse/store/state_candles.py

The plug in merrily handles receiving the candles and not how we saw them.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[movy]

Hey,
Sorry I wasn't able to look at this one yet, as I temporary walked away from live trading back to drawing board so to speak. Was it addressed in 0.45? If not, I'll try to have a look next week.

[saleh-mir]

Hey man. No problem. You've already contributed so much. No it wasn't addressed

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.