Query params lost when retrying a request on token expired?

[ndlarge]

Hi all,
First of all, huge thanks to @jasonacox and @uzlonewolf for this wonderful piece of software!

What am I trying to achieve? Transfer voltage, power, energy consumed etc measurements from a Tuya-compliant WIFI power meter into an influxdb database for further visualization and analysis.

How am I going about it? On a Raspberry Pi, every 10 minutes I run a script that first checks for the latest measurement timestamp already in the influxdb, then calls getdevicelog() with appropriate start and end times. As my power meter is rather talkative, 10 minutes worth of logs is about 1000 records, so it takes ~10 "fetches" to get all the data.

What is my problem? Everything works like a charm in general, but every now and then I see 5-10 minute gaps in my measurement sequences. I checked the Tuya cloud online and the missing data is there, so Tuya is not to blame here.
I turned on debugging and tinituya started logging every single request for me.
And here's what I see:

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): openapi.tuyaeu.com:443
DEBUG:urllib3.connectionpool:https://openapi.tuyaeu.com:443 "GET /v1.0/token?grant_type=1 HTTP/1.1" 200 232
DEBUG:tinytuya.core:GET: URL=https://openapi.tuyaeu.com/v1.0/token?grant_type=1 HEADERS={'secret': 'xxx, 'client_id': xxx', 'sign': 'xxx', 't': '1706537103276', 'sign_method': 'HMAC-SHA256', 'mode': 'cors'} response code=200 text={"result":{"access_token":"xxx","expire_time":1,"refresh_token":"xxx","uid":"xxx"},"success":true,"t":1706537103606,"tid":"xxx"} token=None
DEBUG:tinytuya.core:server_time_offset: 0
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): openapi.tuyaeu.com:443
DEBUG:urllib3.connectionpool:https://openapi.tuyaeu.com:443 "GET /v1.0/devices/xxx/logs?end_time=1706537103649&query_type=1&size=100&start_time=1706536496790&type=7 HTTP/1.1" 200 None
DEBUG:tinytuya.core:GET: URL=https://openapi.tuyaeu.com/v1.0/devices/xxx/logs?end_time=1706537103649&query_type=1&size=100&start_time=1706536496790&type=7 HEADERS={'client_id': 'xxx', 'sign': 'xxx', 't': '1706537103651', 'sign_method': 'HMAC-SHA256', 'mode': 'cors', 'access_token': 'xxx'} response code=200 text={"result":{"current_row_key":"NjFmMDhiMDRia3ppYmFhYjFlY2EzNjJmOTE4OWZiXzkyMjMzNzAzMzAzMTc2NzI4MTlfN18xMzI=","device_id":"xxx","has_next":true,"logs":[{"code":"Current","event_from":"1","event_id":7,"event_time":1706537102988,"status":"1","value":"4598"},{"code":"CurrentB","event_from":"1","event_id":7,"event_time":1706537102963,"status":"1","value":"1763"},
...
{"code":"PowerFactorC","event_from":"1","event_id":7,"event_time":1706537037173,"status":"1","value":"58"},{"code":"ActivePower","event_from":"1","event_id":7,"event_time":1706537035214,"status":"1","value":"706"},{"code":"ActivePowerC","event_from":"1","event_id":7,"event_time":1706537035200,"status":"1","value":"187"}],"next_row_key":"NjFmMDhiMDRia3ppYmFhYjFlY2EzNjJmOTE4OWZiXzkyMjMzNzAzMzAzMTc3NDA2MjBfN18xMTM="},"success":true,"t":1706537103971,"tid":"xxx"} token=xxx
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): openapi.tuyaeu.com:443
DEBUG:urllib3.connectionpool:https://openapi.tuyaeu.com:443 "GET /v1.0/devices/xxx/logs?end_time=1706537103649&query_type=1&size=100&start_row_key=NjFmMDhiMDRia3ppYmFhYjFlY2EzNjJmOTE4OWZiXzkyMjMzNzAzMzAzMTc3NDA2MjBfN18xMTM%3D&start_time=1706536496790&type=7 HTTP/1.1" 200 110
DEBUG:tinytuya.core:GET: URL=https://openapi.tuyaeu.com/v1.0/devices/xxx/logs?end_time=1706537103649&query_type=1&size=100&start_row_key=NjFmMDhiMDRia3ppYmFhYjFlY2EzNjJmOTE4OWZiXzkyMjMzNzAzMzAzMTc3NDA2MjBfN18xMTM%3D&start_time=1706536496790&type=7 HEADERS={'client_id': 'xxx', 'sign': 'xxx', 't': '1706537104063', 'sign_method': 'HMAC-SHA256', 'mode': 'cors', 'access_token': 'xxx'} response code=200 text={"code":1010,"msg":"token invalid","success":false,"t":1706537104337,"tid":"xxx"} token=ceb37b5875dfa7b949ddf199142a0278
DEBUG:tinytuya.core:Token Expired - Try to renew
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): openapi.tuyaeu.com:443
DEBUG:urllib3.connectionpool:https://openapi.tuyaeu.com:443 "GET /v1.0/token?grant_type=1 HTTP/1.1" 200 235
DEBUG:tinytuya.core:GET: URL=https://openapi.tuyaeu.com/v1.0/token?grant_type=1 HEADERS={'secret': 'xxx', 'client_id': 'xxx', 'sign': 'xxx', 't': '1706537104378', 'sign_method': 'HMAC-SHA256', 'mode': 'cors'} response code=200 text={"result":{"access_token":"xxx","expire_time":7200,"refresh_token":"xxx","uid":"xxx"},"success":true,"t":1706537104669,"tid":"xxx"} token=None
DEBUG:tinytuya.core:server_time_offset: 0
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): openapi.tuyaeu.com:443
DEBUG:urllib3.connectionpool:https://openapi.tuyaeu.com:443 "GET /v1.0/devices/xxx/logs HTTP/1.1" 200 117
DEBUG:tinytuya.core:GET: URL=https://openapi.tuyaeu.com/v1.0/devices/xxx/logs HEADERS={'client_id': 'xxx', 'sign': 'xxx', 't': '1706537104711', 'sign_method': 'HMAC-SHA256', 'mode': 'cors', 'access_token': 'xxx'} response code=200 text={"code":40000303,"msg":"Parameter error!","success":false,"t":1706537105016,"tid":"xxx"} token=xxx

As you can see, on the first "fetch" a properly formed URL with all the necessary query params is passed and a perfectly formed response is received.
The second perfectly formed fetch with proper start_row_key is rejected with "1010 token invalid" response (why? that is another question).
And then on retry an URL with no query parameters at all is passed (https://openapi.tuyaeu.com/v1.0/devices/xxx/logs) and obviously the response is "40000303 Parameter error!". And as there's no error handling visible, the function simply stops here and returns what it has (one fetch out of ten).

When I look into the code for _tuyaplatform(), here's what I see:

    def _tuyaplatform(self, uri, action='GET', post=None, ver='v1.0', recursive=False, query=None, content_type=None):
 ...
        # Check to see if token is expired
        if "token invalid" in response.text:
            if recursive is True:
                log.debug("Failed 2nd attempt to renew token - Aborting")
                return None
            log.debug("Token Expired - Try to renew")
            self._gettoken()
            if not self.token:
                log.debug("Failed to renew token")
                return None
            else:
                return self._tuyaplatform(uri, action, post, ver, True)

But uri variable is just pure path with no query parameters, you probably need to pass query as well to the recursive call to the function? Like this:

              return self._tuyaplatform(uri, action, post, ver, True, query)

Would appreciate any feedback.

[jasonacox]

Hi @ndlarge Yes, that does look like a bug. We actually want to pass the entire context to the recursion:

              return self._tuyaplatform(uri, action, post, ver, True, query, content_type)

Thank you!

I'll open an issue. Feel free to submit a PR if you would like, so you can get credit. 😁

One concern: That's a lot of API calls to Tuya Cloud. Are you hitting any rate limits or quotas?

[ndlarge]

Thank you @jasonacox . As for the rate limits or quotas - I have no idea frankly. The Tuya documentation on this is extremely unclear but so far (a few days) I had no indication that I was hitting any limits. Could this constantly expiring token be an indication of me hitting some limits? I don't even know how I am going to know if I hit a limit or a quota (besides stuff stopping working suddenly).