Support premium search APIs

[ddefranza]

With recent API updates, both basic and premium accounts have access to full-history search as long as a development environment is created and specified by the API account holder. Will tweepy be able to access this feature sometime in the near future?

[Dmium]

I'm not a contributor yet but after a glance at the code it looks like a fairly simple change to make it possible and a bit more to make it easy. I might make a PR for this if nobody else is.

Interestingly almost all of the code needed to do it appears to already be there

[Harmon758]

For reference:
https://developer.twitter.com/en/docs/tweets/search/overview/premium
https://twittercommunity.com/t/full-archive-search-now-available-as-a-premium-api/100718
https://blog.twitter.com/developer/en_us/topics/tools/2018/access-the-full-history-of-tweets.html

[drandreaskrueger]

Oh, so there is an issue about this already.
Yes, it would be great to use the Premium search from tweepy.

I had started to just modify some existing code but got stuck at a status_code=422 error, look #1293

[Harmon758]

I've created a branch (https://github.com/tweepy/tweepy/tree/premium-search) and draft PR (#1294) that adds API methods for the data endpoints of both premium search APIs.
Cursor support with those methods is still pending. See #1294.

[drandreaskrueger]

Coooool, nice one, thanks!!

So how would I use this? I assume like that?

source env/bin/activate
pip uninstall tweepy
git clone https://github.com/tweepy/tweepy.git
cd tweepy
git checkout premium-search
python setup.py install

right? When I use python setup.py install it will end up in my virtualenv, as it would with pip, right? And what is the way to fully get rid of it again, if I want to get back to the current stable version 3.8.0 ?

Thanks!

[Harmon758]

You can simply pip install --upgrade git+https://github.com/tweepy/tweepy@premium-search.
See https://pip.pypa.io/en/stable/reference/pip_install/#git.
pip install --force-reinstall --no-dependencies tweepy will install the latest version from PyPI. You can remove the --no-dependencies option if you want to reinstall dependencies as well.
See https://pip.pypa.io/en/stable/reference/pip_install/#options.

[drandreaskrueger]

cool, thanks.

[EliteMasterEric]

@Harmon758 Thank you for developing a pull request for this. The cursor changes required to continue development have been merged, and I would be happy to see continued progress.

[Harmon758]

I've added cursor support, and both methods should now be fully functional.
Feel free to try out and test them in the branch and PR.

[smoufarrege]

Found what appear to be some bugs and missing features when using the search_30_day function in conjunction with paging.

If you try to use a plus sign (+) in the query (i.e. for boolean searches), it crashes with the message "There were errors processing your request: no viable alternative at character '+'" -- I'm guessing a minus sign perhaps certain other characters would cause the same issue.

Also, my original algorithm used tweet_mode='extended' as one of the parameters. As per Twitter, this allows for a full 280 character tweet to be returned, rather than truncating it to the original 140 character limit. It looks like this isn't supported by the new function.

Here's the line in question:

for page in tweepy.Cursor(API.search_30_day, query=query, maxResults=per_page, environment_name="AzureDevelopment").pages(page_count):

And here's a sample query:

bitcoin +btc

[Harmon758]

I'm not sure what you mean by "for boolean searches", as + isn't a valid operator for that (or any other, as far as I'm aware) API/endpoint.
See https://developer.twitter.com/en/docs/tweets/search/guides/premium-operators.
At any rate, this is an issue with Twitter's processing of the character (seemingly stemming from Salesforce/Apex) rather than with the Tweepy branch/PR.

As for extended Tweets, premium APIs are different from standard REST APIs and include an additional extended_tweet field in the payload, represented as an attribute for the corresponding Status object, exactly the same as Status objects from streams.
See https://developer.twitter.com/en/docs/tweets/tweet-updates and https://tweepy.readthedocs.io/en/latest/extended_tweets.html#streaming.

[eunbee31]

Hello, I would like to know what means stale on premium-search branch. Does it mean that the branch is not updated at all? That you will remove it eventually?
Thanks in advance for your reply. Also, good work you have done :)

[Harmon758]

I'm not sure where you're seeing that.
The branch is not stale and should be fully functional. I still plan to release it with Tweepy v3.9.
Once the PR is merged into the master branch, that branch will be deleted.

[eunbee31]

Thanks for your reply. I.m glad to hear that.
I saw it here: https://github.com/tweepy/tweepy/branches

[Harmon758]

The Stale view on that page is simply GitHub showing all branches that no one has committed to in the last three months.
See https://docs.github.com/en/github/administering-a-repository/viewing-branches-in-your-repository.

[jikamens]

Did this make it into 3.9.0? I don't see it listed in the release notes.

[Harmon758]

v3.9.0 was released without #1294 having been merged yet. I had intended to release it with v3.9.0.
I've since fixed and improved the changelog and release notes for v3.9.0.
Since v3.9.0 was also released without originally noting that it would be the last version to support Python 2 (#1253) in its changelog and release notes, a v3.10.0 is now planned for that and this will be released as part of that version.
I've updated the milestones accordingly, and I'll be merging #1294 into master and deleting the premium-search branch shortly.

[stabilus]

just came across this and don't get how to include premium api support, since currently:

• pip install --upgrade git+https://github.com/tweepy/tweepy@premium-search is not working (as branch is not available)
• Add API.search_30_day and API.search_full_archive #1294 is not included in v3.9.0 yet
• v3.10. is not available yet

Any directions on how to proceed?

[Harmon758]

The branch has been deleted since #1294 has been merged into the master branch.
Until v3.10.0 is released, you can install from the master branch.

[stabilus]

The branch has been deleted since #1294 has been merged into the master branch.
Until v3.10.0 is released, you can install from the master branch.

ok. pip install --upgrade git+https://github.com/tweepy/tweepy@master successfully installed.
But how do you now access premium search? In twitter dev, I created a new Dev environment and I have the corresponding label. How do you submit it to the API? How do you call premium search?

[Harmon758]

See the documentation for API.search_30_day and API.search_full_archive and the corresponding documentation for the premium search APIs.

[msklc]

Don't forget to update tweepy with pip install --upgrade git+https://github.com/tweepy/tweepy@master. Then you can use the API with premium feature...

[amillb]

It seems that the premium search is still missing from the standard pip and conda installs with 3.10.0. The pip install --upgrade git+https://github.com/tweepy/tweepy@master method still works, but is there a reason that the Client class is missing from the version on pip?

[Harmon758]

The premium search APIs were implemented with API.search_30_day and API.search_full_archive and released as part of v3.10.0.
Client is used to interface with Twitter API v2, not the premium search APIs, and is expected to be released with v4.0.