Bug: reddit engine

[edisonzf2020]

Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG
Repository: https://github.com/searxng/searxng
Branch: master
Version: 2024.4.29+e45a7cc06

How did you install SearXNG?

What happened?

How To Reproduce

Expected behavior

Screenshots & Logs

Additional context

Technical report

[glanham-jr]

Hi @edisonzf2020! Thanks for the bug report. With this, are you able to provide any details on what error you are seeing, or some screenshots?

[andypiper]

I am seeing this from the reddit engine (and I suspect but cannot confirm that this is what folks have also been raising in this issue and in #3112)

Error: searx.exceptions.SearxEngineAccessDeniedException 
Parameters: ('HTTP error 403',) 
File name: searx/search/processors/online.py:116 
Error Function: _send_http_request 
Code: response = req(params['url'], **request_args)

[JeffAlyanak]

@andypiper Have you tested this using a different IP or outbound proxy to confirm whether this is just your IP being blocked by Reddit?

[JeffAlyanak]

@glanham-jr

Just tested this, and it appears that Reddit returns a 403 for requests missing a desktop browser User-Agent.

I tested by adding Mozilla/5.0 (X11; Linux x86_64; rv:124.0) Gecko/20100101 Firefox/124.0 and changing to another IP and that seems to have worked.

@andypiper Do you have the ability to test from another IP? If so, try adding the following to line 33 of the reddit engine:

searxng/searx/engines/reddit.py

Line 33 in 0fb3f0e

    params['headers']['User-Agent'] = "Mozilla/5.0 (X11; Linux x86_64; rv:124.0) Gecko/20100101 Firefox/124.0"

Should look something like this:

def request(query, params):
    query = urlencode({'q': query, 'limit': page_size})
    params['url'] = search_url.format(query=query)
    params['headers']['User-Agent'] = "Mozilla/5.0 (X11; Linux x86_64; rv:124.0) Gecko/20100101 Firefox/124.0"
    return params

[glanham-jr]

I have a branch which you can test the changes. If the changes look good, the PR should be ready to go.

#3556

[glanham-jr]

@JeffAlyanak so based on @return42 message in my PR, there should already be a User-Agent set. Does the default User Agent not work for you? Or specifically, is it possible to provide the headers on the error request? Do we think it's really an IP issue?

Also note that Reddit blocks VPNs to their ability, so if searxng is running in a VPN there may be a chance that could cause issues.