-
-
Notifications
You must be signed in to change notification settings - Fork 266
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
💡 Feature request: PROXY/SOCKS support #841
Comments
Side note, it may require a semi large rework but we could utilize Arti (GitLab Repo; lib.rs page) instead of a proxy to a localhost TOR server which would require less setup than the latter (a TOR localhost proxy requires you to run TOR in the background while Arti implements all the logic itself, so you don't need anything except for the libreddit program to be running). |
I feel if the instance also hosted TOR guard and relay nodes this shouldn't be an issue, and would actually help other instances/peers on the network. Though there's still the problem of exit nodes... maybe have 2/3 of connections go through the TOR network (1/3 through exit nodes and the other 1/3 through Reddit's hidden service) and the other 1/3 go through clearnet. That way there'd be equal compensation for the use of exit nodes, and adequate compensation for the guard and relay nodes. Though it might be smart to make this as a support layer that libreddit can sit atop instead of something baked directly into the project (as it would facilitate reuse in other privacy respecting frontends). Say a session based connection scrabler, where every session is randomly distributed over TOR, (maybe I2P,) and clearnet, over a predetermined list of servernames ("{old|www}.reddit.com" have their own hidden services "{old|www}.redditto...wfj4ooad.onion" which could also be used for scraping (I'm of course assuming that old.reddit.com and www.reddit.com would have independent rate limits as they're distinct servers but that may be wrong)). Just a thought. |
Hey i've utilized an idea to "educationally-use"-IPV6 #845 imo use an ipv6 per user.... maybe strong server would be needed ... idk |
Time to route different /64 ipv4 addresses - or limit by /48 addresses |
I kind-of dirty patched TOR routing into libreddit with
diff --git a/src/client.rs b/src/client.rs
index 4c174cd..ca3eef6 100644
--- a/src/client.rs
+++ b/src/client.rs
@@ -1,24 +1,20 @@
+use arti_client::*;
+use arti_hyper::*;
use cached::proc_macro::cached;
use futures_lite::{future::Boxed, FutureExt};
-use hyper::client::HttpConnector;
-use hyper::{body, body::Buf, client, header, Body, Client, Method, Request, Response, Uri};
-use hyper_rustls::HttpsConnector;
+use hyper::{body, body::Buf, client, header, Body, Method, Request, Response, Uri};
use libflate::gzip;
-use once_cell::sync::Lazy;
use percent_encoding::{percent_encode, CONTROLS};
use serde_json::Value;
use std::{io, result::Result};
+use tls_api::{TlsConnector as TlsConnectorTrait, TlsConnectorBuilder};
+use tls_api_native_tls::TlsConnector;
use crate::dbg_msg;
use crate::server::RequestExt;
const REDDIT_URL_BASE: &str = "https://www.reddit.com";
-static CLIENT: Lazy<Client<HttpsConnector<HttpConnector>>> = Lazy::new(|| {
- let https = hyper_rustls::HttpsConnectorBuilder::new().with_native_roots().https_only().enable_http1().build();
- client::Client::builder().build(https)
-});
-
/// Gets the canonical path for a resource on Reddit. This is accomplished by
/// making a `HEAD` request to Reddit at the path given in `path`.
///
@@ -75,7 +71,12 @@ async fn stream(url: &str, req: &Request<Body>) -> Result<Response<Body>, String
let uri = url.parse::<Uri>().map_err(|_| "Couldn't parse URL".to_string())?;
// Build the hyper client from the HTTPS connector.
- let client: client::Client<_, hyper::Body> = CLIENT.clone();
+ let client: client::Client<_, hyper::Body> = {
+ let tor_client = TorClient::builder().bootstrap_behavior(BootstrapBehavior::OnDemand).create_unbootstrapped().unwrap();
+ let tls_connector = TlsConnector::builder().unwrap().build().unwrap();
+ let tor_connector = ArtiHttpConnector::new(tor_client, tls_connector);
+ hyper::Client::builder().build(tor_connector)
+ };
let mut builder = Request::get(uri);
@@ -129,7 +130,12 @@ fn request(method: &'static Method, path: String, redirect: bool, quarantine: bo
let url = format!("{}{}", REDDIT_URL_BASE, path);
// Construct the hyper client from the HTTPS connector.
- let client: client::Client<_, hyper::Body> = CLIENT.clone();
+ let client: client::Client<_, hyper::Body> = {
+ let tor_client = TorClient::builder().bootstrap_behavior(BootstrapBehavior::OnDemand).create_unbootstrapped().unwrap();
+ let tls_connector = TlsConnector::builder().unwrap().build().unwrap();
+ let tor_connector = ArtiHttpConnector::new(tor_client, tls_connector);
+ hyper::Client::builder().build(tor_connector)
+ };
// Build request to Reddit. When making a GET, request gzip compression.
// (Reddit doesn't do brotli yet.) Then I just added |
T.: There's https://git.spec.cat/Nyaaori/libreddit also which uses Arti, too. Doesn't seem particularly slow either; we're using't for lr.artemislena.eu currently. |
@artemislena looks great :) thanks for sharing. maybe you could set up a github mirror and open a pull request here so your improvements are available to more people? |
nope. goes nowhere.

… On Sep 15, 2023, at 2:27 PM, Artemis ***@***.***> wrote:
T.: There's https://git.spec.cat/Nyaaori/libreddit also which uses Arti, too. Doesn't seem particularly slow either; we're using't for lr.artemislena.eu currently.
—
Reply to this email directly, view it on GitHub <#841 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APRHFD7MGPG6ZGJ5XFSB7YTX2TB4DANCNFSM6AAAAAA2LEATXE>.
You are receiving this because you are subscribed to this thread.
|

… On Sep 15, 2023, at 2:47 PM, outsider1305 ***@***.***> wrote:
@artemislena <https://github.com/artemislena> looks great :) thanks for sharing. maybe you could set up a github mirror and open a pull request here so your improvements are available to more people?
—
Reply to this email directly, view it on GitHub <#841 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APRHFD7D56CRQN4TIBJOI2TX2TEFNANCNFSM6AAAAAA2LEATXE>.
You are receiving this because you are subscribed to this thread.
|
@avincent98144 ...can you elaborate? i can't really tell what you're trying to say. if you meant to say Tanith's links lead nowhere, that's not true (at least not for me): |
note the first screenshot: the yellow progress bar in the address bar sits where it’s at and never loads further. 3 browsers on 3 devices so it’s REPL or something else, but note the REPL screenshots to confirm it is indeed REPL. Tried over 30 times in 30 minutes.
… On Sep 15, 2023, at 2:54 PM, outsider1305 ***@***.***> wrote:
@avincent98144 <https://github.com/avincent98144> ...can you elaborate? i can't really tell what you're trying to say. if you meant Tanith's links lead nowhere, that's not true (at least not for me):
<https://user-images.githubusercontent.com/64506392/268402026-187b1156-8bff-48df-9bf1-3963dfff17a7.png>
<https://user-images.githubusercontent.com/64506392/268402046-e1112006-1c87-44b9-9ff1-bec0f73bd72c.png>
—
Reply to this email directly, view it on GitHub <#841 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APRHFD73SIOAOXE3JN7LPF3X2TFBPANCNFSM6AAAAAA2LEATXE>.
You are receiving this because you were mentioned.
|
T.: It's not our Forgejo instance, we didn't make the fork, n we don't got enough experience in Rust programming (or enough interest in programming in general) for doing this ^^; I mean sure we could open a PR but we can't provide any support on't, beyond on how ya host't; for the container it's recommended mounting |
As a instance maintainer, running 6 instances behind a loadbalancer, with 6 public IP addresses I'm able to operate a public instance with only 30% of rate-limiting happening from reddits API side throughout a day - while other instances are constantly 425 Rate-limited. That realization brought me to experiment a little bit with piping libreddits' traffic through the TOR network, and it seems to work well enough. For containers, it seems to work reasonably well with David Personettes Tor Proxy. Also setups with
proxychains
work quite well. However, currently the experience is not as great as it could be. If libreddit would support a PROXY configuration and/or a SOCKS proxy it'd be easy to spin up a TOR proxy on a host system and route libreddits traffic directly through the TOR network without using hacks that force all traffic through the TOR network.Suggestion
make a (socks-)proxy settable through well established env-vars
However...
Any other/further thoughts on that?
The text was updated successfully, but these errors were encountered: