Allow client to connect to Tor onion addresses #267

orbitalturtle · 2021-02-15T04:26:15Z

This change allows a client to connect to a .onion address, as long as they're running Tor.

Initially added it to the example client in contrib/client because I'm more familiar with that code, but also added it to the c-lightning plugin since that's what people are probably more likely to use

One note is that the added e2e test require Tor to be installed to run... Not sure if we want to change that.

sr-gi · 2021-02-19T15:35:18Z

Not sure if I'm missing something, but isn't the server part missing here?

I cannot see any changes on the tower side to serve the data trough Tor.

orbitalturtle · 2021-02-19T21:16:27Z

@sr-gi Yeah I only added the client side for now. For now, people wanting to operate a hidden service watchtower would need to set it up manually - which isn't too hard (that's what I did for mine), but I would love to eventually also automate creating the hidden service and such!

sr-gi

Did a first pass.

Both of the clients should have socks_port as a config param, instead of it being a param with defaults on the function calls since, as it is right now, it cannot be changed.

Also feels like the e2e test needs something more than just tor being running, since the tower is not set up to serve data through the given hidden service, is it?

orbitalturtle · 2021-03-03T01:39:30Z

@sr-gi Cool, added the ability to assign the socks_port as a config option for both clients

I think what's tested is enough. In create_hidden_service in the e2e conftest, the hidden service is being connected to the created test tower API at port 9814 on line 103

Then on the client side, for the client to connect to the tower, Tor needs to be running. Then it can connect to the hidden service by tapping into the control port running at port 9050

But let me know if it seems I'm missing something ~

sr-gi · 2021-03-03T12:51:44Z

Then on the client side, for the client to connect to the tower, Tor needs to be running. Then it can connect to the hidden service by tapping into the control port running at port 9050

I may be missing something because I've tried to run the e2e test with tor installed (both running and not running) and it always fail.

Could be there some local setup that is not being pushed? Can you try running the test on a fresh machine to see if it works?

orbitalturtle · 2021-03-03T17:53:21Z

@sr-gi Oooohhhh my guess is that in the torrc config file the ControlPort 9051 and CookieAuthentication 1 are commented out

I'll give it a try on another machine ASAP to see if that's it

orbitalturtle · 2021-03-03T22:24:37Z

@sr-gi Nevermind, I forgot that I already set the test config in conftest run_tor. You also installed the python package "stem"?

sr-gi · 2021-03-04T09:59:55Z

@sr-gi Nevermind, I forgot that I already set the test config in conftest run_tor. You also installed the python package "stem"?

Yep, tried with both but it's not working.

FAILED        [100%]
test/teos/e2e/test_client_e2e.py:563 (test_register_request_to_onion_service)
teosd = (<Process name='Process-1' pid=65778 parent=65771 stopped exitcode=0>, '0220e15b0691e4ca31f7f1ad7cc9b41f2dadbe3c30fc16b5a45eed83e95857c628')
run_tor = None

    def test_register_request_to_onion_service(teosd, run_tor):
        _, teos_id = teosd
    
        run_tor
        sleep(3)
    
        onion_address = create_hidden_service()
        onion_endpoint = f"http://{onion_address}:9814"
    
        # See if a user can connect to the hidden service.
        tmp_user_sk = PrivateKey()
        tmp_user_id = Cryptographer.get_compressed_pk(tmp_user_sk.public_key)
    
>       available_slots, subscription_expiry = teos_client.register(tmp_user_id, teos_id, onion_endpoint, socks_port=9060)

test_client_e2e.py:577: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../contrib/client/teos_client.py:66: in register
    response = process_post_response(post_request(data, register_endpoint, socks_port))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

data = {'public_key': '02cd2afdce43b16cf87e7ae4cfb7c3dbee96b025d0a1603002a4869b2aefc6e611'}
endpoint = 'http://4k45nfkabfn2gx64olvtshzy5jur7tlyeawogwm6q474jw2vqzyyylyd.onion:9814/register'
socks_port = 9060

    def post_request(data, endpoint, socks_port=9050):
        """
        Sends a post request to the tower.
    
        Args:
            data (:obj:`dict`): a dictionary containing the data to be posted.
            endpoint (:obj:`str`): the endpoint to send the post request.
    
        Returns:
            :obj:`dict`: A json-encoded dictionary with the server response if the data can be posted.
    
        Raises:
            :obj:`ConnectionError`: if the client cannot connect to the tower.
        """
    
        try:
            if ".onion" in endpoint:
                proxies = {"http": f"socks5h://127.0.0.1:{socks_port}", "https": f"socks5h://127.0.0.1:{socks_port}"}
    
                return requests.post(url=endpoint, json=data, timeout=15, proxies=proxies)
            else:
                return requests.post(url=endpoint, json=data, timeout=5)
    
        except Timeout:
            message = "Cannot connect to the Eye of Satoshi's API. Connection timeout"
    
        except ConnectionError:
            message = "Cannot connect to the Eye of Satoshi's API. Server cannot be reached"
    
        except (InvalidSchema, MissingSchema, InvalidURL):
            message = "Invalid URL. No schema, or invalid schema, found ({})".format(endpoint)
    
>       raise ConnectionError(message)
E       requests.exceptions.ConnectionError: Invalid URL. No schema, or invalid schema, found (http://4k45nfkabfn2gx64olvtshzy5jur7tlyeawogwm6q474jw2vqzyyylyd.onion:9814/register)

../../../contrib/client/teos_client.py:325: ConnectionError

orbitalturtle · 2021-03-05T04:14:30Z

Interesting! Well I tried it on a new machine & it's working for me. Wonder what it could be. What OS are you on? I was using ubuntu 20.04 for both

sr-gi · 2021-03-05T06:59:30Z

Interesting. I'm testing on osx, will boot a different OS and give it a go.

sr-gi

Here's a second pass.

There's a dependency missing so socks can be used. That's the tor E2E test was failing on my end. Both clients need to add requests[socks] in their requirements.txt
Code also needs formatting.
At test/teos/e2e/conftest.py:85, the config dict has a non-necessary ending comma.
Will need rebase to be merged.
Commits are not signed. Have you changed your setup?

The rest looks good AFAICT.

sr-gi

Some nits here and there. Most of the comments for contrib/client also apply to watchtower-plugin (left a few ack it, but at some point I stopped).

The main thing to check is whether the way we are approaching the Tor connection from the client makes sense.

Codes still needs formatting in contrib/client/teos_client.py and test/teos/e2e/conftest.py

sr-gi · 2021-03-15T10:24:45Z

watchtower-plugin/requirements.txt

@@ -1,5 +1,7 @@
 pyln-client
+pyln-testing
 requests


Same as with contrib/requirements.txt

sr-gi · 2021-03-15T10:26:00Z

contrib/client/requirements.txt

@@ -1,3 +1,4 @@
 cryptography>=2.8
 requests


I'm pretty sure requests[socks] builds on top of requests, so by installing the latter the former is also installed. Could you test it in a fresh virtualenv and see if after removing requests from requirements, things still work?

sr-gi · 2021-03-15T10:28:16Z

contrib/client/teos_client.py

@@ -36,7 +36,7 @@
 logger = logging.getLogger()


-def register(user_id, teos_id, teos_url):
+def register(user_id, teos_id, teos_url, socks_port=9050):


docstrings missing for socks_port. Furthermore, the default value is still there.

sr-gi · 2021-03-15T10:29:06Z

contrib/client/teos_client.py

@@ -115,7 +115,7 @@ def create_appointment(appointment_data):
    return Appointment.from_dict(appointment_data)


-def add_appointment(appointment, user_sk, teos_id, teos_url):
+def add_appointment(appointment, user_sk, teos_id, teos_url, socks_port=9050):


Same as register

sr-gi · 2021-03-15T10:29:13Z

contrib/client/teos_client.py


    return response


-def get_subscription_info(user_sk, teos_id, teos_url):
+def get_subscription_info(user_sk, teos_id, teos_url, socks_port=9050):


Same as register

sr-gi · 2021-03-15T10:56:14Z

contrib/client/teos_client.py

@@ -306,7 +306,12 @@ def post_request(data, endpoint):
    """

    try:
-        return requests.post(url=endpoint, json=data, timeout=5)
+        if ".onion" in endpoint:


I'm wondering whether this is the best approach for using Tor with the client.

With the current approach, if the tower is accessible from a regular address (non-onion) but the user still wants to use Tor, the client will not allow it.

I think it may be best to think about Tor for both sides separately:

The client may want to use Tor to connect to both .onion and regular addresses.

The tower may offer both .onion and regular endpoints.

Therefore, if a client enables Tor on their side, it will use Tor to connect to whatever endpoint they are trying to reach.

If we define the PROXY config option (defaulting to None) then you'll only need to check whether a proxy has been provided here or not, no matter the address type.

sr-gi · 2021-03-15T10:59:06Z

watchtower-plugin/net/http.py

@@ -110,13 +110,21 @@ def post_request(data, endpoint, tower_id):
    """

    try:
-        return requests.post(url=endpoint, json=data, timeout=5)
+        if ".onion" in endpoint:


Same as for contrib/client

sr-gi · 2021-03-15T10:59:30Z

watchtower-plugin/watchtower.py

@@ -35,6 +35,7 @@
    "APPOINTMENTS_FOLDER_NAME": {"value": "appointment_receipts", "type": str, "path": True},
    "TOWERS_DB": {"value": "towers", "type": str, "path": True},
    "PRIVATE_KEY": {"value": "sk.der", "type": str, "path": True},
+    "SOCKS_PORT": {"value": 9050, "type": int},


Same as for contrib/client

sr-gi · 2021-03-15T11:01:56Z

test/teos/e2e/conftest.py

+        controller.authenticate()
+
+        hidden_service_dir = os.path.join(controller.get_conf("DataDirectory", "/tmp"), "onion_test")
+        result = controller.create_hidden_service(hidden_service_dir, 9814, target_port=9814)


It'll be nice to get the config from the config file, in a similar manner that is done for run_teosd.

Here it basically applied to the http port

sr-gi · 2021-03-15T11:02:57Z

test/teos/e2e/conftest.py

+        config={
+            "SocksPort": "9060",
+            "ControlPort": "9061",
+            "DataDirectory": data_dir,


Remove trailing comma

orbitalturtle requested a review from sr-gi February 15, 2021 04:26

orbitalturtle force-pushed the client_tor branch from 9e52dfb to fede666 Compare February 15, 2021 04:34

sr-gi suggested changes Feb 23, 2021

View reviewed changes

orbitalturtle force-pushed the client_tor branch from fede666 to 857f004 Compare March 3, 2021 01:35

sr-gi self-requested a review March 10, 2021 15:03

sr-gi suggested changes Mar 10, 2021

View reviewed changes

orbitalturtle added 4 commits March 13, 2021 17:23

Add support to client for SOCKS5 requests

2a7afae

Unit test posting data to an onion address

2cf5921

Test e2e client connecting to an onion address

51f284c

Add Tor requests to plugin

3f31b74

orbitalturtle force-pushed the client_tor branch from 857f004 to 3f31b74 Compare March 13, 2021 22:24

orbitalturtle requested a review from sr-gi March 13, 2021 22:58

sr-gi suggested changes Mar 15, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow client to connect to Tor onion addresses #267

Allow client to connect to Tor onion addresses #267

orbitalturtle commented Feb 15, 2021

sr-gi commented Feb 19, 2021

orbitalturtle commented Feb 19, 2021 •

edited

sr-gi left a comment

orbitalturtle commented Mar 3, 2021

sr-gi commented Mar 3, 2021

orbitalturtle commented Mar 3, 2021

orbitalturtle commented Mar 3, 2021

sr-gi commented Mar 4, 2021

orbitalturtle commented Mar 5, 2021

sr-gi commented Mar 5, 2021

sr-gi left a comment •

edited

sr-gi left a comment

sr-gi Mar 15, 2021

sr-gi Mar 15, 2021

sr-gi Mar 15, 2021

sr-gi Mar 15, 2021

sr-gi Mar 15, 2021

sr-gi Mar 15, 2021

sr-gi Mar 15, 2021

sr-gi Mar 15, 2021

sr-gi Mar 15, 2021

sr-gi Mar 15, 2021

sr-gi Mar 15, 2021

Allow client to connect to Tor onion addresses #267

Are you sure you want to change the base?

Allow client to connect to Tor onion addresses #267

Conversation

orbitalturtle commented Feb 15, 2021

sr-gi commented Feb 19, 2021

orbitalturtle commented Feb 19, 2021 • edited

sr-gi left a comment

Choose a reason for hiding this comment

orbitalturtle commented Mar 3, 2021

sr-gi commented Mar 3, 2021

orbitalturtle commented Mar 3, 2021

orbitalturtle commented Mar 3, 2021

sr-gi commented Mar 4, 2021

orbitalturtle commented Mar 5, 2021

sr-gi commented Mar 5, 2021

sr-gi left a comment • edited

Choose a reason for hiding this comment

sr-gi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

orbitalturtle commented Feb 19, 2021 •

edited

sr-gi left a comment •

edited