Possible to configure endpoint outbound ZeroMQ channel ports to the forwarder?

[matthewfeickert]

Hi. I'm running into an Issue when trying to do a localhost demo of funcX on Binder where the funcx-workers seem to be getting killed. I've opened a topic for discussion with the Binder team on the JupyterHub Discourse here which also gives reproducible examples. In the discussion @manics has pointed to this section in the Starting an Endpoint docs as a potential problem

The endpoint establishes three outbound ZeroMQ channels to the forwarder (on the three ports returned during registration) to retrieve tasks, send results, and communicate command information.

This might be the problem- if those are three random ports they’ll be blocked.

To test this, is there anyway to configure those ports in the funcX config?

[ZhuozhaoLi]

The ports from endpoint to forwarder (outbound connections) are not open on the endpoint side. The endpoint just connects to the ports open on the forwarder. The forwarder ports are fixed to 55001,55002,55003.

However, there are some ports open for the connections between the endpoint, manager, and workers (all are on the local 127.0.0.1 for your case, not outbound). I am not sure if Binder has limits on local ports or not. But they can be configured via below, where worker_port_range needs to be a range of at least 2（e.g., (54000, 54002)） and interchange_port_range needs to be a range of at least 3.

Config(
    Executors=[
        HighThroughputExecutor(
             worker_port_range=(54000, 55000),
             interchange_port_range=(55000, 56000),
        )
    ]
)

[ZhuozhaoLi]

I checked the logs in ~/.funcx/pyhf/. It does look like the problem is the connections between endpoint and forwarder. For some reasons, binder does not allow connections to the forwarder

2021-07-05 02:59:36.018 funcx_endpoint:159 [INFO]  Attempting connection to client at 54.242.225.131 on ports: 55001,55002,55003
...
...
2021-07-05 02:59:42.075 funcx_endpoint:245 [INFO]  Task incoming on tcp://54.242.225.131:55001
2021-07-05 03:01:37.252 funcx_endpoint:254 [CRITICAL]  [TASK_PULL_THREAD] Missed too many heartbeats. Setting kill event.

[manics]

mybinder.org limits egress network traffic to prevent abuse since it's a free unauthenticated resource. Outbound traffic to only the following ports is allowed

• 80
• 443
• 9418
• 873
• 1094
• 1095
• 16286