Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FluxExecutor exposed as endpoint executor? #1077

Open
vsoch opened this issue Apr 2, 2023 · 0 comments
Open

FluxExecutor exposed as endpoint executor? #1077

vsoch opened this issue Apr 2, 2023 · 0 comments
Labels
enhancement New feature or request

Comments

@vsoch
Copy link

vsoch commented Apr 2, 2023

Hi! I was looking to set up a funcx endpoint, where the funcx-endpoint start X command would be run by the flux broker, thus giving it access to an entire cluster in Kubernetes. This is what I was hoping to use for my config:

from funcx_endpoint.endpoint.utils.config import Config
from funcx_endpoint.executors import FluxExecutor
import os

# Retrieve settings from the environment
working_dir = os.environ.get('FUNCX_WORKDIR')

# The executor will be run under the broker, with access to all its resources
executor = FluxExecutor(working_dir=working_dir)
executor.launch_cmd="{flux} submit {python} {manager} {protocol} {hostname} {port}"
config = Config(executors=[executor])

# For now, visible_to must be a list of URNs for globus auth users or groups, e.g.:
# urn:globus:auth:identity:{user_uuid}
# urn:globus:groups:id:{group_uuid}
meta = {
    "name": "default",
    "description": "The Flux Operator FuncX Endpoint",
    "organization": "Lawrence Livermore National Laboratory",
    "department": "The Harry Potter Closet",
    "public": False,
    "visible_to": [],
}

But I was looking here: https://github.com/funcx-faas/funcX/blob/main/funcx_endpoint/funcx_endpoint/executors/__init__.py and I don't see it as an option. Is this something we could potentially add?

Note that it's already an executor in parsl (alongside the high throughput - why I am hoping we can add it here!)

https://github.com/Parsl/parsl/blob/7a235a4f5e285bc8deb7b57726f01cea001f4c89/parsl/executors/flux/executor.py#L1

To give some context, the endpoint would already be started in a flux instance with an allocated set of resources, so we would just need to receive a task and do flux submit via the Python sdk.

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant