Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dill 0.3.5.1 vs python 3.11 vs globus-compute-sdk 2.2.1 #1197

Open
benclifford opened this issue Jun 28, 2023 · 3 comments
Open

dill 0.3.5.1 vs python 3.11 vs globus-compute-sdk 2.2.1 #1197

benclifford opened this issue Jun 28, 2023 · 3 comments
Labels
bug Something isn't working

Comments

@benclifford
Copy link
Contributor

Describe the bug
I think I have encountered this dill behaviour: uqfoundation/dill#514

Stack trace 1 below gives the traceback.

Following that issue I forcibly violated the globus-compute-sdk packaging requirements using this command line:

$ pip install 'dill>0.3.5.1'

and now get further in execution, to what looks like an execution failure remotely... (presumably because of incompatible dill or python versions between my submit side and the tutorial endpoint - so I think that means that execution proceeded further but still doesn't work. See stack trace 2 below.

To Reproduce
Install globus-compute-sdk 2.2.1 vs python 3.11.2. Run quickstart example.

STACK TRACE 1 - DILL 0.3.5.1 VS PYTHON 3.11.2

benc@parsl-dev-3-11-5704:~/parsl/src/parsl$ python3 parsl2funcx.py 
Unable to register function: add_func
Traceback (most recent call last):
  File "/home/benc/parsl/virtualenv-3.11/lib/python3.11/site-packages/globus_compute_sdk/serialize/facade.py", line 63, in serialize
    return strategy.serialize(data)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benc/parsl/virtualenv-3.11/lib/python3.11/site-packages/globus_compute_sdk/serialize/concretes.py", line 137, in serialize
    x = codecs.encode(dill.dumps(data), "base64").decode()
                      ^^^^^^^^^^^^^^^^
  File "/home/benc/parsl/virtualenv-3.11/lib/python3.11/site-packages/dill/_dill.py", line 364, in dumps
    dump(obj, file, protocol, byref, fmode, recurse, **kwds)#, strictio)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benc/parsl/virtualenv-3.11/lib/python3.11/site-packages/dill/_dill.py", line 336, in dump
    Pickler(file, protocol, **_kwds).dump(obj)
  File "/home/benc/parsl/virtualenv-3.11/lib/python3.11/site-packages/dill/_dill.py", line 620, in dump
    StockPickler.dump(self, obj)
  File "/usr/local/lib/python3.11/pickle.py", line 487, in dump
    self.save(obj)
  File "/usr/local/lib/python3.11/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "/home/benc/parsl/virtualenv-3.11/lib/python3.11/site-packages/dill/_dill.py", line 1963, in save_function
    _save_with_postproc(pickler, (_create_function, (
  File "/home/benc/parsl/virtualenv-3.11/lib/python3.11/site-packages/dill/_dill.py", line 1140, in _save_with_postproc
    pickler.save_reduce(*reduction, obj=obj)
  File "/usr/local/lib/python3.11/pickle.py", line 692, in save_reduce
    save(args)
  File "/usr/local/lib/python3.11/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/pickle.py", line 902, in save_tuple
    save(element)
  File "/usr/local/lib/python3.11/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "/home/benc/parsl/virtualenv-3.11/lib/python3.11/site-packages/dill/_dill.py", line 1187, in save_code
    obj.co_firstlineno, obj.co_lnotab, obj.co_endlinetable,
                                       ^^^^^^^^^^^^^^^^^^^
AttributeError: 'code' object has no attribute 'co_endlinetable'. Did you mean: 'co_linetable'?

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/benc/parsl/src/parsl/parsl2funcx.py", line 15, in <module>
    future = gce.submit(add_func, 5, 10)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benc/parsl/virtualenv-3.11/lib/python3.11/site-packages/globus_compute_sdk/sdk/executor.py", line 322, in submit
    self.register_function(fn)
  File "/home/benc/parsl/virtualenv-3.11/lib/python3.11/site-packages/globus_compute_sdk/sdk/executor.py", line 281, in register_function
    func_reg_id = self.funcx_client.register_function(fn, **reg_kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benc/parsl/virtualenv-3.11/lib/python3.11/site-packages/globus_compute_sdk/sdk/login_manager/decorators.py", line 17, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benc/parsl/virtualenv-3.11/lib/python3.11/site-packages/globus_compute_sdk/sdk/client.py", line 654, in register_function
    data = FunctionRegistrationData(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benc/parsl/virtualenv-3.11/lib/python3.11/site-packages/globus_compute_sdk/sdk/web_client.py", line 44, in __init__
    function_code = _get_packed_code(function, serializer=serializer)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benc/parsl/virtualenv-3.11/lib/python3.11/site-packages/globus_compute_sdk/sdk/web_client.py", line 26, in _get_packed_code
    return serializer.pack_buffers([serializer.serialize(func)])
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benc/parsl/virtualenv-3.11/lib/python3.11/site-packages/globus_compute_sdk/serialize/facade.py", line 66, in serialize
    raise SerializationError(err_msg) from e
globus_compute_sdk.serialize.base.SerializationError: Code serialization strategy DillCode failed

STACK TRACE 2 - DILL 0.3.6 VS TUTORIAL ENDPOINT

benc@parsl-dev-3-11-5704:~/parsl/src/parsl$ python3 parsl2funcx.py 
Traceback (most recent call last):
  File "/home/benc/parsl/src/parsl/parsl2funcx.py", line 18, in <module>
    print(future.result())
          ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
globus_compute_sdk.errors.error_types.TaskExecutionFailed: 
    Traceback (most recent call last):
      File "/usr/local/lib/python3.10/site-packages/globus_compute_sdk/serialize/facade.py", line 116, in unpack_and_deserialize
        deserialized = self.deserialize(current)
      File "/usr/local/lib/python3.10/site-packages/globus_compute_sdk/serialize/facade.py", line 67, in deserialize
        result = self.methods_for_code[header].deserialize(payload)
      File "/usr/local/lib/python3.10/site-packages/globus_compute_sdk/serialize/concretes.py", line 137, in deserialize
        function = dill.loads(codecs.decode(chomped.encode(), "base64"))
      File "/usr/local/lib/python3.10/site-packages/dill/_dill.py", line 387, in loads
        return load(file, ignore, **kwds)
      File "/usr/local/lib/python3.10/site-packages/dill/_dill.py", line 373, in load
        return Unpickler(file, ignore=ignore, **kwds).load()
      File "/usr/local/lib/python3.10/site-packages/dill/_dill.py", line 646, in load
        obj = StockUnpickler.load(self)
      File "/usr/local/lib/python3.10/site-packages/dill/_dill.py", line 805, in _create_code
        return CodeType(args[0], 0, 0, *args[1:])
    TypeError: code expected at most 16 arguments, got 21

Expected behavior
The quickstart tutorial should proceed as documented

Environment
my laptop, in containerised debian-ish dev environment

  • Python version @ client: custom built python 3.11.2
  • globus-compute-sdk version @ client: 2.2.1
  • Python version @ endpoint: tutorial endpoint:
  • globus-compute-endpoint version @ endpoint: tutorial endpoint:

Distributed Environment

  • Where are you running the funcX script from? [e.g. Laptop/Workstation, Login node, Compute node]
    laptop
  • Where does the endpoint run? [e.g. Laptop/Workstation, Login node]
    globus hq
  • What is your endpoint-uuid?
    4b116d3c-1703-4f8f-9f6f-39921e5864df
  • Attach endpoint logs at ~/.globus_compute/<ENDPOINT_NAME> if this is an endpoint issue.
    globus hq should have these alreayd
@benclifford benclifford added the bug Something isn't working label Jun 28, 2023
@benclifford
Copy link
Contributor Author

(this may be a separate issue, or not... but it follows on from the above narrative)

I reset dill to 0.3.5.1

I tried using the CombinedCode serializer like this:

from globus_compute_sdk.serialize import CombinedCode
from globus_compute_sdk import Client

c= Client(code_serialization_strategy=CombinedCode())
from globus_compute_sdk import Executor

def add_func(a, b):
    return a + b

tutorial_endpoint_id = '4b116d3c-1703-4f8f-9f6f-39921e5864df' # Public tutorial endpoint
# ... then create the executor, ...
with Executor(endpoint_id=tutorial_endpoint_id, funcx_client=c) as gce:
    # ... then submit for execution, ...
    future = gce.submit(add_func, 5, 10)

    # ... and finally, wait for the result
    print(future.result())

and receive this AssertionError (at the tutorial endpoint, I think?)

benc@parsl-dev-3-11-5704:~/parsl/src/parsl$ !pyt
python3 parsl2funcx.py 
Traceback (most recent call last):
  File "/home/benc/parsl/src/parsl/parsl2funcx.py", line 18, in <module>
    print(future.result())
          ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
globus_compute_sdk.errors.error_types.TaskExecutionFailed: 
    Traceback (most recent call last):
      File "/usr/local/lib/python3.10/site-packages/globus_compute_sdk/serialize/facade.py", line 116, in unpack_and_deserialize
        deserialized = self.deserialize(current)
      File "/usr/local/lib/python3.10/site-packages/globus_compute_sdk/serialize/facade.py", line 67, in deserialize
        result = self.methods_for_code[header].deserialize(payload)
      File "/usr/local/lib/python3.10/site-packages/globus_compute_sdk/serialize/concretes.py", line 182, in deserialize
        for serial_id, encoded_func in self.get_multiple_payloads(payload):
      File "/usr/local/lib/python3.10/site-packages/globus_compute_sdk/serialize/concretes.py", line 155, in get_multiple_payloads
        assert len(parts) == 2 * len(COMBINED_SERIALIZE_METHODS)
    AssertionError

@benclifford
Copy link
Contributor Author

With DillCodeSource as a serializer, I'm able to get a successful function execution in this environment.

@kommav
Copy link

kommav commented Jul 9, 2023

I have also experienced this:
GlobusCompute Versions:
globus-compute-common 0.2.0
globus-compute-sdk 2.2.3
globus-sdk 3.23.0
MacOS Version: 12.5.1
Python Version: 3.11.4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@benclifford @kommav