Support for recursive workflows

[utf]

Firstly, thank you for the lovely code. The UI interface is particularly nice and easy to get started with.

One roadblock I've run into is support for recursive functions. For example, the following code snippet doesn't work:

import covalent as ct

@ct.lattice
@ct.electron
def fibonacci(smaller, larger, stop_point=1000):
    """Calculate the next number in the Fibonacci sequence.

    If the number is larger than stop_point, the job will stop the workflow
    execution, otherwise, a new job will be submitted to calculate the next number.
    """
    total = smaller + larger

    if total > stop_point:
        return total
    
    return fibonacci(larger, total, stop_point=stop_point)


dispatch_id = ct.dispatch(fibonacci)(1, 1)
result = ct.get_result(dispatch_id=dispatch_id, wait=True)

The error message I see on the UI is:

Traceback (most recent call last): 
File "/Users/alexganose/opt/miniconda3/envs/jobflow/lib/python3.9/site-packages/covalent/executor/utils/wrappers.py",
    line 47, in io_wrapper output = fn(*args, **kwargs)
File "/Users/alexganose/opt/miniconda3/envs/jobflow/lib/python3.9/site-packages/covalent/executor/base.py",
    line 109, in wrapper_fn output = fn(*new_args, **new_kwargs)
File "<ipython-input-1-ed3627cd8ac8>",
    line 16, in fibonacci NameError: name 'fibonacci' is not defined 

Is this something you would consider supporting in the future?

[santoshkumarradha]

Hey @utf!

Thanks for giving Covalent a shot and welcome to Covalent !

Just wanted to let you know that we fully support recursive functions within electrons. It seems like the issue you ran into is a little bug. But, no worries! A small tweak to the example you provided should solve the problem interim.

import covalent as ct


@ct.electron
def fibonacci(smaller, larger, stop_point=1000):
    """Calculate the next number in the Fibonacci sequence.

    If the number is larger than stop_point, the job will stop the workflow
    execution, otherwise, a new job will be submitted to calculate the next number.
    """
    total = smaller + larger

    if total > stop_point:
        return total
    
    return fibonacci(larger, total, stop_point=stop_point)

@ct.lattice
def w(smaller, larger, stop_point=1000):
    return fibonacci(smaller, larger, stop_point=stop_point)

dispatch_id = ct.dispatch(w)(1, 1)

I believe that there may be an issue with the lattice decorator preventing the fibonacci function from being available in the local scope. This certainly warrants a bug report.

CC: @kessler-frost

[santoshkumarradha]

You are correct ! it was prepopulated in namespace, having it in a script/fresh notebook does pipe out the error*. The issue remains for sublattice as well.

[utf]

Thanks for confirming. Out of interest, do you also consider the non-deterministic/inconsistent behaviour of notebooks an issue that you plan to resolve?

[santoshkumarradha]

Hey there, @utf! It looks like my notebook was doing fine before since I didn't have to restart the kernel and the electron function was already in your namespace from the previous cell. But after the restart, it failed in the notebook too.

Could you give us a little more info on your workflow, setup, and the error you ran into? We're really thorough in checking our notebooks before each release by doing manual QA of all features, and we haven't seen any unpredictable or inconsistent issues. We'd really appreciate it if you could help us out by sharing more details.

[santoshkumarradha]

Running from a notebook is a significant flow utilized by our internal researchers, and we would promptly address any issues that arise.

[utf]

I just meant the inconsistency that you experienced yourself. E.g., that sometimes workflows will work fine on one execution but fail when running after the kernel is restarted. It specifically seems to be present when running recursive dynamic workflows, so perhaps it is related to the namespace issue described above.

[santoshkumarradha]

A very quick update,

Pattern 1

Just tried recursion at electron level, that does seem to work (both with and sans jupyter)

import covalent as ct



@ct.electron
def f(x):
    """Calculate the next number in the Fibonacci sequence.

    If the number is larger than stop_point, the job will stop the workflow
    execution, otherwise, a new job will be submitted to calculate the next number.
    """
    x+=1
    return x if x>10 else f(x)


@ct.lattice
def w(x):
    return f(x)

print("Normal result :",w(1))
dispatch_id = ct.dispatch(w)(1)
print("Covalent result :",ct.get_result(dispatch_id,wait=True).result)

Pattern 2

with sublattice

import covalent as ct

@ct.electron
@ct.lattice
def fibonacci(smaller, larger,function, stop_point=1000):
    """Calculate the next number in the Fibonacci sequence.

    If the number is larger than stop_point, the job will stop the workflow
    execution, otherwise, a new job will be submitted to calculate the next number.
    """
    total = smaller + larger

    if total > stop_point:
        return total
    
    return function(larger, total, function,stop_point=stop_point)

@ct.lattice
def workflow(smaller, larger, stop_point=1000):
    return fibonacci(smaller, larger,fibonacci, stop_point=stop_point)

dispatch_id = ct.dispatch(workflow)(1, 1,1000)
result = ct.get_result(dispatch_id=dispatch_id, wait=True)
print(result)

this pattern send the recursive function in as input as well, the bug currently makes it such that the name space inside the inner lattice does not contain the lattice itself. But forcing the lattice as inputs takes away that problem, tho not ideal.

Hope this would help until we solve the bug.