Integration with pyqt

[guixiaocai]

Pypyr seems to be the pipeline integration tool I have been looking for. it is simple and easy to use.

I currently want to integrate pypyr with one of my pipeline tools with a GUI to achieve real-time processing, and the interface can display the results in real time. This capability is extremely important for data scientists, and is integral to processing and analyzing data effectively and efficiently.

In Pyqt, I can easily use signals to trigger when the GUI makes an attempt to load the data and display it. I don't know how to deal with it in pypyr?

[yaythomas]

hi @guixiaocai!

Glad you're finding pypyr simple & easy to use! That's been the plan, so great to hear it's working out that way too 😄

There're a few ways you can integrate pyqt with pypyr - you could:

1. use the pypyr API to run pipelines programatically based on user input

import sys                                                                                          
from PySide2.QtWidgets import QApplication, QPushButton                                                                                                                                                                                                                                                                  

from pypyr import pipelinerunner
                                                                                                    
def say_hello():
    # maybe get the pipeline-name from a textbox, e.g QLineEdit
    # context response here works like a dict - so you can display results etc. like you would a dict
    context = pipelinerunner.run(pipeline_name='pipeline-dir/my-pipe',
                                 args_in=['arbkey=pipe', 'anotherkey=song'])                                                                   
                                                                                                    
# Create the Qt Application                                                                         
app = QApplication(sys.argv)                                                                        
# Create a button, connect it and show it                                                           
button = QPushButton("run pipeline")                                                                    
button.clicked.connect(say_hello)                                                                    
button.show()                                                                                       
# Run the main Qt loop                                                                              
app.exec_()

2. capture stdout and stderr in pyqt.
   The output that you see in the console is pypyr writing to stdout and in the case of errors, to stderr. You can capture stdout and stderr output in pyqt and redirect it to a textedit - here's a stackoverflow showing something similar: https://stackoverflow.com/questions/21071448/redirecting-stdout-and-stderr-to-a-pyqt4-qtextedit-from-a-secondary-thread

[guixiaocai]

Hi Thomas, thank you very much for your reply.

I noticed the pipelinerunner.run function, which can be easily combined with python code. But I still have a small question.

I have a pipeline with several processing steps. Each step is processed according to the size of the input list (this can take quite a while).
My question is, if I want to watch the processing progress or processing results of the list task in real time, how can I return a signal from pypyr to trigger the update of the GUI (at this time, the pipelinerunner.run has not completed the execution and has not returned the context).

In other words, how do I know what step pipelinerunner.run has run to, while it is running?

Looking forward to your reply.
Best regards

Zheyuan Zhang

[yaythomas]

If you just want to see the pypyr output (i.e what you see at the console when you $ pypyr mypipeline), you can capture stdout and stderr from PyQt as suggested in option 2 above. But you would still have to invoke pypyr in a background thread not to lock up the UI.

To do this, use QThread - tutorial here: https://realpython.com/python-pyqt-qthread/

You can use signals and slots to interact between pypyr steps and your GUI by passing in your Signal objects to the pipeline and invoking them from a custom step:

So when you invoke pypyr, it'll go something like this:

class Worker(QObject):
    progress = Signal(str)
    completed = Signal(str)

    @Slot(str)
    def run(self, pipeline_name, input_args):
        context = pipelinerunner.run(pipeline_name=pipeline_name,
                             dict_in={'arbkey': 'pipe',
                                      'signals': self.progress,
                                      'inputs': input_args })


        self.completed.emit("done")

And then somewhere in your pipeline, either in a py step or a custom step:

# save as ./steps/mystep.py. Now can run in pipeline as `name: steps.mystep`
def run_step(context):
    context['signals'].emit("some value here")