Idea for encapsulation in flowchart

[pritamdodeja]

Awesome project! I was looking to do some ascii diagramming to depict the execution flow in some python programs in vim. I am looking to capture the flow of the program as it executes to organize the design better. Specifically, I have the following flowcharge

'Start';                                                                                                                                                                    
'Set working directory';                                                                                                                                                    
'Call pipeline_function passing preprocessing_fn as argument';                                                                                                              
if ('Within pipeline function')                                                                                                                                             
    {                                                                                                                                                                       
    'Call tfxio.BeamRecordsCsvTFXIO with format, columns, schema';                                                                                                          
    'Read using beam.io.ReadFromText as bytes';                                                                                                                             
    'Decode bytes using tfxio';                                                                                                                                             
    'Make a raw dataset with raw data and TensorAdapterConfig()';                                                                                                           
    'Transform raw dataset using tft_beam.AnalyzeAndTransformDataset';                                                                                                      
    'Get transform_fn back from transformation';                                                                                                                            
    'Encode transform data using beam.FlatMapTuple';                                                                                                                        
    'Write it using beam.io.WriteToTFRecord';                                                                                                                               
    'Write transform_fn using tft_beam.WriteTransformFn';                                                                                                                   
    return 'stuff';                                                                                                                                                         
    }                                                      
else {                                                                                                                                                                  
    'Get transform output using tft.TFTransformOutput';                                                                                                                     
    'Get single example from raw data, make batch';                                                                                                                         
    'Get tft_layer using transform_output.transform_features_layer()';                                                                                                      
    'Transform single example using tft_layer(single_example_batched)';                                                                                                     
    'Inspect transformed example to show names, datatypes, and shapes';                                                                                                     
    }                                                                                                                                                                       
    'End';                

This produces the following flowchart:

                   ┌─────┐                                                                                                                                                                                                                                                                                                      
                   │Start│                                                                                                                                                                                                                                                                                                      
                   └──┬──┘                                                                                                                                                                                                                                                                                                      
           ┌──────────▽──────────┐                                                                                                                                                                                                                                                                                              
           │Set working directory│                                                                                                                                                                                                                                                                                              
           └──────────┬──────────┘                                                                                                                                                                                                                                                                                              
      ┌───────────────▽──────────────┐                                                                                                                                                                                                                                                                                          
      │Call pipeline_function passing│                                                                                                                                                                                                                                                                                          
      │preprocessing_fn as argument  │                                                                                                                                                                                                                                                                                          
      └───────────────┬──────────────┘                                                                                                                                                                                                                                                                                          
               ____▽____                 ┌──────────────────────────────┐                                                                                                                                                                                                                                                       
              ╱               ╲                │Call tfxio.BeamRecordsCsvTFXIO│                                                                                                                                                                                                                                                 
             ╱ Within pipeline ╲_______________│with format, columns, schema  │                                                                                                                                                                                                                                                 
             ╲ function        ╱yes            └───────────────┬──────────────┘                                                                                                                                                                                                                                                 
              ╲_______________╱                 ┌──────────────▽──────────────┐                                                                                                                                                                                                                                                 
                      │no                       │Read using                   │                                                                                                                                                                                                                                                 
        ┌─────────────▽────────────┐            │beam.io.ReadFromText as bytes│                                                                                                                                                                                                                                                 
        │Get transform output using│            └──────────────┬──────────────┘                                                                                                                                                                                                                                                 
        │tft.TFTransformOutput     │                    ┌──────▽─────┐                                                                                                                                                                                                                                                          
        └─────────────┬────────────┘                    │Decode bytes│                                                                                                                                                                                                                                                          
          ┌───────────▽───────────┐                     │using tfxio │                                                                                                                                                                                                                                                          
          │Get single example from│                     └──────┬─────┘                                                                                                                                                                                                                                                          
          │raw data, make batch   │            ┌───────────────▽──────────────┐                                                                                                                                                                                                                                                 
          └───────────┬───────────┘            │Make a raw dataset with raw   │                                                                                                                                                                                                                                                 
┌─────────────────────▽─────────────────────┐  │data and TensorAdapterConfig()│                                                                                                                                                                                                                                                 
│Get tft_layer using                        │  └───────────────┬──────────────┘                                                                                                                                                                                                                                                 
│transform_output.transform_features_layer()│┌─────────────────▽─────────────────┐                                                                                                                                                                                                                                              
└─────────────────────┬─────────────────────┘│Transform raw dataset using        │                                                                                                                                                                                                                                              
     ┌────────────────▽────────────────┐     │tft_beam.AnalyzeAndTransformDataset│                                                                                                                                                                                                                                              
     │Transform single example using   │     └─────────────────┬─────────────────┘                                                                                                                                                                                                                                              
     │tft_layer(single_example_batched)│            ┌──────────▽──────────┐                                                                                                                                                                                                                                                     
     └────────────────┬────────────────┘            │Get transform_fn back│                                                                                                                                                                                                                                                     
          ┌───────────▽──────────┐                  │from transformation  │                                                                                                                                                                                                                                                     
          │Inspect transformed   │                  └──────────┬──────────┘                                                                                                                                                                                                                                                     
          │example to show names,│                 ┌───────────▽───────────┐                                                                                                                                                                                                                                                    
          │datatypes, and shapes │                 │Encode transform data  │                                                                                                                                                                                                                                                    
          └───────────┬──────────┘                 │using beam.FlatMapTuple│                                                                                                                                                                                                                                                    
                      │                            └───────────┬───────────┘                                                                                                                                                                                                                                                    
                      │                            ┌───────────▽───────────┐                                                                                                                                                                                                                                                    
                      │                            │Write it using         │                                                                                                                                                                                                                                                    
                      │                            │beam.io.WriteToTFRecord│                                                                                                                                                                                                                                                    
                      │                            └───────────┬───────────┘                                                                                                                                                                                                                                                    
                      │                           ┌────────────▽────────────┐                                                                                                                                                                                                                                                   
                      │                           │Write transform_fn using │                                                                                                                                                                                                                                                   
                      │                           │tft_beam.WriteTransformFn│                                                                                                                                                                                                                                                   
                      │                           └────────────┬────────────┘                                                                                                                                                                                                                                                   
                      │                                     ┌──▽──┐                                                                                                                                                                                                                                                             
                      │                                     │stuff│                                                                                                                                                                                                                                                             
                      │                                     └─────┘                                                                                                                                                                                                                                                             
                      │                                                                                                                                                                                                                                                                                                         
                    ┌─▽─┐                                                                                                                                                                                                                                                                                                       
                    │End│                                                                                                                                                                                                                                                                                                       
                    └───┘                                                                                                           

Right off the bat, this is really good and helpful. As you can see, I am trying to depict what's happening inside the pipeline function using a conditional. Probably not what you intended, and if there's a better way, please let me know. What I'd like to do is to be able to encapsulate/abstract the details that when needed. I was thinking of two ways that this could be done.

The first is to continue the left branch only after "stuff" is returned by the pipeline function. This way, I can fold the text in vim to abstract the details if needed.

A second option is to allow for the branch to be rendered horizontally, so that if I want to go into the details, I can navigate in the text editor horizontally.

Thanks for this excellent project!

[ArthurSonzogni]

Hello!
Sorry for this late reply ;-(

How would user specify in the input what they want to see in the output? Would you had example?

[pritamdodeja]

No problem at all! How about something like this:

diagon Flowchart -expand=horizontal < flow

Horizontal keeps growing the flowchart horizontally till something is returned, and vertical would do vertically. If there are no returns, -expand is ignored or an error is thrown informing the user of an invalid selection.