How to set parameters for calculators?

[lubagov]

Hello!
I have create grpah from pbtxt file, and try to modify calculator parameters in this graph, to change thresholds for BlazePose. I use branch fix-side-packets for my experiments. I generate the missing files from Protobuf.

In proto files, append namespace:

option csharp_namespace = "Mediapipe.Net.Framework.Protobuf";

and next run:

protoc -I=. -I ..\..\..\ --csharp_out=.\cs_out\ .\tensors_to_detections_calculator.proto
protoc -I=. -I ..\..\..\ --csharp_out=.\cs_out\ .\thresholding_calculator.proto

i have attach generated files: CalculatorOptions.zip

GraphPath="mediapipe/modules/pose_landmark/pose_landmark_cpu.pbtxt"

CalculatorGraph GraphTmp = new CalculatorGraph(File.ReadAllText(GraphPath));
CalculatorGraphConfig configTmp = new CalculatorGraphConfig(GraphTmp.Config());

foreach (CalculatorGraphConfig.Types.Node n in configTmp.Node)
{
    if (n.Name.Equals("posedetectioncpu__TensorsToDetectionsCalculator"))
    {                   
        TensorsToDetectionsCalculatorOptions to = n.Options.GetExtension(TensorsToDetectionsCalculatorOptions.Extensions.Ext);
        to ??= new TensorsToDetectionsCalculatorOptions();
        to.MinScoreThresh = 0.99f;
        n.Options.SetExtension(TensorsToDetectionsCalculatorOptions.Extensions.Ext, to);
     }
     if (n.Name.Equals("poselandmarkbyroicpu__tensorstoposelandmarksandsegmentation__ThresholdingCalculator"))
    {
         ThresholdingCalculatorOptions to = n.Options.GetExtension(ThresholdingCalculatorOptions.Extensions.Ext);
         to ??= new ThresholdingCalculatorOptions();
         to.Threshold = 0.99;
         n.Options.SetExtension(ThresholdingCalculatorOptions.Extensions.Ext, to);
    }
}

//Now I'm doing some experiments with the graph:

// create new CalculatorGraph from modifyed config
Graph = new CalculatorGraph(configTmp); 

byte[] cb = configTmp.ToByteArray();
Console.WriteLine("Modified config length:" + cb.Length);  //Length 21759

//test merge graph from copy bytes
CalculatorGraphConfig c2 = new CalculatorGraphConfig();
c2.MergeFrom(cb);
byte[] cb2 = c2.ToByteArray(); 
Console.WriteLine("Merge copy, from modifyed config:" + cb2.Length); //Length 21759

//returned back graph
byte[] cb3 = Graph.Config().ToByteArray();
Console.WriteLine("Config returned back from Mediapipe.Runtime length:" + cb3.Length); //Length 21732

//Original graph that i read from pbtxt file
byte[] cb4 = GraphTmp.Config().ToByteArray();
Console.WriteLine("Original graph that we read from file:" + cb4.Length); //Length 21732

Output in console:

Modified config length:21759
Merge copy, from modifyed config:21759
Config returned back from Mediapipe.Runtime length:21732
Original graph that we read from file:21732

Those. the size of the graph has become smaller, in the version that went to Mediapipe.Runtime and returned back. We've lost extension options.

Another interesting point, is if I want to get values from an extension, I can only get it from the original graph, but not copies (even can't do it from byte array copy with same size). From any copy, this returns null.

TensorsToDetectionsCalculatorOptions? to1 = n.Options.GetExtension(TensorsToDetectionsCalculatorOptions.Extensions.Ext);

And in principle, these changes do not affect the result of the pipeline in any way. those. with a value of 0.99 I shouldn't have any detections at all. But they are exactly the same as before.
Thank you!

[Speykious]

I'm not sure how I can help with this.

I assume you got these 2 proto files in the original mediapipe repository? Because they're not in MediaPipe.NET.Runtime. You might wanna take a look into that repo, since the source of the bindings in C++ are there, it might give you a clue.
For now, this is unsupported. And I think I noticed that we couldn't do this when I started refactoring packets and solutions, since I now try to base myself off the official mediapipe Python API.

If you think you can contribute in this area, we'd appreciate your help :)

[homuler]

@lubagov I think this is related to protocolbuffers/protobuf#9591.
This fix is included in Google.Protobuf 3.20.0.

P.S.
Note that you may need to pass a proper ExtensionRegistry to CalculatorGraphConfig.Parser to call GetExtension.

[lubagov]

Thank you @homuler !

I haven't tested it, but it actually seems to be a Protobuf bug. I tried with C++ application. I also tried saving it in a Python application. Both the C++ and Python versions have no problem, and read Protobuf and extensions well. Yes, the size is actually 21732 bytes, but in the C # implementation it turns out to be larger. As a result, I saved the graph in Python and added the graph loading code from the binary file to Mediapipe.Net:

diff --git a/Mediapipe.Net/Solutions/Solution.cs b/Mediapipe.Net/Solutions/Solution.cs
index 3a5da7f..87069ae 100644
--- a/Mediapipe.Net/Solutions/Solution.cs
+++ b/Mediapipe.Net/Solutions/Solution.cs
@@ -36,9 +36,18 @@ namespace Mediapipe.Net.Solutions
             SidePackets? sidePackets)
         {
             GraphPath = graphPath;
-            Graph = new CalculatorGraph(File.ReadAllText(GraphPath));
             SidePackets = sidePackets;
 
+            string ext = Path.GetExtension(graphPath).ToLower();
+            if ("pbtxt".Equals(ext))
+            {
+                Graph = new CalculatorGraph(File.ReadAllText(GraphPath));
+            }
+            else
+            {
+                Graph = new CalculatorGraph(File.ReadAllBytes(GraphPath));
+            }
+
             GraphOutputs = new Dictionary<string, object?>();
 
             observeStreamHandles = new Dictionary<string, GCHandle>();

And for now this is a good temporary solution. Anyway, it works.

[Speykious]

@lubagov I think this is related to protocolbuffers/protobuf#9591.
This fix is included in Google.Protobuf 3.20.0.

I see. I guess it is time to update the dependency in Mediapipe.Net.Framework.Protobuf.

[sr229]

Hello @lubagov, please let us know if the current revision solves your issue.

[sr229]

Since there is no activity on this issue, I'll be closing this of now.