CEL to ElasticSearch

[TristonianJones]

Feature request checklist

• There are no issues that match the desired change
• The change is large enough it can't be addressed with a simple Pull Request
• If this is a bug, please file a Bug Report.

Change
Support translation from CEL expressions to ElasticSearch filters. At one level, the conversion from CEL to another query language is super useful in its own right. At another, it's highly valuable when used within residual computations where CEL handles some portion and the query engine handles the rest.

[achew22]

@TristonianJones, have got gotten a chance to look at this idea?

I've been considering how you would safely go about it. Is there more to it than defining the set of objects you allow the user to interact with, parsing it, running it through the checker, and finally walking the AST to translate between the two? Are there and unexpected sharp edges that I should be looking out for?

[TristonianJones]

Oh, I have not gotten to this yet. But I think the transformation should be pretty straightforward. The big issue would be if you need to have custom transforms that interact with the source-to-source translation for custom functions.

[mrparkers]

@TristonianJones @achew22 I'm really glad I stumbled upon this issue, this is actually something that my team and I have been working on for another project. We've implemented a package that parses a CEL filter and returns an Elasticsearch query that uses custom types we've implemented within that package (the Elasticsearch SDK does not provide strong types to work with out of the box).

So far, we have ==, !=, &&, ||, .startsWith(), and .contains(). You can check out our work here: https://github.com/rode/grafeas-elasticsearch/blob/main/go/v1beta1/storage/filtering/filtering.go

Right now, we're struggling a bit with adding array indexing (with or without wildcards), as seen here:

cel-go/interpreter/attribute_patterns.go

Lines 37 to 41 in 3ea8bd3

	// Examples:
	//
	// 1. ns.myvar["complex-value"]
	// 2. ns.myvar["complex-value"][0]
	// 3. ns.myvar["complex-value"].*.name

We're trying to learn CEL as we go, which has worked for us so far, but I imagine we've made some mistakes along the way with regards to the filter expressions we're testing against, and how we're parsing the AST 😅

If we could get some assistance with understanding how some of the more complex queries work under the hood, we'd be happy to pull this package out into its own Go module so others can use it / contribute to it.

Thanks!

[TristonianJones]

@mrparkers I noticed that you're playing with grafeas filtering. Have you by chance seen the filter grammar to CEL AST code that I put together long, long ago in a repo far, far away? You might find these to be useful reference points in your own work:
https://github.com/grafeas/grafeas/tree/master/go/filtering/parser
https://github.com/grafeas/grafeas/tree/master/go/filtering/parser/testdata

Admittedly, I hadn't expected to many people to dig into the AST representation of CEL, so there aren't a lot of docs on it. I'm happy to help however I can though.

The attribute_patterns are really intended for marking certain variable and field/element accesses as unknown, so not everything present in the pattern set is supported syntactically by CEL. The basic Expr nodes that the patterns work on though are ident, select, and call(_[_]) expressions. The nice part about any fully qualified expression composed only of those operators is that they surface in the env.Program step as InterpretableAttributes which you can inspect if you create a custom decorator: https://github.com/google/cel-go/blob/master/cel/cel_test.go#L1055. Detectors give good insight into the basic CEL operations of value lookups and function calls, but don't provide a lot of insights into when comprehensions or object creations are encountered during the program plan step. For the guts of how CEL creates its own internal program representation from the AST, the https://github.com/google/cel-go/blob/master/interpreter/planner.go is a wealth of information.

Hopefully, that helps for now. In the end, I would love for there to be a go module for CEL to ElasticSearch generation. I think the community would find that very helpful!

[achew22]

I've been playing with this and I was wondering, is there anything in the CEL language that allows you to specify sorting or an offset, or is that better expressed out of band of the query?