Help required to design a good parser

[renard]

I am willing to create a F5 configuration parser using participle.

I start with simple object (ltm node, as described here https://clouddocs.f5.com/cli/tmsh-reference/latest/modules/ltm/ltm_node.html).

I have no experience in writing parsers and the existing examples do not cover complexe cases (or I didn't understand it :-) ).

For now I am testing a basic subset of ltm node with:

ltm node /Common/10.2.3.4 {
    description "some random text"
    address 10.2.3.4
}

ltm node "/Common/name with space" {
    address 10.2.3.4
    description MyDescription
    fqdn {
        address-family all
        autopopulate enabled
        name host.example.com
    }
}

and

ltm node host.example.com {
    # this is a comment
    address 10.2.3.4
    description "A description can contain (almost) any characters!" # and comment can be in-lined.
    fqdn {
        address-family all
        autopopulate enabled
        name host.example.com
    }
}

For code simplicity and usage I already split each config block in a single string to process on its own (in other word I do not want the parser to process the whole file, but just some parts of it)

I started with a stateless lexer but I quickly ended up with some errors mainly because of description strings which can confuse other tokens.

I guess having a stateful parser will help but I am not able to define it correctly.

Any help would be highly appreciated.

Once I have a simple working example I can do all other parts.

Thanks in advance.

Here is my stateless attempt:

type LtmNode struct {
	Pos     lexer.Position
	Name    string   `"ltm" "node" ( @F5Name | @QF5Name ) "{"`
	Description   string   ` | "Description" @( F5String | QF5string )`
	Address string   `("address" @Ident`
	Logging *Boolean ` | "logging" @("enabled" | "disabled")`
	FQDN    *FQDN    `| "fqdn" "{" @@ "}" )+ "}"`
}

type FQDN struct {
	AddressFamily string   `( "address-family" @Ident`
	Autopopulate  *Boolean ` | "autopopulate" @("enabled" | "disabled")`
	Name          string   ` | "name" @Ident`
	Interval      int      ` | "interval" @Ident`
	DownInterval  int      ` | "down-interval" @Ident )+`
}

var myLexer = lexer.MustSimple([]lexer.Rule{
	{"comment", `#[^\n]*`, nil},
	{"whitespace", `\s+`, nil},
	{"EOL", `[\n\r]+`, nil},
	{"Punct", `[{}]`, nil},
	// {"Ident", `[A-Za-z0-9-]+`, nil},

	{"QF5Name", `"/[A-Za-z0-9/. -]*[A-Za-z0-9/]+"`, nil},
	{"F5Name", `/[A-Za-z0-9/. -]*[A-Za-z0-9/]+`, nil},

	{"Ident", `\b[A-Za-z0-9.:-]+\b`, nil},
	{"QF5String", `".+"`, nil},
	{"F5String", `.+`, nil},

	// {"address", `[A-Za-z0-9][A-Za-z0-9.-]*`, nil},
},
	lexer.MatchLongest(),
)

	obj := &LtmNode{}
	parser := participle.MustBuild(
		obj,
		participle.Lexer(myLexer),
		participle.Unquote("QF5Name"),
		participle.Unquote("QF5String"),
	)

In the same vain here is a statefull attempt:

type LtmNode struct {
	Pos  lexer.Position
	Name string `@LtmNodeName`
	//Description      `"descrip
	Address string   `"address" @Ident`
	Logging *Boolean ` | "logging" @("enabled" | "disabled")`
	FQDN    *FQDN    `| "fqdn" "{" @@ "}" )+ "}"`
}

type FQDN struct {
	AddressFamily string   `( "address-family" @Ident`
	Autopopulate  *Boolean ` | "autopopulate" @("enabled" | "disabled")`
	Name          string   ` | "name" @Ident`
	Interval      int      ` | "interval" @Ident`
	DownInterval  int      ` | "down-interval" @Ident )+`
}

var myLexer = lexer.MustStateful(lexer.Rules{
	"Root": {
		{"LtmNode", `ltm\bnode\b(.+)\b{\b`, lexer.Push("LtmNode")},
		lexer.Include("formating"),
	},
	"formating": {
		{"whitespace", `\s+`, nil},
		//{"EOL", `[\n\r]+`, nil},
		//{"Ident", `\w+`, nil},
	},
	"LtmNode": {
		{"LtmNodeName", `\1`, nil},
		{"LtmNodeEnd", `\b}\b`, lexer.Pop()},
		lexer.Include("formating"),
	},
})

[alecthomas]

It's a pretty good start, you were almost there. The main issue was that your attributes needed to be parsed into a slice. Here's working code that parses all the example inputs you gave.

[renard]

Thanks for the very quick response. You are fantastic!

Just to be sure to understand correctly, why a NodeAttr is required to host all attributes? Could this be possible by just using a LtmNode structure and moving NodeAttr fields in it?

[alecthomas]

No because what you're trying to express is "a sequence of (one of X, Y, or Z) in any order". You cannot express that with just "X, Y, or Z", which is what your suggested structure would provide.

[renard]

Ok fair enough. I think I have all I need to go on. Thanks again for the efficient help.

[renard]

For the record I managed to make NodeAttr into LtmNode: https://go.dev/play/p/DDAgE_A91Hk

Don't know if this is a good practice or not but this solves my needs.

instead of generating

&main.LtmNode{
  Pos: Position{Filename: "example2.f5", Offset: 1, Line: 2, Column: 1},
  Name: "/Common/host.example.com",
  Attrs: []*main.LtmNodeAttr{
    &main.LtmNodeAttr{
      Address: "10.2.3.4",
    },
    &main.LtmNodeAttr{
      Description: "A description can contain (almost) any characters!",
    },
    &main.LtmNodeAttr{
      FQDN: &main.FQDN{
        AddressFamily: "all",
        Autopopulate: &true,
        Name: "host.example.com",
      },
    },
  },
}

It generates

&main.LtmNode{
  Pos: Position{Filename: "example3.f5", Offset: 1, Line: 2, Column: 1},
  Name: "/Common/host.example.com",
  Description: "A description can contain (almost) any characters!",
  Address: "10.2.3.4",
  FQDN: &main.FQDN{
    AddressFamily: "all",
    Autopopulate: &true,
    Name: "host.example.com",
  },
}

Which is exactly what I need and fields can be defined in any order. One drawback, if a field is present twice its content is concatenated (but that is not a big deal for now).