Emit new line tokens

[hbagdi]

Thank you for creating participle!
I've the following lexer:

var hitLexer = lexer.MustSimple([]lexer.SimpleRule{
	{"comment", `[#][^\n]*`},
	{"Ident", `[a-zA-Z_]\w*`},
	{"Path", `/.*`},
	{"Punct", `[[!@#~$%^&*()+_={}\|:;"'<,>.?/]|]`},
	{"whitespace", `[\s]+`},
	{"Marker", `[\-]{3}`},
})

And the following grammar:

type File struct {
	Requests []*Requests `@@*`
}

type Requests struct {
	Id           string `"---" "@"@Ident`
	Method       string `@Ident?`
	Path         string `@Path `
	BodyEncoding string `("~"@Ident)?`
	Body         string `@!Marker* Marker`
}

With the following input:

---
@c3
post
/foo
~yaml
foo: bar
bar: baz
---

This parses to:

&main.File{
  Requests: []*main.Requests{
    {
      Id: "c3",
      Method: "post",
      Path: "/foo",
      BodyEncoding: "yaml",
      Body: "foo:barsf",
    },
  },
}

I need to capture "\n" tokens in the body to preserve formatting, do I need a stateful parser for this and is there an pointer that you could share?

[alecthomas]

Apologies for the very belated response, I somehow missed this.

A couple of things:

1. I suspect your whitespace token is discarding newlines.

2. Once you are able to lex \n, I think your best bet would be to either capture the newline into the corresponding field, or add eg.

    BodyEncodingNL bool `@"\n"`
   

[hbagdi]

I've been trying for a very long time but I can't seem to parse the new line token correctly.
My code is here: https://gist.github.com/hbagdi/9c3a3384b80f396c77348b5d29a93db1

@alecthomas Could you please take a look and let me know what am I doing wrong?
Really appreciate your help.

[alecthomas]

This doesn't look correct:

	Id     string `"@"@Ident`
	NL     string `| @NL`
	Method string `@Ident`
...

If you want to match alternatives in this case you'll need to use () to group the tokens accordingly. It's similar to regex in that regard.