scan_folded_as_literal not behaving plausible with trailing whitespace

[bewee-i]

Config:

formatter:
    type: basic
    scan_folded_as_literal: true

Minimal reproducible example:

foo: >
  lorem ipsum 

(note there is a trailing whitespace after "lorem ipsum ")

is being formatted to

foo: "lorem ipsum \n"

It took me quite a while to figure out that this is the underlying reason why our 20+ line folded strings got turned into single lines. So I guess that this behaviour is for the very least misleading and probably also not desired.

[bewee-i]

This is not limited to trailing whitespaces in the last line of a folded string:

foo: >
  lorem 
  ipsum

is being formated to

foo: "lorem \nipsum\n"

[braydonk]

Thanks for opening an issue. This looks like it's probably caused by the same thing as I've been tracking in #86, which I haven't been able to resolve yet due to its complexity.

[mikelorant]

Can also confirmed this is happening with normal block scalars.

Before

block: |
  key: value (single whitespace appended)

After

block: "key: value \n"

Expected

block: |
  key: value (whitespace removed)

If there is no trailing whitespace, everything words as expected. As per @bewee-i any line with a trailing whitespace will cause this occur, it is not limited to the last line.

[braydonk]

Thank you for the additional example.

On the root issue #86 I added an explanation for why this bug occurs, but I'll copy that explanation here for visibility:

This is caused by the hack around the fact that yaml.v3 doesn't retain plain line-break information. yamlfmt will insert a magic string before being serialized into yaml.v3's node structure. Then after the new output is produced, we use the magic strings to put them back in. However, one place that newline information will actually be retained is in block scalars. So the magic line string being thrown on there messes with the serialization. I can't think of a way around this without somehow getting rid of the hack. I've tried in the past to fix this in yaml.v3 but I came up short. I've been wanting to build my own yaml parser instead to rid myself of yaml.v3 in general, and in that case this and many other fixes/features I've wanted to implement would be possible. Haven't had the time to invest to make that happen though.