I wish I could use yq -s with an external file

[symbl-ay]

I wish I could use yq -s with an external file.

Expressions passed to -s can sometimes get quite involved. If I could store them in a file, and refer to them like I do with yq --from-file, these split expressions can be written in a way that is easier to document and understand.

Current yq:

$: yq --from-file yq.input input.yml -s '(["yml/country", ([$index] | join("") | match("[[:digit:]]+") | ("000" | split("")) *+ (.string | split("")) | [.[-3, -2, -1]] | join("")), (.country | sub("[^[:word:][:space:]]", "") | split(" ") | join("-")), "exports"] | join("-"))'

Desired yq:

$: yq --from-file yq.input input.yml -S yq.split-files
# or...
$: yq --from-file yq.input input.yml --split-from-file yq.split-files

Describe alternatives you've considered

Given some Makefile with these dependencies, which are normal files:

yml: input.yml yq.input yq.split-files
	mkdir -p $@
	$(eval YQ_SPLIT=$(shell sed 's|\$$|\$$$$|g' yq.split-files))
	yq --from-file yq.input $< -s '$(YQ_SPLIT)'

I could create an input.yml file like this:

exports:
- country: Australia
  export_units:
  - trailer
  - tray
  export_goods:
  - meat
- country: United States
  export_units:
  - freedom
  export_goods:
  - freedom

I might use something like this for my yq.input:

.exports[]

And for my yq.split-files:

(
  [
    "yml/country",
    ([$index] | join("") | match("[[:digit:]]+") | ("000" | split("")) *+ (.string | split("")) | [.[-3, -2, -1]] | join("")),
    (.country | sub("[^[:word:][:space:]]", "") | split(" ") | join("-")),
    "exports"
  ] | join("-")
)

And see that there are two files created, yml/country-000-austrailia-exports.yml and yml/country-001-united-states-exports.yml.

Having all my inputs to yq resting on the filesystem allows for me to keep my transformed output up to date with every change to both of these two yq command input "files", which is nice.

Additional context

Formats for style="..." that can transform number inputs into zero-padded (scientifically notated, integer/float strings, etc.) would make this approach a lot less code to write.

[mikefarah]

You can do this already, see https://mikefarah.gitbook.io/yq/usage/split-into-multiple-files

[symbl-ay]

If I try to do that, I get an error. I'm on version 4.24.2.

$> yq --from-file yq.input -s yq.split-files
Error: bad split document expression: parsing expression: Lexer error: could not match text starting at 1:1 failing at 1:2.
        unmatched text: "y"

[mikefarah]

Ah gotcha, an external file 👍🏼

[mikefarah]

Added --split-exp-file flag to the latest release (4.25.1) - give it a try and let me know how it goes for you

[symbl-ay]

[symbl-ay]

Works great thanks for adding this.