Collection of ideas for new functions

[sharkdp]

• More string utilities (e.g. regex functions)
• Formatting functions (echo 3.1415 | map format "{:.2f}")

[cylfoo]

String

• search and match based on regex
• Add regex support for replace input
• Add function endwith a pattern

Arithmetic

• Add mod, pow, div and log etc. functions

Filesystem

• Add function mv to move and rename files
• More functions to filter files based on file mode bits and ownship

[sharkdp]

@LeeLeeee Sounds great. Especially the mv function, which will be really useful (we should add cp as well)

[cylfoo]

@sharkdp What do you think if we can add du or something like this?

[sharkdp]

@sharkdp What do you think if we can add du or something like this?

In which sense? Like a filesize :: Path -> Int function?

---

By the way, I just realized that the coreutils du commands works very nicely with functools:

> du | filter -c1 greater_than 8000 | map -c2 abspath
8340	/tmp/demo/folder/subdirectory
8408	/tmp/demo/folder
8480	/tmp/demo

[cylfoo]

Yeah, more precisely it is filesize :: Path -> String.

$ find . | map basename | filter startswith test_ | map filesize mb 
test_file1 1MB 
test_file2 5.2MB 

It is fantastic that linux commands are work very well with these functions. 👍

[sharkdp]

filesize is now supported.

[sharkdp]

• fst and snd for array-index access (alias for index 1 and index 2)
• replicate
• reverse

[ghost]

Does split currently work with variable size of whitespace between words? if not, does it make sense to add it? Something like below by default if no separator is provided, then it would be a nice replacement for awk

➜  ~ echo "a b         ccc " | ~/repos/shell-functools/ft/map split | map at 2
ccc

[sharkdp]

Does split currently work with variable size of whitespace between words?

It does not (yet). We could either provide a split_whitespace function which would simply call .split() without argument and/or provide a split_regex function which could be called with map split_regex '\s+'.

[vegarsti]

The reverse function, do you mean on array level, or on string level?

[sharkdp]

I think array-reverse can be useful. Not sure if we need a string-reverse function.

[guilhermeleobas]

Hey @sharkdp, what about creating a max as well as min function in shell-functools? I can work on something if there is interest.

max :: Array -> Int
find . -name cnt.txt | xargs cat | max

[sharkdp]

@guilhermeleobas Sounds great!

[Jake6329]

These days, new unix/linux CLI tools are coming out supporting JSON outputs. Would it make sense to create a json function that can parse JSON outputs?

[sharkdp]

These days, new unix/linux CLI tools are coming out supporting JSON outputs. Would it make sense to create a json function that can parse JSON outputs?

I've been thinking about JSON support in the past, so this sounds interesting. What would this function do, exactly?

[Jake6329]

I can imagine there would a command named json which could be used like this:

$ curl http://echo.jsontest.com/key1/value1 | json key1

The curl command outputs {"key1": "value1"}. The whole command would output value1. The argument to json command would specify the path to the value you want. It could support the full json path spec or just a small part of it.

And then there would also a function called json which could be used like this:

$ curl <url that returns json in each line> | map json <json path>

In this case, the assumption is that the curl outputs a json-encoded string in each line, which is also a proposed standard format called jsonlines. And the map will transform each line with the json function.

$ <command that outputs json> | json <path to an array of objects> | map json <path in each object>

In the above command, the json command narrows data down to an array of objects, and then map json transforms each json into something. I think the path you specify could be some kind of template so that you can transform json into anything, which sounds a bit too ambitious.

[sharkdp]

Do you know jq? I think this could probably play well in combination with shell-functools.

[Jake6329]

@sharkdp jq seems to cover my use cases. Cool, thanks.

[sharkdp]

More functions that should be implemented:

• div/divide
• sum/product (working on arrays)
• matches => similar to starts_width/ends_with

[jdhedden]

Offering

For README.md:

### Usage of `lambda`

The `lambda` *function* allows you to create custom functions using a
definition string to Python's *lambda* function.
``` bash
> seq 3 | map lambda 'x : x+x'
2
4
6

> echo -e 'a:b\n2:4' | map split ':' | map lambda 'x, y : x+y'
ab
6

> seq -3 3 | map lambda 'x : abs(x)'
3
2
1
0
1
2
3

> echo -e "uppercase\nLOWERCASE\ncAMELcASE" | map lambda "x : x.swapcase()"
UPPERCASE
lowercase
CamelCase

For ft/ft/functions.py:

@register("lambda")
@typed(None, None)
def ft_lambda(definition, inp):
    _lambda = 'lambda ' + definition.value
    if type(inp) == list:
        args = list(map(lambda v: v.value, inp))
        return [TypedValue(str((eval(_lambda))(*args)), T_STRING)]
    else:
        return (eval(_lambda))(inp)

[jdhedden]

Offering

For README.md:

### Usage of `shell`

The `shell` *function* allows you to create custom functions using arbitrary
*shell* commands.  If the command string contains pairs curly braces `{}`, the
arguments to the function are formatted into it, otherwise they are appended
to it.
``` bash
> /usr/bin/ls -1 ft/ft/*.py | map shell 'grep -H -c def' | map split ':' | map format '{} has {} functions'
ft/ft/command.py has 14 functions
ft/ft/error.py has 1 functions
ft/ft/functions.py has 60 functions
ft/ft/__init__.py has 0 functions
ft/ft/internal.py has 4 functions
ft/ft/termcolor.py has 2 functions
ft/ft/test_command.py has 5 functions
ft/ft/types.py has 21 functions
ft/ft/version.py has 0 functions

> echo -e 'a\tb\n2\t4' | map shell 'echo "{} and {}"'
a and b
2 and 4

Total lines of text

find | filter is_file | map shell 'cat {} | wc -l' | foldl1 add
5040

For ft/ft/functions.py:

@register("shell")
@typed(None, None)
def shell(command, inp):
    if type(inp) == list:
        args = list(map(lambda v: str(v.value), inp))
    else:
        args = [str(inp)]

    command = command.value
    if '{}' in command:
        command = command.format(*args)
    else:
        command += ' '+' '.join(args)

    result = os.popen(command).read().rstrip('\r\n')

    if type(inp) == list:
        return [TypedValue(result, T_STRING)]
    else:
        return result

[jdhedden]

Offering

For README.md:

### Usage of `regex`

The `regex` *function* permits filtering based on Python supported regular
expressions.  For case-insensitive filtering, use the `(?i:...)` syntax:
``` bash
> /usr/bin/ls | filter regex '(?i:[a-c]+)'
LICENSE
make
path
README.md

For ft/ft/functions.py:

import re

@register("regex")
@typed(T_STRING, T_BOOL)
def contains(pattern, inp):
    pattern = dynamic_cast(T_STRING, pattern).value
    return re.search(pattern, inp)

[jdhedden]

Offering of new command array to create or flatten arrays

For README.md:

### Usage of `array`

The `array` command is used to take *lines of input*, and convert them into an
`array`.
``` bash
> seq 3 | array
1	2	3

Given an numerical argument, array will make arrays with that number of
columns:

> seq 12 | array 5
1	2	3	4	5
6	7	8	9	10
11	12

Given an argument of '0' (or a non-numerical argument), array will convert
arrays back into lines of input:

> echo -e 'aa\tbb\tcc\tdd' | array 0
aa
bb
cc
dd

For ft/array (new):

#!/usr/bin/bash

if [[ -z $1 || $1 -gt 0 ]]; then
    if [[ -n $1 ]]; then
        max=$1
    else
        max=1000000000000000
    fi

    cnt=0
    while read; do
        if [[ $cnt -gt 0 ]]; then
            result+=$'\t'
        fi
        result+=$REPLY

        (( cnt++ ))
        if [[ $cnt -eq $max ]]; then
            echo "$result"
            result=
            cnt=0
        fi
    done

    # Remaining partial line
    if [[ -n $result ]]; then
        echo "$result"
    fi

else
    # array 0
    sed 's@\t@\n@g'
fi

# EOF

[goyalyashpal]

hi! can u expand on that? like what would these do or what would be their input be like.

• div/divide
• sum/product (working on arrays)
  - @ sharkdp at Collection of ideas for new functions #7 (comment)