Go lacks an ergonomically composable idiomatic iterator utility library. Package it
builds
on top of a (rangefunc experiment)[https://go.dev/wiki/RangefuncExperiment] for
go 1.22.
The design goals are
- Minimal library providing just enough.
- Idiomatic Go: that means use generic functions and use
rangefunc
experiment under the hood. - Type safe: so use generics everywhere.
- Provide a composable API where practical
- Supports Go builtin types: slices/maps/channels
Non goals are
- Support of every iterator utility only because it exists in lo/lodash/Rust/Haskell/Scala/Python
- Every possible permutation of primitives provides by the
it
itself.
Don't forget to install go 1.22 and
export GOEXPERIMENT=rangefunc
The it
provides methods that can be chained together. These are a better show
case to developers.
n := []string{"aa", "aaa", "aaaaaaa", "a"}
slice := it.NewMapable[string, int](it.FromSlice(n)).
Map(func(s string) int { return len(s) }).
Index().
Filter2(func(index int, _ int) bool { return index <= 1 }).
Values().
Slice()
fmt.Println(slice)
// Output: [2 3]
All this can be done using simple functions and an explicit sequence passing. Go compiler will catch the unused variables and type mismatches helping the developer in this form too.
n := []string{"aa", "aaa", "aaaaaaa", "a"}
seq0 := it.FromSlice(n)
seq1 := it.Map(seq0, func(s string) int { return len(s) })
seq2 := it.Index(seq1)
seq3 := it.Filter2(seq2, func(index int, _ int) bool { return index <= 1 })
seq4 := it.Values(seq3)
slice := it.AsSlice(seq4)
fmt.Println(slice)
// Output: [2 3]
In order to limit the sequence, use a Filter
.
n := []string{"aa", "aaa", "aaaaaaa", "a"}
res := it.NewChain(it.FromSlice(n)).
Filter(func(s string) bool { return len(s) >= 2 }).
Filter(func(s string) bool { return len(s) >= 3 }).
Slice()
fmt.Println(res)
// Output: [aaa aaaaaaa]
n := []string{"aa", "aaa", "aaaaaaa", "a"}
s0 := it.FromSlice(n)
s1 := it.Filter(s0, func(s string) bool { return len(s) >= 2 })
slice := it.AsSlice(s1)
fmt.Println(slice)
// Output: [aa aaa aaaaaaa]
Every Go developer is familiar with iterating through a slice and two variable range form.
for index, value := range slice {}
The it
does implement the Index
/IndexFrom
functions, which adds the index into the sequence.
n := []string{"aa", "aaa", "aaaaaaa", "a"}
s0 := it.FromSlice(n)
for index, value := range it.Index(s0) {
fmt.Println(index, value)
}
// Output:
// 0 aa
// 1 aaa
// 2 aaaaaaa
// 3 a
The way how this is implemented is that it
returns iter.Seq2[int, T]
. This
type uses functions suffixed by 2
. That means Filter2
is used in this
example. The Values
functions returns the second value later on.
n := []string{"aa", "aaa", "aaaaaaa", "a"}
res := it.NewChain(it.FromSlice(n)).
Index().
Filter2(func(index int, _ string) bool { return index <= 1 }).
Values().
Slice()
fmt.Println(res)
// Output: [aa aaa]
Map transforms one type into another. This is easy to do in Go as a simple
function. Much harder to do via method. The Mapable
allows the developer to
use a Map
in a method chain.
n := []string{"aa", "aaa", "aaaaaaa", "a"}
// maps string->int and int->string
res := it.NewMapable[string, int](it.FromSlice(n)).
Map(func(s string) int { return len(s) }).
Filter(func(i int) bool { return i >= 2 }).
Map(func(i int) string { return "string(" + strconv.Itoa(i) + ")" }).
Slice()
fmt.Println(res)
// Output: [string(2) string(3) string(7)]
There are only two drawbacks
- Developer has to specify type parameters in advance
- It supports only two types - supporting more would lead to confusing API
However good old functions have no such limitation and can be used instead.
n := []string{"aa", "aaa", "aaaaaaa", "a"}
// maps string->int->float32
s0 := it.FromSlice(n)
s1 := it.Map(s0, func(s string) int { return len(s) })
s2 := it.Map(s1, func(i int) float32 { return float32(i) })
s3 := it.Map(s2, func(f float32) string { return strconv.FormatFloat(float64(f), 'E', 4, 32) })
slice := it.AsSlice(s3)
fmt.Println(slice)
// Output: [2.0000E+00 3.0000E+00 7.0000E+00 1.0000E+00]
Sometimes there is no 1:1 transformation between T
and V
and the mapping
can fail. The it
provides a mapping function from iter.Seq[T]
into
iter.Seq2[K, V]
, which can solve this problem.
n := []string{"forty-two", "42"}
s0 := it.FromSlice(n)
s1 := it.MapSeq2(s0, strconv.Atoi)
for value, error := range s1 {
fmt.Println(value, error)
}
// Output:
// 0 strconv.Atoi: parsing "forty-two": invalid syntax
// 42 <nil>
Since this is a very common operation in Go the specialised MapError
function
exists as an exception to the permutation rule above.
n := []string{"forty-two", "42"}
s0 := it.FromSlice(n)
s1 := it.MapError(s0, strconv.Atoi)
for value, error := range s1 {
fmt.Println(value, error)
}
// Output:
// 0 strconv.Atoi: parsing "forty-two": invalid syntax
// 42 <nil>
And can be done inside a method chain too.
n := []string{"forty-two", "42"}
c := it.NewMapable[string, int](it.FromSlice(n)).
MapError(strconv.Atoi)
for value, error := range c.Seq2() {
fmt.Println(value, error)
}
// Output:
// 0 strconv.Atoi: parsing "forty-two": invalid syntax
// 42 <nil>
Reduce is a common functional operation, except it returns a single value. It
allows the developer to implement operation Count
.
m := []int{1, 2, 3, 4, 5, 6, 7}
count := it.NewChain(it.FromSlice(m)).
Reduce(func(a, _ int) int { return a + 1 }, 0)
fmt.Println(count)
// Output: 7
All other operations can work on a single item at the time. Not sort - it first pulls all items to the slice, sorts them and then pushes the values to the iterator.
n := []string{"aa", "aaa", "aaaaaaa", "a"}
s0 := it.FromSlice(n)
s1 := it.Sort(s0, func(slice []string) { slices.SortFunc(slice, strings.Compare) })
slice := it.AsSlice(s1)
fmt.Println(slice)
// Output: [a aa aaa aaaaaaa]
It accepts type SortFunc[T any] func([]T)
instead of a less
function. This
allows the developer to specify exactly how the sequence should be
sorted. For example use a slices.SortStableFunc
to get a stable sort.
n := []string{"aa", "aaa", "aaaaaaa", "a"}
s0 := it.FromSlice(n)
s1 := it.Sort(s0, func(slice []string) { slices.SortStableFunc(slice, strings.Compare) })
slice := it.AsSlice(s1)
fmt.Println(slice)
// Output: [a aa aaa aaaaaaa]
Simply iterate backward - it must collect the slice first.
n := []string{"aa", "aaa", "aaaaaaa", "a"}
s0 := it.FromSlice(n)
s1 := it.Sort(s0, func(slice []string) { slices.SortFunc(slice, strings.Compare) })
s2 := it.Reverse(s1)
slice := it.AsSlice(s2)
fmt.Println(slice)
// Output: [aaaaaaa aaa aa a]
Most operations does have the alternative working on iter.Seq2[K, V]
. In
order to distinguish between the types, all functions and structs has a suffix 2
, so
it is clear if method works with a single value or a pair.
Filtering
m := map[string]int{"one": 0, "two": 1, "three": 2}
s0 := it.From2Slice(m)
s1 := it.Filter2(s0, func(k string, v int) bool { return v >= 1 })
s2 := it.Sort2(s1, func(slice []string) { slices.SortFunc(slice, strings.Compare) })
for k, v := range s2 {
fmt.Println(k, v)
}
// Output:
// three 2
// two 1
Note the sort is mandatory - the order of a range loop was changed from time to time.
m := map[string]int{"one": 0, "two": 1, "three": 2}
s0 := it.From2Slice(m)
s1 := it.Map2(s0, func(k string, v int) (int, string) { return v, k })
s2 := it.Sort2(s1, slices.Sort)
for k, v := range s2 {
fmt.Println(k, v)
}
// Output:
// 0 one
// 1 two
// 2 three
Sometimes developer only one of K
, V
, so Keys
and Values
do not
have 2
suffix as they return itre.Seq
.
m := map[string]int{"one": 0, "two": 1, "three": 2}
s0 := it.From2Slice(m)
s1 := it.Keys(s0)
s2 := it.Sort(s1, slices.Sort)
for s := range s2 {
fmt.Println(s)
}
// Output:
// one
// three
// two
m := map[string]int{"one": 0, "two": 1, "three": 2}
s0 := it.From2Slice(m)
s1 := it.Values(s0)
s2 := it.Sort(s1, slices.Sort)
for n := range s2 {
fmt.Println(n)
}
// Output:
// 0
// 1
// 2
And the developer can get values back as a map - note the K
must be
comparable
otherwise the type system will not allow one to construct a map. This
is the reason Chain2
does not have AsMap
method. Doing so would impose the
constraint to both K
and V
and limiting the usability of a Chain2
.
invalid map key type K (missing comparable constraint)
m := map[string]int{"one": 0, "two": 1, "three": 2}
s0 := it.From2Slice(m)
s1 := it.Filter2(s0, func(_ string, v int) bool { return v == 2 })
m2 := it.AsMap(s1)
for k, v := range m2 {
fmt.Println(k, v)
}
// Output:
// three 2
All operations above can be chained.
m := map[string]int{"one": 0, "two": 1, "three": 2}
chain2 := it.NewChain2(it.From2Slice(m)).
Filter2(func(_ string, v int) bool { return v == 2 })
for k, v := range chain2.Seq2() {
fmt.Println(k, v)
}
// Output:
// three 2
Some crazy and not so crazy ideas to explore
As suggested by jerf on Reddit, it is interesting to compare rangefunc with plain range
loop. First set of benchmarks was added in #8
One of the coolest (keep in mind it was a late night one) ideas may be breaking the chain. The prototype exists in ideas_test.go, just not sure if it is actually a good idea. It is definitely doable and possible in Go.
package it_test
import (
"fmt"
"github.com/gomoni/it"
)
type pusher struct {
stack chan string
}
func (y *pusher) push(s string) {
y.stack <- s
}
func (y pusher) seq() func(func(string) bool) {
return func(yield func(string) bool) {
for {
select {
case s, open := <-y.stack:
if !open || !yield(s) {
return
}
}
}
}
}
func (y pusher) wait() {
<-y.stack
}
func Example_break_da_chain() {
n := []string{"aa", "aaa", "aaaaaaa", "a"}
// create a method chain
chain := it.NewChain(it.FromSlice(n)).
Filter(func(s string) bool { return true })
// break it - with some syntax sugar
p := pusher{stack: make(chan string)}
defer p.wait()
go func() {
defer close(p.stack)
for s := range chain.Seq() {
p.push(s)
}
}()
// continue here
chain2 := it.NewChain(p.seq()).
Filter(func(s string) bool { return len(s) > 2 })
slice := chain2.Slice()
fmt.Println(slice)
// Output: [aaa aaaaaaa]
}
Some options
- don't do that
- provide
FilterContext
et all it/itctx
package
See https://go.dev/blog/deconstructing-type-parameters and func Clone[S ~[]E, E any](s S) S
. The it
has the same problem
func PrintSorted(ms MySlice) string {
c := it.NewChain(it.FromSlice(ms)).Sort(slices.Sort).Slice()
- return c.String() // FAILS TO COMPILE
+ return MySlice(c).String() // this works
}
This is not a solution as the original S
got lost.
func fromSlice[S ~[]E, E any](slice S) iter.Seq[E] {
Possible solutions:
- ignore it - it is easy to go back to the original type
- do not return iter.Seq and have a wrapper structure of type
Seq[S ~[]E, E any]
Inspiration and other cool projects.
https://pkg.go.dev/github.com/samber/lo
Is the most used lodash-style library for Go.
Pros
- most favorite
- type safe due usage of generics
- one can iterate over slices, maps or channels
- helper function on everything
Cons
- operate on top of simple functions
- each pass allocates new slice or map
- methods can't be chanined together
- every callback got mandatory int argument
An example is FilterMap
function. It can be implemented as Filter
and
Map
, yet it's not easy to do in lo
https://pkg.go.dev/github.com/samber/lo#pkg-functions
https://github.com/novalagung/gubrak
Less popular, provides nicer looking API than lo. Implements own iterator type,
so methods can be arbitrary chained. Harder to use due prevalent interface{}
usage.
Pros
- nicer API
- methods can be chained
- provides own iterator type
Cons
- not type safe due usage of
interface{}
- API actually harder to use
- https://docs.python.org/3/library/itertools.html
- https://doc.rust-lang.org/book/ch13-02-iterators.html
- https://hackage.haskell.org/package/base-4.19.0.0/docs/Data-List.html#g:2
Documentation generated using https://github.com/dave/rebecca
go install github.com/dave/rebecca/cmd/becca@latest
becca -package github.com/gomoni/it