Go-mapreduce

tl;dr

I've put together an easy to use Map/Reduce package for Go. It manages the creation of background workers (goroutines) and the channels that connect them.

Go-mapreduce

go-mapreduce is a simple Go package that makes it super easy to apply the Map/Reduce pattern in Go using goroutines and channels.

In order to use the package you'll need to implement both a map function and a reduce function (implement interfaces) and then pass those into the Run function.

Currently it only allows you to specify the number of mappers as part of the configuration passed into Run, but I plan to build it out to be more intelligent (dynamically throttle workers, report statistics etc...).

Example

import (  
    "fmt"
    gmr "github.com/ahamidi/go-mapreduce"
)

type MapReducer struct{}

// Example Map Function
func (mr *MapReducer) map(in chan interface{}, out chan interface{}) {  
    for v := range in {
        out <- v
    }
}

// Example Reduce Function
func (mr *MapReducer) reduce(in chan interface{}) interface{} {  
    res := 0
    for v := range in {
        res += v.(int)
    }
    return res
}

func main() {  
    conf := gmr.NewMapReduceConfig()

    // Feed input channel
    go func(in chan interface{}) {
        for i := 0; i < 100; i++ {
            in <- i
        }
        close(in)
    }(conf.InChan)

    var mr MapReducer

    result, _ := gmr.Run(&mr, conf)

    fmt.Println(result)
    // Output:
    // 4950
}