Friday, July 29, 2011

Map Reduce Example in F#

A while back I blogged about the canonical map reduce example (as seen in the Hadoop user manual) of counting words. Today I noticed that an alias for F#'s List.fold function is List.reduce*. I had already seen, so I put two and two together, and pretty soon I had a tiny program up and running in F# Interactive. Enjoy:
/// maps an input into a tuple of input and count
let mapper x =
(x, 1)

/// reduces a list of input and count tuples, summing the counts
let reducer (a:(string * int) list) (x:string * int) =
if List.length a > 0 && fst (List.head a) = fst x then
(fst (List.head a), snd (List.head a) + snd x)::List.tail a

/// maps the mapper function over a list of input
let map xs = (mapper) xs

/// maps the reducer function over a list of input and count tuples
let reduce (xs:(string * int) list) =
List.fold (reducer) [] xs

/// maps the input, sorts the intermediate data, and reduces the results
let mapReduce xs =
map xs
|> List.sort
|> reduce

It turns out the functional programming appears to be made for this pattern!

No comments: