Three days back at
Google I/O 2013 Cloud Keynote conference, Google announced that they are getting
rid of initial MapReduce processing engine (published at early 2004). When followers are adopting MapReduce
paradigm into their products and services like Hadoop based platform or Processor
Core based parallel processing implementation, Google is already started
looking for next gen solution of parallel processing called
"FlumeJava". A paper was published by Google Lab on FlumeJava in
early 2010 by Craig Chambers, Ashish Raniwala, Frances Perry, Stephen Adams,
Robert R. Henry, Robert Bradshaw and Nathan Weizenbaum. In I/O' 13 conference
developer advocate of Cloud Compute Engine announced that they have successfuly
replaced MapReduce initial Engine and FlumeJava already is in production.
MapReduce and similar
systems significantly ease the task of writing data-parallel code. However,
many real-world computations require a pipeline of MapReduces, and programming
and managing such pipelines can be difficult. FlumeJava, a Java library that makes
it easy to develop, test, and run efficient data-parallel pipelines.
1. Parallel collections
and their operations present a simple, high-level, uniform abstraction over
different data representations and execution strategies.
2. To enable parallel operations
to run efficiently it is internally constructing an execution plan dataflow
graph
3. When final outcome of
parallel processing outcome is needed FlemeJava first optimizes operations of
underlying sub operations.
4. Combination of
high-level abstractions for parallel computation, deferred evaluation and
optimization, and efficient parallel sub-operations, easy-to-use system that
approaches the efficiency of optimized pipelines.
Please note that
FlumeJava is not a complementary tool of MapReduce, it's developed to optimize
several pipelined MapReduce jobs in an optimized way. Research shows that Graph
based Algorithmic optimization is far more optimized in terms of time
complexity than hand held optimization of several MapReduce processing.
I can say, yet another
tool that came out from Google's basket which will open new era of parallel
processing techniques which will encourage and will bring fresh air to dev
communities in near future..... :)
