mapreduce - Is spark more efficient than map-reduce for running a bunch of sequential processing tasks? -


i have mapreduce job works fine. wonder can benefit rewriting spark , significant difference, twice of previous performance example? mapreduce job doesn't have reducer transforms data, there no sophisticated computations, there no iterative operations on same input data, seems spark in-memory features useless in case.

what worries me mapreduce don't need set amount of memory , number of executors needed job done, calculated size of input data. while in case spark have use spark.executor.memory , num-executors, dynamic allocation has troubles on our cluster.


Comments