mapreduce - Is spark more efficient than map-reduce for running a bunch of sequential processing tasks? -
i have mapreduce job works fine. wonder can benefit rewriting spark , significant difference, twice of previous performance example? mapreduce job doesn't have reducer transforms data, there no sophisticated computations, there no iterative operations on same input data, seems spark in-memory features useless in case.
what worries me mapreduce don't need set amount of memory , number of executors needed job done, calculated size of input data. while in case spark have use spark.executor.memory
, num-executors
, dynamic allocation has troubles on our cluster.
Comments
Post a Comment