Major Description
TestSecureShuffle is very slow. We need to experiment with it.
There is already an optimization like:
conf.setInt(TezRuntimeConfiguration.TEZ_RUNTIME_SHUFFLE_FETCH_FAILURES_LIMIT, 2);
Maybe it is worth checking if lowering the number of failed attempts can help. In case of shuffle failures, sooner or later the reducer and mapper task is retried, so lowering:
TezConfiguration.TEZ_AM_TASK_MAX_FAILED_ATTEMPTS
might help, theoretically, as we want the DAG to fail as fast as possible.
Attachments
Issue Links
- links to
