Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
0.9.2, 0.10.3
-
None
-
None
Description
A Tez DAG fails to initialize itself when an Application Master is timely preempted.
The problem typically happens with Map Join(Broadcast Hash Join) of Hive when the broadcast edge is multi-staged. In the following case, the smaller side includes one aggregation, and the condition is satisfied.
CREATE TABLE small AS SELECT 1 AS id; CREATE TABLE big AS SELECT 1 AS id UNION ALL SELECT 2 AS id UNION ALL SELECT 3 AS id; SELECT * FROM big JOIN (SELECT id, count(*) AS num FROM small GROUP BY id) s ON big.id = s.id
Once it happens, a retried AM fails to configure the Map Join vertex. In the following case, Map 1 never starts.
---------------------------------------------------------------------------------------------- VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED ---------------------------------------------------------------------------------------------- Map 2 .......... container SUCCEEDED 1 1 0 0 0 1 Reducer 3 ...... container SUCCEEDED 1 1 0 0 0 0 Map 1 container INITIALIZING -1 0 0 -1 0 0 ----------------------------------------------------------------------------------------------
Tez starts Map 2 and Map 1 once their splits are configured. The hang issue happens when an AM is retried before it starts Reducer 3.
Attachments
Attachments
Issue Links
- links to
