[TEZ-4541] Remove or limit evergrowing DAG collections from DAGAppMaster

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: László Bodor
• Reporter: László Bodor
• Created: 20/Feb/24 13:57
• Updated: 16/Apr/24 11:25

Description

TEZ-1495 introduced a dag id collection (here) to track all dag ids only to be able to give a different response ( ? )

https://github.com/apache/tez/blob/f8c2e11d0b469748ea95381e7021266e25e5ac89/tez-dag/src/main/java/org/apache/tez/dag/api/client/DAGClientHandler.java#L101-L111

    if (!currentDAGIdStr.equals(dagIdStr)) {
      if (getAllDagIDs().contains(dagIdStr)) {
        LOG.debug("Looking for finished dagId {} current dag is {}", dagIdStr, currentDAGIdStr);
        throw new DAGNotRunningException("DAG " + dagIdStr + " Not running, current dag is " +
            currentDAGIdStr);
      } else {
        LOG.warn("Current DAGID : " + currentDAGIdStr + ", Looking for string (not found): " +
            dagIdStr + ", dagIdObj: " + dagId);
        throw new TezException("Unknown dagId: " + dagIdStr);
      }
    }

I can see that DAGNotRunningException is used by the DAGClientImpl to handle edge cases (infer dag completion if the dag is not present as current dag but present in the dag ids collection), which is fine, so maybe instead of removing this collection we might want to limit its size, e.g. to 500, to make DAGAppMaster respond as expected for a certain amount of time (hence not breaking current contract)

Issue Links

• is related to: TEZ-1495 ATS integration for TezClient
• links to: GitHub Pull Request #346