[HIVE-20103] WM: Only Aggregate DAG counters if at least one is used

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0, 4.0.0
• Fix Version/s: 4.0.0-alpha-1
• Component/s: llap
• Labels: Branch3Candidate
• Assignee: Gopal Vijayaraghavan
• Reporter: Gopal Vijayaraghavan
• Created: 06/Jul/18 07:02
• Updated: 17/Nov/22 08:54
• Resolved: 09/Jul/18 16:33

Description

        status = dagClient.getDAGStatus(EnumSet.of(StatusGetOpts.GET_COUNTERS), checkInterval);
        TezCounters dagCounters = status.getDAGCounters();
...
        if (dagCounters != null && wmContext != null) {
          Set<String> desiredCounters = wmContext.getSubscribedCounters();
          if (desiredCounters != null && !desiredCounters.isEmpty()) {
            Map<String, Long> currentCounters = getCounterValues(dagCounters, vertexNames, vertexProgressMap,
              desiredCounters, done);

Skip collecting DAG counters unless there at least one desired counter in wmContext.

The AM has a hard-lock around the counters, so the current jstacks are full of

   java.lang.Thread.State: RUNNABLE
        at java.lang.String.intern(Native Method)
        at org.apache.hadoop.util.StringInterner.weakIntern(StringInterner.java:71)
        at org.apache.tez.common.counters.GenericCounter.<init>(GenericCounter.java:50)
        at org.apache.tez.common.counters.TezCounters$GenericGroup.newCounter(TezCounters.java:65)
        at org.apache.tez.common.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:92)
        at org.apache.tez.common.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:104)
        - locked <0x00007efb3ac7af38> (a org.apache.tez.common.counters.TezCounters$GenericGroup)
        at org.apache.tez.common.counters.AbstractCounterGroup.aggrAllCounters(AbstractCounterGroup.java:204)
        at org.apache.tez.common.counters.AbstractCounters.aggrAllCounters(AbstractCounters.java:372)
        - eliminated <0x00007efb3ac64ee8> (a org.apache.tez.common.counters.TezCounters)
        at org.apache.tez.common.counters.AbstractCounters.incrAllCounters(AbstractCounters.java:357)
        - locked <0x00007efb3ac64ee8> (a org.apache.tez.common.counters.TezCounters)
        at org.apache.tez.dag.app.dag.impl.TaskImpl.getCounters(TaskImpl.java:462)
        at org.apache.tez.dag.app.dag.impl.VertexImpl.aggrTaskCounters(VertexImpl.java:1342)
        at org.apache.tez.dag.app.dag.impl.VertexImpl.getAllCounters(VertexImpl.java:1202)
        at org.apache.tez.dag.app.dag.impl.DAGImpl.aggrTaskCounters(DAGImpl.java:755)
        at org.apache.tez.dag.app.dag.impl.DAGImpl.getAllCounters(DAGImpl.java:704)
        at org.apache.tez.dag.app.dag.impl.DAGImpl.getDAGStatus(DAGImpl.java:901)
        at org.apache.tez.dag.app.dag.impl.DAGImpl.getDAGStatus(DAGImpl.java:940)
        at org.apache.tez.dag.api.client.DAGClientHandler.getDAGStatus(DAGClientHandler.java:73)

Issue Links

• relates to: TEZ-3967 DAGImpl: dag lock is unfair and can starve the writers