Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-20103

WM: Only Aggregate DAG counters if at least one is used

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0, 4.0.0
    • 4.0.0
    • llap

    Description

              status = dagClient.getDAGStatus(EnumSet.of(StatusGetOpts.GET_COUNTERS), checkInterval);
              TezCounters dagCounters = status.getDAGCounters();
      ...
              if (dagCounters != null && wmContext != null) {
                Set<String> desiredCounters = wmContext.getSubscribedCounters();
                if (desiredCounters != null && !desiredCounters.isEmpty()) {
                  Map<String, Long> currentCounters = getCounterValues(dagCounters, vertexNames, vertexProgressMap,
                    desiredCounters, done);
      

      Skip collecting DAG counters unless there at least one desired counter in wmContext.

      The AM has a hard-lock around the counters, so the current jstacks are full of

         java.lang.Thread.State: RUNNABLE
              at java.lang.String.intern(Native Method)
              at org.apache.hadoop.util.StringInterner.weakIntern(StringInterner.java:71)
              at org.apache.tez.common.counters.GenericCounter.<init>(GenericCounter.java:50)
              at org.apache.tez.common.counters.TezCounters$GenericGroup.newCounter(TezCounters.java:65)
              at org.apache.tez.common.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:92)
              at org.apache.tez.common.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:104)
              - locked <0x00007efb3ac7af38> (a org.apache.tez.common.counters.TezCounters$GenericGroup)
              at org.apache.tez.common.counters.AbstractCounterGroup.aggrAllCounters(AbstractCounterGroup.java:204)
              at org.apache.tez.common.counters.AbstractCounters.aggrAllCounters(AbstractCounters.java:372)
              - eliminated <0x00007efb3ac64ee8> (a org.apache.tez.common.counters.TezCounters)
              at org.apache.tez.common.counters.AbstractCounters.incrAllCounters(AbstractCounters.java:357)
              - locked <0x00007efb3ac64ee8> (a org.apache.tez.common.counters.TezCounters)
              at org.apache.tez.dag.app.dag.impl.TaskImpl.getCounters(TaskImpl.java:462)
              at org.apache.tez.dag.app.dag.impl.VertexImpl.aggrTaskCounters(VertexImpl.java:1342)
              at org.apache.tez.dag.app.dag.impl.VertexImpl.getAllCounters(VertexImpl.java:1202)
              at org.apache.tez.dag.app.dag.impl.DAGImpl.aggrTaskCounters(DAGImpl.java:755)
              at org.apache.tez.dag.app.dag.impl.DAGImpl.getAllCounters(DAGImpl.java:704)
              at org.apache.tez.dag.app.dag.impl.DAGImpl.getDAGStatus(DAGImpl.java:901)
              at org.apache.tez.dag.app.dag.impl.DAGImpl.getDAGStatus(DAGImpl.java:940)
              at org.apache.tez.dag.api.client.DAGClientHandler.getDAGStatus(DAGClientHandler.java:73)
      

      Attachments

        1. HIVE-20103.1.patch
          2 kB
          Gopal Vijayaraghavan

        Issue Links

          Activity

            People

              gopalv Gopal Vijayaraghavan
              gopalv Gopal Vijayaraghavan
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: