Hive
  1. Hive
  2. HIVE-2106

Increase the number of operator counter

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:
      None

      Description

      Currently Hadoop counters have to be defined as enum (hardcoded) and we support up to 400 counters now. This limit the number of operators to 100 (each operator has 4 counters). We need to increase the hadoop counters or change the Hive code to use Hadoop 0.20 API.

        Activity

        Hide
        Namit Jain added a comment -

        Committed. Thanks Ning

        Show
        Namit Jain added a comment - Committed. Thanks Ning
        Hide
        Ning Zhang added a comment -

        HIVE-2108 is filed for moving to Hadoop 0.20 API.

        Show
        Ning Zhang added a comment - HIVE-2108 is filed for moving to Hadoop 0.20 API.
        Hide
        Namit Jain added a comment -

        Carl, the comment is OK - we can remove it once we move to the new API
        Ning, can you file a new jira for that.

        +1 for this

        Show
        Namit Jain added a comment - Carl, the comment is OK - we can remove it once we move to the new API Ning, can you file a new jira for that. +1 for this
        Hide
        Namit Jain added a comment -

        Ning, can you change the comment ?

        Show
        Namit Jain added a comment - Ning, can you change the comment ?
        Hide
        Carl Steinbach added a comment -

        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java
        <https://reviews.apache.org/r/579/#comment807>

        Is this comment still accurate? If so, can we remove this entirely since we no longer support 0.17?

        Show
        Carl Steinbach added a comment - trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java < https://reviews.apache.org/r/579/#comment807 > Is this comment still accurate? If so, can we remove this entirely since we no longer support 0.17?
        Hide
        Ning Zhang added a comment -
        Show
        Ning Zhang added a comment - https://reviews.apache.org/r/579/
        Hide
        Ning Zhang added a comment -

        Attaching a simple fix that increase the number of operator counters to 1000. As a follow up we should explore the possibility to use Hadoop 0.20 API.

        Show
        Ning Zhang added a comment - Attaching a simple fix that increase the number of operator counters to 1000. As a follow up we should explore the possibility to use Hadoop 0.20 API.

          People

          • Assignee:
            Ning Zhang
            Reporter:
            Ning Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development