Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-4318

OperatorHooks hit performance even when not used

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Component/s: Query Processor
    • Labels:
      None
    • Environment:

      Ubuntu LXC (64 bit)

      Description

      Operator Hooks inserted into Operator.java cause a performance hit even when it is not being used.

      For a count(1) query tested with & without the operator hook calls.

      with
      2013-04-09 07:33:58,920 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 84.07 sec
      Total MapReduce CPU Time Spent: 1 minutes 24 seconds 70 msec
      OK
      28800991
      Time taken: 40.407 seconds, Fetched: 1 row(s)
      
      without
      2013-04-09 07:33:02,355 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 68.48 sec
      ...
      Total MapReduce CPU Time Spent: 1 minutes 8 seconds 480 msec
      OK
      28800991
      Time taken: 35.907 seconds, Fetched: 1 row(s)
      

      The effect is multiplied by the number of operators in the pipeline that has to forward the row - the more operators there are the, the slower the query.

      The modification made to test this was

      Operator.java
      --- ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java
      +++ ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java
      @@ -526,16 +526,16 @@ public void process(Object row, int tag) throws HiveException {
             return;
           }
           OperatorHookContext opHookContext = new OperatorHookContext(this, row, tag);
      -    preProcessCounter();
      -    enterOperatorHooks(opHookContext);
      +    //preProcessCounter();
      +    //enterOperatorHooks(opHookContext);
           processOp(row, tag);
      -    exitOperatorHooks(opHookContext);
      -    postProcessCounter();
      +    //exitOperatorHooks(opHookContext);
      +    //postProcessCounter();
         }
      

        Attachments

        1. HIVE-4318.3.patch
          70 kB
          Gunther Hagleitner
        2. HIVE-4318.patch.pam.txt
          3 kB
          Pamela Vagata
        3. HIVE-4318.2.patch
          74 kB
          Gunther Hagleitner
        4. HIVE-4318.1.patch
          70 kB
          Gunther Hagleitner

          Issue Links

            Activity

              People

              • Assignee:
                hagleitn Gunther Hagleitner
                Reporter:
                gopalv Gopal Vijayaraghavan
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: