[HIVE-4318] OperatorHooks hit performance even when not used

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.11.0
• Component/s: Query Processor

Environment

Ubuntu LXC (64 bit)

• Assignee: Gunther Hagleitner
• Reporter: Gopal Vijayaraghavan
• Created: 09/Apr/13 07:37
• Updated: 16/May/13 21:09
• Resolved: 20/Apr/13 03:33

Description

Operator Hooks inserted into Operator.java cause a performance hit even when it is not being used.

For a count(1) query tested with & without the operator hook calls.

with

2013-04-09 07:33:58,920 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 84.07 sec
Total MapReduce CPU Time Spent: 1 minutes 24 seconds 70 msec
OK
28800991
Time taken: 40.407 seconds, Fetched: 1 row(s)

without

2013-04-09 07:33:02,355 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 68.48 sec
...
Total MapReduce CPU Time Spent: 1 minutes 8 seconds 480 msec
OK
28800991
Time taken: 35.907 seconds, Fetched: 1 row(s)

The effect is multiplied by the number of operators in the pipeline that has to forward the row - the more operators there are the, the slower the query.

The modification made to test this was

Operator.java

--- ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java
+++ ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java
@@ -526,16 +526,16 @@ public void process(Object row, int tag) throws HiveException {
       return;
     }
     OperatorHookContext opHookContext = new OperatorHookContext(this, row, tag);
-    preProcessCounter();
-    enterOperatorHooks(opHookContext);
+    //preProcessCounter();
+    //enterOperatorHooks(opHookContext);
     processOp(row, tag);
-    exitOperatorHooks(opHookContext);
-    postProcessCounter();
+    //exitOperatorHooks(opHookContext);
+    //postProcessCounter();
   }

Issue Links

• is blocked by: HIVE-4378 Counters hit performance even when not used