Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-9779

ATSHook does not log the end user if doAs=false (it logs the hs2 server user)

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.13.0, 0.14.0
    • Fix Version/s: 1.2.0
    • Component/s: HiveServer2
    • Labels:

      Description

      When doAs=false, ATSHook should log the end username in ATS instead of logging the hiveserver2 user's name.
      The way things are, it is not possible for an admin to identify which query is being run by which user. The end user information is already available in the HookContext.

      1. HIVE-9779-testing.xlsx
        28 kB
        Abdelrahman Shettia
      2. HIVE-9779.2.patch
        4 kB
        Abdelrahman Shettia

        Activity

        Hide
        ashettia Abdelrahman Shettia added a comment -

        Hi,

        I am attaching a patch called 9979.001.patch and the build was successful.

        [INFO] ------------------------------------------------------------------------
        [INFO] Reactor Summary:
        [INFO] 
        [INFO] Hive .............................................. SUCCESS [  3.914 s]
        [INFO] Hive Shims Common ................................. SUCCESS [  4.288 s]
        [INFO] Hive Shims 0.20S .................................. SUCCESS [  1.105 s]
        [INFO] Hive Shims 0.23 ................................... SUCCESS [  3.423 s]
        [INFO] Hive Shims Scheduler .............................. SUCCESS [  0.766 s]
        [INFO] Hive Shims ........................................ SUCCESS [  0.838 s]
        [INFO] Hive Common ....................................... SUCCESS [  5.020 s]
        [INFO] Hive Serde ........................................ SUCCESS [  6.324 s]
        [INFO] Hive Metastore .................................... SUCCESS [ 16.112 s]
        [INFO] Hive Ant Utilities ................................ SUCCESS [  0.426 s]
        [INFO] Spark Remote Client ............................... SUCCESS [  5.445 s]
        [INFO] Hive Query Language ............................... SUCCESS [ 43.637 s]
        [INFO] Hive Service ...................................... SUCCESS [  3.268 s]
        [INFO] Hive Accumulo Handler ............................. SUCCESS [  1.568 s]
        [INFO] Hive JDBC ......................................... SUCCESS [  6.045 s]
        [INFO] Hive Beeline ...................................... SUCCESS [  1.164 s]
        [INFO] Hive CLI .......................................... SUCCESS [  1.062 s]
        [INFO] Hive Contrib ...................................... SUCCESS [  1.150 s]
        [INFO] Hive HBase Handler ................................ SUCCESS [  3.252 s]
        [INFO] Hive HCatalog ..................................... SUCCESS [  0.339 s]
        [INFO] Hive HCatalog Core ................................ SUCCESS [  1.757 s]
        [INFO] Hive HCatalog Pig Adapter ......................... SUCCESS [  1.115 s]
        [INFO] Hive HCatalog Server Extensions ................... SUCCESS [  0.829 s]
        [INFO] Hive HCatalog Webhcat Java Client ................. SUCCESS [  0.743 s]
        [INFO] Hive HCatalog Webhcat ............................. SUCCESS [  4.511 s]
        [INFO] Hive HCatalog Streaming ........................... SUCCESS [  0.911 s]
        [INFO] Hive HWI .......................................... SUCCESS [  0.678 s]
        [INFO] Hive ODBC ......................................... SUCCESS [  0.429 s]
        [INFO] Hive Shims Aggregator ............................. SUCCESS [  0.083 s]
        [INFO] Hive TestUtils .................................... SUCCESS [  0.176 s]
        [INFO] Hive Packaging .................................... SUCCESS [  0.668 s]
        [INFO] ------------------------------------------------------------------------
        [INFO] BUILD SUCCESS
        [INFO] ------------------------------------------------------------------------
        [INFO] Total time: 02:02 min
        

        Thanks
        -Rahman

        Show
        ashettia Abdelrahman Shettia added a comment - Hi, I am attaching a patch called 9979.001.patch and the build was successful. [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Hive .............................................. SUCCESS [ 3.914 s] [INFO] Hive Shims Common ................................. SUCCESS [ 4.288 s] [INFO] Hive Shims 0.20S .................................. SUCCESS [ 1.105 s] [INFO] Hive Shims 0.23 ................................... SUCCESS [ 3.423 s] [INFO] Hive Shims Scheduler .............................. SUCCESS [ 0.766 s] [INFO] Hive Shims ........................................ SUCCESS [ 0.838 s] [INFO] Hive Common ....................................... SUCCESS [ 5.020 s] [INFO] Hive Serde ........................................ SUCCESS [ 6.324 s] [INFO] Hive Metastore .................................... SUCCESS [ 16.112 s] [INFO] Hive Ant Utilities ................................ SUCCESS [ 0.426 s] [INFO] Spark Remote Client ............................... SUCCESS [ 5.445 s] [INFO] Hive Query Language ............................... SUCCESS [ 43.637 s] [INFO] Hive Service ...................................... SUCCESS [ 3.268 s] [INFO] Hive Accumulo Handler ............................. SUCCESS [ 1.568 s] [INFO] Hive JDBC ......................................... SUCCESS [ 6.045 s] [INFO] Hive Beeline ...................................... SUCCESS [ 1.164 s] [INFO] Hive CLI .......................................... SUCCESS [ 1.062 s] [INFO] Hive Contrib ...................................... SUCCESS [ 1.150 s] [INFO] Hive HBase Handler ................................ SUCCESS [ 3.252 s] [INFO] Hive HCatalog ..................................... SUCCESS [ 0.339 s] [INFO] Hive HCatalog Core ................................ SUCCESS [ 1.757 s] [INFO] Hive HCatalog Pig Adapter ......................... SUCCESS [ 1.115 s] [INFO] Hive HCatalog Server Extensions ................... SUCCESS [ 0.829 s] [INFO] Hive HCatalog Webhcat Java Client ................. SUCCESS [ 0.743 s] [INFO] Hive HCatalog Webhcat ............................. SUCCESS [ 4.511 s] [INFO] Hive HCatalog Streaming ........................... SUCCESS [ 0.911 s] [INFO] Hive HWI .......................................... SUCCESS [ 0.678 s] [INFO] Hive ODBC ......................................... SUCCESS [ 0.429 s] [INFO] Hive Shims Aggregator ............................. SUCCESS [ 0.083 s] [INFO] Hive TestUtils .................................... SUCCESS [ 0.176 s] [INFO] Hive Packaging .................................... SUCCESS [ 0.668 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 02:02 min Thanks -Rahman
        Hide
        thejas Thejas M Nair added a comment -

        hookContext.getUgi() will always be set. We should use hookContext.getUserName() as long as it is set.
        Can you also fix the indentation ?
        Were you able to verify that this change works with Yarn ATS ?

        +            if (hookContext.getUserName() == null){
        +                 user = hookContext.getUserName();
        +            }
        +            else       {
        +                 user = hookContext.getUgi().getUserName(); 
        +            }
        
        
        Show
        thejas Thejas M Nair added a comment - hookContext.getUgi() will always be set. We should use hookContext.getUserName() as long as it is set. Can you also fix the indentation ? Were you able to verify that this change works with Yarn ATS ? + if (hookContext.getUserName() == null ){ + user = hookContext.getUserName(); + } + else { + user = hookContext.getUgi().getUserName(); + }
        Hide
        ashettia Abdelrahman Shettia added a comment -

        Hi Thejas M Nair,

        I have uploaded the patch file called 9979.002.patch. And here are the testing results:

        Beeline 
        
         curl http://127.0.0.1:8188/ws/v1/timeline/HIVE_QUERY_ID?primaryFilter=requestuser:hue
        {"entities":[{"events":[{"timestamp":1425088884579,"eventtype":"QUERY_COMPLETED","eventinfo":{}},{"timestamp":0,"eventtype":"QUERY_SUBMITTED","eventinfo":{}}],"entitytype":"HIVE_QUERY_ID","entity":"hive_20150228020000_e5581a7d-b0cf-4348-87af-a95c5405602c","starttime":0,"domain":"DEFAULT","otherinfo":{"QUERY":"{\"queryText\":\"select count(1) from poke1 \",\"queryPlan\":{\"STAGE PLANS\":{\"Stage-1\":{\"Map Reduce\":{\"Reduce Operator Tree:\":{\"Group By Operator\":{\"mode:\":\"mergepartial\",\"aggregations:\":[\"count(VALUE._col0)\"],\"outputColumnNames:\":[\"_col0\"],\"children\":{\"File Output Operator\":{\"compressed:\":\"false\",\"table:\":{\"serde:\":\"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe\",\"input format:\":\"org.apache.hadoop.mapred.TextInputFormat\",\"output format:\":\"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat\"}}}}},\"Map Operator Tree:\":[{\"TableScan\":{\"alias:\":\"poke1\",\"children\":{\"Select Operator\":{\"children\":{\"Group By Operator\":{\"mode:\":\"hash\",\"aggregations:\":[\"count(1)\"],\"outputColumnNames:\":[\"_col0\"],\"children\":{\"Reduce Output Operator\":{\"sort order:\":\"\",\"value expressions:\":\"_col0 (type: bigint)\"}}}}}}}}]}},\"Stage-0\":{\"Fetch Operator\":{\"limit:\":\"-1\",\"Processor Tree:\":{\"ListSink\":{}}}}},\"STAGE DEPENDENCIES\":{\"Stage-1\":{\"ROOT STAGE\":\"TRUE\"},\"Stage-0\":{\"DEPENDENT STAGES\":\"Stage-1\"}}}}","TEZ":false,"STATUS":true,"MAPRED":true},"relatedentities":{},"primaryfilters":{"operationid":["l3T-7AI4Su-1m44E2RL05A"],"requestuser":["hue"],"user":["hive"]}}]}
        
         curl http://127.0.0.1:8188/ws/v1/timeline/HIVE_QUERY_ID?primaryFilter=requestuser:hue
        {"entities":[{"events":[{"timestamp":1425088884579,"eventtype":"QUERY_COMPLETED","eventinfo":{}},{"timestamp":0,"eventtype":"QUERY_SUBMITTED","eventinfo":{}}],"entitytype":"HIVE_QUERY_ID","entity":"hive_20150228020000_e5581a7d-b0cf-4348-87af-a95c5405602c","starttime":0,"domain":"DEFAULT","otherinfo":{"QUERY":"{\"queryText\":\"select count(1) from poke1 \",\"queryPlan\":{\"STAGE PLANS\":{\"Stage-1\":{\"Map Reduce\":{\"Reduce Operator Tree:\":{\"Group By Operator\":{\"mode:\":\"mergepartial\",\"aggregations:\":[\"count(VALUE._col0)\"],\"outputColumnNames:\":[\"_col0\"],\"children\":{\"File Output Operator\":{\"compressed:\":\"false\",\"table:\":{\"serde:\":\"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe\",\"input format:\":\"org.apache.hadoop.mapred.TextInputFormat\",\"output format:\":\"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat\"}}}}},\"Map Operator Tree:\":[{\"TableScan\":{\"alias:\":\"poke1\",\"children\":{\"Select Operator\":{\"children\":{\"Group By Operator\":{\"mode:\":\"hash\",\"aggregations:\":[\"count(1)\"],\"outputColumnNames:\":[\"_col0\"],\"children\":{\"Reduce Output Operator\":{\"sort order:\":\"\",\"value expressions:\":\"_col0 (type: bigint)\"}}}}}}}}]}},\"Stage-0\":{\"Fetch Operator\":{\"limit:\":\"-1\",\"Processor Tree:\":{\"ListSink\":{}}}}},\"STAGE DEPENDENCIES\":{\"Stage-1\":{\"ROOT STAGE\":\"TRUE\"},\"Stage-0\":{\"DEPENDENT STAGES\":\"Stage-1\"}}}}","TEZ":false,"STATUS":true,"MAPRED":true},"relatedentities":{},"primaryfilters":{"operationid":["l3T-7AI4Su-1m44E2RL05A"],"requestuser":["hue"],"user":["hive"]}}]}
        
        Hive CLI :
        curl http://127.0.0.1:8188/ws/v1/timeline/HIVE_QUERY_ID?primaryFilter=user:hue
        {"entities":[{"events":[{"timestamp":1425091326014,"eventtype":"QUERY_COMPLETED","eventinfo":{}},{"timestamp":1425091289943,"eventtype":"QUERY_SUBMITTED","eventinfo":{}}],"entitytype":"HIVE_QUERY_ID","entity":"hue_20150228024141_3e6c0287-84b2-453b-ab19-67dddf87d42e","starttime":1425091289943,"domain":"DEFAULT","otherinfo":{"QUERY":"{\"queryText\":\"select count(1) from poke1 \",\"queryPlan\":{\"STAGE PLANS\":{\"Stage-1\":{\"Map Reduce\":{\"Reduce Operator Tree:\":{\"Group By Operator\":{\"mode:\":\"mergepartial\",\"aggregations:\":[\"count(VALUE._col0)\"],\"outputColumnNames:\":[\"_col0\"],\"children\":{\"File Output Operator\":{\"compressed:\":\"false\",\"table:\":{\"serde:\":\"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe\",\"input format:\":\"org.apache.hadoop.mapred.TextInputFormat\",\"output format:\":\"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat\"}}}}},\"Map Operator Tree:\":[{\"TableScan\":{\"alias:\":\"poke1\",\"children\":{\"Select Operator\":{\"children\":{\"Group By Operator\":{\"mode:\":\"hash\",\"aggregations:\":[\"count(1)\"],\"outputColumnNames:\":[\"_col0\"],\"children\":{\"Reduce Output Operator\":{\"sort order:\":\"\",\"value expressions:\":\"_col0 (type: bigint)\"}}}}}}}}]}},\"Stage-0\":{\"Fetch Operator\":{\"limit:\":\"-1\",\"Processor Tree:\":{\"ListSink\":{}}}}},\"STAGE DEPENDENCIES\":{\"Stage-1\":{\"ROOT STAGE\":\"TRUE\"},\"Stage-0\":{\"DEPENDENT STAGES\":\"Stage-1\"}}}}","TEZ":false,"STATUS":true,"MAPRED":true},"relatedentities":{},"primaryfilters":{"requestuser":["hue"],"user":["hue"]}},{"events":[{"timestamp":1425089222614,"eventtype":"QUERY_COMPLETED","eventinfo":{}},{"timestamp":1425089185488,"eventtype":"QUERY_SUBMITTED","eventinfo":{}}],"entitytype":"HIVE_QUERY_ID","entity":"hue_20150228020606_9280ae8c-a2f1-4547-897f-b34100fb0d37","starttime":1425089185488,"domain":"DEFAULT","otherinfo":{"QUERY":"{\"queryText\":\"select count(1) from poke1 \",\"queryPlan\":{\"STAGE PLANS\":{\"Stage-1\":{\"Map Reduce\":{\"Reduce Operator Tree:\":{\"Group By Operator\":{\"mode:\":\"mergepartial\",\"aggregations:\":[\"count(VALUE._col0)\"],\"outputColumnNames:\":[\"_col0\"],\"children\":{\"File Output Operator\":{\"compressed:\":\"false\",\"table:\":{\"serde:\":\"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe\",\"input format:\":\"org.apache.hadoop.mapred.TextInputFormat\",\"output format:\":\"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat\"}}}}},\"Map Operator Tree:\":[{\"TableScan\":{\"alias:\":\"poke1\",\"children\":{\"Select Operator\":{\"children\":{\"Group By Operator\":{\"mode:\":\"hash\",\"aggregations:\":[\"count(1)\"],\"outputColumnNames:\":[\"_col0\"],\"children\":{\"Reduce Output Operator\":{\"sort order:\":\"\",\"value expressions:\":\"_col0 (type: bigint)\"}}}}}}}}]}},\"Stage-0\":{\"Fetch Operator\":{\"limit:\":\"-1\",\"Processor Tree:\":{\"ListSink\":{}}}}},\"STAGE DEPENDENCIES\":{\"Stage-1\":{\"ROOT STAGE\":\"TRUE\"},\"Stage-0\":{\"DEPENDENT STAGES\":\"Stage-1\"}}}}","TEZ":false,"STATUS":true,"MAPRED":true},"relatedentities":{},"primaryfilters":{"requestuser":["hue"],"user":["hue"]}},{"events":[{"timestamp":1425085219623,"eventtype":"QUERY_COMPLETED","eventinfo":{}},{"timestamp":1425085173742,"eventtype":"QUERY_SUBMITTED","eventinfo":{}}],"entitytype":"HIVE_QUERY_ID","entity":"hive_20150228005959_8b59a353-5fd5-4886-af76-a076aa8c3470","starttime":1425085173742,"domain":"DEFAULT","otherinfo":{"QUERY":"{\"queryText\":\"select count(1) from poke1 \",\"queryPlan\":{\"STAGE PLANS\":{\"Stage-1\":{\"Tez\":{\"DagName:\":\"hive_20150228005959_8b59a353-5fd5-4886-af76-a076aa8c3470:1\",\"Vertices:\":{\"Reducer 2\":{\"Reduce Operator Tree:\":{\"Group By Operator\":{\"mode:\":\"mergepartial\",\"aggregations:\":[\"count(VALUE._col0)\"],\"outputColumnNames:\":[\"_col0\"],\"children\":{\"File Output Operator\":{\"Statistics:\":\"Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE\",\"compressed:\":\"false\",\"table:\":{\"serde:\":\"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe\",\"input format:\":\"org.apache.hadoop.mapred.TextInputFormat\",\"output format:\":\"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat\"}}},\"Statistics:\":\"Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE\"}}},\"Map 1\":{\"Map Operator Tree:\":[{\"TableScan\":{\"alias:\":\"poke1\",\"children\":{\"Select Operator\":{\"children\":{\"Group By Operator\":{\"mode:\":\"hash\",\"aggregations:\":[\"count(1)\"],\"outputColumnNames:\":[\"_col0\"],\"children\":{\"Reduce Output Operator\":{\"sort order:\":\"\",\"value expressions:\":\"_col0 (type: bigint)\",\"Statistics:\":\"Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE\"}},\"Statistics:\":\"Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE\"}},\"Statistics:\":\"Num rows: 0 Data size: 0 Basic stats: NONE Column stats: COMPLETE\"}},\"Statistics:\":\"Num rows: 0 Data size: 0 Basic stats: NONE Column stats: COMPLETE\"}}]}},\"Edges:\":{\"Reducer 2\":{\"parent\":\"Map 1\",\"type\":\"SIMPLE_EDGE\"}}}},\"Stage-0\":{\"Fetch Operator\":{\"limit:\":\"-1\",\"Processor Tree:\":{\"ListSink\":{}}}}},\"STAGE DEPENDENCIES\":{\"Stage-1\":{\"ROOT STAGE\":\"TRUE\"},\"Stage-0\":{\"DEPENDENT STAGES\":\"Stage-1\"}}}}","TEZ":true,"STATUS":true,"MAPRED":false},"relatedentities":{},"primaryfilters":{"operationid":["8K0lnZIwTnOV3_4Bp_0cqA"],"user":["hive","hue"]}}]}
        
        curl http://127.0.0.1:8188/ws/v1/timeline/HIVE_QUERY_ID?primaryFilter=requestuser:hive
        {"entities":[{"events":[{"timestamp":1425091144316,"eventtype":"QUERY_COMPLETED","eventinfo":{}},{"timestamp":1425091105837,"eventtype":"QUERY_SUBMITTED","eventinfo":{}}],"entitytype":"HIVE_QUERY_ID","entity":"hive_20150228023838_e2030858-4927-4ae6-97e4-afbad68c43f8","starttime":1425091105837,"domain":"DEFAULT","otherinfo":{"QUERY":"{\"queryText\":\"select count(1) from poke1 \",\"queryPlan\":{\"STAGE PLANS\":{\"Stage-1\":{\"Map Reduce\":{\"Reduce Operator Tree:\":{\"Group By Operator\":{\"mode:\":\"mergepartial\",\"aggregations:\":[\"count(VALUE._col0)\"],\"outputColumnNames:\":[\"_col0\"],\"children\":{\"File Output Operator\":{\"compressed:\":\"false\",\"table:\":{\"serde:\":\"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe\",\"input format:\":\"org.apache.hadoop.mapred.TextInputFormat\",\"output format:\":\"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat\"}}}}},\"Map Operator Tree:\":[{\"TableScan\":{\"alias:\":\"poke1\",\"children\":{\"Select Operator\":{\"children\":{\"Group By Operator\":{\"mode:\":\"hash\",\"aggregations:\":[\"count(1)\"],\"outputColumnNames:\":[\"_col0\"],\"children\":{\"Reduce Output Operator\":{\"sort order:\":\"\",\"value expressions:\":\"_col0 (type: bigint)\"}}}}}}}}]}},\"Stage-0\":{\"Fetch Operator\":{\"limit:\":\"-1\",\"Processor Tree:\":{\"ListSink\":{}}}}},\"STAGE DEPENDENCIES\":{\"Stage-1\":{\"ROOT STAGE\":\"TRUE\"},\"Stage-0\":{\"DEPENDENT STAGES\":\"Stage-1\"}}}}","TEZ":false,"STATUS":true,"MAPRED":true},"relatedentities":{},"primaryfilters":{"requestuser":["hive"],"user":["hive"]}}]}
        

        Please let me know if you have questions.

        Thanks
        -Rahman

        Show
        ashettia Abdelrahman Shettia added a comment - Hi Thejas M Nair , I have uploaded the patch file called 9979.002.patch. And here are the testing results: Beeline curl http: //127.0.0.1:8188/ws/v1/timeline/HIVE_QUERY_ID?primaryFilter=requestuser:hue { "entities" :[{ "events" :[{ "timestamp" :1425088884579, "eventtype" : "QUERY_COMPLETED" , "eventinfo" :{}},{ "timestamp" :0, "eventtype" : "QUERY_SUBMITTED" , "eventinfo" :{}}], "entitytype" : "HIVE_QUERY_ID" , "entity" : "hive_20150228020000_e5581a7d-b0cf-4348-87af-a95c5405602c" , "starttime" :0, "domain" : "DEFAULT" , "otherinfo" :{ "QUERY" : "{\" queryText\ ":\" select count(1) from poke1 \ ",\" queryPlan\ ":{\" STAGE PLANS\ ":{\" Stage-1\ ":{\" Map Reduce\ ":{\" Reduce Operator Tree:\ ":{\" Group By Operator\ ":{\" mode:\ ":\" mergepartial\ ",\" aggregations:\ ":[\" count(VALUE._col0)\ "],\" outputColumnNames:\ ":[\" _col0\ "],\" children\ ":{\" File Output Operator\ ":{\" compressed:\ ":\" false \ ",\" table:\ ":{\" serde:\ ":\" org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe\ ",\" input format:\ ":\" org.apache.hadoop.mapred.TextInputFormat\ ",\" output format:\ ":\" org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat\ "}}}}},\" Map Operator Tree:\ ":[{\" TableScan\ ":{\" alias:\ ":\" poke1\ ",\" children\ ":{\" Select Operator\ ":{\" children\ ":{\" Group By Operator\ ":{\" mode:\ ":\" hash\ ",\" aggregations:\ ":[\" count(1)\ "],\" outputColumnNames:\ ":[\" _col0\ "],\" children\ ":{\" Reduce Output Operator\ ":{\" sort order:\ ":\" \ ",\" value expressions:\ ":\" _col0 (type: bigint)\ "}}}}}}}}]}},\" Stage-0\ ":{\" Fetch Operator\ ":{\" limit:\ ":\" -1\ ",\" Processor Tree:\ ":{\" ListSink\ ":{}}}}},\" STAGE DEPENDENCIES\ ":{\" Stage-1\ ":{\" ROOT STAGE\ ":\" TRUE\ "},\" Stage-0\ ":{\" DEPENDENT STAGES\ ":\" Stage-1\ "}}}}" , "TEZ" : false , "STATUS" : true , "MAPRED" : true }, "relatedentities" :{}, "primaryfilters" :{ "operationid" :[ "l3T-7AI4Su-1m44E2RL05A" ], "requestuser" :[ "hue" ], "user" :[ "hive" ]}}]} curl http: //127.0.0.1:8188/ws/v1/timeline/HIVE_QUERY_ID?primaryFilter=requestuser:hue { "entities" :[{ "events" :[{ "timestamp" :1425088884579, "eventtype" : "QUERY_COMPLETED" , "eventinfo" :{}},{ "timestamp" :0, "eventtype" : "QUERY_SUBMITTED" , "eventinfo" :{}}], "entitytype" : "HIVE_QUERY_ID" , "entity" : "hive_20150228020000_e5581a7d-b0cf-4348-87af-a95c5405602c" , "starttime" :0, "domain" : "DEFAULT" , "otherinfo" :{ "QUERY" : "{\" queryText\ ":\" select count(1) from poke1 \ ",\" queryPlan\ ":{\" STAGE PLANS\ ":{\" Stage-1\ ":{\" Map Reduce\ ":{\" Reduce Operator Tree:\ ":{\" Group By Operator\ ":{\" mode:\ ":\" mergepartial\ ",\" aggregations:\ ":[\" count(VALUE._col0)\ "],\" outputColumnNames:\ ":[\" _col0\ "],\" children\ ":{\" File Output Operator\ ":{\" compressed:\ ":\" false \ ",\" table:\ ":{\" serde:\ ":\" org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe\ ",\" input format:\ ":\" org.apache.hadoop.mapred.TextInputFormat\ ",\" output format:\ ":\" org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat\ "}}}}},\" Map Operator Tree:\ ":[{\" TableScan\ ":{\" alias:\ ":\" poke1\ ",\" children\ ":{\" Select Operator\ ":{\" children\ ":{\" Group By Operator\ ":{\" mode:\ ":\" hash\ ",\" aggregations:\ ":[\" count(1)\ "],\" outputColumnNames:\ ":[\" _col0\ "],\" children\ ":{\" Reduce Output Operator\ ":{\" sort order:\ ":\" \ ",\" value expressions:\ ":\" _col0 (type: bigint)\ "}}}}}}}}]}},\" Stage-0\ ":{\" Fetch Operator\ ":{\" limit:\ ":\" -1\ ",\" Processor Tree:\ ":{\" ListSink\ ":{}}}}},\" STAGE DEPENDENCIES\ ":{\" Stage-1\ ":{\" ROOT STAGE\ ":\" TRUE\ "},\" Stage-0\ ":{\" DEPENDENT STAGES\ ":\" Stage-1\ "}}}}" , "TEZ" : false , "STATUS" : true , "MAPRED" : true }, "relatedentities" :{}, "primaryfilters" :{ "operationid" :[ "l3T-7AI4Su-1m44E2RL05A" ], "requestuser" :[ "hue" ], "user" :[ "hive" ]}}]} Hive CLI : curl http: //127.0.0.1:8188/ws/v1/timeline/HIVE_QUERY_ID?primaryFilter=user:hue { "entities" :[{ "events" :[{ "timestamp" :1425091326014, "eventtype" : "QUERY_COMPLETED" , "eventinfo" :{}},{ "timestamp" :1425091289943, "eventtype" : "QUERY_SUBMITTED" , "eventinfo" :{}}], "entitytype" : "HIVE_QUERY_ID" , "entity" : "hue_20150228024141_3e6c0287-84b2-453b-ab19-67dddf87d42e" , "starttime" :1425091289943, "domain" : "DEFAULT" , "otherinfo" :{ "QUERY" : "{\" queryText\ ":\" select count(1) from poke1 \ ",\" queryPlan\ ":{\" STAGE PLANS\ ":{\" Stage-1\ ":{\" Map Reduce\ ":{\" Reduce Operator Tree:\ ":{\" Group By Operator\ ":{\" mode:\ ":\" mergepartial\ ",\" aggregations:\ ":[\" count(VALUE._col0)\ "],\" outputColumnNames:\ ":[\" _col0\ "],\" children\ ":{\" File Output Operator\ ":{\" compressed:\ ":\" false \ ",\" table:\ ":{\" serde:\ ":\" org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe\ ",\" input format:\ ":\" org.apache.hadoop.mapred.TextInputFormat\ ",\" output format:\ ":\" org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat\ "}}}}},\" Map Operator Tree:\ ":[{\" TableScan\ ":{\" alias:\ ":\" poke1\ ",\" children\ ":{\" Select Operator\ ":{\" children\ ":{\" Group By Operator\ ":{\" mode:\ ":\" hash\ ",\" aggregations:\ ":[\" count(1)\ "],\" outputColumnNames:\ ":[\" _col0\ "],\" children\ ":{\" Reduce Output Operator\ ":{\" sort order:\ ":\" \ ",\" value expressions:\ ":\" _col0 (type: bigint)\ "}}}}}}}}]}},\" Stage-0\ ":{\" Fetch Operator\ ":{\" limit:\ ":\" -1\ ",\" Processor Tree:\ ":{\" ListSink\ ":{}}}}},\" STAGE DEPENDENCIES\ ":{\" Stage-1\ ":{\" ROOT STAGE\ ":\" TRUE\ "},\" Stage-0\ ":{\" DEPENDENT STAGES\ ":\" Stage-1\ "}}}}" , "TEZ" : false , "STATUS" : true , "MAPRED" : true }, "relatedentities" :{}, "primaryfilters" :{ "requestuser" :[ "hue" ], "user" :[ "hue" ]}},{ "events" :[{ "timestamp" :1425089222614, "eventtype" : "QUERY_COMPLETED" , "eventinfo" :{}},{ "timestamp" :1425089185488, "eventtype" : "QUERY_SUBMITTED" , "eventinfo" :{}}], "entitytype" : "HIVE_QUERY_ID" , "entity" : "hue_20150228020606_9280ae8c-a2f1-4547-897f-b34100fb0d37" , "starttime" :1425089185488, "domain" : "DEFAULT" , "otherinfo" :{ "QUERY" : "{\" queryText\ ":\" select count(1) from poke1 \ ",\" queryPlan\ ":{\" STAGE PLANS\ ":{\" Stage-1\ ":{\" Map Reduce\ ":{\" Reduce Operator Tree:\ ":{\" Group By Operator\ ":{\" mode:\ ":\" mergepartial\ ",\" aggregations:\ ":[\" count(VALUE._col0)\ "],\" outputColumnNames:\ ":[\" _col0\ "],\" children\ ":{\" File Output Operator\ ":{\" compressed:\ ":\" false \ ",\" table:\ ":{\" serde:\ ":\" org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe\ ",\" input format:\ ":\" org.apache.hadoop.mapred.TextInputFormat\ ",\" output format:\ ":\" org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat\ "}}}}},\" Map Operator Tree:\ ":[{\" TableScan\ ":{\" alias:\ ":\" poke1\ ",\" children\ ":{\" Select Operator\ ":{\" children\ ":{\" Group By Operator\ ":{\" mode:\ ":\" hash\ ",\" aggregations:\ ":[\" count(1)\ "],\" outputColumnNames:\ ":[\" _col0\ "],\" children\ ":{\" Reduce Output Operator\ ":{\" sort order:\ ":\" \ ",\" value expressions:\ ":\" _col0 (type: bigint)\ "}}}}}}}}]}},\" Stage-0\ ":{\" Fetch Operator\ ":{\" limit:\ ":\" -1\ ",\" Processor Tree:\ ":{\" ListSink\ ":{}}}}},\" STAGE DEPENDENCIES\ ":{\" Stage-1\ ":{\" ROOT STAGE\ ":\" TRUE\ "},\" Stage-0\ ":{\" DEPENDENT STAGES\ ":\" Stage-1\ "}}}}" , "TEZ" : false , "STATUS" : true , "MAPRED" : true }, "relatedentities" :{}, "primaryfilters" :{ "requestuser" :[ "hue" ], "user" :[ "hue" ]}},{ "events" :[{ "timestamp" :1425085219623, "eventtype" : "QUERY_COMPLETED" , "eventinfo" :{}},{ "timestamp" :1425085173742, "eventtype" : "QUERY_SUBMITTED" , "eventinfo" :{}}], "entitytype" : "HIVE_QUERY_ID" , "entity" : "hive_20150228005959_8b59a353-5fd5-4886-af76-a076aa8c3470" , "starttime" :1425085173742, "domain" : "DEFAULT" , "otherinfo" :{ "QUERY" : "{\" queryText\ ":\" select count(1) from poke1 \ ",\" queryPlan\ ":{\" STAGE PLANS\ ":{\" Stage-1\ ":{\" Tez\ ":{\" DagName:\ ":\" hive_20150228005959_8b59a353-5fd5-4886-af76-a076aa8c3470:1\ ",\" Vertices:\ ":{\" Reducer 2\ ":{\" Reduce Operator Tree:\ ":{\" Group By Operator\ ":{\" mode:\ ":\" mergepartial\ ",\" aggregations:\ ":[\" count(VALUE._col0)\ "],\" outputColumnNames:\ ":[\" _col0\ "],\" children\ ":{\" File Output Operator\ ":{\" Statistics:\ ":\" Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE\ ",\" compressed:\ ":\" false \ ",\" table:\ ":{\" serde:\ ":\" org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe\ ",\" input format:\ ":\" org.apache.hadoop.mapred.TextInputFormat\ ",\" output format:\ ":\" org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat\ "}}},\" Statistics:\ ":\" Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE\ "}}},\" Map 1\ ":{\" Map Operator Tree:\ ":[{\" TableScan\ ":{\" alias:\ ":\" poke1\ ",\" children\ ":{\" Select Operator\ ":{\" children\ ":{\" Group By Operator\ ":{\" mode:\ ":\" hash\ ",\" aggregations:\ ":[\" count(1)\ "],\" outputColumnNames:\ ":[\" _col0\ "],\" children\ ":{\" Reduce Output Operator\ ":{\" sort order:\ ":\" \ ",\" value expressions:\ ":\" _col0 (type: bigint)\ ",\" Statistics:\ ":\" Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE\ "}},\" Statistics:\ ":\" Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE\ "}},\" Statistics:\ ":\" Num rows: 0 Data size: 0 Basic stats: NONE Column stats: COMPLETE\ "}},\" Statistics:\ ":\" Num rows: 0 Data size: 0 Basic stats: NONE Column stats: COMPLETE\ "}}]}},\" Edges:\ ":{\" Reducer 2\ ":{\" parent\ ":\" Map 1\ ",\" type\ ":\" SIMPLE_EDGE\ "}}}},\" Stage-0\ ":{\" Fetch Operator\ ":{\" limit:\ ":\" -1\ ",\" Processor Tree:\ ":{\" ListSink\ ":{}}}}},\" STAGE DEPENDENCIES\ ":{\" Stage-1\ ":{\" ROOT STAGE\ ":\" TRUE\ "},\" Stage-0\ ":{\" DEPENDENT STAGES\ ":\" Stage-1\ "}}}}" , "TEZ" : true , "STATUS" : true , "MAPRED" : false }, "relatedentities" :{}, "primaryfilters" :{ "operationid" :[ "8K0lnZIwTnOV3_4Bp_0cqA" ], "user" :[ "hive" , "hue" ]}}]} curl http: //127.0.0.1:8188/ws/v1/timeline/HIVE_QUERY_ID?primaryFilter=requestuser:hive { "entities" :[{ "events" :[{ "timestamp" :1425091144316, "eventtype" : "QUERY_COMPLETED" , "eventinfo" :{}},{ "timestamp" :1425091105837, "eventtype" : "QUERY_SUBMITTED" , "eventinfo" :{}}], "entitytype" : "HIVE_QUERY_ID" , "entity" : "hive_20150228023838_e2030858-4927-4ae6-97e4-afbad68c43f8" , "starttime" :1425091105837, "domain" : "DEFAULT" , "otherinfo" :{ "QUERY" : "{\" queryText\ ":\" select count(1) from poke1 \ ",\" queryPlan\ ":{\" STAGE PLANS\ ":{\" Stage-1\ ":{\" Map Reduce\ ":{\" Reduce Operator Tree:\ ":{\" Group By Operator\ ":{\" mode:\ ":\" mergepartial\ ",\" aggregations:\ ":[\" count(VALUE._col0)\ "],\" outputColumnNames:\ ":[\" _col0\ "],\" children\ ":{\" File Output Operator\ ":{\" compressed:\ ":\" false \ ",\" table:\ ":{\" serde:\ ":\" org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe\ ",\" input format:\ ":\" org.apache.hadoop.mapred.TextInputFormat\ ",\" output format:\ ":\" org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat\ "}}}}},\" Map Operator Tree:\ ":[{\" TableScan\ ":{\" alias:\ ":\" poke1\ ",\" children\ ":{\" Select Operator\ ":{\" children\ ":{\" Group By Operator\ ":{\" mode:\ ":\" hash\ ",\" aggregations:\ ":[\" count(1)\ "],\" outputColumnNames:\ ":[\" _col0\ "],\" children\ ":{\" Reduce Output Operator\ ":{\" sort order:\ ":\" \ ",\" value expressions:\ ":\" _col0 (type: bigint)\ "}}}}}}}}]}},\" Stage-0\ ":{\" Fetch Operator\ ":{\" limit:\ ":\" -1\ ",\" Processor Tree:\ ":{\" ListSink\ ":{}}}}},\" STAGE DEPENDENCIES\ ":{\" Stage-1\ ":{\" ROOT STAGE\ ":\" TRUE\ "},\" Stage-0\ ":{\" DEPENDENT STAGES\ ":\" Stage-1\ "}}}}" , "TEZ" : false , "STATUS" : true , "MAPRED" : true }, "relatedentities" :{}, "primaryfilters" :{ "requestuser" :[ "hive" ], "user" :[ "hive" ]}}]} Please let me know if you have questions. Thanks -Rahman
        Hide
        ashettia Abdelrahman Shettia added a comment -

        The identification is fixed and the testing results show that this change works with Yarn ATS.

        Thanks
        -Rahman

        Show
        ashettia Abdelrahman Shettia added a comment - The identification is fixed and the testing results show that this change works with Yarn ATS. Thanks -Rahman
        Hide
        thejas Thejas M Nair added a comment -

        Abdelrahman Shettia Thanks a lot for the patch and testing it out. The changes look good!
        Can you please also mention if you had doAs=true/false in the above tests, and what you user you ran above 3 tests as ?

        Show
        thejas Thejas M Nair added a comment - Abdelrahman Shettia Thanks a lot for the patch and testing it out. The changes look good! Can you please also mention if you had doAs=true/false in the above tests, and what you user you ran above 3 tests as ?
        Hide
        ashettia Abdelrahman Shettia added a comment -

        I have uploaded excel sheet called HIVE-9779-testing. It has all the details of the test cases.

        Thanks
        -Rahman

        Show
        ashettia Abdelrahman Shettia added a comment - I have uploaded excel sheet called HIVE-9779 -testing. It has all the details of the test cases. Thanks -Rahman
        Hide
        thejas Thejas M Nair added a comment -

        +1

        Show
        thejas Thejas M Nair added a comment - +1
        Hide
        hiveqa Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12702033/HIVE-9779.2.patch

        ERROR: -1 due to 2 failed/errored test(s), 7586 tests executed
        Failed tests:

        TestCustomAuthentication - did not produce a TEST-*.xml file
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby3_map
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2924/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2924/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2924/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 2 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12702033 - PreCommit-HIVE-TRUNK-Build

        Show
        hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12702033/HIVE-9779.2.patch ERROR: -1 due to 2 failed/errored test(s), 7586 tests executed Failed tests: TestCustomAuthentication - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby3_map Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2924/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2924/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2924/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12702033 - PreCommit-HIVE-TRUNK-Build
        Hide
        thejas Thejas M Nair added a comment -

        Patch committed to trunk. Thanks for the contribution Abdelrahman Shettia !

        Show
        thejas Thejas M Nair added a comment - Patch committed to trunk. Thanks for the contribution Abdelrahman Shettia !
        Hide
        sushanth Sushanth Sowmyan added a comment -

        This issue has been fixed and released as part of the 1.2.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

        Show
        sushanth Sushanth Sowmyan added a comment - This issue has been fixed and released as part of the 1.2.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

          People

          • Assignee:
            ashettia Abdelrahman Shettia
            Reporter:
            ashettia Abdelrahman Shettia
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development