[HIVE-26211] "hive.server2.webui.max.historic.queries" should be avoided to be set too large, otherwise it will cause blocking - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Open
Priority: Blocker
Resolution: Unresolved
Affects Version/s: 3.1.2
Fix Version/s: None
Component/s: Clients
Labels:
None
Environment:

Hive 3.1.2
os.name=Linux
os.arch=amd64
os.version=5.4.0-72-generic
java.version=1.8.0_162
java.vendor=Oracle Corporation

Description

In hive-default.xml.template

<property>
  <name>hive.server2.webui.max.historic.queries</name>
  <value>25</value>
  <description>The maximum number of past queries to show in HiverSever2 WebUI.</description>
</property>

Set hive.server2.webui.max.historic.queries to a relatively large value, take 20000000 as an example, start hiveserver2, it can start hiveserver normally, and logging without exception.

liky@ljq1:/usr/local/hive/conf$ hiveserver2 
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2022-05-09 20:03:41: Starting HiveServer2
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = 0b419706-4026-4a8b-80fe-b79fecbccd4f
Hive Session ID = 0f9e28d7-0081-4b2f-a743-4093c38c152d

Next, if you use beeline as a client to connect to hive and send a request for database related operations, for example, if you query all the databases, after successfully executing "show databases", beeline blocks and no other operations can be performed.

liky@ljq1:/opt/hive$ beeline
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Beeline version 3.1.2 by Apache Hive
beeline> !connect jdbc:hive2://192.168.1.194:10000/default
Connecting to jdbc:hive2://192.168.1.194:10000/default
Enter username for jdbc:hive2://192.168.1.194:10000/default: hive
Enter password for jdbc:hive2://192.168.1.194:10000/default: *****
Connected to: Apache Hive (version 3.1.2)
Driver: Hive JDBC (version 3.1.2)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://192.168.1.194:10000/default> show databases
. . . . . . . . . . . . . . . . . . . . . .> ;
INFO : Compiling command(queryId=liky_20220509202542_15382019-f07b-40ff-840d-1f720df77d8b): show databases
INFO : Concurrency mode is disabled, not creating a lock manager
INFO : Semantic Analysis Completed (retrial = false)
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:database_name, type:string, comment:from deserializer)], properties:null)
INFO : Completed compiling command(queryId=liky_20220509202542_15382019-f07b-40ff-840d-1f720df77d8b); Time taken: 0.393 seconds
INFO : Concurrency mode is disabled, not creating a lock manager
INFO : Executing command(queryId=liky_20220509202542_15382019-f07b-40ff-840d-1f720df77d8b): show databases
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing command(queryId=liky_20220509202542_15382019-f07b-40ff-840d-1f720df77d8b); Time taken: 0.109 seconds
INFO : OK
INFO : Concurrency mode is disabled, not creating a lock manager
----------------

database_name
----------------

default      
----------------
1 row selected (1.374 seconds)

Also, on the hiveserver side, a runtime null pointer exception is thrown, and the observation log throws no warnings or errors.

liky@ljq1:/usr/local/hive/conf$ hiveserver2 
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Error: Could not find or load main class org.apache.hadoop.hbase.util.GetJavaProperty
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hbase-2.2.2/hbase/maven/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2022-05-09 20:03:41: Starting HiveServer2
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = 0b419706-4026-4a8b-80fe-b79fecbccd4f
Hive Session ID = 0f9e28d7-0081-4b2f-a743-4093c38c152d
OK
Exception in thread "HiveServer2-Handler-Pool: Thread-41" java.lang.RuntimeException: java.lang.NullPointerException
      at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:89)
      at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
      at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
      at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
      at com.sun.proxy.$Proxy42.close(Unknown Source)
      at org.apache.hive.service.cli.session.SessionManager.closeSession(SessionManager.java:552)
      at org.apache.hive.service.cli.CLIService.closeSession(CLIService.java:241)
      at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService$1.deleteContext(ThriftBinaryCLIService.java:141)
      at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:300)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
      at org.apache.hive.service.cli.operation.OperationManager.getQueryId(OperationManager.java:189)
      at org.apache.hive.service.cli.operation.OperationManager.removeOperation(OperationManager.java:206)
      at org.apache.hive.service.cli.operation.OperationManager.closeOperation(OperationManager.java:287)
      at org.apache.hive.service.cli.session.HiveSessionImpl.close(HiveSessionImpl.java:751)
      at org.apache.hive.service.cli.session.HiveSessionImplwithUGI.close(HiveSessionImplwithUGI.java:93)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
      ... 14 more

If you open another beeline as a client to initiate a database operation request to the hiveserver, the beeline will also block after executing a request and the hiveserver will throw the same exception again.

If you call java API write a program to establish a jdbc connection as a client to access hive, take JDBCDemo as an example.

import demo.utils.JDBCUtils;
public class JDBCDemo{
    public static void main(String[] args) throws Exception {
        JDBCUtils.init();
        JDBCUtils.createDatabase();
        JDBCUtils.showDatabases();
        JDBCUtils.createTable();
        JDBCUtils.showTables();
        JDBCUtils.descTable();
        JDBCUtils.loadData();
        JDBCUtils.selectData();
        JDBCUtils.countData();
        JDBCUtils.dropDatabase();
        JDBCUtils.dropTable();
        JDBCUtils.destory();
    }
}

Starting the hiveserver and running the written program will block directly after the first database operation request, and the terminal will not throw any hints, in line with beeline's blocking node, the hiveserver throws the same exceptions as before.

liky@ljq1:~/hive_jdbc_test$ ./startJDBC_0.sh 
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/liky/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.1/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/liky/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Running: drop database if exists hive_jdbc_test
Running: create database hive_jdbc_test



^Cliky@ljq1:~/hive_jdbc_test$

In summary, when the value of hive.server2.webui.max.historic.queries is set to a larger value, it blocks all clients that initiate the request, and the hive server side only throws a runtime null pointer exception, and the observation log throws no related warnings and errors, causing some impact.

Attachments

Activity

People

Assignee:: ECFuzz

Reporter:: ECFuzz

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Dates

Created:: 09/May/22 13:18

Updated:: 06/Nov/22 13:57