1. Hive
  2. HIVE-5172

TUGIContainingTransport returning null transport, causing intermittent SocketTimeoutException on hive client and NullPointerException in TUGIBasedProcessor on the server


    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.9.0, 0.10.0, 0.11.0
    • Fix Version/s: None
    • Component/s: Metastore
    • Labels:


      We are running into frequent problem using HCatalog 0.4.1 (Hive Metastore Server 0.9) where we get connection reset or connection timeout errors on the client and NullPointerException in TUGIBasedProcessor on the server.

      hive client logs:
      org.apache.thrift.transport.TTransportException: Read timed out
      at org.apache.thrift.transport.TTransport.readAll(
      at org.apache.thrift.protocol.TBinaryProtocol.readAll(
      at org.apache.thrift.protocol.TBinaryProtocol.readI32(
      at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(
      at org.apache.thrift.TServiceClient.receiveBase(
      at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(
      at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(
      at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.openStore(
      at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(
      at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(
      at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(
      at org.apache.hadoop.hive.ql.metadata.Hive.getTable(
      at org.apache.hadoop.hive.ql.metadata.Hive.getTable(
      at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(
      at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(
      at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(
      at org.apache.hadoop.hive.ql.Driver.compile(
      at org.apache.hadoop.hive.ql.Driver.compile(
      at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(
      at org.apache.hadoop.hive.cli.CliDriver.processCmd(
      at org.apache.hadoop.hive.cli.CliDriver.processLine(
      at org.apache.hadoop.hive.cli.CliDriver.processLine(
      at org.apache.hadoop.hive.cli.CliDriver.main(
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(
      at java.lang.reflect.Method.invoke(
      at org.apache.hadoop.util.RunJar.main(
      Caused by: Read timed out
      at Method)
      ... 31 more
      hive metastore server logs:
      2013-07-26 06:34:52,853 ERROR server.TThreadPoolServer ( - Error occurred during processing of message.
              at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.setIpAddress(
              at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(
              at org.apache.thrift.server.TThreadPoolServer$
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
      at java.util.concurrent.ThreadPoolExecutor$

      Adding some extra debug log messages in TUGIBasedProcessor, noticed that the TUGIContainingTransport is null which results in NullPointerException on the server.

      Further drilling into TUGIContainingTransport noticed that getTransport() returns a null which causes the above

      Further corelating with GC logs observed that that error always hits when the CMS GC has just kicked in
      (but does not happen after every GC)

      Put some debugging code in TUGIContainingTransport.getTransport() and I tracked it down to

      public TUGIContainingTransport getTransport(TTransport trans) {
            // UGI information is not available at connection setup time, it will be set later
            // via set_ugi() rpc.
            transMap.putIfAbsent(trans, new TUGIContainingTransport(trans));
            //return transMap.get(trans); //<-change
            TUGIContainingTransport retTrans = transMap.get(trans);
            if ( retTrans == null ) {
                   LOGGER.error ("cannot find transport that was in map !!")
      	     return retTrans;
             }  else {
                   LOGGER.debug ("found transport in map")
                   return retTrans;    

      When we run this in our test environment, see that we run into the problem just after GC runs,
      and "cannot find transport that was in the map!!" message gets logged.

      Could the GC be collecting entries from transMap, just before the we get it

      Tried a minor change which seems to work

      public TUGIContainingTransport getTransport(TTransport trans) {
         TUGIContainingTransport tugiTrans = transMap.get(trans);
          if ( tugiTrans == null ) {
            // UGI information is not available at connection setup time, it will be set later
            // via set_ugi() rpc.
             tugiTrans = new TUGIContainingTransport(trans); //get a new TUGIContainingTransport
              transMap.putIfAbsent(trans, tugiTrans);
         return tugiTrans;

      My questions for hive and thrift experts

      1.) Do we need to use a ConcurrentMap

      ConcurrentMap<TTransport, TUGIContainingTransport> transMap = new MapMaker().weakKeys().weakValues().makeMap();

      It does use == to compare keys (which might be the problem), also in this case we cant rely on the trans to be always there in the transMap, even after a put, so in that case change above probably makes sense

      2.) Is it better idea to use WeakHashMap with WeakReference value instead ? (was looking at org.apache.thrift.transport.TSaslServerTransport, especially change made by THRIFT-1468 as an example)


      private static Map<TTransport, WeakReference<TUGIContainingTransport>> transMap = Collections.synchronizedMap(new WeakHashMap<TTransport, WeakReference<TUGIContainingTransport>>());

      getTransport() would be something like

      public TUGIContainingTransport getTransport(TTransport trans) {
            WeakReference<TUGIContainingTransport> ret = transMap.get(trans);
            if (ret == null || ret.get() == null) {
              ret = new WeakReference<TUGIContainingTransport>(new TUGIContainingTransport(trans));  
              transMap.put(trans, ret); // No need for putIfAbsent().
               // Concurrent calls to getTransport() will pass in different TTransports.
            return ret.get();

      3.) Is it absolutely essential that successive calls to get getTransport() need to retrieve the same TUGIContainingTransport object or its ok to create a new one, if its not available in the map ?

        Issue Links

        There are no Sub-Tasks for this issue.


          No work has yet been logged on this issue.


            • Assignee:
            • Votes:
              0 Vote for this issue
              6 Start watching this issue


              • Created: