Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-9478

Fix race conditions during the initialization of Configuration related to deprecatedKeyMap

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.3.0
    • Component/s: conf
    • Labels:
      None
    • Environment:
    • Target Version/s:

      Description

      When we lanuch the client appliation which use kerberos security,the FileSystem can't be create because the exception ' java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.security.SecurityUtil'.

      I check the exception stack trace,it maybe caused by the unsafe get operation of the deprecatedKeyMap which used by the org.apache.hadoop.conf.Configuration.

      So I write a simple test case:

      import org.apache.hadoop.conf.Configuration;
      import org.apache.hadoop.fs.FileSystem;
      import org.apache.hadoop.hdfs.HdfsConfiguration;

      public class HTest {
      public static void main(String[] args) throws Exception

      { Configuration conf = new Configuration(); conf.addResource("core-site.xml"); conf.addResource("hdfs-site.xml"); FileSystem fileSystem = FileSystem.get(conf); System.out.println(fileSystem); System.exit(0); }

      }

      Then I launch this test case many times,the following exception is thrown:

      Exception in thread "TGT Renewer for XXX" java.lang.ExceptionInInitializerError
      at org.apache.hadoop.security.UserGroupInformation.getTGT(UserGroupInformation.java:719)
      at org.apache.hadoop.security.UserGroupInformation.access$1100(UserGroupInformation.java:77)
      at org.apache.hadoop.security.UserGroupInformation$1.run(UserGroupInformation.java:746)
      at java.lang.Thread.run(Thread.java:662)
      Caused by: java.lang.ArrayIndexOutOfBoundsException: 16
      at java.util.HashMap.getEntry(HashMap.java:345)
      at java.util.HashMap.containsKey(HashMap.java:335)
      at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1989)
      at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1867)
      at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1785)
      at org.apache.hadoop.conf.Configuration.get(Configuration.java:712)
      at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:731)
      at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1047)
      at org.apache.hadoop.security.SecurityUtil.<clinit>(SecurityUtil.java:76)
      ... 4 more
      Exception in thread "main" java.io.IOException: Couldn't create proxy provider class org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
      at org.apache.hadoop.hdfs.NameNodeProxies.createFailoverProxyProvider(NameNodeProxies.java:453)
      at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:133)
      at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:436)
      at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:403)
      at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:125)
      at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2262)
      at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:86)
      at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2296)
      at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2278)
      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:316)
      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:162)
      at HTest.main(HTest.java:11)
      Caused by: java.lang.reflect.InvocationTargetException
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
      at org.apache.hadoop.hdfs.NameNodeProxies.createFailoverProxyProvider(NameNodeProxies.java:442)
      ... 11 more
      Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.security.SecurityUtil
      at org.apache.hadoop.net.NetUtils.createSocketAddrForHost(NetUtils.java:231)
      at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:211)
      at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:159)
      at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:148)
      at org.apache.hadoop.hdfs.DFSUtil.getAddressesForNameserviceId(DFSUtil.java:452)
      at org.apache.hadoop.hdfs.DFSUtil.getAddresses(DFSUtil.java:434)
      at org.apache.hadoop.hdfs.DFSUtil.getHaNnRpcAddresses(DFSUtil.java:496)
      at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.<init>(ConfiguredFailoverProxyProvider.java:88)
      ... 16 more

      If the HashMap used at multi-thread enviroment,not only the put operation be synchronized,the get operation(eg. containKey) should be synchronzied too.

      The simple solution is trigger the init of SecurityUtil before creating the FileSystem,but I think it's should be synchronized for get of deprecatedKeyMap.

      Thanks.

        Attachments

        1. HADOOP-9478.001.patch
          22 kB
          Colin McCabe
        2. HADOOP-9478.002.patch
          31 kB
          Colin McCabe
        3. HADOOP-9478.003.patch
          86 kB
          Colin McCabe
        4. HADOOP-9478.004.patch
          90 kB
          Colin McCabe
        5. HADOOP-9478.005.patch
          90 kB
          Colin McCabe
        6. hadoop-9478-1.patch
          1 kB
          Andrew Wang
        7. hadoop-9478-2.patch
          7 kB
          Andrew Wang

          Activity

            People

            • Assignee:
              cmccabe Colin McCabe
              Reporter:
              d0ngw Dongyong Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: