Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-20540 [umbrella] Hadoop 3 compatibility
  3. HBASE-20244

NoSuchMethodException when retrieving private method decryptEncryptedDataEncryptionKey from DFSClient

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 2.0.0, 2.0.1
    • 3.0.0-alpha-1, 2.1.0, 2.0.2
    • wal
    • None
    • Reviewed
    • Hide
      HDFS-12574 made an incompatible change to HdfsKMSUtil with different method signature.
      This issue uses reflection to try known method signatures in order to work with hadoop releases with and without HDFS-12574
      Show
      HDFS-12574 made an incompatible change to HdfsKMSUtil with different method signature. This issue uses reflection to try known method signatures in order to work with hadoop releases with and without HDFS-12574

    Description

      I was running unit test against hadoop 3.0.1 RC and saw the following in test output:

      ERROR [RS-EventLoopGroup-3-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(267): Couldn't properly initialize access to HDFS internals. Please update  your WAL Provider to not make use of the 'asyncfs' provider. See HBASE-16110 for more information.
      java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo)
        at java.lang.Class.getDeclaredMethod(Class.java:2130)
        at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:232)
        at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.<clinit>(FanOutOneBlockAsyncDFSOutputSaslHelper.java:262)
        at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:661)
        at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:118)
        at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:720)
        at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:715)
        at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
        at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)
        at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)
        at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
        at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
        at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
        at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:306)
        at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:341)
        at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633)
        at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
      

      The private method was moved by HDFS-12574 to HdfsKMSUtil with different signature.

      To accommodate the above method movement, it seems we need to call the following method of DFSClient :

        public KeyProvider getKeyProvider() throws IOException {
      

      Since the new decryptEncryptedDataEncryptionKey method has this signature:

        static KeyVersion decryptEncryptedDataEncryptionKey(FileEncryptionInfo
              feInfo, KeyProvider keyProvider) throws IOException {
      

      Attachments

        1. 20244.v1.txt
          4 kB
          Ted Yu
        2. 20244.v1.txt
          4 kB
          Ted Yu
        3. 20244.v1.txt
          4 kB
          Ted Yu
        4. HBASE-20244.patch
          11 kB
          Duo Zhang
        5. HBASE-20244-v1.patch
          10 kB
          Duo Zhang

        Issue Links

          Activity

            People

              yuzhihong@gmail.com Ted Yu
              yuzhihong@gmail.com Ted Yu
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: