Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-20540 [umbrella] Hadoop 3 compatibility
  3. HBASE-20244

NoSuchMethodException when retrieving private method decryptEncryptedDataEncryptionKey from DFSClient

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.0.0, 2.0.1
    • Fix Version/s: 3.0.0, 2.1.0, 2.0.2, 2.2.0
    • Component/s: wal
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      HDFS-12574 made an incompatible change to HdfsKMSUtil with different method signature.
      This issue uses reflection to try known method signatures in order to work with hadoop releases with and without HDFS-12574
      Show
      HDFS-12574 made an incompatible change to HdfsKMSUtil with different method signature. This issue uses reflection to try known method signatures in order to work with hadoop releases with and without HDFS-12574

      Description

      I was running unit test against hadoop 3.0.1 RC and saw the following in test output:

      ERROR [RS-EventLoopGroup-3-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(267): Couldn't properly initialize access to HDFS internals. Please update  your WAL Provider to not make use of the 'asyncfs' provider. See HBASE-16110 for more information.
      java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo)
        at java.lang.Class.getDeclaredMethod(Class.java:2130)
        at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:232)
        at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.<clinit>(FanOutOneBlockAsyncDFSOutputSaslHelper.java:262)
        at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:661)
        at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:118)
        at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:720)
        at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:715)
        at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
        at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)
        at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)
        at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
        at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
        at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
        at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:306)
        at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:341)
        at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633)
        at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
      

      The private method was moved by HDFS-12574 to HdfsKMSUtil with different signature.

      To accommodate the above method movement, it seems we need to call the following method of DFSClient :

        public KeyProvider getKeyProvider() throws IOException {
      

      Since the new decryptEncryptedDataEncryptionKey method has this signature:

        static KeyVersion decryptEncryptedDataEncryptionKey(FileEncryptionInfo
              feInfo, KeyProvider keyProvider) throws IOException {
      

        Attachments

        1. 20244.v1.txt
          4 kB
          Ted Yu
        2. 20244.v1.txt
          4 kB
          Ted Yu
        3. 20244.v1.txt
          4 kB
          Ted Yu
        4. HBASE-20244.patch
          11 kB
          Duo Zhang
        5. HBASE-20244-v1.patch
          10 kB
          Duo Zhang

          Issue Links

            Activity

              People

              • Assignee:
                yuzhihong@gmail.com Ted Yu
                Reporter:
                yuzhihong@gmail.com Ted Yu
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: