Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-7021

Impala query fails with BlockMissingException while HDFS throws SSLHandshakeException

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Cannot Reproduce
    • Impala 2.12.0
    • None
    • Backend, Security
    • None
    • ghx-label-9

    Description

      Impala query failed with status

      Status: Disk I/O error: Error reading from HDFS file: hdfs://vc1512.halxg.cloudera.com:8020/user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba750000005b_76476061_data.0.parq Error(255): Unknown error 255 Root cause: BlockMissingException: Could not obtain block: BP-1933019065-10.17.221.22-1485905285703:blk_1084803316_11063749 file=/user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba750000005b_76476061_data.0.parq
      

      Impala log

      I0513 19:50:07.248292 150900 coordinator.cc:1092] Release admission control resources for query_id=f14b2814eea7b58b:c2eeddb400000000
      W0513 19:50:11.313998 109914 ShortCircuitCache.java:826] ShortCircuitCache(0x188f4ffc): could not load 1084803339_BP-1933019065-10.17.221.22-1485905285703 due to InvalidToken exception.
      Java exception follows:
      org.apache.hadoop.security.token.SecretManager$InvalidToken: access control error while attempting to set up short-circuit access to /user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba7500000004_131907241_data.0.parq
              at org.apache.hadoop.hdfs.BlockReaderFactory.requestFileDescriptors(BlockReaderFactory.java:653)
              at org.apache.hadoop.hdfs.BlockReaderFactory.createShortCircuitReplicaInfo(BlockReaderFactory.java:552)
              at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:804)
              at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:738)
              at org.apache.hadoop.hdfs.BlockReaderFactory.getBlockReaderLocal(BlockReaderFactory.java:485)
              at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355)
              at org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1219)
              at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1159)
              at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1533)
              at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1492)
              at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92)
      W0513 19:50:11.317173 109914 ShortCircuitCache.java:826] ShortCircuitCache(0x188f4ffc): could not load 1084803339_BP-1933019065-10.17.221.22-1485905285703 due to InvalidToken exception.
      Java exception follows:
      org.apache.hadoop.security.token.SecretManager$InvalidToken: access control error while attempting to set up short-circuit access to /user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba7500000004_131907241_data.0.parq
              at org.apache.hadoop.hdfs.BlockReaderFactory.requestFileDescriptors(BlockReaderFactory.java:653)
              at org.apache.hadoop.hdfs.BlockReaderFactory.createShortCircuitReplicaInfo(BlockReaderFactory.java:552)
              at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:804)
              at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:738)
              at org.apache.hadoop.hdfs.BlockReaderFactory.getBlockReaderLocal(BlockReaderFactory.java:485)
              at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355)
              at org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1219)
              at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1159)
              at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1533)
              at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1492)
              at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92)
      W0513 19:50:11.318079 109914 DFSInputStream.java:1275] Connection failure: Failed to connect to /10.17.187.36:20002 for file /user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba7500000004_131907241_data.0.parq for block BP-1933019065-10.17.221.22-1485905285703:blk_1084803339_11063772:org.apache.hadoop.security.token.SecretManager$InvalidToken: access control error while attempting to set up short-circuit access to /user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba7500000004_131907241_data.0.parq
      

      Impala error log

      hdfsPread: FSDataInputStream#read error:
      BlockMissingException: Could not obtain block: BP-1933019065-10.17.221.22-1485905285703:blk_1084803339_11063772 file=/user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba7500000004_131907241_data.0.parqorg.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1933019065-10.17.221.22-1485905285703:blk_1084803339_11063772 file=/user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba7500000004_131907241_data.0.parq
              at org.apache.hadoop.hdfs.DFSInputStream.refetchLocations(DFSInputStream.java:1048)
              at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:1031)
              at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:1010)
              at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1156)
              at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1533)
              at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1492)
              at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92)
      

      NN log

      2018-05-13 19:51:54,857 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 11 millisecond(s).
      2018-05-13 19:51:57,409 WARN org.mortbay.log: EXCEPTION
      javax.net.ssl.SSLHandshakeException: no cipher suites in common
      	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
      	at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949)
      	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302)
      	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:292)
      	at sun.security.ssl.ServerHandshaker.chooseCipherSuite(ServerHandshaker.java:1045)
      	at sun.security.ssl.ServerHandshaker.clientHello(ServerHandshaker.java:741)
      	at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:224)
      	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
      	at sun.security.ssl.Handshaker.process_record(Handshaker.java:961)
      	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062)
      	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
      	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
      	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
      	at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:723)
      	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
      2018-05-13 19:52:00,591 INFO BlockStateChange: BLOCK* processReport 0x422a616a86ad5ba4: discarded non-initial block report from DatanodeRegistration(10.17.187.39, datanodeUuid=42d8151a-7aaf-4f78-90d8-7d1bda177d5e, infoPort=0, infoSecurePort=20004, ipcPort=20001, storageInfo=lv=-56;cid=cluster2;nsid=606873798;c=0) because namenode still in startup phase
      

      Datanode log

      2018-05-13 19:50:08,946 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: va1030.halxg.cloudera.com:20002:DataXceiver error processing un
      known operation  src: /10.17.246.46:44103 dst: /10.17.187.40:20002
      javax.security.sasl.SaslException: DIGEST-MD5: IO error acquiring password [Caused by org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncr
      yptionKeyException: Can't re-compute encryption key for nonce, since the required block key (keyID=1802676210) doesn't exist. Current key: 180
      2676198]
              at com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:598)
              at com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244)
              at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslParticipant.evaluateChallengeOrResponse(SaslParticipant.java:115)
              at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:372)
              at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getEncryptedStreams(SaslDataTransferServer.java:178)
              at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.receive(SaslDataTransferServer.java:110)
              at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:193)
              at java.lang.Thread.run(Thread.java:745)
      Caused by: org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException: Can't re-compute encryption key for nonce, since the re
      quired block key (keyID=1802676210) doesn't exist. Current key: 1802676198
              at org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.retrieveDataEncryptionKey(BlockTokenSecretManager.java:430)
              at org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.retrieveDataEncryptionKey(BlockPoolTokenSecretManager.java:130)
              at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getEncryptionKeyFromUserName(SaslDataTransferServer.java:267)
              at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.access$000(SaslDataTransferServer.java:71)
              at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer$1.apply(SaslDataTransferServer.java:175)
              at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer$SaslServerCallbackHandler.handle(SaslDataTransferServer.java:239)
              at com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589)
              ... 7 more
      2018-05-13 19:50:08,949 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: va1030.halxg.cloudera.com:20002:DataXceiver error processing unknown operation  src: /10.17.246.46:44104 dst: /10.17.187.40:20002
      

      Attachments

        Activity

          People

            Unassigned Unassigned
            mmokhtar Mostafa Mokhtar
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: