Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-1062

PutHDFS will pass files to success when they were not successfully written with hadoop client misconfiguration

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.3.0
    • Fix Version/s: 0.4.0
    • Component/s: Extensions
    • Labels:
    • Environment:
      hadoop-2.6.0 with hadoop.security.authentication=kerberos and dfs.data.transfer.protection=privacy

      Description

      PutHDFS will create an empty file, but the data it is attempting to write will fail with this stack trace, and the flow file gets routed to success.

      2015-10-24 11:16:19,278 WARN [Thread-4674] org.apache.hadoop.hdfs.DFSClient DataStreamer Exception
      java.lang.IllegalArgumentException: null
              at javax.security.auth.callback.NameCallback.<init>(NameCallback.java:90) ~[na:1.8.0_45]
              at com.sun.security.sasl.digest.DigestMD5Client.processChallenge(DigestMD5Client.java:324) ~[na:1.8.0_45]
              at com.sun.security.sasl.digest.DigestMD5Client.evaluateChallenge(DigestMD5Client.java:220) ~[na:1.8.0_45]
              at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslParticipant.evaluateChallengeOrResponse(SaslParticipant.java:113) ~[na:na]
              at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:451) ~[na:na]
              at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getSaslStreams(SaslDataTransferClient.java:390) ~[na:na]
              at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:262) ~[na:na]
              at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:210) ~[na:na]
              at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.socketSend(SaslDataTransferClient.java:182) ~[na:na]
              at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1413) ~[na:na]
              at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361) ~[na:na]
              at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588) ~[na:na]
      2015-10-24 11:16:19,303 INFO [Timer-Driven Process Thread-2] o.apache.nifi.processors.hadoop.PutHDFS PutHDFS[id=3c30a474-86b4-45fc-b771-95d7a1b5054d] copied StandardFlowFileRecord[uuid=5a3deada-9739-474f-a83d-0447ad5aefd9,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1445694733273-1, container=default, section=1], offset=87, length=29],offset=0,name=x11.txt,size=29] to HDFS at /use/hdfs/x11.txt in 56 milliseconds at a rate of 511 bytes/sec
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ozhurakousky Oleg Zhurakousky
                Reporter:
                tkurc Tony Kurc
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: