Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-2435

GetHDFSSequenceFile sequence reader throwing away authenticated filesystem and creating its own

    XMLWordPrintableJSON

    Details

      Description

      Hadoop Kerberos has a way of accessing things as an authenticated user
      UserGroupInformation.doAs
      NIFI is doing that to get access to the filesystem in order to basically tell hadoop to use the correct credentials so our processor does that correctly and then passes the results into the sequence reader unfortunately, the sequence reader then throws away our authenticated filesystem and creates its own without wrapping in a UGI.doAs

      More context:
      The constructor used for org.apache.hadoop.io.SequenceFile.Reader in both KeyValueReader and ValueReader is deprecated and has been turned into a callthrough that only uses the passed in FileSystem to get the qualified path[1]. I've modified those classes to do that logic and then wrapped the callsite in the processor in a UGI.doAs if it is configured to use a UGI

      [1]: https://github.com/apache/hadoop/commit/0c462b223f151208ff7bd5148cee0e436c23d795#diff-e70823f17863445da52875999e4b19a2L1592

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                bryanrosander@gmail.com Bryan Rosander
                Reporter:
                digitalplummer Juan Sequeiros
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: