Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14872

CryptoInputStream should implement unbuffer

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.6.4
    • 3.1.0, 2.10.0, 2.9.1, 3.0.1
    • fs, security
    • None

    Description

      Discovered in IMPALA-5909.

      Opening an encrypted HDFS file returns a chain of wrapped input streams:

      HdfsDataInputStream
        CryptoInputStream
          DFSInputStream
      

      If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, FSDataInputStream#unbuffer will be called:

          try {
            ((CanUnbuffer)in).unbuffer();
          } catch (ClassCastException e) {
            throw new UnsupportedOperationException("this stream does not " +
                "support unbuffering.");
          }
      

      If the in class does not implement CanUnbuffer, UOE will be thrown. If the application is not careful, tons of UOEs will show up in logs.

      In comparison, opening an non-encrypted HDFS file returns this chain:

      HdfsDataInputStream
        DFSInputStream
      

      DFSInputStream implements CanUnbuffer.

      It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons:

      • Release buffer, cache, or any other resource when instructed
      • Able to call its wrapped DFSInputStream unbuffer
      • Avoid the UOE described above. Applications may not handle the UOE very well.

      Attachments

        1. HADOOP-14872.013.patch
          24 kB
          John Zhuge
        2. HADOOP-14872.012.patch
          20 kB
          John Zhuge
        3. HADOOP-14872.011.patch
          20 kB
          John Zhuge
        4. HADOOP-14872.010.patch
          20 kB
          John Zhuge
        5. HADOOP-14872.009.patch
          17 kB
          John Zhuge
        6. HADOOP-14872.008.patch
          17 kB
          John Zhuge
        7. HADOOP-14872.007.patch
          15 kB
          John Zhuge
        8. HADOOP-14872.006.patch
          14 kB
          John Zhuge
        9. HADOOP-14872.005.patch
          14 kB
          John Zhuge
        10. HADOOP-14872.004.patch
          9 kB
          John Zhuge
        11. HADOOP-14872.003.patch
          6 kB
          John Zhuge
        12. HADOOP-14872.002.patch
          5 kB
          John Zhuge
        13. HADOOP-14872.001.patch
          5 kB
          John Zhuge

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            jzhuge John Zhuge
            jzhuge John Zhuge
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment