Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14872

CryptoInputStream should implement unbuffer

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.4
    • Fix Version/s: 3.1.0, 2.10.0, 2.9.1, 3.0.1
    • Component/s: fs, security
    • Labels:
      None
    • Target Version/s:

      Description

      Discovered in IMPALA-5909.

      Opening an encrypted HDFS file returns a chain of wrapped input streams:

      HdfsDataInputStream
        CryptoInputStream
          DFSInputStream
      

      If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, FSDataInputStream#unbuffer will be called:

          try {
            ((CanUnbuffer)in).unbuffer();
          } catch (ClassCastException e) {
            throw new UnsupportedOperationException("this stream does not " +
                "support unbuffering.");
          }
      

      If the in class does not implement CanUnbuffer, UOE will be thrown. If the application is not careful, tons of UOEs will show up in logs.

      In comparison, opening an non-encrypted HDFS file returns this chain:

      HdfsDataInputStream
        DFSInputStream
      

      DFSInputStream implements CanUnbuffer.

      It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons:

      • Release buffer, cache, or any other resource when instructed
      • Able to call its wrapped DFSInputStream unbuffer
      • Avoid the UOE described above. Applications may not handle the UOE very well.

        Attachments

        1. HADOOP-14872.001.patch
          5 kB
          John Zhuge
        2. HADOOP-14872.002.patch
          5 kB
          John Zhuge
        3. HADOOP-14872.003.patch
          6 kB
          John Zhuge
        4. HADOOP-14872.004.patch
          9 kB
          John Zhuge
        5. HADOOP-14872.005.patch
          14 kB
          John Zhuge
        6. HADOOP-14872.006.patch
          14 kB
          John Zhuge
        7. HADOOP-14872.007.patch
          15 kB
          John Zhuge
        8. HADOOP-14872.008.patch
          17 kB
          John Zhuge
        9. HADOOP-14872.009.patch
          17 kB
          John Zhuge
        10. HADOOP-14872.010.patch
          20 kB
          John Zhuge
        11. HADOOP-14872.011.patch
          20 kB
          John Zhuge
        12. HADOOP-14872.012.patch
          20 kB
          John Zhuge
        13. HADOOP-14872.013.patch
          24 kB
          John Zhuge

          Issue Links

            Activity

              People

              • Assignee:
                jzhuge John Zhuge
                Reporter:
                jzhuge John Zhuge
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: