Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14346

CryptoOutputStream throws IOException on flush() if stream is closed

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.3
    • Fix Version/s: 2.9.0, 3.0.0-alpha4, 2.8.2
    • Component/s: None
    • Labels:
      None
    • Flags:
      Patch

      Description

      With Druid and Hadoop 2.7.3 I see that calls to flush() on a closed CryptoOutputStream throws IOException.

      Although flushing closed stream is dodgy, the culprit is actually CryptoOutputStream, according to the java documentation it look actually authorised.

      The OutputStream java doc (http://docs.oracle.com/javase/8/docs/api/java/io/OutputStream.html#flush()) shows that flush should behave like close(), which allows being called on a closed stream without exception (as opposed to write() and others which clearly says should throw an exception)

      1. HADOOP-14346-1.patch
        0.7 kB
        Pierre Lacave
      2. CryptoOutputStream.flush-no-throw-on-closed.patch
        0.7 kB
        Pierre Lacave

        Activity

        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - 2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11639 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11639/)
        HADOOP-14346. CryptoOutputStream throws IOException if stream is closed. (stevel: rev 61cda39e4eec00678706d96e90026455ecfc0a90)

        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoOutputStream.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11639 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11639/ ) HADOOP-14346 . CryptoOutputStream throws IOException if stream is closed. (stevel: rev 61cda39e4eec00678706d96e90026455ecfc0a90) (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoOutputStream.java
        Hide
        stevel@apache.org Steve Loughran added a comment -

        +1
        Committed to branch-2.8+, thanks.

        Show
        stevel@apache.org Steve Loughran added a comment - +1 Committed to branch-2.8+, thanks.
        Hide
        stevel@apache.org Steve Loughran added a comment -

        Actually, I'm thinking of applying it

        1. there's ambiguity in what things are meant to do
        2. if it works on non-crypto streams, it's clear that normal codepath ignores the operation when closed
        3. which means it's a visible change in behaviour

        We hit a similar issue with thread safety: java.io says "no thread safety", HDFS had it, HBase relied on it, so this class follows the HDFS rules (HADOOP-11710). This is similar

        Show
        stevel@apache.org Steve Loughran added a comment - Actually, I'm thinking of applying it there's ambiguity in what things are meant to do if it works on non-crypto streams, it's clear that normal codepath ignores the operation when closed which means it's a visible change in behaviour We hit a similar issue with thread safety: java.io says "no thread safety", HDFS had it, HBase relied on it, so this class follows the HDFS rules ( HADOOP-11710 ). This is similar
        Hide
        pierre.lacave Pierre Lacave added a comment -

        so we'll fix the druid case in druid (cleaning up those late flush), feel free to close this issue if you think the current CryptoOutputStream flush behaviour is fine.

        Thanks

        Show
        pierre.lacave Pierre Lacave added a comment - so we'll fix the druid case in druid (cleaning up those late flush), feel free to close this issue if you think the current CryptoOutputStream flush behaviour is fine. Thanks
        Hide
        pierre.lacave Pierre Lacave added a comment -

        I agree the flush on closed stream shouldn't happen in the first place, will see with Druid what update can be done on their side.

        Show
        pierre.lacave Pierre Lacave added a comment - I agree the flush on closed stream shouldn't happen in the first place, will see with Druid what update can be done on their side.
        Hide
        stevel@apache.org Steve Loughran added a comment -

        One thing to consider here is: is it the case that you may call flush() on a closed stream? The docs don't explicitly say you can, or that you can't. They just don't say that some form of IOE will be raised if called on a closed stream

        Show
        stevel@apache.org Steve Loughran added a comment - One thing to consider here is: is it the case that you may call flush() on a closed stream? The docs don't explicitly say you can, or that you can't. They just don't say that some form of IOE will be raised if called on a closed stream
        Hide
        stevel@apache.org Steve Loughran added a comment -

        Looks like a sign of a larger problem: a lot of subclasses of flush appear to assume their stream is open.

        Show
        stevel@apache.org Steve Loughran added a comment - Looks like a sign of a larger problem: a lot of subclasses of flush appear to assume their stream is open.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 21s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 13m 19s trunk passed
        +1 compile 15m 12s trunk passed
        +1 checkstyle 0m 35s trunk passed
        +1 mvnsite 1m 5s trunk passed
        +1 mvneclipse 0m 20s trunk passed
        -1 findbugs 1m 21s hadoop-common-project/hadoop-common in trunk has 17 extant Findbugs warnings.
        +1 javadoc 0m 49s trunk passed
        +1 mvninstall 0m 38s the patch passed
        +1 compile 14m 16s the patch passed
        +1 javac 14m 16s the patch passed
        +1 checkstyle 0m 36s the patch passed
        +1 mvnsite 1m 1s the patch passed
        +1 mvneclipse 0m 20s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 30s the patch passed
        +1 javadoc 0m 48s the patch passed
        -1 unit 7m 53s hadoop-common in the patch failed.
        +1 asflicense 0m 34s The patch does not generate ASF License warnings.
        62m 34s



        Reason Tests
        Failed junit tests hadoop.security.TestKDiag



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ac17dc
        JIRA Issue HADOOP-14346
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12864684/HADOOP-14346-1.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 37f9c4437891 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / fda86ef
        Default Java 1.8.0_121
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12168/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12168/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12168/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12168/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 13m 19s trunk passed +1 compile 15m 12s trunk passed +1 checkstyle 0m 35s trunk passed +1 mvnsite 1m 5s trunk passed +1 mvneclipse 0m 20s trunk passed -1 findbugs 1m 21s hadoop-common-project/hadoop-common in trunk has 17 extant Findbugs warnings. +1 javadoc 0m 49s trunk passed +1 mvninstall 0m 38s the patch passed +1 compile 14m 16s the patch passed +1 javac 14m 16s the patch passed +1 checkstyle 0m 36s the patch passed +1 mvnsite 1m 1s the patch passed +1 mvneclipse 0m 20s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 30s the patch passed +1 javadoc 0m 48s the patch passed -1 unit 7m 53s hadoop-common in the patch failed. +1 asflicense 0m 34s The patch does not generate ASF License warnings. 62m 34s Reason Tests Failed junit tests hadoop.security.TestKDiag Subsystem Report/Notes Docker Image:yetus/hadoop:0ac17dc JIRA Issue HADOOP-14346 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12864684/HADOOP-14346-1.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 37f9c4437891 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / fda86ef Default Java 1.8.0_121 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12168/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12168/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12168/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12168/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 patch 0m 7s HADOOP-14346 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



        Subsystem Report/Notes
        JIRA Issue HADOOP-14346
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12864651/CryptoOutputStream.flush-no-throw-on-closed.patch
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12166/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 7s HADOOP-14346 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HADOOP-14346 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12864651/CryptoOutputStream.flush-no-throw-on-closed.patch Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12166/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

          People

          • Assignee:
            pierre.lacave Pierre Lacave
            Reporter:
            pierre.lacave Pierre Lacave
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development