Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8386

Improve synchronization of 'streamer' reference in DFSOutputStream

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: hdfs-client
    • Labels:
      None
    • Target Version/s:

      Description

      Presently DFSOutputStream#streamer object reference is accessed inconsistently with respect to synchronization. It would be good to improve this part. This has been noticed when implementing the erasure coding feature.

      Please refer the related discussion thread in the jira HDFS-8294 for more details.

      1. HDFS-8386-00.patch
        17 kB
        Rakesh R
      2. HDFS-8386-01.patch
        17 kB
        Rakesh R

        Issue Links

          Activity

          Hide
          rakeshr Rakesh R added a comment -

          The patch is just a simple refactor by getting the streamer object reference through synchronized getStreamer() function call instead of directly using it.

          Show
          rakeshr Rakesh R added a comment - The patch is just a simple refactor by getting the streamer object reference through synchronized getStreamer() function call instead of directly using it.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 43s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 31s There were no new javac warning messages.
          +1 javadoc 9m 41s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 15s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 3m 5s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 native 3m 13s Pre-build of native portion
          -1 hdfs tests 165m 2s Tests failed in hadoop-hdfs.
              208m 4s  



          Reason Tests
          Failed unit tests hadoop.tracing.TestTraceAdmin
            hadoop.tools.TestHdfsConfigFields



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12732497/HDFS-8386-00.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / fcd0702
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10950/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10950/testReport/
          Java 1.7.0_55
          uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10950/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 43s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 31s There were no new javac warning messages. +1 javadoc 9m 41s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 15s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 3m 5s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 13s Pre-build of native portion -1 hdfs tests 165m 2s Tests failed in hadoop-hdfs.     208m 4s   Reason Tests Failed unit tests hadoop.tracing.TestTraceAdmin   hadoop.tools.TestHdfsConfigFields Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732497/HDFS-8386-00.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / fcd0702 hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10950/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10950/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10950/console This message was automatically generated.
          Hide
          rakeshr Rakesh R added a comment -

          I haven't included unit test case as this is minor code refactoring. Also, test case failures are unrelated to the patch. Could some one help to review the proposed changes. Thanks!

          Show
          rakeshr Rakesh R added a comment - I haven't included unit test case as this is minor code refactoring. Also, test case failures are unrelated to the patch. Could some one help to review the proposed changes. Thanks!
          Hide
          andrew.wang Andrew Wang added a comment -

          Patch looks good thanks for working on this Rakesh. One question: should we take this opportunity to tighten the visibility of streamer to private?

          +1 pending though

          Show
          andrew.wang Andrew Wang added a comment - Patch looks good thanks for working on this Rakesh. One question: should we take this opportunity to tighten the visibility of streamer to private? +1 pending though
          Hide
          rakeshr Rakesh R added a comment -

          Thank you Andrew Wang for the interest.

          streamer is used by the erasure coding element DFSStripedOutputStream.java#L283. Considering this usage, can we keep the visibility same as protected. But I feel it is possible to to make the visibility to private, following is one proposal to make it private. I hope the below code snippet will be helpful to know about the proposed changes. If you agree I'm happy to do the changes accordingly. Any thoughts?

          1. Expose update and getter API in DFSOutputStream.java:
              protected synchronized void updateStreamer(DataStreamer streamer) {
                this.streamer = streamer;
              }
            
              protected synchronized DataStreamer getStreamer() {
                return streamer;
              }
            
          2. Modify the logic to access streamer in DFSStripedOutputStream.java like,
              private synchronized StripedDataStreamer getCurrentStreamer() {
                return (StripedDataStreamer)super.getStreamer();
              }
            
              private synchronized StripedDataStreamer setCurrentStreamer(int i) {
                super.updateStreamer(streamers.get(i));
                return getCurrentStreamer();
              }
            
          Show
          rakeshr Rakesh R added a comment - Thank you Andrew Wang for the interest. streamer is used by the erasure coding element DFSStripedOutputStream.java#L283 . Considering this usage, can we keep the visibility same as protected . But I feel it is possible to to make the visibility to private , following is one proposal to make it private . I hope the below code snippet will be helpful to know about the proposed changes. If you agree I'm happy to do the changes accordingly. Any thoughts? Expose update and getter API in DFSOutputStream.java: protected synchronized void updateStreamer(DataStreamer streamer) { this .streamer = streamer; } protected synchronized DataStreamer getStreamer() { return streamer; } Modify the logic to access streamer in DFSStripedOutputStream.java like, private synchronized StripedDataStreamer getCurrentStreamer() { return (StripedDataStreamer) super .getStreamer(); } private synchronized StripedDataStreamer setCurrentStreamer( int i) { super .updateStreamer(streamers.get(i)); return getCurrentStreamer(); }
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks Rakesh R for the update. I like the getter and setter proposal #1, it seems cleaner. Could we name it setStreamer instead of updateStreamer too, to follow the normal naming convention? Otherwise LGTM.

          Show
          andrew.wang Andrew Wang added a comment - Thanks Rakesh R for the update. I like the getter and setter proposal #1, it seems cleaner. Could we name it setStreamer instead of updateStreamer too, to follow the normal naming convention? Otherwise LGTM.
          Hide
          rakeshr Rakesh R added a comment -

          Thanks Andrew Wang. I've uploaded another patch by exposing setter/getter APIs.

          Show
          rakeshr Rakesh R added a comment - Thanks Andrew Wang . I've uploaded another patch by exposing setter/getter APIs.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 13s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 39s There were no new javac warning messages.
          +1 javadoc 9m 56s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 19s The applied patch generated 1 new checkstyle issues (total was 40, now 39).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 37s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 3m 17s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 20s Pre-build of native portion
          -1 hdfs tests 163m 19s Tests failed in hadoop-hdfs.
              210m 39s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12736978/HDFS-8386-01.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 03fb5c6
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11210/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11210/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11210/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11210/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 13s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 39s There were no new javac warning messages. +1 javadoc 9m 56s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 19s The applied patch generated 1 new checkstyle issues (total was 40, now 39). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 37s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 3m 17s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 20s Pre-build of native portion -1 hdfs tests 163m 19s Tests failed in hadoop-hdfs.     210m 39s   Reason Tests Failed unit tests hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12736978/HDFS-8386-01.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 03fb5c6 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11210/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11210/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11210/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11210/console This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for the contribution Rakesh, I've committed this to trunk and branch-2.

          Show
          andrew.wang Andrew Wang added a comment - Thanks for the contribution Rakesh, I've committed this to trunk and branch-2.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7945 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7945/)
          HDFS-8386. Improve synchronization of 'streamer' reference in DFSOutputStream. Contributed by Rakesh R. (wang: rev efc510a570cf880e7df1b69932aa41932658ee51)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7945 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7945/ ) HDFS-8386 . Improve synchronization of 'streamer' reference in DFSOutputStream. Contributed by Rakesh R. (wang: rev efc510a570cf880e7df1b69932aa41932658ee51) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #217 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/217/)
          HDFS-8386. Improve synchronization of 'streamer' reference in DFSOutputStream. Contributed by Rakesh R. (wang: rev efc510a570cf880e7df1b69932aa41932658ee51)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #217 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/217/ ) HDFS-8386 . Improve synchronization of 'streamer' reference in DFSOutputStream. Contributed by Rakesh R. (wang: rev efc510a570cf880e7df1b69932aa41932658ee51) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #947 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/947/)
          HDFS-8386. Improve synchronization of 'streamer' reference in DFSOutputStream. Contributed by Rakesh R. (wang: rev efc510a570cf880e7df1b69932aa41932658ee51)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #947 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/947/ ) HDFS-8386 . Improve synchronization of 'streamer' reference in DFSOutputStream. Contributed by Rakesh R. (wang: rev efc510a570cf880e7df1b69932aa41932658ee51) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2163 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2163/)
          HDFS-8386. Improve synchronization of 'streamer' reference in DFSOutputStream. Contributed by Rakesh R. (wang: rev efc510a570cf880e7df1b69932aa41932658ee51)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2163 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2163/ ) HDFS-8386 . Improve synchronization of 'streamer' reference in DFSOutputStream. Contributed by Rakesh R. (wang: rev efc510a570cf880e7df1b69932aa41932658ee51) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #215 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/215/)
          HDFS-8386. Improve synchronization of 'streamer' reference in DFSOutputStream. Contributed by Rakesh R. (wang: rev efc510a570cf880e7df1b69932aa41932658ee51)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #215 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/215/ ) HDFS-8386 . Improve synchronization of 'streamer' reference in DFSOutputStream. Contributed by Rakesh R. (wang: rev efc510a570cf880e7df1b69932aa41932658ee51) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2145 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2145/)
          HDFS-8386. Improve synchronization of 'streamer' reference in DFSOutputStream. Contributed by Rakesh R. (wang: rev efc510a570cf880e7df1b69932aa41932658ee51)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2145 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2145/ ) HDFS-8386 . Improve synchronization of 'streamer' reference in DFSOutputStream. Contributed by Rakesh R. (wang: rev efc510a570cf880e7df1b69932aa41932658ee51) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #206 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/206/)
          HDFS-8386. Improve synchronization of 'streamer' reference in DFSOutputStream. Contributed by Rakesh R. (wang: rev efc510a570cf880e7df1b69932aa41932658ee51)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #206 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/206/ ) HDFS-8386 . Improve synchronization of 'streamer' reference in DFSOutputStream. Contributed by Rakesh R. (wang: rev efc510a570cf880e7df1b69932aa41932658ee51) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          In trunk, the streamer synchronization bug does not exist since streamer is only initialized in constructors.

          Also, simply replacing streamer with getStreamer() may not be really fix synchronization bugs if there is any. For example, in adjustChunkBoundary() below, would it work if getStreamer() returns a different object each time?

             protected void adjustChunkBoundary() {
          -    if (streamer.getAppendChunk() &&
          -        streamer.getBytesCurBlock() % bytesPerChecksum == 0) {
          -      streamer.setAppendChunk(false);
          +    if (getStreamer().getAppendChunk() &&
          +        getStreamer().getBytesCurBlock() % bytesPerChecksum == 0) {
          +      getStreamer().setAppendChunk(false);
                 resetChecksumBufSize();
               }
           
          -    if (!streamer.getAppendChunk()) {
          -      int psize = Math.min((int)(blockSize- streamer.getBytesCurBlock()),
          +    if (!getStreamer().getAppendChunk()) {
          +      int psize = Math.min((int)(blockSize- getStreamer().getBytesCurBlock()),
                     dfsClient.getConf().getWritePacketSize());
                 computePacketChunkSize(psize, bytesPerChecksum);
               }
          

          I suggest revert the committed patch since it may potentially hide some synchronization bugs in the HDFS-7285 branch.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - In trunk, the streamer synchronization bug does not exist since streamer is only initialized in constructors. Also, simply replacing streamer with getStreamer() may not be really fix synchronization bugs if there is any. For example, in adjustChunkBoundary() below, would it work if getStreamer() returns a different object each time? protected void adjustChunkBoundary() { - if (streamer.getAppendChunk() && - streamer.getBytesCurBlock() % bytesPerChecksum == 0) { - streamer.setAppendChunk( false ); + if (getStreamer().getAppendChunk() && + getStreamer().getBytesCurBlock() % bytesPerChecksum == 0) { + getStreamer().setAppendChunk( false ); resetChecksumBufSize(); } - if (!streamer.getAppendChunk()) { - int psize = Math .min(( int )(blockSize- streamer.getBytesCurBlock()), + if (!getStreamer().getAppendChunk()) { + int psize = Math .min(( int )(blockSize- getStreamer().getBytesCurBlock()), dfsClient.getConf().getWritePacketSize()); computePacketChunkSize(psize, bytesPerChecksum); } I suggest revert the committed patch since it may potentially hide some synchronization bugs in the HDFS-7285 branch.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          It is fine to do refactoring in trunk. However, I think we should remove “synchronized" from getStreamer() since it may unnecessarily block the caller and, as a result, slow down write performance in trunk.

          Let's also remove setStreamer(..) which is currently not used. We may add it in the HDFS-7285 branch and see how to do synchronization correctly.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - It is fine to do refactoring in trunk. However, I think we should remove “synchronized" from getStreamer() since it may unnecessarily block the caller and, as a result, slow down write performance in trunk. Let's also remove setStreamer(..) which is currently not used. We may add it in the HDFS-7285 branch and see how to do synchronization correctly.
          Hide
          rakeshr Rakesh R added a comment -

          Thank you Tsz Wo Nicholas Sze for the detailed explanation. I got your point. How about raising a trivial improvement jira task and put a patch over there to do the suggested changes in trunk?. Also, I've raised HDFS-8550 separate jira to correct the synchronization part in HDFS-7285 branch.

          Show
          rakeshr Rakesh R added a comment - Thank you Tsz Wo Nicholas Sze for the detailed explanation. I got your point. How about raising a trivial improvement jira task and put a patch over there to do the suggested changes in trunk?. Also, I've raised HDFS-8550 separate jira to correct the synchronization part in HDFS-7285 branch.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8032 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8032/)
          HDFS-8606. Cleanup DFSOutputStream by removing unwanted changes from HDFS-8386. Contributed by Rakesh R (szetszwo: rev d4929f448f95815af99100780a08b172e0262c17)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8032 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8032/ ) HDFS-8606 . Cleanup DFSOutputStream by removing unwanted changes from HDFS-8386 . Contributed by Rakesh R (szetszwo: rev d4929f448f95815af99100780a08b172e0262c17) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #961 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/961/)
          HDFS-8606. Cleanup DFSOutputStream by removing unwanted changes from HDFS-8386. Contributed by Rakesh R (szetszwo: rev d4929f448f95815af99100780a08b172e0262c17)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #961 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/961/ ) HDFS-8606 . Cleanup DFSOutputStream by removing unwanted changes from HDFS-8386 . Contributed by Rakesh R (szetszwo: rev d4929f448f95815af99100780a08b172e0262c17) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #231 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/231/)
          HDFS-8606. Cleanup DFSOutputStream by removing unwanted changes from HDFS-8386. Contributed by Rakesh R (szetszwo: rev d4929f448f95815af99100780a08b172e0262c17)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #231 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/231/ ) HDFS-8606 . Cleanup DFSOutputStream by removing unwanted changes from HDFS-8386 . Contributed by Rakesh R (szetszwo: rev d4929f448f95815af99100780a08b172e0262c17) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #220 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/220/)
          HDFS-8606. Cleanup DFSOutputStream by removing unwanted changes from HDFS-8386. Contributed by Rakesh R (szetszwo: rev d4929f448f95815af99100780a08b172e0262c17)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #220 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/220/ ) HDFS-8606 . Cleanup DFSOutputStream by removing unwanted changes from HDFS-8386 . Contributed by Rakesh R (szetszwo: rev d4929f448f95815af99100780a08b172e0262c17) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #229 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/229/)
          HDFS-8606. Cleanup DFSOutputStream by removing unwanted changes from HDFS-8386. Contributed by Rakesh R (szetszwo: rev d4929f448f95815af99100780a08b172e0262c17)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #229 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/229/ ) HDFS-8606 . Cleanup DFSOutputStream by removing unwanted changes from HDFS-8386 . Contributed by Rakesh R (szetszwo: rev d4929f448f95815af99100780a08b172e0262c17) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2177 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2177/)
          HDFS-8606. Cleanup DFSOutputStream by removing unwanted changes from HDFS-8386. Contributed by Rakesh R (szetszwo: rev d4929f448f95815af99100780a08b172e0262c17)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2177 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2177/ ) HDFS-8606 . Cleanup DFSOutputStream by removing unwanted changes from HDFS-8386 . Contributed by Rakesh R (szetszwo: rev d4929f448f95815af99100780a08b172e0262c17) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #2159 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2159/)
          HDFS-8606. Cleanup DFSOutputStream by removing unwanted changes from HDFS-8386. Contributed by Rakesh R (szetszwo: rev d4929f448f95815af99100780a08b172e0262c17)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2159 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2159/ ) HDFS-8606 . Cleanup DFSOutputStream by removing unwanted changes from HDFS-8386 . Contributed by Rakesh R (szetszwo: rev d4929f448f95815af99100780a08b172e0262c17) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

            People

            • Assignee:
              rakeshr Rakesh R
              Reporter:
              rakeshr Rakesh R
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development