Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-10320

Replace FSDataInputStream#read with readFully in Log Aggregation

Details

    Description

      Have observed Log Aggregation code has used FSDataInputStream#read instead of readFully in multiple places like below. One of the place is fixed by YARN-8106.
      This Jira targets to fix at all other places.

      LogAggregationIndexedFileController#loadUUIDFromLogFile

                byte[] b = new byte[uuid.length];
                int actual = fsDataInputStream.read(b);
                if (actual != uuid.length || Arrays.equals(b, uuid)) {
      

      Attachments

        1. YARN-10320-001.patch
          4 kB
          Tanu Ajmera
        2. YARN-10320-002.patch
          3 kB
          Tanu Ajmera

        Issue Links

          Activity

            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 0s Docker mode activated.
            -1 patch 0m 13s YARN-10320 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



            Subsystem Report/Notes
            JIRA Issue YARN-10320
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/13006724/YARN-10320-001.patch
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/26233/console
            versions git=2.17.1
            Powered by Apache Yetus 0.12.0 https://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 13s YARN-10320 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue YARN-10320 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/13006724/YARN-10320-001.patch Console output https://builds.apache.org/job/PreCommit-YARN-Build/26233/console versions git=2.17.1 Powered by Apache Yetus 0.12.0 https://yetus.apache.org This message was automatically generated.
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 2m 22s Docker mode activated.
                  Prechecks
            +1 dupname 0m 0s No case conflicting files found.
            +1 @author 0m 0s The patch does not contain any @author tags.
            -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
                  trunk Compile Tests
            +1 mvninstall 23m 23s trunk passed
            +1 compile 0m 41s trunk passed
            +1 checkstyle 0m 30s trunk passed
            +1 mvnsite 0m 44s trunk passed
            +1 shadedclient 17m 6s branch has no errors when building and testing our client artifacts.
            +1 javadoc 0m 47s trunk passed
            0 spotbugs 1m 51s Used deprecated FindBugs config; considering switching to SpotBugs.
            +1 findbugs 1m 48s trunk passed
                  Patch Compile Tests
            +1 mvninstall 0m 39s the patch passed
            +1 compile 0m 36s the patch passed
            +1 javac 0m 36s the patch passed
            +1 checkstyle 0m 24s the patch passed
            +1 mvnsite 0m 38s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 shadedclient 15m 20s patch has no errors when building and testing our client artifacts.
            +1 javadoc 0m 43s the patch passed
            +1 findbugs 1m 52s the patch passed
                  Other Tests
            +1 unit 3m 52s hadoop-yarn-common in the patch passed.
            +1 asflicense 0m 32s The patch does not generate ASF License warnings.
            71m 59s



            Subsystem Report/Notes
            Docker ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/PreCommit-YARN-Build/26235/artifact/out/Dockerfile
            JIRA Issue YARN-10320
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/13006730/YARN-10320-002.patch
            Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
            uname Linux 81d9ae82fa42 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality personality/hadoop.sh
            git revision trunk / cd188ea9f0e
            Default Java Private Build-1.8.0_252-8u252-b09-1~18.04-b09
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/26235/testReport/
            Max. process+thread count 308 (vs. ulimit of 5500)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/26235/console
            versions git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
            Powered by Apache Yetus 0.12.0 https://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 2m 22s Docker mode activated.       Prechecks +1 dupname 0m 0s No case conflicting files found. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       trunk Compile Tests +1 mvninstall 23m 23s trunk passed +1 compile 0m 41s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 0m 44s trunk passed +1 shadedclient 17m 6s branch has no errors when building and testing our client artifacts. +1 javadoc 0m 47s trunk passed 0 spotbugs 1m 51s Used deprecated FindBugs config; considering switching to SpotBugs. +1 findbugs 1m 48s trunk passed       Patch Compile Tests +1 mvninstall 0m 39s the patch passed +1 compile 0m 36s the patch passed +1 javac 0m 36s the patch passed +1 checkstyle 0m 24s the patch passed +1 mvnsite 0m 38s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 15m 20s patch has no errors when building and testing our client artifacts. +1 javadoc 0m 43s the patch passed +1 findbugs 1m 52s the patch passed       Other Tests +1 unit 3m 52s hadoop-yarn-common in the patch passed. +1 asflicense 0m 32s The patch does not generate ASF License warnings. 71m 59s Subsystem Report/Notes Docker ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/PreCommit-YARN-Build/26235/artifact/out/Dockerfile JIRA Issue YARN-10320 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/13006730/YARN-10320-002.patch Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 81d9ae82fa42 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality personality/hadoop.sh git revision trunk / cd188ea9f0e Default Java Private Build-1.8.0_252-8u252-b09-1~18.04-b09 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/26235/testReport/ Max. process+thread count 308 (vs. ulimit of 5500) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/26235/console versions git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1 Powered by Apache Yetus 0.12.0 https://yetus.apache.org This message was automatically generated.
            BilwaST Bilwa S T added a comment -

            Thanks for patch tanu.ajmera

            I think you need to replace read with readFully in below code too

             while ((len = in.read(buf)) != -1) {
                      //If buffer contents within fileLength, write
                      if (len < bytesLeft) {
                        outputStreamState.getOutputStream().write(buf, 0, len);
                        bytesLeft-=len;
                      } else {
                        //else only write contents within fileLength, then exit early
                        outputStreamState.getOutputStream().write(buf, 0,
                            (int)bytesLeft);
                        break;
                      }
                    }
            

             

            BilwaST Bilwa S T added a comment - Thanks for patch tanu.ajmera I think you need to replace read with readFully in below code too while ((len = in.read(buf)) != -1) { //If buffer contents within fileLength, write if (len < bytesLeft) { outputStreamState.getOutputStream().write(buf, 0, len); bytesLeft-=len; } else { // else only write contents within fileLength, then exit early outputStreamState.getOutputStream().write(buf, 0, ( int )bytesLeft); break ; } }  
            groot Ashutosh Gupta added a comment -

            This issue seems to be pending for a while. Taking it up.

            groot Ashutosh Gupta added a comment - This issue seems to be pending for a while. Taking it up.

            People

              groot Ashutosh Gupta
              prabhujoseph Prabhu Joseph
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 10m
                  1h 10m