Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15109

TestDFSIO -read -random doesn't work on file sized 4GB

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.0.0-beta1
    • 3.1.0
    • fs, test
    • None

    Description

      TestDFSIO -read -random throws IllegalArgumentException on 4GB file. The cause is:

      private long nextOffset(long current) {
            if(skipSize == 0)
              return rnd.nextInt((int)(fileSize));
            if(skipSize > 0)
              return (current < 0) ? 0 : (current + bufferSize + skipSize);
            // skipSize < 0
            return (current < 0) ? Math.max(0, fileSize - bufferSize) :
                                   Math.max(0, current + skipSize);
          }
        }
      

      When filesize exceeds signed int, (int)(filesize) will be negative and cause Random.nextInt throws IllegalArgumentException("n must be positive").

      Attachments

        1. HADOOP-15109.002.patch
          2 kB
          Ajay Kumar
        2. Screen Shot 2017-12-11 at 3.17.22 PM.png
          112 kB
          Ajay Kumar
        3. HADOOP-15109.001.patch
          1 kB
          Ajay Kumar

        Activity

          ajayydv Ajay Kumar added a comment -

          zhoutai.zt, Attached patch to address the issue. Tested it locally with 5Gb file for random reads.

          ajayydv Ajay Kumar added a comment - zhoutai.zt , Attached patch to address the issue. Tested it locally with 5Gb file for random reads.
          zhoutai.zt zhoutai.zt added a comment -

          Thanks ajayydv.

          Another way to generate a bounded random long.

          ThreadLocalRandom.current().nextLong(fileSize)
          
          zhoutai.zt zhoutai.zt added a comment - Thanks ajayydv . Another way to generate a bounded random long. ThreadLocalRandom.current().nextLong(fileSize)
          genericqa genericqa added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 18m 33s trunk passed
          +1 compile 0m 26s trunk passed
          +1 checkstyle 0m 18s trunk passed
          +1 mvnsite 0m 29s trunk passed
          +1 shadedclient 10m 36s branch has no errors when building and testing our client artifacts.
          +1 findbugs 0m 27s trunk passed
          +1 javadoc 0m 15s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 27s the patch passed
          +1 compile 0m 23s the patch passed
          +1 javac 0m 23s the patch passed
          -0 checkstyle 0m 15s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: The patch generated 1 new + 46 unchanged - 1 fixed = 47 total (was 47)
          +1 mvnsite 0m 26s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 shadedclient 11m 14s patch has no errors when building and testing our client artifacts.
          +1 findbugs 0m 32s the patch passed
          +1 javadoc 0m 15s the patch passed
                Other Tests
          -1 unit 118m 56s hadoop-mapreduce-client-jobclient in the patch failed.
          -1 asflicense 0m 28s The patch generated 1 ASF License warnings.
          164m 19s



          Reason Tests
          Failed junit tests hadoop.mapreduce.v2.TestUberAM



          Subsystem Report/Notes
          Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639
          JIRA Issue HADOOP-15109
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12901567/HADOOP-15109.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
          uname Linux 8c5b05437ccf 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2316f52
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_151
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13814/artifact/out/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13814/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13814/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/13814/artifact/out/patch-asflicense-problems.txt
          Max. process+thread count 1187 (vs. ulimit of 5000)
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13814/console
          Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 18m 33s trunk passed +1 compile 0m 26s trunk passed +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 29s trunk passed +1 shadedclient 10m 36s branch has no errors when building and testing our client artifacts. +1 findbugs 0m 27s trunk passed +1 javadoc 0m 15s trunk passed       Patch Compile Tests +1 mvninstall 0m 27s the patch passed +1 compile 0m 23s the patch passed +1 javac 0m 23s the patch passed -0 checkstyle 0m 15s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: The patch generated 1 new + 46 unchanged - 1 fixed = 47 total (was 47) +1 mvnsite 0m 26s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 11m 14s patch has no errors when building and testing our client artifacts. +1 findbugs 0m 32s the patch passed +1 javadoc 0m 15s the patch passed       Other Tests -1 unit 118m 56s hadoop-mapreduce-client-jobclient in the patch failed. -1 asflicense 0m 28s The patch generated 1 ASF License warnings. 164m 19s Reason Tests Failed junit tests hadoop.mapreduce.v2.TestUberAM Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 JIRA Issue HADOOP-15109 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12901567/HADOOP-15109.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 8c5b05437ccf 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 2316f52 maven version: Apache Maven 3.3.9 Default Java 1.8.0_151 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13814/artifact/out/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13814/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13814/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/13814/artifact/out/patch-asflicense-problems.txt Max. process+thread count 1187 (vs. ulimit of 5000) modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13814/console Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          ajayydv Ajay Kumar added a comment -

          zhoutai.zt, yes this is more readable and easy to understand. Updating patch for same.

          ajayydv Ajay Kumar added a comment - zhoutai.zt , yes this is more readable and easy to understand. Updating patch for same.
          genericqa genericqa added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 12m 14s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 19m 34s trunk passed
          +1 compile 0m 32s trunk passed
          +1 checkstyle 0m 22s trunk passed
          +1 mvnsite 0m 38s trunk passed
          +1 shadedclient 11m 13s branch has no errors when building and testing our client artifacts.
          +1 findbugs 0m 28s trunk passed
          +1 javadoc 0m 16s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 28s the patch passed
          +1 compile 0m 24s the patch passed
          +1 javac 0m 24s the patch passed
          -0 checkstyle 0m 16s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: The patch generated 1 new + 46 unchanged - 1 fixed = 47 total (was 47)
          +1 mvnsite 0m 26s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 shadedclient 11m 12s patch has no errors when building and testing our client artifacts.
          +1 findbugs 0m 32s the patch passed
          +1 javadoc 0m 14s the patch passed
                Other Tests
          -1 unit 136m 4s hadoop-mapreduce-client-jobclient in the patch failed.
          -1 asflicense 0m 28s The patch generated 1 ASF License warnings.
          195m 18s



          Reason Tests
          Failed junit tests hadoop.mapreduce.v2.TestUberAM



          Subsystem Report/Notes
          Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639
          JIRA Issue HADOOP-15109
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12901728/HADOOP-15109.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
          uname Linux 81a742a74b63 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8bb83a8
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_151
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13820/artifact/out/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13820/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13820/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/13820/artifact/out/patch-asflicense-problems.txt
          Max. process+thread count 1191 (vs. ulimit of 5000)
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13820/console
          Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 12m 14s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 19m 34s trunk passed +1 compile 0m 32s trunk passed +1 checkstyle 0m 22s trunk passed +1 mvnsite 0m 38s trunk passed +1 shadedclient 11m 13s branch has no errors when building and testing our client artifacts. +1 findbugs 0m 28s trunk passed +1 javadoc 0m 16s trunk passed       Patch Compile Tests +1 mvninstall 0m 28s the patch passed +1 compile 0m 24s the patch passed +1 javac 0m 24s the patch passed -0 checkstyle 0m 16s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: The patch generated 1 new + 46 unchanged - 1 fixed = 47 total (was 47) +1 mvnsite 0m 26s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 11m 12s patch has no errors when building and testing our client artifacts. +1 findbugs 0m 32s the patch passed +1 javadoc 0m 14s the patch passed       Other Tests -1 unit 136m 4s hadoop-mapreduce-client-jobclient in the patch failed. -1 asflicense 0m 28s The patch generated 1 ASF License warnings. 195m 18s Reason Tests Failed junit tests hadoop.mapreduce.v2.TestUberAM Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 JIRA Issue HADOOP-15109 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12901728/HADOOP-15109.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 81a742a74b63 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 8bb83a8 maven version: Apache Maven 3.3.9 Default Java 1.8.0_151 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13820/artifact/out/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13820/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13820/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/13820/artifact/out/patch-asflicense-problems.txt Max. process+thread count 1191 (vs. ulimit of 5000) modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13820/console Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          zhoutai.zt zhoutai.zt added a comment -

          Thanks ajayydv.
          The second patch looks good to me, +1.

          zhoutai.zt zhoutai.zt added a comment - Thanks ajayydv . The second patch looks good to me, +1.
          ajayydv Ajay Kumar added a comment -

          zhoutai.zt, thanks for review.

          ajayydv Ajay Kumar added a comment - zhoutai.zt , thanks for review.
          vagarychen Chen Liang added a comment -

          +1 on v002 patch, the license warning and the unit test failure are related. I've committed to trunk, thanks zhoutai.zt for reporting this and ajayydv for the contribution!

          vagarychen Chen Liang added a comment - +1 on v002 patch, the license warning and the unit test failure are related. I've committed to trunk, thanks zhoutai.zt for reporting this and ajayydv for the contribution!
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13397 (See https://builds.apache.org/job/Hadoop-trunk-Commit/13397/)
          HADOOP-15109. TestDFSIO -read -random doesn't work on file sized 4GB. (cliang: rev c7a4dda3c5571e64c216810f8eb1a824c9b8f6f8)

          • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestDFSIO.java
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13397 (See https://builds.apache.org/job/Hadoop-trunk-Commit/13397/ ) HADOOP-15109 . TestDFSIO -read -random doesn't work on file sized 4GB. (cliang: rev c7a4dda3c5571e64c216810f8eb1a824c9b8f6f8) (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestDFSIO.java

          People

            ajayydv Ajay Kumar
            zhoutai.zt zhoutai.zt
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: