Description
TestDFSIO -read -random throws IllegalArgumentException on 4GB file. The cause is:
private long nextOffset(long current) { if(skipSize == 0) return rnd.nextInt((int)(fileSize)); if(skipSize > 0) return (current < 0) ? 0 : (current + bufferSize + skipSize); // skipSize < 0 return (current < 0) ? Math.max(0, fileSize - bufferSize) : Math.max(0, current + skipSize); } }
When filesize exceeds signed int, (int)(filesize) will be negative and cause Random.nextInt throws IllegalArgumentException("n must be positive").
Attachments
Attachments
- HADOOP-15109.001.patch
- 1 kB
- Ajay Kumar
- Screen Shot 2017-12-11 at 3.17.22 PM.png
- 112 kB
- Ajay Kumar
- HADOOP-15109.002.patch
- 2 kB
- Ajay Kumar
Activity
Thanks ajayydv.
Another way to generate a bounded random long.
ThreadLocalRandom.current().nextLong(fileSize)
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 12s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 18m 33s | trunk passed |
+1 | compile | 0m 26s | trunk passed |
+1 | checkstyle | 0m 18s | trunk passed |
+1 | mvnsite | 0m 29s | trunk passed |
+1 | shadedclient | 10m 36s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 0m 27s | trunk passed |
+1 | javadoc | 0m 15s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 0m 27s | the patch passed |
+1 | compile | 0m 23s | the patch passed |
+1 | javac | 0m 23s | the patch passed |
-0 | checkstyle | 0m 15s | hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: The patch generated 1 new + 46 unchanged - 1 fixed = 47 total (was 47) |
+1 | mvnsite | 0m 26s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 11m 14s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 0m 32s | the patch passed |
+1 | javadoc | 0m 15s | the patch passed |
Other Tests | |||
-1 | unit | 118m 56s | hadoop-mapreduce-client-jobclient in the patch failed. |
-1 | asflicense | 0m 28s | The patch generated 1 ASF License warnings. |
164m 19s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.mapreduce.v2.TestUberAM |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12901567/HADOOP-15109.001.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux 8c5b05437ccf 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 2316f52 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_151 |
findbugs | v3.1.0-RC1 |
checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/13814/artifact/out/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt |
unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/13814/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt |
Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/13814/testReport/ |
asflicense | https://builds.apache.org/job/PreCommit-HADOOP-Build/13814/artifact/out/patch-asflicense-problems.txt |
Max. process+thread count | 1187 (vs. ulimit of 5000) |
modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient |
Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/13814/console |
Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
zhoutai.zt, yes this is more readable and easy to understand. Updating patch for same.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 12m 14s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 19m 34s | trunk passed |
+1 | compile | 0m 32s | trunk passed |
+1 | checkstyle | 0m 22s | trunk passed |
+1 | mvnsite | 0m 38s | trunk passed |
+1 | shadedclient | 11m 13s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 0m 28s | trunk passed |
+1 | javadoc | 0m 16s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 0m 28s | the patch passed |
+1 | compile | 0m 24s | the patch passed |
+1 | javac | 0m 24s | the patch passed |
-0 | checkstyle | 0m 16s | hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: The patch generated 1 new + 46 unchanged - 1 fixed = 47 total (was 47) |
+1 | mvnsite | 0m 26s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 11m 12s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 0m 32s | the patch passed |
+1 | javadoc | 0m 14s | the patch passed |
Other Tests | |||
-1 | unit | 136m 4s | hadoop-mapreduce-client-jobclient in the patch failed. |
-1 | asflicense | 0m 28s | The patch generated 1 ASF License warnings. |
195m 18s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.mapreduce.v2.TestUberAM |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12901728/HADOOP-15109.002.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux 81a742a74b63 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 8bb83a8 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_151 |
findbugs | v3.1.0-RC1 |
checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/13820/artifact/out/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt |
unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/13820/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt |
Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/13820/testReport/ |
asflicense | https://builds.apache.org/job/PreCommit-HADOOP-Build/13820/artifact/out/patch-asflicense-problems.txt |
Max. process+thread count | 1191 (vs. ulimit of 5000) |
modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient |
Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/13820/console |
Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
+1 on v002 patch, the license warning and the unit test failure are related. I've committed to trunk, thanks zhoutai.zt for reporting this and ajayydv for the contribution!
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13397 (See https://builds.apache.org/job/Hadoop-trunk-Commit/13397/)
HADOOP-15109. TestDFSIO -read -random doesn't work on file sized 4GB. (cliang: rev c7a4dda3c5571e64c216810f8eb1a824c9b8f6f8)
- (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestDFSIO.java
zhoutai.zt, Attached patch to address the issue. Tested it locally with 5Gb file for random reads.