Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
0.20.205.0
-
Ubuntu, hadoop cloudera CDH3U2, Oracle SUN JDK 6U30
Description
LineReader reader = new LineReader(in, 4096);
...
Text text = new Text();
while((reader.readLine(text)) > 0) {
...
text.clear();
}
}
Even the clear() method is called each time, some bytes are still not filled as zero.
So, when reader.readLine(text) is called in a loop, some bytes are dirty which was from last call.
Attachments
Attachments
- HADOOP-7940.patch
- 1 kB
- Csaba Miklos
Issue Links
- is superceded by
-
HADOOP-8323 Revert HADOOP-7940 and improve javadocs and test for Text.clear()
- Closed
Activity
Index: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java
===================================================================
— hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java (revision 1294407)
+++ hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java (working copy)
@@ -239,6 +239,7 @@
*/
public void clear()
/*
Index: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java
===================================================================
— hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java (revision 1294407)
+++ hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java (working copy)
@@ -192,6 +192,16 @@
assertTrue(text.find("\u20ac", 5)==11);
}
+ public void testClear()
{ + Text text = new Text(); + assertEquals("", text.toString()); + assertEquals(0, text.getBytes().length); + text = new Text("abcd\u20acbdcd\u20ac"); + text.clear(); + assertEquals("", text.toString()); + assertEquals(0, text.getBytes().length); + }+
public void testFindAfterUpdatingContents() throws Exception {
Text text = new Text("abcd");
text.set("a".getBytes());
Hi Csaba,
Thanks for contributing! Mind attaching a patch against trunk? (attach files and be sure to check "Grant license to ASF for inclusion in ASF works"). This is necessary to incorporate your change into the code. More info here: http://wiki.apache.org/hadoop/HowToContribute
Thanks,
Eli
Hi Eli!
I've read HowToContribute page. This patch for the trunk. I've also attached the patch to the issue.
Best regards,
Csaba
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12516354/HADOOP-7940.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.fs.viewfs.TestViewFsTrash
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/638//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/638//console
This message is automatically generated.
+1, TestViewFsTrash failure is unrelated to this patch (There's already a separate bug report available for its failing though). Test fails without fix.
Committing to trunk and branch-0.23 shortly.
Integrated in Hadoop-Hdfs-trunk-Commit #1878 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1878/)
HADOOP-7940. The Text.clear() method does not clear the bytes as intended. Contributed by Csaba Miklos. (Revision 1295061)
Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295061
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java
Integrated in Hadoop-Common-trunk-Commit #1804 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1804/)
HADOOP-7940. The Text.clear() method does not clear the bytes as intended. Contributed by Csaba Miklos. (Revision 1295061)
Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295061
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java
Committed to branch-0.23 and trunk. Thanks a lot for your contribution Csaba! We're hoping to see many more
Integrated in Hadoop-Hdfs-0.23-Commit #602 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/602/)
svn merge -c 1295061. Merge HADOOP-7940 into branch-0.23. (Csaba Miklos via harsh). (Revision 1295064)
Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295064
Files :
- /hadoop/common/branches/branch-0.23
- /hadoop/common/branches/branch-0.23/hadoop-common-project
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job
- /hadoop/common/branches/branch-0.23/hadoop-project
- /hadoop/common/branches/branch-0.23/hadoop-project/src/site
Integrated in Hadoop-Common-0.23-Commit #614 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/614/)
svn merge -c 1295061. Merge HADOOP-7940 into branch-0.23. (Csaba Miklos via harsh). (Revision 1295064)
Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295064
Files :
- /hadoop/common/branches/branch-0.23
- /hadoop/common/branches/branch-0.23/hadoop-common-project
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job
- /hadoop/common/branches/branch-0.23/hadoop-project
- /hadoop/common/branches/branch-0.23/hadoop-project/src/site
Integrated in Hadoop-Mapreduce-trunk-Commit #1812 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1812/)
HADOOP-7940. The Text.clear() method does not clear the bytes as intended. Contributed by Csaba Miklos. (Revision 1295061)
Result = ABORTED
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295061
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java
Integrated in Hadoop-Mapreduce-0.23-Commit #614 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/614/)
svn merge -c 1295061. Merge HADOOP-7940 into branch-0.23. (Csaba Miklos via harsh). (Revision 1295064)
Result = ABORTED
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295064
Files :
- /hadoop/common/branches/branch-0.23
- /hadoop/common/branches/branch-0.23/hadoop-common-project
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job
- /hadoop/common/branches/branch-0.23/hadoop-project
- /hadoop/common/branches/branch-0.23/hadoop-project/src/site
Integrated in Hadoop-Hdfs-trunk #970 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/970/)
HADOOP-7940. The Text.clear() method does not clear the bytes as intended. Contributed by Csaba Miklos. (Revision 1295061)
Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295061
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java
Integrated in Hadoop-Hdfs-0.23-Build #183 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/183/)
svn merge -c 1295061. Merge HADOOP-7940 into branch-0.23. (Csaba Miklos via harsh). (Revision 1295064)
Result = FAILURE
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295064
Files :
- /hadoop/common/branches/branch-0.23
- /hadoop/common/branches/branch-0.23/hadoop-common-project
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job
- /hadoop/common/branches/branch-0.23/hadoop-project
- /hadoop/common/branches/branch-0.23/hadoop-project/src/site
Integrated in Hadoop-Mapreduce-0.23-Build #211 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/211/)
svn merge -c 1295061. Merge HADOOP-7940 into branch-0.23. (Csaba Miklos via harsh). (Revision 1295064)
Result = FAILURE
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295064
Files :
- /hadoop/common/branches/branch-0.23
- /hadoop/common/branches/branch-0.23/hadoop-common-project
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job
- /hadoop/common/branches/branch-0.23/hadoop-project
- /hadoop/common/branches/branch-0.23/hadoop-project/src/site
Integrated in Hadoop-Mapreduce-trunk #1005 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1005/)
HADOOP-7940. The Text.clear() method does not clear the bytes as intended. Contributed by Csaba Miklos. (Revision 1295061)
Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295061
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java
toString, getLength and other Text methods behave correctly after clear(). This will hurt the performance of the class when calling append often and then clear because you wont be able to use the byte array size that you had previously scaled up to.
I dont think there was any expectation that getBytes would return a 0 length array, it says right in the javadoc: "Returns the raw bytes; however, only data up to
{@link #getLength()}is valid." which in this case is valid, 0 of the bytes are valid
This will likely hurt the performance problem described in HADOOP-6109.
I was looking at the javadoc in Hadoop 1.0.2. The more recent javadoc adds: "Please use
{@link #copyBytes()}if you need the returned array to be precisely the length of the data." which makes it even clearer than you must pay attention to the length if you use getBytes()
Jim - I'm actually not convinced this hurts at all. I've decided not to have this reverted in second thought, and I've explained the reasoning at HADOOP-8323.
Clear call ought to clear memory and thats what this change actually intends to do (though the test case may lead you astray - filed HADOOP-8324).
The current javadocs indeed cover the getBytes usage behavior as you've pointed out. So if you'd like to keep the size in its increased state, why clear() it?
Integrated in Hadoop-Hdfs-trunk-Commit #2268 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2268/)
HADOOP-8323. Revert HADOOP-7940, cause it may cause a performance regression. (harsh) (Revision 1334603)
Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1334603
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java
Integrated in Hadoop-Common-trunk-Commit #2193 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2193/)
HADOOP-8323. Revert HADOOP-7940, cause it may cause a performance regression. (harsh) (Revision 1334603)
Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1334603
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java
Note: This was reverted by HADOOP-8323. Please see that ticket for more information.
Integrated in Hadoop-Mapreduce-trunk-Commit #2211 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2211/)
HADOOP-8323. Revert HADOOP-7940, cause it may cause a performance regression. (harsh) (Revision 1334603)
Result = ABORTED
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1334603
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java
Integrated in Hadoop-Hdfs-trunk #1036 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1036/)
HADOOP-8323. Revert HADOOP-7940, cause it may cause a performance regression. (harsh) (Revision 1334603)
Result = FAILURE
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1334603
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java
Integrated in Hadoop-Mapreduce-trunk #1071 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1071/)
HADOOP-8323. Revert HADOOP-7940, cause it may cause a performance regression. (harsh) (Revision 1334603)
Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1334603
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java
Hey Aaron,
Would you be interested in contributing a fix and a test case for this as well?
Thanks,
Harsh