Hadoop Common
  1. Hadoop Common
  2. HADOOP-10855

Allow Text to be read with a known length

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.6.0
    • Component/s: io
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      For the native task work (MAPREDUCE-2841) it is useful to be able to store strings in a different fashion than the default (varint-prefixed) serialization. We should provide a "read" method in Text which takes an already-known length to support this use case while still providing Text objects back to the user.

      1. hadoop-10855.txt
        3 kB
        Todd Lipcon
      2. hadoop-10855.txt
        3 kB
        Todd Lipcon
      3. hadoop-10855.txt
        3 kB
        Todd Lipcon

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1840 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1840/)
          HADOOP-10855. Allow Text to be read with a known Length. Contributed by Todd Lipcon. (todd: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1612731)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1840 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1840/ ) HADOOP-10855 . Allow Text to be read with a known Length. Contributed by Todd Lipcon. (todd: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1612731 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1813 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1813/)
          HADOOP-10855. Allow Text to be read with a known Length. Contributed by Todd Lipcon. (todd: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1612731)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1813 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1813/ ) HADOOP-10855 . Allow Text to be read with a known Length. Contributed by Todd Lipcon. (todd: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1612731 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #621 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/621/)
          HADOOP-10855. Allow Text to be read with a known Length. Contributed by Todd Lipcon. (todd: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1612731)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #621 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/621/ ) HADOOP-10855 . Allow Text to be read with a known Length. Contributed by Todd Lipcon. (todd: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1612731 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #5946 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5946/)
          HADOOP-10855. Allow Text to be read with a known Length. Contributed by Todd Lipcon. (todd: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1612731)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #5946 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5946/ ) HADOOP-10855 . Allow Text to be read with a known Length. Contributed by Todd Lipcon. (todd: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1612731 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java
          Hide
          Aaron T. Myers added a comment -

          +1, the patch looks good to me. I agree that the test failures are unrelated - they're in orthogonal parts of the code and I also just ran them all locally and they passed with this patch applied.

          Thanks a lot, Todd.

          Show
          Aaron T. Myers added a comment - +1, the patch looks good to me. I agree that the test failures are unrelated - they're in orthogonal parts of the code and I also just ran them all locally and they passed with this patch applied. Thanks a lot, Todd.
          Hide
          Todd Lipcon added a comment -

          Pretty confident these failures are unrelated. Seems to be something that went wrong with the local FS.

          Show
          Todd Lipcon added a comment - Pretty confident these failures are unrelated. Seems to be something that went wrong with the local FS.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12656306/hadoop-10855.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

          org.apache.hadoop.fs.shell.TestCopyPreserveFlag
          org.apache.hadoop.fs.TestSymlinkLocalFSFileContext
          org.apache.hadoop.fs.shell.TestTextCommand
          org.apache.hadoop.ipc.TestIPC
          org.apache.hadoop.fs.TestSymlinkLocalFSFileSystem
          org.apache.hadoop.fs.shell.TestPathData
          org.apache.hadoop.fs.TestDFVariations

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4305//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4305//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12656306/hadoop-10855.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.fs.shell.TestCopyPreserveFlag org.apache.hadoop.fs.TestSymlinkLocalFSFileContext org.apache.hadoop.fs.shell.TestTextCommand org.apache.hadoop.ipc.TestIPC org.apache.hadoop.fs.TestSymlinkLocalFSFileSystem org.apache.hadoop.fs.shell.TestPathData org.apache.hadoop.fs.TestDFVariations +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4305//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4305//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          woops, my patch wasn't relative to the right dir... take 3.

          Show
          Todd Lipcon added a comment - woops, my patch wasn't relative to the right dir... take 3.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12656290/hadoop-10855.txt
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4304//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12656290/hadoop-10855.txt against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4304//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          Oops, noticed a silly typo in a comment.

          Show
          Todd Lipcon added a comment - Oops, noticed a silly typo in a comment.
          Hide
          Todd Lipcon added a comment -

          Attached patch implements readWithKnownLength(). I also refactored the common code out from the existing read methods to call this new one after deserializing the length. Added a simple new unit test to verify.

          Show
          Todd Lipcon added a comment - Attached patch implements readWithKnownLength(). I also refactored the common code out from the existing read methods to call this new one after deserializing the length. Added a simple new unit test to verify.

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development