Hadoop Common
  1. Hadoop Common
  2. HADOOP-7651

Hadoop Record compiler generates Java files with erroneous byte-array lengths for fields trailing a 'ustring' field

    Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.20.2
    • Fix Version/s: None
    • Component/s: record
    • Tags:
      Hadoop Record Compiler

      Description

      Hadoop Record compiler produces Java files from a DDL file. If a DDL file has a class that contains a 'ustring' field, then the generated 'compareRaw()' function for this record is erroneous in computing the length of remaining bytes after the logic of computing the buffer segment for a 'ustring' field.

      Below is a line in a generated 'compareRaw()' function for a record class with a 'ustring' field :
      s1+=i1; s2+=i2; l1-=i1; l1-=i2;
      This line shoud be corrected by changing the last 'l1' to 'l2':
      s1+=i1; s2+=i2; l1-=i1; l2-=i2;

      To fix this bug, one should correct the 'genCompareBytes()' function in the 'JString.java' file of the package 'org.apache.hadoop.record.compiler' by changing the line below to the ensuing line. There is only one digit difference:

      cb.append("s1+=i1; s2+=i2; l1-=i1; l1-=i2;\n");

      cb.append("s1+=i1; s2+=i2; l1-=i1; l2-=i2;\n");

      This bug is serious as it will always crash unserializing a record with a simple definition like the one below
      class PairStringDouble

      { ustring first; double second; }

      Unserializing a record of this class will throw an exception as the 'second' field does not have 8 bytes for a double value due to the erroneous length computation for the remaining buffer.

      Both Hadoop 0.20 and 0.21 have this bug.

      1. 7651-trunk.patch
        0.8 kB
        Milind Bhandarkar
      2. 7651-22.patch
        0.7 kB
        Milind Bhandarkar

        Activity

        Hide
        Milind Bhandarkar added a comment -

        Attaching simple fix.

        Show
        Milind Bhandarkar added a comment - Attaching simple fix.
        Hide
        Milind Bhandarkar added a comment -

        Thanks for the bug report, Hung-Chih. Attaching two version of patches. The trunk version applies to 0.23, and trunk. The 0.22 version applies to others.

        Show
        Milind Bhandarkar added a comment - Thanks for the bug report, Hung-Chih. Attaching two version of patches. The trunk version applies to 0.23, and trunk. The 0.22 version applies to others.
        Hide
        Hung-chih Yang added a comment -

        Thanks Milind. Nice to hear from an old colleague.

        Hung-chih

        Show
        Hung-chih Yang added a comment - Thanks Milind. Nice to hear from an old colleague. Hung-chih
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12497074/7651-22.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/247//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12497074/7651-22.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/247//console This message is automatically generated.
        Hide
        Milind Bhandarkar added a comment -

        Looks like the almighty hadoop-qa tried to apply the 0.22 patch to trunk.

        Show
        Milind Bhandarkar added a comment - Looks like the almighty hadoop-qa tried to apply the 0.22 patch to trunk.
        Hide
        Konstantin Shvachko added a comment -

        Milind, do you really need to change "stat" to full path "/usr/bin/stat" for MacOS in 0.22 patch? I'd rather keep it as it was before.

        Show
        Konstantin Shvachko added a comment - Milind, do you really need to change "stat" to full path "/usr/bin/stat" for MacOS in 0.22 patch? I'd rather keep it as it was before.
        Hide
        Milind Bhandarkar added a comment -

        Konstantin, sorry my mistake. I generated that patch from a tree modified for another issue. Let me upload a new patch for 0.22.

        Show
        Milind Bhandarkar added a comment - Konstantin, sorry my mistake. I generated that patch from a tree modified for another issue. Let me upload a new patch for 0.22.
        Hide
        Milind Bhandarkar added a comment -

        canceling patch for 0.22.

        Show
        Milind Bhandarkar added a comment - canceling patch for 0.22.
        Hide
        Milind Bhandarkar added a comment -

        Attaching correct 0.22 patch.

        Show
        Milind Bhandarkar added a comment - Attaching correct 0.22 patch.
        Hide
        Milind Bhandarkar added a comment -

        submitted new 0.22 patch.

        Show
        Milind Bhandarkar added a comment - submitted new 0.22 patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12499462/7651-22.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/300//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12499462/7651-22.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/300//console This message is automatically generated.
        Hide
        Konstantin Shvachko added a comment -

        Milind could you please resubmit your patch for trunk as is. Then Jenkins will pick it up instead of the 0.22 version. Could you run test target for 0.22 locally.

        Show
        Konstantin Shvachko added a comment - Milind could you please resubmit your patch for trunk as is. Then Jenkins will pick it up instead of the 0.22 version. Could you run test target for 0.22 locally.
        Hide
        Milind Bhandarkar added a comment -

        Hung-Chih, since I am currently busy in a lot of other work, can you report the results of your manual testing here, after applying the appropriate patch ? Thanks.

        Show
        Milind Bhandarkar added a comment - Hung-Chih, since I am currently busy in a lot of other work, can you report the results of your manual testing here, after applying the appropriate patch ? Thanks.
        Hide
        Harsh J added a comment -

        Since Avro has replaced the Hadoop RCC, shouldn't we just close this?

        Show
        Harsh J added a comment - Since Avro has replaced the Hadoop RCC, shouldn't we just close this?
        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 patch 0m 0s The patch command could not apply the patch during dryrun.



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12499462/7651-22.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / f1a152c
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6263/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12499462/7651-22.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / f1a152c Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6263/console This message was automatically generated.

          People

          • Assignee:
            Milind Bhandarkar
            Reporter:
            Hung-chih Yang
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:

              Time Tracking

              Estimated:
              Original Estimate - 1m
              1m
              Remaining:
              Remaining Estimate - 1m
              1m
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development