Hadoop Common
  1. Hadoop Common
  2. HADOOP-7651

Hadoop Record compiler generates Java files with erroneous byte-array lengths for fields trailing a 'ustring' field

    Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.20.2
    • Fix Version/s: None
    • Component/s: record
    • Labels:
    • Tags:
      Hadoop Record Compiler

      Description

      Hadoop Record compiler produces Java files from a DDL file. If a DDL file has a class that contains a 'ustring' field, then the generated 'compareRaw()' function for this record is erroneous in computing the length of remaining bytes after the logic of computing the buffer segment for a 'ustring' field.

      Below is a line in a generated 'compareRaw()' function for a record class with a 'ustring' field :
      s1+=i1; s2+=i2; l1-=i1; l1-=i2;
      This line shoud be corrected by changing the last 'l1' to 'l2':
      s1+=i1; s2+=i2; l1-=i1; l2-=i2;

      To fix this bug, one should correct the 'genCompareBytes()' function in the 'JString.java' file of the package 'org.apache.hadoop.record.compiler' by changing the line below to the ensuing line. There is only one digit difference:

      cb.append("s1+=i1; s2+=i2; l1-=i1; l1-=i2;\n");

      cb.append("s1+=i1; s2+=i2; l1-=i1; l2-=i2;\n");

      This bug is serious as it will always crash unserializing a record with a simple definition like the one below
      class PairStringDouble

      { ustring first; double second; }

      Unserializing a record of this class will throw an exception as the 'second' field does not have 8 bytes for a double value due to the erroneous length computation for the remaining buffer.

      Both Hadoop 0.20 and 0.21 have this bug.

      1. 7651-22.patch
        0.7 kB
        Milind Bhandarkar
      2. 7651-trunk.patch
        0.8 kB
        Milind Bhandarkar

        Activity

        Hung-chih Yang created issue -
        Milind Bhandarkar made changes -
        Field Original Value New Value
        Assignee Milind Bhandarkar [ milindb ]
        Milind Bhandarkar made changes -
        Status Open [ 1 ] In Progress [ 3 ]
        Milind Bhandarkar made changes -
        Status In Progress [ 3 ] Open [ 1 ]
        Hide
        Milind Bhandarkar added a comment -

        Attaching simple fix.

        Show
        Milind Bhandarkar added a comment - Attaching simple fix.
        Milind Bhandarkar made changes -
        Attachment 7651-trunk.patch [ 12497073 ]
        Attachment 7651-22.patch [ 12497074 ]
        Hide
        Milind Bhandarkar added a comment -

        Thanks for the bug report, Hung-Chih. Attaching two version of patches. The trunk version applies to 0.23, and trunk. The 0.22 version applies to others.

        Show
        Milind Bhandarkar added a comment - Thanks for the bug report, Hung-Chih. Attaching two version of patches. The trunk version applies to 0.23, and trunk. The 0.22 version applies to others.
        Milind Bhandarkar made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Affects Version/s 0.22.0 [ 12314296 ]
        Affects Version/s 0.23.0 [ 12315569 ]
        Affects Version/s 0.24.0 [ 12317652 ]
        Fix Version/s 0.20.204.1 [ 12317846 ]
        Fix Version/s 0.21.1 [ 12315270 ]
        Fix Version/s 0.22.0 [ 12314296 ]
        Fix Version/s 0.23.0 [ 12315569 ]
        Fix Version/s 0.24.0 [ 12317652 ]
        Hide
        Hung-chih Yang added a comment -

        Thanks Milind. Nice to hear from an old colleague.

        Hung-chih

        Show
        Hung-chih Yang added a comment - Thanks Milind. Nice to hear from an old colleague. Hung-chih
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12497074/7651-22.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/247//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12497074/7651-22.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/247//console This message is automatically generated.
        Hide
        Milind Bhandarkar added a comment -

        Looks like the almighty hadoop-qa tried to apply the 0.22 patch to trunk.

        Show
        Milind Bhandarkar added a comment - Looks like the almighty hadoop-qa tried to apply the 0.22 patch to trunk.
        Hide
        Konstantin Shvachko added a comment -

        Milind, do you really need to change "stat" to full path "/usr/bin/stat" for MacOS in 0.22 patch? I'd rather keep it as it was before.

        Show
        Konstantin Shvachko added a comment - Milind, do you really need to change "stat" to full path "/usr/bin/stat" for MacOS in 0.22 patch? I'd rather keep it as it was before.
        Hide
        Milind Bhandarkar added a comment -

        Konstantin, sorry my mistake. I generated that patch from a tree modified for another issue. Let me upload a new patch for 0.22.

        Show
        Milind Bhandarkar added a comment - Konstantin, sorry my mistake. I generated that patch from a tree modified for another issue. Let me upload a new patch for 0.22.
        Hide
        Milind Bhandarkar added a comment -

        canceling patch for 0.22.

        Show
        Milind Bhandarkar added a comment - canceling patch for 0.22.
        Milind Bhandarkar made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Milind Bhandarkar made changes -
        Attachment 7651-22.patch [ 12497074 ]
        Hide
        Milind Bhandarkar added a comment -

        Attaching correct 0.22 patch.

        Show
        Milind Bhandarkar added a comment - Attaching correct 0.22 patch.
        Milind Bhandarkar made changes -
        Attachment 7651-22.patch [ 12499462 ]
        Hide
        Milind Bhandarkar added a comment -

        submitted new 0.22 patch.

        Show
        Milind Bhandarkar added a comment - submitted new 0.22 patch.
        Milind Bhandarkar made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12499462/7651-22.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/300//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12499462/7651-22.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/300//console This message is automatically generated.
        Hide
        Konstantin Shvachko added a comment -

        Milind could you please resubmit your patch for trunk as is. Then Jenkins will pick it up instead of the 0.22 version. Could you run test target for 0.22 locally.

        Show
        Konstantin Shvachko added a comment - Milind could you please resubmit your patch for trunk as is. Then Jenkins will pick it up instead of the 0.22 version. Could you run test target for 0.22 locally.
        Hide
        Milind Bhandarkar added a comment -

        Hung-Chih, since I am currently busy in a lot of other work, can you report the results of your manual testing here, after applying the appropriate patch ? Thanks.

        Show
        Milind Bhandarkar added a comment - Hung-Chih, since I am currently busy in a lot of other work, can you report the results of your manual testing here, after applying the appropriate patch ? Thanks.
        Arun C Murthy made changes -
        Fix Version/s 0.22.0 [ 12314296 ]
        Fix Version/s 0.21.1 [ 12315270 ]
        Fix Version/s 0.23.0 [ 12315569 ]
        Fix Version/s 0.20.204.1 [ 12317846 ]
        Harsh J made changes -
        Fix Version/s 0.24.0 [ 12317652 ]
        Hide
        Harsh J added a comment -

        Since Avro has replaced the Hadoop RCC, shouldn't we just close this?

        Show
        Harsh J added a comment - Since Avro has replaced the Hadoop RCC, shouldn't we just close this?
        Harsh J made changes -
        Affects Version/s 0.20.2 [ 12314203 ]
        Affects Version/s 0.21.0 [ 12313563 ]
        Affects Version/s 0.22.0 [ 12314296 ]
        Affects Version/s 0.23.0 [ 12315569 ]
        Affects Version/s 0.20.203.0 [ 12316064 ]
        Affects Version/s 0.24.0 [ 12317652 ]

          People

          • Assignee:
            Milind Bhandarkar
            Reporter:
            Hung-chih Yang
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Time Tracking

              Estimated:
              Original Estimate - 1m
              1m
              Remaining:
              Remaining Estimate - 1m
              1m
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development