Hive
  1. Hive
  2. HIVE-6016

Hadoop23Shims has a bug in listLocatedStatus impl.

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.13.0
    • Fix Version/s: 0.13.0
    • Component/s: Shims
    • Labels:
      None

      Description

      Prashant and I discovered that the implementation of the wrapping Iterator in listLocatedStatus at https://github.com/apache/hive/blob/2d2f89c21618341987c1257a88691981f1f606c7/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java#L350-L393 is broken.

      Basically, if you had files (a,b,_s) , with a filter that is supposed to filter out _s, we expect an output result of (a,b). Instead, we get (a,b,null), with hasNext looking at the next value to see if it's null, and using that to decide if it has any more entries, and thus, (a,b,_s) becomes (a,b).

      There's a boundary condition on the very first pick, which causes a (_s,a,b) to result in (_s,a,b), bypassing the filter, and thus, we wind up with a resultant unfiltered (_s,a,b) which orc breaks on.

      The effect of this bug is that Orc will not be able to read directories where there is a _SUCCESS file, say, as the first entry returned by the FileStatus.

      1. HIVE-6016.1.patch
        1 kB
        Prasanth Jayachandran

        Issue Links

          Activity

          Hide
          Prasanth Jayachandran added a comment -

          There is a correction to the description.

          I think only (_s,a,b) is a problem. The logic seems not to apply the PathFilter for first file alone. For other cases it works fine as there is a while loop in next() which keeps iterating to next valid file by applying filter. So in case of (a,_s,b), first file is a for which no filter is applied. For the next file _s filter is applied and next becomes null. But the while continues to next valid file in which case its b. So finally only (a,b) is returned. The iterator will not return null under any case.

          Show
          Prasanth Jayachandran added a comment - There is a correction to the description. I think only (_s,a,b) is a problem. The logic seems not to apply the PathFilter for first file alone. For other cases it works fine as there is a while loop in next() which keeps iterating to next valid file by applying filter. So in case of (a,_s,b), first file is a for which no filter is applied. For the next file _s filter is applied and next becomes null. But the while continues to next valid file in which case its b. So finally only (a,b) is returned. The iterator will not return null under any case.
          Hide
          Sushanth Sowmyan added a comment -

          Thanks for the correction, Prashanth, I've edited the bug report to remove that case.

          Show
          Sushanth Sowmyan added a comment - Thanks for the correction, Prashanth, I've edited the bug report to remove that case.
          Hide
          Prasanth Jayachandran added a comment -

          This should fix hcatalog unit test failure TestOrcDynamicPartitioned in hadoop2.

          Show
          Prasanth Jayachandran added a comment - This should fix hcatalog unit test failure TestOrcDynamicPartitioned in hadoop2.
          Hide
          Prasanth Jayachandran added a comment -

          Making it as patch available for precommit tests.

          Show
          Prasanth Jayachandran added a comment - Making it as patch available for precommit tests.
          Hide
          Sushanth Sowmyan added a comment -

          Patch looks good to me. +1.

          Paging Ashutosh Chauhan/Owen O'Malley for another review.

          Show
          Sushanth Sowmyan added a comment - Patch looks good to me. +1. Paging Ashutosh Chauhan / Owen O'Malley for another review.
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12618327/HIVE-6016.1.patch

          SUCCESS: +1 4763 tests passed

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/619/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/619/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12618327

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12618327/HIVE-6016.1.patch SUCCESS: +1 4763 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/619/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/619/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12618327
          Hide
          Ashutosh Chauhan added a comment -

          Instead of doing filter.accept() logic twice (after this patch), it seems like its enough to just do it once in outer loop(as introduced in this patch). Shall we remove existing filter.accept() from inner loop?

          Show
          Ashutosh Chauhan added a comment - Instead of doing filter.accept() logic twice (after this patch), it seems like its enough to just do it once in outer loop(as introduced in this patch). Shall we remove existing filter.accept() from inner loop?
          Hide
          Ashutosh Chauhan added a comment -

          Sorry, I was confused those are not 2 loops, but constructor and overloaded method. Patch looks good. +1

          Show
          Ashutosh Chauhan added a comment - Sorry, I was confused those are not 2 loops, but constructor and overloaded method. Patch looks good. +1
          Hide
          Ashutosh Chauhan added a comment -

          Committed to trunk. Thanks, Prasanth!

          Show
          Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Prasanth!

            People

            • Assignee:
              Prasanth Jayachandran
              Reporter:
              Sushanth Sowmyan
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development