Hive
  1. Hive
  2. HIVE-2138

Exception when no splits returned from index

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: Indexing
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Running a query that uses indexing but doesn't return any results give an exception.

       java.lang.IllegalArgumentException: Can not create a Path from an empty string
      at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
      at org.apache.hadoop.fs.Path.<init>(Path.java:90)
      at org.apache.hadoop.util.StringUtils.stringToPath(StringUtils.java:224)
      at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:282)
      at org.apache.hadoop.hive.ql.index.HiveIndexedInputFormat.getSplits(HiveIndexedInputFormat.java:123) 

      This could potentially be fixed by creating a new empty file to use for the splits.

      Once this is fixed, the index_auto_test_if_used.q can be used.

      1. index_auto_test_if_used.q
        0.9 kB
        Russell Melick
      2. HIVE-2138.1.patch
        14 kB
        Syed S. Albiz
      3. HIVE-2138.2.patch
        19 kB
        Syed S. Albiz
      4. HIVE-2138.3.patch
        10 kB
        Syed S. Albiz
      5. HIVE-2138.4.patch
        8 kB
        Syed S. Albiz
      6. HIVE-2138.5.patch
        8 kB
        Syed S. Albiz

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #881 (See https://builds.apache.org/job/Hive-trunk-h0.21/881/)
          HIVE-2138. Exception when no splits returned from index.
          (Syed Albiz via jvs)

          jvs : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1155059
          Files :

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexedInputFormat.java
          • /hive/trunk/ql/src/test/results/clientpositive/index_auto_empty.q.out
          • /hive/trunk/ql/src/test/queries/clientpositive/index_auto_empty.q
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #881 (See https://builds.apache.org/job/Hive-trunk-h0.21/881/ ) HIVE-2138 . Exception when no splits returned from index. (Syed Albiz via jvs) jvs : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1155059 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexedInputFormat.java /hive/trunk/ql/src/test/results/clientpositive/index_auto_empty.q.out /hive/trunk/ql/src/test/queries/clientpositive/index_auto_empty.q
          Hide
          John Sichi added a comment -

          Committed. Thanks Syed!

          Show
          John Sichi added a comment - Committed. Thanks Syed!
          Hide
          John Sichi added a comment -

          Giving it another go now.

          Show
          John Sichi added a comment - Giving it another go now.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1255/
          -----------------------------------------------------------

          (Updated 2011-08-05 19:05:05.682867)

          Review request for hive, John Sichi and Yongqiang He.

          Changes
          -------

          updated testcases with new output

          Summary
          -------

          return an empty array of input splits when we get back no results from the hiveIndexResult

          This addresses bug HIVE-2138.
          https://issues.apache.org/jira/browse/HIVE-2138

          Diffs (updated)


          ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexedInputFormat.java 1a1ecd7
          ql/src/test/queries/clientpositive/index_auto_empty.q PRE-CREATION
          ql/src/test/results/clientpositive/index_auto_empty.q.out PRE-CREATION

          Diff: https://reviews.apache.org/r/1255/diff

          Testing
          -------

          added index_auto_test_if_used.q to the test suite

          Thanks,

          Syed

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1255/ ----------------------------------------------------------- (Updated 2011-08-05 19:05:05.682867) Review request for hive, John Sichi and Yongqiang He. Changes ------- updated testcases with new output Summary ------- return an empty array of input splits when we get back no results from the hiveIndexResult This addresses bug HIVE-2138 . https://issues.apache.org/jira/browse/HIVE-2138 Diffs (updated) ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexedInputFormat.java 1a1ecd7 ql/src/test/queries/clientpositive/index_auto_empty.q PRE-CREATION ql/src/test/results/clientpositive/index_auto_empty.q.out PRE-CREATION Diff: https://reviews.apache.org/r/1255/diff Testing ------- added index_auto_test_if_used.q to the test suite Thanks, Syed
          Hide
          Syed S. Albiz added a comment -

          regenerated patch with updated testcase output

          Show
          Syed S. Albiz added a comment - regenerated patch with updated testcase output
          Hide
          John Sichi added a comment -

          index_auto_empty needs its log updated:

          [junit] < PREHOOK: type: ALTERINDEX_REBUILD
          [junit]
          [junit] > PREHOOK: type: QUERY
          [junit] 23c23
          [junit] < POSTHOOK: type: ALTERINDEX_REBUILD
          [junit]

          Show
          John Sichi added a comment - index_auto_empty needs its log updated: [junit] < PREHOOK: type: ALTERINDEX_REBUILD [junit] — [junit] > PREHOOK: type: QUERY [junit] 23c23 [junit] < POSTHOOK: type: ALTERINDEX_REBUILD [junit] —
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1255/
          -----------------------------------------------------------

          (Updated 2011-08-04 21:56:31.633797)

          Review request for hive, John Sichi and Yongqiang He.

          Changes
          -------

          removed stale testcase

          Summary
          -------

          return an empty array of input splits when we get back no results from the hiveIndexResult

          This addresses bug HIVE-2138.
          https://issues.apache.org/jira/browse/HIVE-2138

          Diffs (updated)


          ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexedInputFormat.java 1a1ecd7
          ql/src/test/queries/clientpositive/index_auto_empty.q PRE-CREATION
          ql/src/test/results/clientpositive/index_auto_empty.q.out PRE-CREATION

          Diff: https://reviews.apache.org/r/1255/diff

          Testing
          -------

          added index_auto_test_if_used.q to the test suite

          Thanks,

          Syed

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1255/ ----------------------------------------------------------- (Updated 2011-08-04 21:56:31.633797) Review request for hive, John Sichi and Yongqiang He. Changes ------- removed stale testcase Summary ------- return an empty array of input splits when we get back no results from the hiveIndexResult This addresses bug HIVE-2138 . https://issues.apache.org/jira/browse/HIVE-2138 Diffs (updated) ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexedInputFormat.java 1a1ecd7 ql/src/test/queries/clientpositive/index_auto_empty.q PRE-CREATION ql/src/test/results/clientpositive/index_auto_empty.q.out PRE-CREATION Diff: https://reviews.apache.org/r/1255/diff Testing ------- added index_auto_test_if_used.q to the test suite Thanks, Syed
          Hide
          Syed S. Albiz added a comment -

          removed old testcase

          Show
          Syed S. Albiz added a comment - removed old testcase
          Hide
          John Sichi added a comment -

          index_auto_test_if_used.q should not be in this patch.

          Show
          John Sichi added a comment - index_auto_test_if_used.q should not be in this patch.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1255/
          -----------------------------------------------------------

          (Updated 2011-08-02 23:44:30.658334)

          Review request for hive, John Sichi and Yongqiang He.

          Changes
          -------

          replace test case with something more robust

          Summary
          -------

          return an empty array of input splits when we get back no results from the hiveIndexResult

          This addresses bug HIVE-2138.
          https://issues.apache.org/jira/browse/HIVE-2138

          Diffs (updated)


          ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexedInputFormat.java 1a1ecd7
          ql/src/test/queries/clientpositive/index_auto_empty.q PRE-CREATION
          ql/src/test/queries/clientpositive/index_auto_test_if_used.q PRE-CREATION
          ql/src/test/results/clientpositive/index_auto_empty.q.out PRE-CREATION

          Diff: https://reviews.apache.org/r/1255/diff

          Testing
          -------

          added index_auto_test_if_used.q to the test suite

          Thanks,

          Syed

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1255/ ----------------------------------------------------------- (Updated 2011-08-02 23:44:30.658334) Review request for hive, John Sichi and Yongqiang He. Changes ------- replace test case with something more robust Summary ------- return an empty array of input splits when we get back no results from the hiveIndexResult This addresses bug HIVE-2138 . https://issues.apache.org/jira/browse/HIVE-2138 Diffs (updated) ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexedInputFormat.java 1a1ecd7 ql/src/test/queries/clientpositive/index_auto_empty.q PRE-CREATION ql/src/test/queries/clientpositive/index_auto_test_if_used.q PRE-CREATION ql/src/test/results/clientpositive/index_auto_empty.q.out PRE-CREATION Diff: https://reviews.apache.org/r/1255/diff Testing ------- added index_auto_test_if_used.q to the test suite Thanks, Syed
          Hide
          Syed S. Albiz added a comment -

          replace testcase with a robust testcase to check for empty splits

          Show
          Syed S. Albiz added a comment - replace testcase with a robust testcase to check for empty splits
          Hide
          John Sichi added a comment -

          +1, will commit when tests pass.

          Show
          John Sichi added a comment - +1, will commit when tests pass.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1255/
          -----------------------------------------------------------

          (Updated 2011-08-02 18:48:23.995744)

          Review request for hive, John Sichi and Yongqiang He.

          Changes
          -------

          addressed comments

          Summary
          -------

          return an empty array of input splits when we get back no results from the hiveIndexResult

          This addresses bug HIVE-2138.
          https://issues.apache.org/jira/browse/HIVE-2138

          Diffs (updated)


          ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexedInputFormat.java 1a1ecd7
          ql/src/test/queries/clientpositive/index_auto_test_if_used.q PRE-CREATION
          ql/src/test/results/clientpositive/index_auto_test_if_used.q.out PRE-CREATION

          Diff: https://reviews.apache.org/r/1255/diff

          Testing
          -------

          added index_auto_test_if_used.q to the test suite

          Thanks,

          Syed

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1255/ ----------------------------------------------------------- (Updated 2011-08-02 18:48:23.995744) Review request for hive, John Sichi and Yongqiang He. Changes ------- addressed comments Summary ------- return an empty array of input splits when we get back no results from the hiveIndexResult This addresses bug HIVE-2138 . https://issues.apache.org/jira/browse/HIVE-2138 Diffs (updated) ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexedInputFormat.java 1a1ecd7 ql/src/test/queries/clientpositive/index_auto_test_if_used.q PRE-CREATION ql/src/test/results/clientpositive/index_auto_test_if_used.q.out PRE-CREATION Diff: https://reviews.apache.org/r/1255/diff Testing ------- added index_auto_test_if_used.q to the test suite Thanks, Syed
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1255/#review1265
          -----------------------------------------------------------

          ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexedInputFormat.java
          <https://reviews.apache.org/r/1255/#comment2898>

          We're not actually generating a dummy file now...

          ql/src/test/queries/clientpositive/index_auto_test_if_used.q
          <https://reviews.apache.org/r/1255/#comment2900>

          Could you add a note that this test relies on the fact that we currently attempt to use indexes even when they are out of date, and that once we fix that by detecting stale indexes, we'll need another way to verify that the index actually had an effect?

          ql/src/test/queries/clientpositive/index_auto_test_if_used.q
          <https://reviews.apache.org/r/1255/#comment2899>

          Reissue query to demonstrate that results are returned after index lookup is disabled.

          • John

          On 2011-08-02 02:35:11, Syed Albiz wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/1255/

          -----------------------------------------------------------

          (Updated 2011-08-02 02:35:11)

          Review request for hive, John Sichi and Yongqiang He.

          Summary

          -------

          return an empty array of input splits when we get back no results from the hiveIndexResult

          This addresses bug HIVE-2138.

          https://issues.apache.org/jira/browse/HIVE-2138

          Diffs

          -----

          ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexedInputFormat.java 1a1ecd7

          ql/src/test/queries/clientpositive/index_auto_test_if_used.q PRE-CREATION

          ql/src/test/results/clientpositive/index_auto_test_if_used.q.out PRE-CREATION

          Diff: https://reviews.apache.org/r/1255/diff

          Testing

          -------

          added index_auto_test_if_used.q to the test suite

          Thanks,

          Syed

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1255/#review1265 ----------------------------------------------------------- ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexedInputFormat.java < https://reviews.apache.org/r/1255/#comment2898 > We're not actually generating a dummy file now... ql/src/test/queries/clientpositive/index_auto_test_if_used.q < https://reviews.apache.org/r/1255/#comment2900 > Could you add a note that this test relies on the fact that we currently attempt to use indexes even when they are out of date, and that once we fix that by detecting stale indexes, we'll need another way to verify that the index actually had an effect? ql/src/test/queries/clientpositive/index_auto_test_if_used.q < https://reviews.apache.org/r/1255/#comment2899 > Reissue query to demonstrate that results are returned after index lookup is disabled. John On 2011-08-02 02:35:11, Syed Albiz wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1255/ ----------------------------------------------------------- (Updated 2011-08-02 02:35:11) Review request for hive, John Sichi and Yongqiang He. Summary ------- return an empty array of input splits when we get back no results from the hiveIndexResult This addresses bug HIVE-2138 . https://issues.apache.org/jira/browse/HIVE-2138 Diffs ----- ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexedInputFormat.java 1a1ecd7 ql/src/test/queries/clientpositive/index_auto_test_if_used.q PRE-CREATION ql/src/test/results/clientpositive/index_auto_test_if_used.q.out PRE-CREATION Diff: https://reviews.apache.org/r/1255/diff Testing ------- added index_auto_test_if_used.q to the test suite Thanks, Syed
          Hide
          John Sichi added a comment -

          A few comments added in Review Board.

          Show
          John Sichi added a comment - A few comments added in Review Board.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1255/
          -----------------------------------------------------------

          Review request for hive, John Sichi and Yongqiang He.

          Summary
          -------

          return an empty array of input splits when we get back no results from the hiveIndexResult

          This addresses bug HIVE-2138.
          https://issues.apache.org/jira/browse/HIVE-2138

          Diffs


          ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexedInputFormat.java 1a1ecd7
          ql/src/test/queries/clientpositive/index_auto_test_if_used.q PRE-CREATION
          ql/src/test/results/clientpositive/index_auto_test_if_used.q.out PRE-CREATION

          Diff: https://reviews.apache.org/r/1255/diff

          Testing
          -------

          added index_auto_test_if_used.q to the test suite

          Thanks,

          Syed

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1255/ ----------------------------------------------------------- Review request for hive, John Sichi and Yongqiang He. Summary ------- return an empty array of input splits when we get back no results from the hiveIndexResult This addresses bug HIVE-2138 . https://issues.apache.org/jira/browse/HIVE-2138 Diffs ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexedInputFormat.java 1a1ecd7 ql/src/test/queries/clientpositive/index_auto_test_if_used.q PRE-CREATION ql/src/test/results/clientpositive/index_auto_test_if_used.q.out PRE-CREATION Diff: https://reviews.apache.org/r/1255/diff Testing ------- added index_auto_test_if_used.q to the test suite Thanks, Syed
          Hide
          Syed S. Albiz added a comment -

          fix + testcase

          Show
          Syed S. Albiz added a comment - fix + testcase
          Hide
          John Sichi added a comment -

          Syed, if you get time to look into this one, it's a good one to fix for getting automatic index usage production-ready.

          Show
          John Sichi added a comment - Syed, if you get time to look into this one, it's a good one to fix for getting automatic index usage production-ready.
          Hide
          Russell Melick added a comment -

          Attached test file (index_auto_test_if_used) that should test whether indexes are used.

          Show
          Russell Melick added a comment - Attached test file (index_auto_test_if_used) that should test whether indexes are used.

            People

            • Assignee:
              Syed S. Albiz
              Reporter:
              Russell Melick
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development