Hive
  1. Hive
  2. HIVE-1889

add an option (hive.index.compact.file.ignore.hdfs) to ignore HDFS location stored in index files.

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.7.0
    • Component/s: Indexing
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In the current code, we use the whole hdfs path in index files to choose splits to run mapreduce job.

      If the data got moved or the name of the cluster got changed, the index data should still be usable.

      1. HIVE-1889.2.patch
        5 kB
        He Yongqiang
      2. HIVE-1889.1.patch
        4 kB
        He Yongqiang

        Issue Links

          Activity

          He Yongqiang created issue -
          He Yongqiang made changes -
          Field Original Value New Value
          Attachment HIVE-1889.1.patch [ 12467610 ]
          He Yongqiang made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Namit Jain added a comment -

          Can you add the new parameter in HiveConf.java and hive-default.xml ?

          Show
          Namit Jain added a comment - Can you add the new parameter in HiveConf.java and hive-default.xml ?
          Namit Jain made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Carl Steinbach made changes -
          Issue Type Improvement [ 4 ] New Feature [ 2 ]
          Component/s Indexing [ 12313710 ]
          Hide
          Carl Steinbach added a comment -

          @Yongqiang: can you please add the name of the new property to this ticket so that people can find it via search?

          Also, do we lose anything in terms of functionality or performance by ignoring the HDFS path information?
          If the answer is no then we should probably just ignore this information by default instead of adding a new
          configuration property.

          Show
          Carl Steinbach added a comment - @Yongqiang: can you please add the name of the new property to this ticket so that people can find it via search? Also, do we lose anything in terms of functionality or performance by ignoring the HDFS path information? If the answer is no then we should probably just ignore this information by default instead of adding a new configuration property.
          He Yongqiang made changes -
          Summary add an option to ignore HDFS location stored in index files. add an option (hive.index.compact.file.ignore.hdfs) to ignore HDFS location stored in index files.
          He Yongqiang made changes -
          Attachment HIVE-1889.2.patch [ 12467654 ]
          Hide
          He Yongqiang added a comment -

          @Carl,
          added the option name to the title of this jira.

          Show
          He Yongqiang added a comment - @Carl, added the option name to the title of this jira.
          Namit Jain made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Namit Jain added a comment -

          +1

          Show
          Namit Jain added a comment - +1
          Hide
          Namit Jain added a comment -

          Committed. Thanks Yongqiang

          Show
          Namit Jain added a comment - Committed. Thanks Yongqiang
          Namit Jain made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          Carl Steinbach made changes -
          Fix Version/s 0.7.0 [ 12315150 ]
          Carl Steinbach made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Lefty Leverenz made changes -
          Link This issue relates to HIVE-417 [ HIVE-417 ]

            People

            • Assignee:
              He Yongqiang
              Reporter:
              He Yongqiang
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development