Hive
  1. Hive
  2. HIVE-1813

Hive should be able to run on multiple data centers

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Currently, hive assumes a single metastore and the HADOOP_HOME is passed as a environment variable.

      It would be desirable to support hive on top of multiple data centers (dfs + mr).

      For eg. there could be 2 metastores: primary and secondary. They would have different dfs's , and there will be a
      dfs->mr mapping maintained by the metastore.

      Hive would be enhanced to support multiple metastores and all operations (ddl + query) would span multiple metastores.

      Different consistency pluggable policies can be employed - for eg. if a table/partition can be present in both the metastores with different
      last modification times, either the last one can be used or an error can be thrown.

      It will be upto the application (outside hive) to copy the data from one metastore to another, and to maintain consistency inside.

        Issue Links

          Activity

          Hide
          Namit Jain added a comment -

          The data can be copied from one dfs to another using distcp - later on a wrapper can be developed in hive for the same.
          Something like:

          alter table <T> partition <P> copy <src> to <dst>;
          alter table <T> partition <P> move <src> to <dst>;

          Show
          Namit Jain added a comment - The data can be copied from one dfs to another using distcp - later on a wrapper can be developed in hive for the same. Something like: alter table <T> partition <P> copy <src> to <dst>; alter table <T> partition <P> move <src> to <dst>;

            People

            • Assignee:
              Unassigned
              Reporter:
              Namit Jain
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:

                Development