Hive
  1. Hive
  2. HIVE-3764

Support metastore version consistency check

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0, 0.9.0, 0.10.0, 0.11.0
    • Fix Version/s: 0.12.0
    • Component/s: Metastore
    • Labels:
      None

      Description

      Today there's no version/compatibility information stored in hive metastore. Also the datanucleus configuration property to automatically create missing tables is enabled by default. If you happen to start an older or newer hive or don't run the correct upgrade scripts during migration, the metastore would end up corrupted. The autoCreate schema is not always sufficient to upgrade metastore when migrating to newer release. It's not supported with all databases. Besides the migration often involves altering existing table, changing or moving data etc.

      Hence it's very useful to have some consistency check to make sure that hive is using correct metastore and for production systems the schema is not automatically by running hive.

      1. HIVE-3764.1.patch
        39 kB
        Prasad Mujumdar
      2. HIVE-3764.2.patch
        40 kB
        Prasad Mujumdar
      3. HIVE-3764-12.3.patch
        40 kB
        Prasad Mujumdar
      4. HIVE-3764-trunk.3.patch
        163 kB
        Prasad Mujumdar

        Issue Links

          Activity

          Brett Randall made changes -
          Link This issue is related to HIVE-5606 [ HIVE-5606 ]
          Ashutosh Chauhan made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Thejas M Nair made changes -
          Link This issue relates to HIVE-5427 [ HIVE-5427 ]
          Kousuke Saruta made changes -
          Link This issue is duplicated by HIVE-5374 [ HIVE-5374 ]
          Kousuke Saruta made changes -
          Link This issue is duplicated by HIVE-5374 [ HIVE-5374 ]
          Thejas M Nair made changes -
          Fix Version/s 0.12.0 [ 12324312 ]
          Fix Version/s 0.13.0 [ 12324986 ]
          Ashutosh Chauhan made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 0.13.0 [ 12324986 ]
          Fix Version/s 0.12.0 [ 12324312 ]
          Resolution Fixed [ 1 ]
          Prasad Mujumdar made changes -
          Attachment HIVE-3764-trunk.3.patch [ 12604007 ]
          Prasad Mujumdar made changes -
          Attachment HIVE-3764-12.3.patch [ 12604006 ]
          Prasad Mujumdar made changes -
          Attachment HIVE-3764-trunk.2.patch [ 12604000 ]
          Prasad Mujumdar made changes -
          Attachment HIVE-3764-12.2.patch [ 12603999 ]
          Prasad Mujumdar made changes -
          Attachment HIVE-3764-trunk.2.patch [ 12604000 ]
          Prasad Mujumdar made changes -
          Attachment HIVE-3764-12.2.patch [ 12603999 ]
          Prasad Mujumdar made changes -
          Attachment HIVE-3764.2.patch [ 12603758 ]
          Brock Noland made changes -
          Attachment HIVE-3764.4.patch [ 12603014 ]
          Prasad Mujumdar made changes -
          Link This issue blocks HIVE-5301 [ HIVE-5301 ]
          Prasad Mujumdar made changes -
          Description Today there's no version/compatibility information stored in hive metastore. Also the datanucleus configuration property to automatically create missing tables is enabled by default. If you happen to start an older or newer hive or don't run the correct upgrade scripts during migration, the metastore would end up corrupted. The autoCreate schema is not always sufficient to upgrade metastore when migrating to newer release. It's not supported with all databases. Besides the migration often involves altering existing table, changing or moving data etc.

          Hence it's very useful to have some consistency check to make sure that hive is using correct metastore and for production systems the schema is not automatically by running hive.

          Besides it would be helpful to add a tool that can leverage this version information to figure out the required set of upgrade scripts, and execute those against the configured metastore. Now that Hive includes Beeline client, it can be used to execute the scripts.
          Today there's no version/compatibility information stored in hive metastore. Also the datanucleus configuration property to automatically create missing tables is enabled by default. If you happen to start an older or newer hive or don't run the correct upgrade scripts during migration, the metastore would end up corrupted. The autoCreate schema is not always sufficient to upgrade metastore when migrating to newer release. It's not supported with all databases. Besides the migration often involves altering existing table, changing or moving data etc.

          Hence it's very useful to have some consistency check to make sure that hive is using correct metastore and for production systems the schema is not automatically by running hive.
          Prasad Mujumdar made changes -
          Attachment HIVE-3764-0.13-addional-file.patch [ 12602976 ]
          Prasad Mujumdar made changes -
          Attachment HIVE-3764.4.patch [ 12602975 ]
          Prasad Mujumdar made changes -
          Attachment HIVE-3764.1.patch [ 12603543 ]
          Brock Noland made changes -
          Attachment HIVE-3764.4.patch [ 12603014 ]
          Prasad Mujumdar made changes -
          Attachment HIVE-3764-0.13-addional-file.patch [ 12602976 ]
          Prasad Mujumdar made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Prasad Mujumdar made changes -
          Attachment HIVE-3764.4.patch [ 12602975 ]
          Prasad Mujumdar made changes -
          Fix Version/s 0.12.0 [ 12324312 ]
          Affects Version/s 0.11.0 [ 12323587 ]
          Affects Version/s 0.10.0 [ 12320745 ]
          Affects Version/s 0.9.0 [ 12317742 ]
          Affects Version/s 0.8.0 [ 12316178 ]
          Prasad Mujumdar made changes -
          Attachment HIVE-3764-1.patch [ 12555725 ]
          Prasad Mujumdar made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Prasad Mujumdar made changes -
          Description Today there's no version/compatibility information stored in hive metastore. Also the datanucleus configuration property to automatically create missing tables is enabled by default. If you happen to start an older or newer hive or don't run the correct upgrade scripts during migration, the metastore would end up corrupted. The autoCreate schema is not always sufficient to upgrade metastore when migrating to newer release. It's not supported with all databases. Besides the migration often involves altering existing table, changing or moving data etc.

          Hence it's very useful to have some consistency check to make sure that hive is using correct metastore and for production systems the schema is not automatically by running hive.
          Today there's no version/compatibility information stored in hive metastore. Also the datanucleus configuration property to automatically create missing tables is enabled by default. If you happen to start an older or newer hive or don't run the correct upgrade scripts during migration, the metastore would end up corrupted. The autoCreate schema is not always sufficient to upgrade metastore when migrating to newer release. It's not supported with all databases. Besides the migration often involves altering existing table, changing or moving data etc.

          Hence it's very useful to have some consistency check to make sure that hive is using correct metastore and for production systems the schema is not automatically by running hive.

          Besides it would be helpful to add a tool that can leverage this version information to figure out the required set of upgrade scripts, and execute those against the configured metastore. Now that Hive includes Beeline client, it can be used to execute the scripts.
          Ashutosh Chauhan made changes -
          Fix Version/s 0.10.0 [ 12320745 ]
          Prasad Mujumdar made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Fix Version/s 0.10.0 [ 12320745 ]
          Prasad Mujumdar made changes -
          Field Original Value New Value
          Attachment HIVE-3764-1.patch [ 12555725 ]
          Prasad Mujumdar created issue -

            People

            • Assignee:
              Prasad Mujumdar
              Reporter:
              Prasad Mujumdar
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development