HBase
  1. HBase
  2. HBASE-4389

Address lots of issues with migration from 90 to trunk

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Later
    • Affects Version/s: 0.92.0
    • Fix Version/s: None
    • Component/s: master, migration
    • Labels:
      None
    • Release Note:
      Unscheduling. It doesn't look like anything is working on this.

      Description

      Looking over the migration code that removes HTD from HRI, there are lots of issues. This JIRA is to redo this code in a way that will be less bug prone, and also future proof.

        Activity

        Hide
        Todd Lipcon added a comment -

        After a quick pass through the migration code, here are the various issues I see:

        • HRegionInfo didn't have its VERSION incremented. Hence exception catching is used to try to determine which version is being read.
        • A single "migrated" boolean flag is used in ROOT to indicate that META has been updated to the new format. This leaves us no room for future migrations. "migrated" is not a boolean. It should instead be "migratedToVersion" or something
        • Migration should be idempotent - ie even if the "migratedToVersion" flag didn't get updated, migration should be able to re-run without crashing
        • Duplicated code between updateRootWithNewRegionInfo and updateMetaWithNewRegionInfo
        • Each region that is processed results in a call to createTableDescriptor, which results in calls to the NN - this will take a long time on a big cluster, and is unnecessary
        • No sanity checking that all of the HTDs for a table are equal
        • Migration code should ideally be moved to a separate class, instead of mixed with the non-migration code paths
        Show
        Todd Lipcon added a comment - After a quick pass through the migration code, here are the various issues I see: HRegionInfo didn't have its VERSION incremented. Hence exception catching is used to try to determine which version is being read. A single "migrated" boolean flag is used in ROOT to indicate that META has been updated to the new format. This leaves us no room for future migrations. "migrated" is not a boolean. It should instead be "migratedToVersion" or something Migration should be idempotent - ie even if the "migratedToVersion" flag didn't get updated, migration should be able to re-run without crashing Duplicated code between updateRootWithNewRegionInfo and updateMetaWithNewRegionInfo Each region that is processed results in a call to createTableDescriptor, which results in calls to the NN - this will take a long time on a big cluster, and is unnecessary No sanity checking that all of the HTDs for a table are equal Migration code should ideally be moved to a separate class, instead of mixed with the non-migration code paths
        Hide
        stack added a comment -

        Patch in hbase-4388 addresses some of above.

        + It increments HRI VERSION (and makes HRI able to deserialize the old format that has HTD)
        + We no longer have a migrated flag but a version in root for meta table (a short)
        + Migration should be idempotent now; we look at each row and check its HRI VERSION and if not current, do update
        + Duplicated code has been removed
        + We no longer do createTableDescriptor on each HRI scanned; just on the first.

        Outstanding:

        + Migration code has been coalesced some but still code spread about
        + There is no sanity checking that all HTDs for a table are equal. Not sure what we'd do if they were not equal other than warn. We wouldn't know which of them all to take. Patch in hbase-4388 takes the first regions HTD. This is usually the first updated; later updates could fail so its the first is likely the 'latest' iteration on HTD I'd guess.

        Show
        stack added a comment - Patch in hbase-4388 addresses some of above. + It increments HRI VERSION (and makes HRI able to deserialize the old format that has HTD) + We no longer have a migrated flag but a version in root for meta table (a short) + Migration should be idempotent now; we look at each row and check its HRI VERSION and if not current, do update + Duplicated code has been removed + We no longer do createTableDescriptor on each HRI scanned; just on the first. Outstanding: + Migration code has been coalesced some but still code spread about + There is no sanity checking that all HTDs for a table are equal. Not sure what we'd do if they were not equal other than warn. We wouldn't know which of them all to take. Patch in hbase-4388 takes the first regions HTD. This is usually the first updated; later updates could fail so its the first is likely the 'latest' iteration on HTD I'd guess.
        Hide
        stack added a comment -

        Making this major rather than critical since a good few of the listed items have been addressed by hbase-4388.

        Show
        stack added a comment - Making this major rather than critical since a good few of the listed items have been addressed by hbase-4388.
        Hide
        Lars Hofhansl added a comment -

        Are we trying to fix these? Moving to 0.94.1 for now.

        Show
        Lars Hofhansl added a comment - Are we trying to fix these? Moving to 0.94.1 for now.
        Hide
        stack added a comment -

        Old issue, no longer pertinent. Abandoned.

        Show
        stack added a comment - Old issue, no longer pertinent. Abandoned.

          People

          • Assignee:
            Unassigned
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development