Uploaded image for project: 'Sentry'
  1. Sentry
  2. SENTRY-1295

Investigate malformed paths in HMS db

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Paths in HMS are expected to be in one of these forms:

      • hdfs://hostname:port/path
      • hdfs:///path
      • /path, in which case, scheme will be constructed from FileSystem.getDefaultURI
      • URIs with non hdfs scheme will just be ignored

      I came across atleast 2 sentry users where HMS did have paths which do not comply with above rules and hence HMS plugin initialization for pathupdates failed. See sentry-1260 and sentry-1270 for details on how these errors surface.

      With 1260 and 1270 we should have more information on what these malformed paths were. But we should continue to investigate and fix the root cause, It would most likely be in HMS code base. Until then, here is how you can diagnose and fix it manually:

      Look for malformed paths in HMS : Look in DBS as well as SDS tables.

      SELECT "NAME", "DB_LOCATION_URI" FROM "DBS" WHERE NOT "DB_LOCATION_URI" LIKE 'hdfs://%/%';
      NAME | DB_LOCATION_URI 
      -----------+--------------------
      db_name | hdfs://nameservice1
      (1 row)
      

      Fix it manually updating the HMS location

      UPDATE DBS
      SET DB_LOCATION_URI='hdfs://nameservice1/user/hive/warehouse/db_name.db'
      WHERE DB_ID=12345;
      

      Lets track occurrences of these malformed paths here:

      • hdfs://nameservice1 : Not sure why would any one create a db/table in root directory? Should we accept this in Sentry?

      What does SKEWED_COL_VALUE_LOC_MAP.location in HMS correspond to? Double check if there are any malformed paths here?

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                colin Colin
                Reporter:
                sravya Sravya Tirukkovalur
              • Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: