Uploaded image for project: 'Sentry'
  1. Sentry
  2. SENTRY-1898

Sentry no longer supports creating more than ~15 partitions at once

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.0.0
    • Component/s: Sentry
    • Labels:
      None

      Description

      Vamsee reported: I notice this problem on our stress cluster where we are creating a huge amounts of Hive metadata (Databases, Tables, and Partitions). During this process, multiple partitions of the table are being created at one go using:
      ALTER TABLE <TB> ADD PARTITION <PART1> ADD PARTITION <PART2>...
      I notice that when we add more than 20 partitions here in one sql statement, Sentry fails to process the transaction as we cross the 4000 character threshold on PATH change table.

      Every partition entry is of format:
      {"1":

      {"str":"test_hivemd_db_2017_08_15_08_14_27_2_0_8.test_hivemd_table_2017_08_15_08_14_27_2_0_0"}

      ,"2":

      {"lst":["lst",1,["str",6,"user","hive","warehouse","test_hivemd_db_2017_08_15_08_14_27_2_0_8.db","test_hivemd_table_2017_08_15_08_14_27_2_0_0","part=test_hivemd_partition_2017_08_15_08_14_27_2_0_1"]]}

      ,"3":{"lst":["lst",0]}}
      ~ 327 characters
      So to fit in 4000 characters, we are limited to 10-15 partitions to be created at one go.

      We should allow the big path update from hive.

        Attachments

        1. SENTRY-1898.002.patch
          8 kB
          Na Li
        2. SENTRY-1898.001.patch
          8 kB
          Na Li

          Issue Links

            Activity

              People

              • Assignee:
                linaataustin Na Li
                Reporter:
                linaataustin Na Li
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: