Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-18885

DbNotificationListener has a deadlock between Java and DB locks (2.x line)

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.2
    • Fix Version/s: 2.4.0, 2.3.3
    • Component/s: Hive, Metastore
    • Labels:
      None

      Description

      You can see the problem from looking at the code, but it actually created severe problems for real life Hive user.

      When alter table has cascade option it does the following:

               msdb.openTransaction()
                ...
                List<Partition> parts = msdb.getPartitions(dbname, name, -1);
                for (Partition part : parts) {
                  List<FieldSchema> oldCols = part.getSd().getCols();
                  part.getSd().setCols(newt.getSd().getCols());
                  String oldPartName = Warehouse.makePartName(oldt.getPartitionKeys(), part.getValues());
                  updatePartColumnStatsForAlterColumns(msdb, part, oldPartName, part.getValues(), oldCols, part);
                  msdb.alterPartition(dbname, name, part.getValues(), part);
                }
       

      So it walks all partitions (and this may be huge list) and does some non-trivial operations in one single uber-transaction.

      When DbNotificationListener is enabled, it adds an event for each partition, all while
      holding a row lock on NOTIFICATION_SEQUENCE table. As a result, while this is happening no other write DDL can proceed. This can sometimes cause DB lock timeouts which cause HMS level operation retries which make things even worse.

      In one particular case this pretty much made HMS unusable.

        Attachments

        1. HIVE-18885.04-branch-2.patch
          6 kB
          Vihang Karajgaonkar
        2. HIVE-18885.03-branch-2.patch
          5 kB
          Vihang Karajgaonkar
        3. HIVE-18885.02.branch-2.patch
          3 kB
          Vihang Karajgaonkar
        4. HIVE-18885.01.branch-2.patch
          3 kB
          Vihang Karajgaonkar

          Issue Links

            Activity

              People

              • Assignee:
                vihangk1 Vihang Karajgaonkar
                Reporter:
                akolb Alex Kolbasov
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: