Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: 0.20.1, 0.90.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      From Tatsuya Kawano up on hbase-user@

      50 client threads who try to put millions of records, autoFlush(false), flushCommits() on every 5,000 put. After inserting about 3 million records, a deadlock occurred on a region server who has both the table and index regions loaded.

      I have attached a full thread dump of the deadlocked region server, and you can see IPC Server handlers are blocked in org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.updateIndex().

      I found the flowing FIXME comment on updateIndex() method, and it seems this is the deadlock I'm having.

        // FIXME: This call takes place in an RPC, and requires an RPC. This makes for
        // a likely deadlock if the number of RPCs we are trying to serve is >= the
        // number of handler threads.
        private void updateIndex(IndexSpecification indexSpec, byte[] row,
            SortedMap<byte[], byte[]> columnValues) throws IOException {
      

      I use HBase 0.20.1 and my region servers were running with 10 RPC handler threads on each (default).

      Maybe you can workaround this by adding more RPC handlers (increase the value of "hbase.regionserver.handler.count" in hbase-site.xml)

      Opening this issue to track the FIXME.

      1. thread.dump.gz
        4 kB
        Andrew Purtell

        Issue Links

          Activity

          Andrew Purtell created issue -
          Hide
          ryan rawson added a comment -

          I doubt we could fix this in 0.20.x, (especially not in 0.20.1), since it would require a completely different RPC architecture....

          I think this might be fixable in 0.22.

          Show
          ryan rawson added a comment - I doubt we could fix this in 0.20.x, (especially not in 0.20.1), since it would require a completely different RPC architecture.... I think this might be fixable in 0.22.
          ryan rawson made changes -
          Field Original Value New Value
          Fix Version/s 0.22.0 [ 12314223 ]
          Hide
          Andrew Purtell added a comment -

          Attached thread dump provided by user for reference.

          Show
          Andrew Purtell added a comment - Attached thread dump provided by user for reference.
          Andrew Purtell made changes -
          Attachment thread.dump.gz [ 12422313 ]
          Andrew Purtell made changes -
          Link This issue is related to HBASE-2182 [ HBASE-2182 ]
          Hide
          stack added a comment -

          indexed has been removed from hbase.

          Show
          stack added a comment - indexed has been removed from hbase.
          stack made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Gary Helmling added a comment -

          Re-opening to fix up resolution and fix version

          Show
          Gary Helmling added a comment - Re-opening to fix up resolution and fix version
          Gary Helmling made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Hide
          Gary Helmling added a comment -

          Re-resolving as won't fix since indexed is moved out. Also dropping fix version – we shouldn't be showing this against 0.92.

          Show
          Gary Helmling added a comment - Re-resolving as won't fix since indexed is moved out. Also dropping fix version – we shouldn't be showing this against 0.92.
          Gary Helmling made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Fix Version/s 0.92.0 [ 12314223 ]
          Resolution Won't Fix [ 2 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              Andrew Purtell
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development