Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-11654

WAL Splitting dirs are not deleted after replay.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.98.4
    • Fix Version/s: 0.98.5
    • Component/s: master, wal
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I build a small cluster (20 nodes, several hundred regions) with hbase-0.98.4. And I found some splitting directories in /hbase/WALs/ today, which is very strange because those logs should have been replayed and deleted. Even though the ZK nodes of the dead RS had been deleted, these splitting directories still can cause a serious trouble for cluster restart. It resplitted and replayed all the splitting directories every time I restart my cluster, and cost a huge amount of time. Can't imagine what could happened if it's a cluster with hundreds of nodes and tens of thousands of regions.

      Found 56 items
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:21 /hbase/WALs/hdpdev1.cm6.tbsite.net,60020,1406714828440-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:43 /hbase/WALs/hdpdev1.cm6.tbsite.net,60020,1406716991836-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-08-02 19:16 /hbase/WALs/hdpdev1.cm6.tbsite.net,60020,1406778815585
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:26 /hbase/WALs/hdpdev10.cm6.tbsite.net,60020,1406526862752-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:43 /hbase/WALs/hdpdev10.cm6.tbsite.net,60020,1406716933471-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-08-02 19:02 /hbase/WALs/hdpdev10.cm6.tbsite.net,60020,1406778815536
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:26 /hbase/WALs/hdpdev11.cm6.tbsite.net,60020,1406526862802-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:43 /hbase/WALs/hdpdev11.cm6.tbsite.net,60020,1406716992986-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-08-02 19:14 /hbase/WALs/hdpdev11.cm6.tbsite.net,60020,1406778815552
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:15 /hbase/WALs/hdpdev12.cm6.tbsite.net,60020,1406526862752-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:43 /hbase/WALs/hdpdev12.cm6.tbsite.net,60020,1406716992874-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-08-02 19:20 /hbase/WALs/hdpdev12.cm6.tbsite.net,60020,1406778816074
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:27 /hbase/WALs/hdpdev13.cm6.tbsite.net,60020,1406526862832-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:43 /hbase/WALs/hdpdev13.cm6.tbsite.net,60020,1406716992753-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-08-02 19:15 /hbase/WALs/hdpdev13.cm6.tbsite.net,60020,1406857929773
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:26 /hbase/WALs/hdpdev14.cm6.tbsite.net,60020,1406526862736-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:43 /hbase/WALs/hdpdev14.cm6.tbsite.net,60020,1406716992923-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-08-02 19:25 /hbase/WALs/hdpdev14.cm6.tbsite.net,60020,1406778815595
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:19 /hbase/WALs/hdpdev15.cm6.tbsite.net,60020,1406526862821-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:43 /hbase/WALs/hdpdev15.cm6.tbsite.net,60020,1406716993082-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-08-02 19:15 /hbase/WALs/hdpdev15.cm6.tbsite.net,60020,1406778815578
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:21 /hbase/WALs/hdpdev16.cm6.tbsite.net,60020,1406526862816-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:43 /hbase/WALs/hdpdev16.cm6.tbsite.net,60020,1406716992787-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-08-02 19:15 /hbase/WALs/hdpdev16.cm6.tbsite.net,60020,1406778816006
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:43 /hbase/WALs/hdpdev17.cm6.tbsite.net,60020,1406716992814-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-08-02 19:26 /hbase/WALs/hdpdev17.cm6.tbsite.net,60020,1406778815579
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:43 /hbase/WALs/hdpdev18.cm6.tbsite.net,60020,1406716993051-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-08-02 19:24 /hbase/WALs/hdpdev18.cm6.tbsite.net,60020,1406778815587
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:23 /hbase/WALs/hdpdev19.cm6.tbsite.net,60020,1406526862720-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:43 /hbase/WALs/hdpdev19.cm6.tbsite.net,60020,1406716992736-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-08-02 19:22 /hbase/WALs/hdpdev19.cm6.tbsite.net,60020,1406865567732
      drwxr-xr-x - hadoop hadoop 0 2014-08-02 19:21 /hbase/WALs/hdpdev2.cm6.tbsite.net,60020,1406778815846
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:27 /hbase/WALs/hdpdev20.cm6.tbsite.net,60020,1406714346484-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:43 /hbase/WALs/hdpdev20.cm6.tbsite.net,60020,1406716991741-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-08-02 19:27 /hbase/WALs/hdpdev20.cm6.tbsite.net,60020,1406778815555
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:25 /hbase/WALs/hdpdev3.cm6.tbsite.net,60020,1406714830504-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:43 /hbase/WALs/hdpdev3.cm6.tbsite.net,60020,1406716992137-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-08-02 19:24 /hbase/WALs/hdpdev3.cm6.tbsite.net,60020,1406778815585
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:21 /hbase/WALs/hdpdev4.cm6.tbsite.net,60020,1406714829881-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:43 /hbase/WALs/hdpdev4.cm6.tbsite.net,60020,1406716992118-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-08-02 19:13 /hbase/WALs/hdpdev4.cm6.tbsite.net,60020,1406864942962
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:10 /hbase/WALs/hdpdev5.cm6.tbsite.net,60020,1406526862790-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:22 /hbase/WALs/hdpdev5.cm6.tbsite.net,60020,1406715762598-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:43 /hbase/WALs/hdpdev5.cm6.tbsite.net,60020,1406716991309-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-07-31 15:39 /hbase/WALs/hdpdev5.cm6.tbsite.net,60020,1406778815529-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-08-02 19:13 /hbase/WALs/hdpdev5.cm6.tbsite.net,60020,1406941782379
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:43 /hbase/WALs/hdpdev6.cm6.tbsite.net,60020,1406716992903-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-08-02 19:19 /hbase/WALs/hdpdev6.cm6.tbsite.net,60020,1406778815530
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:24 /hbase/WALs/hdpdev7.cm6.tbsite.net,60020,1406526862796-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:43 /hbase/WALs/hdpdev7.cm6.tbsite.net,60020,1406716993002-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-08-02 19:24 /hbase/WALs/hdpdev7.cm6.tbsite.net,60020,1406778815785
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:43 /hbase/WALs/hdpdev8.cm6.tbsite.net,60020,1406716991377-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-08-02 19:21 /hbase/WALs/hdpdev8.cm6.tbsite.net,60020,1406778815557
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:28 /hbase/WALs/hdpdev9.cm6.tbsite.net,60020,1406716099285-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-07-30 18:43 /hbase/WALs/hdpdev9.cm6.tbsite.net,60020,1406716991336-splitting
      drwxr-xr-x - hadoop hadoop 0 2014-08-02 19:10 /hbase/WALs/hdpdev9.cm6.tbsite.net,60020,1406778815554

        Attachments

        1. HBASE-11654.patch
          0.7 kB
          Victor Xu

          Activity

            People

            • Assignee:
              victorunique Victor Xu
              Reporter:
              victorunique Victor Xu
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: