Without filter (WAL files proccesed and removed) ================================================================================================================= 1. [hadoop@hnode2 ~]$ hadoop fs -ls -R /hbase/WALs drwxr-xr-x - hbase supergroup 0 2015-11-03 10:14 /hbase/WALs/hnode2,16000,1446541683349-splitting -rw-r--r-- 3 hbase supergroup 83 2015-11-03 10:13 /hbase/WALs/hnode2,16000,1446541683349-splitting/hnode2%2C16000%2C1446541683349.meta.1446541987763.meta hnode2252C16000252C1446541683349.meta.1446541987763.meta 2. 2015-11-03 10:18:41,832 INFO [hnode1:16000.activeMasterManager] master.MasterFileSystem: Log folder hdfs://P3cluster/hbase/WALs/hnode2,16000,1446541683349-splitting doesn't belong to a known region server, splitting 3. 2015-11-03 10:18:43,350 DEBUG [hnode1:16000.activeMasterManager] procedure2.ProcedureExecutor: Procedure ServerCrashProcedure serverName=hnode2,16000,1446541683349, shouldSplitWal=true, carryingMeta=false id=855 state=RUNNABLE:SERVER_CRASH_START added to the store. 2015-11-03 10:18:43,353 INFO [ProcedureExecutor-0] procedure.ServerCrashProcedure: Start processing crashed hnode2,16000,1446541683349 2015-11-03 10:18:43,491 DEBUG [hnode1:16000.activeMasterManager] procedure2.ProcedureExecutor: Procedure ServerCrashProcedure serverName=hnode1,16000,1446541629249, shouldSplitWal=true, carryingMeta=false id=856 state=RUNNABLE:SERVER_CRASH_START added to the store. 2015-11-03 10:18:43,618 DEBUG [ProcedureExecutor-0] procedure.ServerCrashProcedure: Splitting logs from hnode2,16000,1446541683349; region count=0 2015-11-03 10:18:43,622 INFO [ProcedureExecutor-0] master.SplitLogManager: dead splitlog workers [hnode2,16000,1446541683349] 2015-11-03 10:18:43,635 INFO [ProcedureExecutor-0] master.SplitLogManager: Started splitting 1 logs in [hdfs://P3cluster/hbase/WALs/hnode2,16000,1446541683349-splitting] for [hnode2,16000,1446541683349] 2015-11-03 10:18:43,715 INFO [SplitLogWorker-hnode1:16000] wal.DefaultWALProvider: ***Returning TRUE for: WALs%2Fhnode2%2C16000%2C1446541683349-splitting%2Fhnode2%252C16000%252C1446541683349.meta.1446541987763.meta 2015-11-03 10:18:43,747 DEBUG [main-EventThread] coordination.SplitLogManagerCoordination: put up splitlog task at znode /hbase/splitWAL/WALs%2Fhnode2%2C16000%2C1446541683349-splitting%2Fhnode2%252C16000%252C1446541683349.meta.1446541987763.meta 2015-11-03 10:18:43,768 INFO [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase/splitWAL/WALs%2Fhnode2%2C16000%2C1446541683349-splitting%2Fhnode2%252C16000%252C1446541683349.meta.1446541987763.meta acquired by hnode5,16020,1446455348779 2015-11-03 10:18:43,785 INFO [SplitLogWorker-hnode1:16000] wal.DefaultWALProvider: ***Returning TRUE for: WALs%2Fhnode2%2C16000%2C1446541683349-splitting%2Fhnode2%252C16000%252C1446541683349.meta.1446541987763.meta 2015-11-03 10:18:43,876 INFO [hnode1,16000,1446542195515_splitLogManager__ChoreService_1] master.SplitLogManager: total tasks = 2 unassigned = 0 tasks={/hbase/splitWAL/WALs%2Fhnode1%2C16000%2C1446541629249-splitting%2Fhnode1%252C16000%252C1446541629249.meta.1446541695580.meta=last_update = 1446542323792 last_version = 2 cur_worker_name = hnode4,16020,1446455348794 status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0, /hbase/splitWAL/WALs%2Fhnode2%2C16000%2C1446541683349-splitting%2Fhnode2%252C16000%252C1446541683349.meta.1446541987763.meta=last_update = 1446542323768 last_version = 2 cur_worker_name = hnode5,16020,1446455348779 status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0} 2015-11-03 10:18:47,823 INFO [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase/splitWAL/WALs%2Fhnode2%2C16000%2C1446541683349-splitting%2Fhnode2%252C16000%252C1446541683349.meta.1446541987763.meta entered state: DONE hnode5,16020,1446455348779 2015-11-03 10:18:47,842 INFO [main-EventThread] wal.WALSplitter: Archived processed log hdfs://P3cluster/hbase/WALs/hnode2,16000,1446541683349-splitting/hnode2%2C16000%2C1446541683349.meta.1446541987763.meta to hdfs://P3cluster/hbase/oldWALs/hnode2%2C16000%2C1446541683349.meta.1446541987763.meta 2015-11-03 10:18:47,845 INFO [main-EventThread] coordination.SplitLogManagerCoordination: Done splitting /hbase/splitWAL/WALs%2Fhnode2%2C16000%2C1446541683349-splitting%2Fhnode2%252C16000%252C1446541683349.meta.1446541987763.meta 2015-11-03 10:18:47,849 INFO [ProcedureExecutor-0] master.SplitLogManager: finished splitting (more than or equal to) 83 bytes in 1 log files in [hdfs://P3cluster/hbase/WALs/hnode2,16000,1446541683349-splitting] in 4214ms **With filter (files not removed and not processed) ========================================================================================================== 2015-11-03 10:14:32,059 INFO [hnode2:16000.activeMasterManager] master.MasterFileSystem: Log folder hdfs://P3cluster/hbase/WALs/hnode2,16000,1446541683349 doesn't belong to a known region server, splitting 2015-11-03 10:14:33,686 DEBUG [hnode2:16000.activeMasterManager] procedure2.ProcedureExecutor: Procedure ServerCrashProcedure serverName=hnode2,16000,1446541683349, shouldSplitWal=true, carryingMeta=false id=852 state=RUNNABLE:SERVER_CRASH_START added to the store. 2015-11-03 10:14:33,921 DEBUG [ProcedureExecutor-1] procedure.ServerCrashProcedure: Splitting logs from hnode2,16000,1446541683349; region count=0 2015-11-03 10:14:33,944 DEBUG [ProcedureExecutor-1] master.MasterFileSystem: Renamed region directory: hdfs://P3cluster/hbase/WALs/hnode2,16000,1446541683349-splitting 2015-11-03 10:14:33,944 INFO [ProcedureExecutor-1] master.SplitLogManager: dead splitlog workers [hnode2,16000,1446541683349] 2015-11-03 10:14:33,968 INFO [ProcedureExecutor-1] wal.DefaultWALProvider: ***Returning FALSE for: hnode2%2C16000%2C1446541683349.1446541986629 2015-11-03 10:14:33,969 INFO [ProcedureExecutor-1] wal.DefaultWALProvider: ***Returning TRUE for: hnode2%2C16000%2C1446541683349.meta.1446541987763.meta 2015-11-03 10:14:33,969 INFO [ProcedureExecutor-1] master.SplitLogManager: Started splitting 1 logs in [hdfs://P3cluster/hbase/WALs/hnode2,16000,1446541683349-splitting] for [hnode2,16000,1446541683349] 2015-11-03 10:14:34,065 INFO [SplitLogWorker-hnode2:16000] wal.DefaultWALProvider: ***Returning FALSE for: WALs%2Fhnode2%2C16000%2C1446541683349-splitting%2Fhnode2%252C16000%252C1446541683349.1446541986629 2015-11-03 10:14:34,080 DEBUG [main-EventThread] coordination.SplitLogManagerCoordination: put up splitlog task at znode /hbase/splitWAL/WALs%2Fhnode2%2C16000%2C1446541683349-splitting%2Fhnode2%252C16000%252C1446541683349.1446541986629 2015-11-03 10:14:34,123 INFO [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase/splitWAL/WALs%2Fhnode2%2C16000%2C1446541683349-splitting%2Fhnode2%252C16000%252C1446541683349.1446541986629 acquired by hnode5,16020,1446455348779 2015-11-03 10:14:34,135 INFO [main-EventThread] coordination.ZkSplitLogWorkerCoordination: task /hbase/splitWAL/WALs%2Fhnode2%2C16000%2C1446541683349-splitting%2Fhnode2%252C16000%252C1446541683349.1446541986629 preempted from hnode2,16000,1446542058824, current task state and owner=OWNED hnode5,16020,1446455348779 2015-11-03 10:14:34,963 INFO [hnode2,16000,1446542058824_splitLogManager__ChoreService_1] master.SplitLogManager: total tasks = 1 unassigned = 0 tasks={/hbase/splitWAL/WALs%2Fhnode2%2C16000%2C1446541683349-splitting%2Fhnode2%252C16000%252C1446541683349.1446541986629=last_update = 1446542074134 last_version = 2 cur_worker_name = hnode5,16020,1446455348779 status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0} 2015-11-03 10:14:38,183 INFO [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase/splitWAL/WALs%2Fhnode2%2C16000%2C1446541683349-splitting%2Fhnode2%252C16000%252C1446541683349.1446541986629 entered state: DONE hnode5,16020,1446455348779 2015-11-03 10:14:38,199 INFO [main-EventThread] wal.WALSplitter: Archived processed log hdfs://P3cluster/hbase/WALs/hnode2,16000,1446541683349-splitting/hnode2%2C16000%2C1446541683349.1446541986629 to hdfs://P3cluster/hbase/oldWALs/hnode2%2C16000%2C1446541683349.1446541986629 2015-11-03 10:14:38,202 INFO [main-EventThread] coordination.SplitLogManagerCoordination: Done splitting /hbase/splitWAL/WALs%2Fhnode2%2C16000%2C1446541683349-splitting%2Fhnode2%252C16000%252C1446541683349.1446541986629 2015-11-03 10:14:38,209 WARN [ProcedureExecutor-1] master.SplitLogManager: Returning success without actually splitting and deleting all the log files in path hdfs://P3cluster/hbase/WALs/hnode2,16000,1446541683349-splitting: [FileStatus{path=hdfs://P3cluster/hbase/WALs/hnode2,16000,1446541683349-splitting/hnode2%2C16000%2C1446541683349.meta.1446541987763.meta; isDirectory=false; length=83; replication=3; blocksize=268435456; modification_time=1446541987767; access_time=1446541987767; owner=hbase; group=supergroup; permission=rw-r--r--; isSymlink=false}] org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.PathIsNotEmptyDirectoryException): `/hbase/WALs/hnode2,16000,1446541683349-splitting is non empty': Directory is not empty 2015-11-03 10:14:38,210 INFO [ProcedureExecutor-1] master.SplitLogManager: finished splitting (more than or equal to) 83 bytes in 1 log files in [hdfs://P3cluster/hbase/WALs/hnode2,16000,1446541683349-splitting] in 4241ms 2015-11-03 10:14:38,422 INFO [ProcedureExecutor-0] procedure.ServerCrashProcedure: Finished processing of crashed hnode2,16000,1446541683349 2015-11-03 10:14:38,526 DEBUG [ProcedureExecutor-0] procedure2.ProcedureExecutor: Procedure completed in 4.8450sec: ServerCrashProcedure serverName=hnode2,16000,1446541683349, shouldSplitWal=true, carryingMeta=false id=852 state=FINISHED