hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 19:50:22,094 DEBUG [M_LOG_REPLAY_OPS-e1120:36020-9] master.MasterFileSystem: Renamed region directory: hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 19:50:22,096 INFO [M_LOG_REPLAY_OPS-e1120:36020-9] master.SplitLogManager: started splitting 17 logs in [hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting] hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 19:50:22,271 INFO [RS_LOG_REPLAY_OPS-e1120:36020-1] wal.HLogSplitter: Splitting hlog: hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838086718, length=179756004 hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 19:50:22,288 INFO [RS_LOG_REPLAY_OPS-e1120:36020-1] util.FSHDFSUtils: Recovering lease on dfs file hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838086718 hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 19:50:22,288 INFO [RS_LOG_REPLAY_OPS-e1120:36020-1] util.FSHDFSUtils: recoverLease=true, attempt=0 on file=hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838086718 after 0ms hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 19:50:24,243 INFO [main-EventThread] wal.HLogSplitter: Archived processed log hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838824207 to hdfs://e1120:35802/hbase/oldWALs/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838824207 hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 19:50:24,543 INFO [main-EventThread] wal.HLogSplitter: Archived processed log hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838907152 to hdfs://e1120:35802/hbase/oldWALs/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838907152 hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 19:50:26,975 INFO [main-EventThread] wal.HLogSplitter: Archived processed log hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838364443 to hdfs://e1120:35802/hbase/oldWALs/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838364443 hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 19:50:26,984 INFO [main-EventThread] wal.HLogSplitter: Archived processed log hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838997349 to hdfs://e1120:35802/hbase/oldWALs/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838997349 hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 19:50:27,392 INFO [main-EventThread] wal.HLogSplitter: Archived processed log hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396839016640 to hdfs://e1120:35802/hbase/oldWALs/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396839016640 hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 19:50:28,419 INFO [RS_LOG_REPLAY_OPS-e1120:36020-1] wal.HLogSplitter: Processed 36 edits across 5 regions; log file=hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838086718 is corrupted = false progress failed = false hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 19:50:28,441 INFO [main-EventThread] wal.HLogSplitter: Archived processed log hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838086718 to hdfs://e1120:35802/hbase/oldWALs/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838086718 hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 19:50:31,648 INFO [main-EventThread] wal.HLogSplitter: Archived processed log hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838792519 to hdfs://e1120:35802/hbase/oldWALs/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838792519 hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 19:50:33,389 INFO [main-EventThread] wal.HLogSplitter: Archived processed log hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838726151 to hdfs://e1120:35802/hbase/oldWALs/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838726151 hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 19:50:33,489 INFO [main-EventThread] wal.HLogSplitter: Archived processed log hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838095497 to hdfs://e1120:35802/hbase/oldWALs/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838095497 hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 19:50:36,921 INFO [main-EventThread] wal.HLogSplitter: Archived processed log hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838571138 to hdfs://e1120:35802/hbase/oldWALs/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838571138 hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 19:50:38,512 INFO [main-EventThread] wal.HLogSplitter: Archived processed log hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396839009656 to hdfs://e1120:35802/hbase/oldWALs/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396839009656 hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 19:50:38,820 INFO [main-EventThread] wal.HLogSplitter: Archived processed log hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838788499 to hdfs://e1120:35802/hbase/oldWALs/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838788499 hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 19:50:39,778 INFO [main-EventThread] wal.HLogSplitter: Archived processed log hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838055602 to hdfs://e1120:35802/hbase/oldWALs/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838055602 hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 19:50:49,357 INFO [main-EventThread] wal.HLogSplitter: Archived processed log hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838119462 to hdfs://e1120:35802/hbase/oldWALs/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838119462 hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 19:52:23,358 INFO [RS_LOG_REPLAY_OPS-e1120:36020-0] wal.HLogSplitter: Splitting hlog: hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838130980, length=133872798 hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 19:52:23,392 INFO [RS_LOG_REPLAY_OPS-e1120:36020-0] util.FSHDFSUtils: Recovering lease on dfs file hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838130980 hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 19:52:23,393 INFO [RS_LOG_REPLAY_OPS-e1120:36020-0] util.FSHDFSUtils: recoverLease=true, attempt=0 on file=hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838130980 after 1ms hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 19:52:56,726 INFO [ActiveMasterManager] master.MasterFileSystem: Log folder hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting doesn't belong to a known region server, splitting hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 19:52:57,362 INFO [ActiveMasterManager] master.SplitLogManager: hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting is empty dir, no logs to split hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 19:52:57,368 INFO [ActiveMasterManager] master.SplitLogManager: started splitting 1 logs in [hdfs://e1120:35802/hbase/WALs/e1319.halxg.cloudera.com,36020,1396830103512-splitting, hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396834215357-splitting, hdfs://e1120:35802/hbase/WALs/e1319.halxg.cloudera.com,36020,1396838060105-splitting, hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting, hdfs://e1120:35802/hbase/WALs/e1321.halxg.cloudera.com,36020,1396838082154-splitting, hdfs://e1120:35802/hbase/WALs/e1120.halxg.cloudera.com,36020,1396838229159-splitting] hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 19:53:02,531 INFO [RS_LOG_REPLAY_OPS-e1120:36020-1] wal.HLogSplitter: Splitting hlog: hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838130980, length=133872798 hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 19:53:02,534 INFO [RS_LOG_REPLAY_OPS-e1120:36020-1] util.FSHDFSUtils: Recovering lease on dfs file hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838130980 hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 19:53:02,536 INFO [RS_LOG_REPLAY_OPS-e1120:36020-1] util.FSHDFSUtils: recoverLease=true, attempt=0 on file=hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838130980 after 2ms hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 19:53:02,552 WARN [ActiveMasterManager] master.SplitLogManager: returning success without actually splitting and deleting all the log files in path hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 19:53:02,563 INFO [ActiveMasterManager] master.SplitLogManager: finished splitting (more than or equal to) 9 bytes in 1 log files in [hdfs://e1120:35802/hbase/WALs/e1319.halxg.cloudera.com,36020,1396830103512-splitting, hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396834215357-splitting, hdfs://e1120:35802/hbase/WALs/e1319.halxg.cloudera.com,36020,1396838060105-splitting, hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting, hdfs://e1120:35802/hbase/WALs/e1321.halxg.cloudera.com,36020,1396838082154-splitting, hdfs://e1120:35802/hbase/WALs/e1120.halxg.cloudera.com,36020,1396838229159-splitting] in 5195ms hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 19:53:03,575 INFO [M_LOG_REPLAY_OPS-e1120:36020-2] master.SplitLogManager: started splitting 3 logs in [hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting] hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 20:24:28,634 INFO [RS_LOG_REPLAY_OPS-e1120:36020-0] wal.HLogSplitter: Splitting hlog: hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838124343, length=133855324 hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 20:24:28,652 INFO [RS_LOG_REPLAY_OPS-e1120:36020-0] util.FSHDFSUtils: Recovering lease on dfs file hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838124343 hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 20:24:28,653 INFO [RS_LOG_REPLAY_OPS-e1120:36020-0] util.FSHDFSUtils: recoverLease=true, attempt=0 on file=hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838124343 after 1ms hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 20:39:30,583 WARN [RS_LOG_REPLAY_OPS-e1120:36020-1] regionserver.SplitLogWorker: log splitting of WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838130980 failed, returning error hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 21:05:30,113 WARN [RS_LOG_REPLAY_OPS-e1120:36020-0] regionserver.SplitLogWorker: log splitting of WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838124343 failed, returning error hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 21:28:09,619 INFO [main-EventThread] wal.HLogSplitter: Archived processed log hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838124343 to hdfs://e1120:35802/hbase/oldWALs/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838124343 hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 21:28:10,240 WARN [M_LOG_REPLAY_OPS-e1120:36020-2] master.SplitLogManager: error while splitting logs in [hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting] installed = 3 but only 1 done hbase-jxiang-master-e1120.halxg.cloudera.com.log:Caused by: java.io.IOException: error or interrupted while splitting logs in [hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting] Task = installed = 3 done = 1 error = 2 hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 21:28:10,244 INFO [M_LOG_REPLAY_OPS-e1120:36020-8] master.SplitLogManager: started splitting 2 logs in [hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting] hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 21:28:10,326 INFO [RS_LOG_REPLAY_OPS-e1120:36020-1] wal.HLogSplitter: Splitting hlog: hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838130980, length=133872798 hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 21:28:10,331 INFO [RS_LOG_REPLAY_OPS-e1120:36020-1] util.FSHDFSUtils: Recovering lease on dfs file hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838130980 hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 21:28:10,332 INFO [RS_LOG_REPLAY_OPS-e1120:36020-1] util.FSHDFSUtils: recoverLease=true, attempt=0 on file=hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838130980 after 1ms hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 21:28:12,172 INFO [RS_LOG_REPLAY_OPS-e1120:36020-1] wal.HLogSplitter: Processed 16 edits across 1 regions; log file=hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838130980 is corrupted = false progress failed = false hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 21:28:12,193 INFO [main-EventThread] wal.HLogSplitter: Archived processed log hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838130980 to hdfs://e1120:35802/hbase/oldWALs/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838130980 hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 21:28:12,876 INFO [main-EventThread] wal.HLogSplitter: Archived processed log hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838110950 to hdfs://e1120:35802/hbase/oldWALs/e1320.halxg.cloudera.com%2C36020%2C1396838053131.1396838110950 hbase-jxiang-master-e1120.halxg.cloudera.com.log:2014-04-06 21:28:12,918 INFO [M_LOG_REPLAY_OPS-e1120:36020-8] master.SplitLogManager: finished splitting (more than or equal to) 290293794 bytes in 2 log files in [hdfs://e1120:35802/hbase/WALs/e1320.halxg.cloudera.com,36020,1396838053131-splitting] in 2674ms