Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-3412

HLogSplitter should handle missing HLogs

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.90.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In build #48 (https://hudson.apache.org/hudson/job/hbase-0.90/48/), TestReplication failed because of missing rows on the slave cluster. The reason is that a region server that was killed was able to archive a log at the same time the master was trying to recover it:

      [MASTER_META_SERVER_OPERATIONS-vesta.apache.org:47907-0] util.FSUtils(625):
       Recovering file hdfs://localhost:50121/user/hudson/.logs/vesta.apache.org,58598,1294117333857/vesta.apache.org%3A58598.1294117406909
      ...
      [RegionServer:0;vesta.apache.org,58598,1294117333857.logRoller] wal.HLog(740):
       moving old hlog file /user/hudson/.logs/vesta.apache.org,58598,1294117333857/vesta.apache.org%3A58598.1294117406909
       whose highest sequenceid is 422 to /user/hudson/.oldlogs/vesta.apache.org%3A58598.1294117406909
      ...
      [MASTER_META_SERVER_OPERATIONS-vesta.apache.org:47907-0] master.MasterFileSystem(204):
       Failed splitting hdfs://localhost:50121/user/hudson/.logs/vesta.apache.org,58598,1294117333857
       java.io.IOException: Failed to open hdfs://localhost:50121/user/hudson/.logs/vesta.apache.org,58598,1294117333857/vesta.apache.org%3A58598.1294117406909 for append
      Caused by: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException:
       org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException:
       No lease on /user/hudson/.logs/vesta.apache.org,58598,1294117333857/vesta.apache.org%3A58598.1294117406909
       File does not exist. [Lease.  Holder: DFSClient_-986975908, pendingcreates: 1]
      

      We should probably just handle the fact that a file could have been archived (maybe even check in .oldlogs to be sure) and move on to the next log.

        Attachments

        1. HBASE-3412.patch
          4 kB
          Jean-Daniel Cryans
        2. HBASE-3412-2.patch
          5 kB
          Jean-Daniel Cryans

          Activity

            People

            • Assignee:
              jdcryans Jean-Daniel Cryans
              Reporter:
              jdcryans Jean-Daniel Cryans
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: