Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-5843

Improve HBase MTTR - Mean Time To Recover

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.95.2
    • 0.96.0
    • None
    • None

    Description

      A part of the approach is described here: https://docs.google.com/document/d/1z03xRoZrIJmg7jsWuyKYl6zNournF_7ZHzdi0qz_B4c/edit

      The ideal target is:

      • failure impact client applications only by an added delay to execute a query, whatever the failure.
      • this delay is always inferior to 1 second.

      We're not going to achieve that immediately...

      Priority will be given to the most frequent issues.
      Short term:

      • software crash
      • standard administrative tasks as stop/start of a cluster.

      Attachments

        Issue Links

          is related to

          Sub-task - The sub-task of the issue HBASE-2108 [HA] hbase cluster should be able to ride over hdfs 'safe mode' flip and namenode restart/move

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HDFS-2296 If read error while lease is being recovered, client reverts to stale view on block info

          • Critical - Crashes, loss of data, severe memory leak.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. HBASE-3809 .META. may not come back online if > number of executors servers crash and one of those > number of executors was carrying meta

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-4177 Handling read failures during recovery‏ - when HMaster calls Namenode recovery, recovery may be a failure leading to read failure while splitting logs

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-6401 HBase may lose edits after a crash if used with HDFS 1.0.3 or older

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-2958 When hbase.hlog.split.skip.errors is set to false, we fail the split but thats it

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-6140 Make distributed log splitting faster by changing call site of tmp log renaming

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-7390 Add extra test cases for assignement on the region server and fix the related issues

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-6328 FSHDFSUtils#recoverFileLease tries to rethrow InterruptedException but actually shallows it

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-7407 TestMasterFailover under tests some cases and over tests some others

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-8216 Be able to differentiate Power failures from Rack switch reboot

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-6175 TestFSUtils flaky on hdfs getFileStatus method

          • Trivial - Cosmetic problem like misspelt words or misaligned text.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-6356 printStackTrace in FSUtils

          • Trivial - Cosmetic problem like misspelt words or misaligned text.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HBASE-2315 BookKeeper for write-ahead logging

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-1075 Separately configure connect timeouts from read timeouts in data path

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-1094 Intelligent block placement policy to decrease probability of block loss

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-3706 Add the possibility to mark a node as 'low priority' for writes in the DFSClient

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. ZOOKEEPER-922 enable faster timeout of sessions in case of unexpected socket disconnect

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-4642 Allow lease recovery for multiple paths to be issued in one request

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ZOOKEEPER-1147 Add support for local sessions

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-3702 Add an option for NOT writing the blocks locally if there is a datanode on the same box as the client

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-6134 Improvement for split-worker to speed up distributed log splitting

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-1316 ZooKeeper: use native threads to avoid GC stalls (JNI integration)

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-2183 Ride over restart

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-6295 Possible performance improvement in client batch operations: presplit and send in background

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-6508 [0.89-fb] Filter out edits at log split time

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-3703 Decrease the datanode failure detection time

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-6490 'dfs.client.block.write.retries' value could be increased in HBase

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Wish - General wishlist item. ZOOKEEPER-702 GSoC 2010: Failure Detector Model

          • Major - Major loss of function.
          • Open

          Umbrella - An overarching type made of sub-tasks HBASE-1111 [performance] Crash recovery takes way too long

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HBASE-6060 Regions's in OPENING state from failed regionservers takes a long time to recover

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-8144 pseudoSortByDistance in NetworkTopology doesn't work properly if no local node and first node is local rack node

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. ZOOKEEPER-1495 ZK client hangs when using a function not available on the server.

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-5970 Improve the AssignmentManager#updateTimer and speed up handling opened event

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. HBASE-7386 Investigate providing some supervisor support for znode deletion

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed
          requires

          Sub-task - The sub-task of the issue HDFS-3912 Detecting and avoiding stale datanodes for writing

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-6737 NullPointerException at regionserver.wal.SequenceFileLogWriter.append

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-6738 Too aggressive task resubmission from the distributed log manager

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-6364 Powering down the server host holding the .META. table causes HBase Client to take excessively long to recover and connect to reassigned .META. table

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-6713 Stopping META/ROOT RS may take 50mins when some region is splitting

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-6736 Distributed Split: a split tasks can be mark as DONE but keep unassigned

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-6970 hbase-deamon.sh creates/updates pid file even when that start failed.

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-7271 Have a single executor for all zkWorkers in the assignment manager

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-7756 Strange code in ServerCallable#shouldRetry

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-7989 Client with a cache info on a dead server will wait for 20s before trying another one.

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-8204 Don't use hdfs append during lease recovery

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-5992 Generalization of region move implementation + manage draining servers in bulk assign

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-6156 Improve multiop performances in HTable#flushCommits

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-6315 ipc.HBaseClient should support address change as does hdfs

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-6878 DistributerLogSplit can fail to resubmit a task done if there is an exception during the log archiving

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-7815 Too subtle behavior for HConnection#getRegionLocation reload parameter and performance risk

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-5902 Some scripts are not executable

          • Trivial - Cosmetic problem like misspelt words or misaligned text.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HBASE-4755 HBase based block placement in DFS

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HBASE-7006 [MTTR] Improve Region Server Recovery Time - Distributed Log Replay

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HBASE-7590 Add a costless notifications mechanism from master to regionservers & clients

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HDFS-2576 Namenode should have a favored nodes hint to enable clients to have control over block placement.

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-3705 Add the possibility to mark a node as 'low priority' for read in the DFSClient

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-6870 HTable#coprocessorExec always scan the whole table

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-7213 Have HLog files for .META. and -ROOT- edits only

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-5844 Delete the region servers znode after a regions server crash

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-5924 In the client code, don't wait for all the requests to be executed before resubmitting a request in error.

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-5930 Limits the amount of time an edit can live in the memstore.

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-6309 [MTTR] Do NN operations outside of the ZK EventThread in SplitLogManager

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-6435 Reading WAL files after a recovery leads to time lost in HDFS timeouts when using dead datanodes

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-6751 Too many retries, leading a a delay to read the HLog after a datanode failure

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-6752 On region server failure, serve writes and timeranged reads during the log split

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-6772 Make the Distributed Split HDFS Location aware

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-6773 Make the dfs replication factor configurable per table

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-6774 Immediate assignment of regions that don't have entries in HLog

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-6783 Make read short circuit the default

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-7246 Assignment#nodeChildrenChanged calls listChildrenAndWatchForNewChildren, overloading master & zookeper needlessly

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-7247 Assignment performances decreased by 50% because of regionserver.OpenRegionHandler#tickleOpening

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-7327 Assignment Timeouts: Remove the code from the master

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-7334 We should expire the zk session for crashed servers rather than deleting ephemeral znodes

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-4721 Speed up lease/block recovery when DN fails and a block goes into recovery

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-5859 Optimize the rolling restart script

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-5877 When a query fails because the region has moved, let the regionserver return the new address to the client

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-5926 Delete the master znode after a master crash

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-5939 Add an autorestart option in the start scripts

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-5998 Bulk assignment: regionserver optimization by using a temporary cache for table descriptors when receveing an open regions request

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-6058 Use ZK 3.4 API 'multi' in bulk assignment

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-6109 Improve RIT performances during assignment on large clusters

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-6290 Add a function a mark a server as dead and start the recovery the process

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-4754 Add an API in the namenode to mark a datanode as stale

          • Critical - Crashes, loss of data, severe memory leak.
          • Patch Available
          supercedes

          Umbrella - An overarching type made of sub-tasks HBASE-1111 [performance] Crash recovery takes way too long

          • Major - Major loss of function.
          • Closed

          Activity

            People

              nkeywal Nicolas Liochon
              nkeywal Nicolas Liochon
              Votes:
              0 Vote for this issue
              Watchers:
              37 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: