HBase
  1. HBase
  2. HBASE-5712

Parallelize load of .regioninfo files in diagnostic/repair portion of hbck.

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.90.7, 0.92.2, 0.94.0, 0.95.2
    • Fix Version/s: 0.94.0, 0.95.0
    • Component/s: hbck
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      On heavily loaded hdfs's some dfs nodes may not respond quickly and backs off for 60s before attempting to read data from another datanode. Portions of the information gathered from hdfs (.regioninfo files) are loaded serially. With HBase with clusters with 100's, or 1000's, or 10000's regions encountering these 60s delay blocks progress and can be very painful.

      There is already some parallelization of portions of the hdfs information load operations and the goal here is move the reading of .regioninfos into the parallelized sections..

      1. hbase-5712-v2.patch
        10 kB
        Jonathan Hsieh
      2. hbase-5712-90-v2.patch
        9 kB
        Jonathan Hsieh
      3. hbase-5712-90.patch
        9 kB
        Jonathan Hsieh
      4. hbase-5712.patch
        10 kB
        Jonathan Hsieh

        Issue Links

          Activity

          Jonathan Hsieh created issue -
          Jonathan Hsieh made changes -
          Field Original Value New Value
          Component/s hbck [ 12315702 ]
          Jonathan Hsieh made changes -
          Assignee Jonathan Hsieh [ jmhsieh ]
          Jonathan Hsieh made changes -
          Attachment hbase-5712.patch [ 12524375 ]
          Jonathan Hsieh made changes -
          Attachment hbase-5712-90.patch [ 12524376 ]
          Jonathan Hsieh made changes -
          Attachment hbase-5712.patch [ 12524375 ]
          Jonathan Hsieh made changes -
          Attachment hbase-5712.patch [ 12524377 ]
          Jonathan Hsieh made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 0.90.7 [ 12319481 ]
          Affects Version/s 0.92.2 [ 12319888 ]
          Affects Version/s 0.94.0 [ 12316419 ]
          Affects Version/s 0.96.0 [ 12320040 ]
          Jonathan Hsieh made changes -
          Parent HBASE-5628 [ 12547917 ]
          Issue Type Sub-task [ 7 ] Improvement [ 4 ]
          Jonathan Hsieh made changes -
          Link This issue relates to HBASE-5628 [ HBASE-5628 ]
          Jonathan Hsieh made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 0.90.7 [ 12319481 ]
          Fix Version/s 0.92.2 [ 12319888 ]
          Fix Version/s 0.94.0 [ 12316419 ]
          Fix Version/s 0.96.0 [ 12320040 ]
          Resolution Fixed [ 1 ]
          Jonathan Hsieh made changes -
          Attachment hbase-5712-v2.patch [ 12525042 ]
          Attachment hbase-5712-90-v2.patch [ 12525043 ]
          Lars Hofhansl made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          stack made changes -
          Fix Version/s 0.95.0 [ 12324094 ]
          Fix Version/s 0.94.0 [ 12316419 ]
          Fix Version/s 0.90.7 [ 12319481 ]
          Fix Version/s 0.92.2 [ 12319888 ]
          Fix Version/s 0.96.0 [ 12320040 ]
          Lars Hofhansl made changes -
          Fix Version/s 0.94.0 [ 12316419 ]

            People

            • Assignee:
              Jonathan Hsieh
              Reporter:
              Jonathan Hsieh
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development