Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-11339 HBase MOB
  3. HBASE-12487

Explicitly flush the file name in sweep job

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: hbase-11339
    • Fix Version/s: hbase-11339, 2.0.0
    • Component/s: regionserver, Scanners
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently in the implementation of sweeper, we need to remove the mob files that are not referenced by hbase any more.

      1. List the names of all the existing files and write them to a seq file.
      2. In each reducer, write the visited mob file names to a seq file.
      3. After the mapreduce is done, remove the files that are existent in step1 but not in step2 (those are the unused/unreferenced files).

      Currently the flush of the writer depends on the IOUtils.closeStream(writer), if this close operation fails silently, the file names won't be written to seq files, some files that are still referenced by hbase will be archived after the mapreduce is finished.
      We should explicitly invoke write.hflush() to flush out the user buffer.

        Attachments

        1. HBASE-12487.diff
          1 kB
          Jingcheng Du

          Activity

            People

            • Assignee:
              jingcheng.du@intel.com Jingcheng Du
              Reporter:
              jingcheng.du@intel.com Jingcheng Du
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: