Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-18640

Move mapreduce out of hbase-server into separate hbase-mapreduce module

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.0.0-alpha-3, 2.0.0
    • None
    • None
    • Hide
      - Moves all org.apache.hadoop.hbase.mapreduce.* (except LoadIncrementalHFiles) and org.apache.hadoop.hbase.mapred.* classes from hbase-server module to new hbase-mapreduce module.
      - Also moves following tools from hbase-server module to hbase-mapreduce module: CompactionTool, ExportSnapshot, PerformanceEvaluation, LoadTestTool
      - Very minor breakages in LoadTestTool(LimitedPrivate HBaseInterfaceAudience.TOOLS)
      Show
      - Moves all org.apache.hadoop.hbase.mapreduce.* (except LoadIncrementalHFiles) and org.apache.hadoop.hbase.mapred.* classes from hbase-server module to new hbase-mapreduce module. - Also moves following tools from hbase-server module to hbase-mapreduce module: CompactionTool, ExportSnapshot, PerformanceEvaluation, LoadTestTool - Very minor breakages in LoadTestTool(LimitedPrivate HBaseInterfaceAudience.TOOLS)

    Description

      (Couldn't find another dedicated jira, so creating new one).
      Uploaded patch which is moving ~60 files to the new module. Few notes:

      • The classes remaining in hbase-server are the ones which are intensively coupled with visibility labels/wal/filesystem/hfile. These can not be migrated to new module until corresponding subcomponents are untangled out of hbase-server into their own separate modules.
      • Almost all mapreduce tests uses HBaseTestingUtil, so they can't be moved to hbase-mapreduce module. Given these dependency constraints, one way would be having a separate module for tests:
        hbase-mapreduce <---- hbase-server <------- hbase-mapreduce-tests
        Imo, this makes sense and looks fine.
        The only issue is - yetus' pre-commit. It won't run tests in hbase-mapreduce-tests module if something changed in just hbase-mapreduce. However, yetus' limitation shouldn't warrant against the idea.
        So i'd say that we should go that way, unless there are better suggestions.

      Attachments

        1. HBASE-18640.master.001.patch
          182 kB
          Apekshit Sharma
        2. HBASE-18640.master.002.patch
          172 kB
          Apekshit Sharma
        3. HBASE-18640.master.003.patch
          239 kB
          Apekshit Sharma
        4. HBASE-18640.master.004.patch
          278 kB
          Apekshit Sharma
        5. HBASE-18640.master.003.patch
          239 kB
          Michael Stack
        6. HBASE-18640.master.004.patch
          278 kB
          Michael Stack
        7. HBASE-18640.master.005.patch
          295 kB
          Apekshit Sharma
        8. HBASE-18640.master.006.patch
          232 kB
          Apekshit Sharma
        9. HBASE-18640.master.007.patch
          237 kB
          Apekshit Sharma
        10. HBASE-18640.master.008.patch
          254 kB
          Apekshit Sharma
        11. HBASE-18640.branch-2.001.patch
          254 kB
          Apekshit Sharma

        Issue Links

          breaks

          Umbrella - An overarching type made of sub-tasks HBASE-18696 Fix the problems when introducing the new hbase-mapreduce module

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-11843 MapReduce classes shouldn't be in hbase-server

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          relates to

          Sub-task - The sub-task of the issue HBASE-14910 Create base hbase-mapred and hbase-io modules, move HTableInputFormat and HTableOutputFormat

          • Major - Major loss of function.
          • Closed
          links to

          Web Link Review Board (master)

          Activity

            People

              appy Apekshit Sharma
              appy Apekshit Sharma
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: