Hive
  1. Hive
  2. HIVE-3934

Put tag in value for join with map reduce

    Details

      Description

      While trying to facilitate hash-based map reduce, I found that for join with map reduce in hive, the tag is appended to the key writable. This is quite a hinder for facilitating other runtime map reduce implementation of map reduce computation model such as hash-based map reduce. For example, whent the tag was in the key, there are some special things must be cared, such as,

      1. HiveKey must handles specially for the hash code for properly partition the keys between the reduce.
      2. The key in map reduce's view is actually key + tag and which makes map reduce sort a compulsory to satisfy the need of hive to group the key in reduce side. This disables or hinders hash-based map reduce because group by key + tag will make no sense to hive.
      3. ExecReducer must check the real key boundary by stripping out the tag for startGroup and endGroup calls to the operator. While without the tag, each reduce call is a natural key boundary.

      Considering append the tag as the last byte to the value writable which can avoid all the above things and fit naturually to map reduce computation model.

      I see the code in JoinOperator which will generate join results ealier which assumes on the fact that the tag is sorted. This only useful when there are many many rows with the same key in both join tables which is not a compulsory for most cases.

      Let's disucss the possibiblity of "tag in value" approach.

        Activity

        There are no comments yet on this issue.

          People

          • Assignee:
            Unassigned
            Reporter:
            Jerry Chen
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Time Tracking

              Estimated:
              Original Estimate - 168h
              168h
              Remaining:
              Remaining Estimate - 168h
              168h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development