Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-2167 CUBE operation in Pig
  3. PIG-2831

MR-Cube implementation (Distributed cubing for holistic measures)

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Implementing distributed cube materialization on holistic measure based on MR-Cube approach as described in http://arnab.org/files/mrcube.pdf.
      Primary steps involved:
      1) Identify if the measure is holistic or not
      2) Determine algebraic attribute (can be detected automatically for few cases, if automatic detection fails user should hint the algebraic attribute)
      3) Modify MRPlan to insert a sampling job which executes naive cube algorithm and generates annotated cube lattice (contains large group partitioning information)
      4) Modify plan to distribute annotated cube lattice to all mappers using distributed cache
      5) Execute actual cube materialization on full dataset
      6) Modify MRPlan to insert a post process job for combining the results of actual cube materialization job
      7) OOM exception handling

        Attachments

        1. PIG-2831.9.git.patch
          287 kB
          Prasanth Jayachandran
        2. PIG-2831.8.git.patch
          287 kB
          Prasanth Jayachandran
        3. PIG-2831.7.git.patch
          282 kB
          Prasanth Jayachandran
        4. PIG-2831.6.git.patch
          272 kB
          Prasanth Jayachandran
        5. PIG-2831.5.git.patch
          272 kB
          Prasanth Jayachandran
        6. PIG-2831.4.git.patch
          242 kB
          Prasanth Jayachandran
        7. PIG-2831.3.git.patch
          136 kB
          Prasanth Jayachandran
        8. PIG-2831.2.git.patch
          136 kB
          Prasanth Jayachandran
        9. PIG-2831.1.git.patch
          132 kB
          Prasanth Jayachandran

          Activity

            People

            • Assignee:
              prasanth_j Prasanth Jayachandran
              Reporter:
              prasanth_j Prasanth Jayachandran
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: