Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-2167 CUBE operation in Pig
  3. PIG-2831

MR-Cube implementation (Distributed cubing for holistic measures)


    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:


      Implementing distributed cube materialization on holistic measure based on MR-Cube approach as described in http://arnab.org/files/mrcube.pdf.
      Primary steps involved:
      1) Identify if the measure is holistic or not
      2) Determine algebraic attribute (can be detected automatically for few cases, if automatic detection fails user should hint the algebraic attribute)
      3) Modify MRPlan to insert a sampling job which executes naive cube algorithm and generates annotated cube lattice (contains large group partitioning information)
      4) Modify plan to distribute annotated cube lattice to all mappers using distributed cache
      5) Execute actual cube materialization on full dataset
      6) Modify MRPlan to insert a post process job for combining the results of actual cube materialization job
      7) OOM exception handling


        1. PIG-2831.1.git.patch
          132 kB
          Prasanth Jayachandran
        2. PIG-2831.2.git.patch
          136 kB
          Prasanth Jayachandran
        3. PIG-2831.3.git.patch
          136 kB
          Prasanth Jayachandran
        4. PIG-2831.4.git.patch
          242 kB
          Prasanth Jayachandran
        5. PIG-2831.5.git.patch
          272 kB
          Prasanth Jayachandran
        6. PIG-2831.6.git.patch
          272 kB
          Prasanth Jayachandran
        7. PIG-2831.7.git.patch
          282 kB
          Prasanth Jayachandran
        8. PIG-2831.8.git.patch
          287 kB
          Prasanth Jayachandran
        9. PIG-2831.9.git.patch
          287 kB
          Prasanth Jayachandran



            • Assignee:
              prasanth_j Prasanth Jayachandran
              prasanth_j Prasanth Jayachandran
            • Votes:
              0 Vote for this issue
              4 Start watching this issue


              • Created: