Pig
  1. Pig
  2. PIG-1219

Extra listStatus call to the namenode in WeightedRangePartitioner

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: 0.6.0
    • Fix Version/s: 0.6.0
    • Component/s: impl
    • Labels:
      None

      Description

      We call FileLocalizer.openDFSFile in WeightedRangePartitioner to open quantile file. openDFSFile internally will check the existence of the quantile file, which adds burden to hdfs namenode. We shall remove this extra check.

      1. PIG-1219-1.patch
        5 kB
        Daniel Dai
      2. PIG-1219-2.patch
        5 kB
        Daniel Dai
      3. PIG-1219-3.patch
        6 kB
        Daniel Dai

        Activity

        Daniel Dai made changes -
        Resolution Won't Fix [ 2 ]
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Daniel Dai made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Daniel Dai made changes -
        Attachment PIG-1219-3.patch [ 12434850 ]
        Daniel Dai made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Daniel Dai made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Daniel Dai made changes -
        Attachment PIG-1219-2.patch [ 12434747 ]
        Daniel Dai made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Daniel Dai made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Daniel Dai made changes -
        Attachment PIG-1219-1.patch [ 12434733 ]
        Daniel Dai made changes -
        Field Original Value New Value
        Summary Extra call to the namenode in WeightedRangePartitioner Extra listStatus call to the namenode in WeightedRangePartitioner
        Daniel Dai created issue -

          People

          • Assignee:
            Daniel Dai
            Reporter:
            Daniel Dai
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development