Pig
  1. Pig
  2. PIG-1219

Extra listStatus call to the namenode in WeightedRangePartitioner

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: 0.6.0
    • Fix Version/s: 0.6.0
    • Component/s: impl
    • Labels:
      None

      Description

      We call FileLocalizer.openDFSFile in WeightedRangePartitioner to open quantile file. openDFSFile internally will check the existence of the quantile file, which adds burden to hdfs namenode. We shall remove this extra check.

      1. PIG-1219-1.patch
        5 kB
        Daniel Dai
      2. PIG-1219-2.patch
        5 kB
        Daniel Dai
      3. PIG-1219-3.patch
        6 kB
        Daniel Dai

        Activity

          People

          • Assignee:
            Daniel Dai
            Reporter:
            Daniel Dai
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development