Uploaded image for project: 'Hama'
  1. Hama
  2. HAMA-949

File splits based on number of input files

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 0.6.4
    • Fix Version/s: 0.7.0
    • Component/s: None
    • Labels:
      None

      Description

      I've create multiple input files considering max task capacity of cluster, but it wasn't able to run. Because, currently file splits are determined based on number of blocks.

      I don't know why below code has been removed. What if add this again?

          // take the short circuit path if we have already partitioned
          if (numSplits == files.length) {
            for (FileStatus file : files) {
              if (file != null) {
                splits.add(new FileSplit(file.getPath(), 0, file.getLen(),
                    new String[0]));
              }
            }
            return splits.toArray(new FileSplit[splits.size()]);
          }
      

      https://www.mail-archive.com/commits@hama.apache.org/msg00319.html

        Attachments

        1. patch.txt
          3 kB
          Edward J. Yoon

          Activity

            People

            • Assignee:
              udanax Edward J. Yoon
              Reporter:
              udanax Edward J. Yoon
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: