Uploaded image for project: 'Crunch'
  1. Crunch
  2. CRUNCH-636

Make replication factor for temporary files configurable

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.0.0
    • Component/s: None
    • Labels:
      None

      Description

      As of now, Crunch does not allow having different replication factor for temporary files and non-temporary files (e.g. final output data of leaf nodes) at the same time. If a user has a large amount of data (say hundreds a of gigabytes) to process, they might want to have lower replication factor for large temporary files between Crunch jobs.

      We could make this configurable via a new setting (e.g. crunch.tmp.dir.replication).

        Attachments

        1. CRUNCH-636.01.patch
          15 kB
          Attila Sasvari
        2. CRUNCH-636.02.patch
          15 kB
          Attila Sasvari
        3. CRUNCH-636.03.patch
          20 kB
          Attila Sasvari
        4. CRUNCH-636.04.patch
          20 kB
          Attila Sasvari
        5. CRUNCH-636.04-amendment.patch
          4 kB
          Attila Sasvari
        6. test.WordCount_2017-03-08_16.31.55.737_jobplan.dot.png
          112 kB
          Attila Sasvari
        7. test.WordCount_2017-03-08_16.31.55.737.log
          8 kB
          Attila Sasvari

          Activity

            People

            • Assignee:
              asasvari Attila Sasvari
              Reporter:
              asasvari Attila Sasvari
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: