Uploaded image for project: 'REEF (Retired)'
  1. REEF (Retired)
  2. REEF-345 Complete implementation for YARN AM HA
  3. REEF-483

Add configuration in allowing other ways to preserve running evaluators aside from DFS

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.12
    • REEF Driver, REEF.NET Driver
    • None

    Description

      Currently the way to preserve running evaluators in the driver is via a file in the DFS. We should add a configuration to allow other ways to preserve running evaluators other than using the DFS.

      Merged from REEF-85:
      During REEF-80, we discovered that the Driver today keeps a log of all Evaluator (de-)allocations on HDFS for use during a Driver restart situation. This is happening even for Drivers that don't make use of the Driver restart feature. We should refactor this code to be modular, not to impose this cost on all Drivers.

      Attachments

        Issue Links

          is blocked by

          Sub-task - The sub-task of the issue REEF-514 Move Driver restart configuration options to a separate configuration

          • Major - Major loss of function.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue REEF-85 Refactor the Driver restart code to be modular

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          links to

          Web Link Link to PR

          Activity

            People

              afchung90 Andrew Chung
              afchung90 Andrew Chung
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: