Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-9100

Add tests for GpuResourceAllocator and do minor code cleanup

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3.0, 3.2.1, 3.1.3
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Add tests for GpuResourceAllocator and do minor code cleanup

      • Improved log and exception messages
      • Added some new debug logs
      • Some methods are named like *Copy, these are returning copies of internal data structures. The word "copy" is just a noise in their name, so they have been renamed. Additionally, the copied data structures modified to be immutable.
      • The waiting loop in method assignGpus were decoupled into a new class, RetryCommand.

      Some more words about the new class RetryCommand:
      There are some similar waiting loops in the code in: AMRMClient, AMRMClientAsync and even in GenericTestUtils (see waitFor method). RetryCommand could be a future replacement of these duplicated code, as it gives a solution to this waiting loop problem in a generic way.
      The only downside of the usage of RetryCommand in GpuResourceAllocator (startGpuAssignmentLoop) is the ugly exception handling part, but that's solely because how Java deals with checked exceptions vs. lambdas. If there's a cleaner way to solve the exception handling, I'm open for any suggestions.

        Attachments

        1. YARN-9100.001.patch
          43 kB
          Szilard Nemeth
        2. YARN-9100.002.patch
          43 kB
          Szilard Nemeth
        3. YARN-9100.003.patch
          43 kB
          Szilard Nemeth
        4. YARN-9100.branch-3.1.001.patch
          33 kB
          Szilard Nemeth
        5. YARN-9100.branch-3.1.002.patch
          33 kB
          Peter Bacsko
        6. YARN-9100.branch-3.2.001.patch
          33 kB
          Szilard Nemeth
        7. YARN-9100.branch-3.2.002.patch
          33 kB
          Peter Bacsko
        8. YARN-9100-004.patch
          33 kB
          Peter Bacsko
        9. YARN-9100-005.patch
          33 kB
          Peter Bacsko
        10. YARN-9100-006.patch
          33 kB
          Peter Bacsko
        11. YARN-9100-007.patch
          33 kB
          Szilard Nemeth
        12. YARN-9100-008.patch
          33 kB
          Szilard Nemeth
        13. YARN-9100-009.patch
          33 kB
          Peter Bacsko

          Activity

            People

            • Assignee:
              pbacsko Peter Bacsko
              Reporter:
              snemeth Szilard Nemeth
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: