Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15679

ShutdownHookManager shutdown time needs to be configurable & extended

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0, 3.0.0
    • Fix Version/s: 2.9.2, 2.8.5, 3.0.4, 3.1.2
    • Component/s: util
    • Labels:
      None

      Description

      HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging shutdowns. But the timeout is too short for applications where a large flush of data is needed on shutdown.

      A key example of this is Spark apps which save their history to object stores, where the file close() call triggers an upload of the final local cached block of data (could be 32+MB), and then execute the final mutipart commit.

      Proposed

      1. make the default sleep time 30s, not 10s
      2. make it configurable with a time duration property (with minimum time of 1s.?)

        Attachments

        1. HADOOP-15679-001.patch
          24 kB
          Steve Loughran
        2. HADOOP-15679-002.patch
          27 kB
          Steve Loughran
        3. HADOOP-15679-002.patch
          27 kB
          Steve Loughran
        4. HADOOP-15679-003.patch
          26 kB
          Steve Loughran
        5. HADOOP-15679-branch-2-001.patch
          26 kB
          Steve Loughran
        6. HADOOP-15679-branch-2-001.patch
          26 kB
          Steve Loughran
        7. HADOOP-15679-branch-2-003.patch
          26 kB
          Steve Loughran
        8. HADOOP-15679-branch-2-003.patch
          26 kB
          Steve Loughran
        9. HADOOP-15679-branch-2-004.patch
          26 kB
          Steve Loughran
        10. HADOOP-15679-branch-2-004.patch
          26 kB
          Steve Loughran
        11. HADOOP-15679-branch-2.8-005.patch
          27 kB
          Steve Loughran
        12. HADOOP-15679-branch-2.8-005.patch
          27 kB
          Steve Loughran

          Issue Links

            Activity

              People

              • Assignee:
                stevel@apache.org Steve Loughran
                Reporter:
                stevel@apache.org Steve Loughran
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: