Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7298

Distcp doesn't close the job after the job is completed

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.0
    • 3.4.0
    • distcp
    • None
    • Reviewed

    Description

      Distcp doesn't close the job after the job is completed. This leads to leaked Truststore Reloader Threads.

      The fix is to close the job once it is complete. job.close internally calls yarnClient.close(), which then calls timelineConnector.serviceStop() . This destroys the sslFactory cleaning up the ReloadingX509TrustManager.

      Without the patch for each distcp job, a new ReloadingX509TrustManager is created which creates a new thread. These threads are never killed and they remain like that till HS2 is restarted. With the close, the thread will be cleaned up once the job is completed.

      Attachments

        1. MAPREDUCE-7298.02.patch
          4 kB
          Aasha Medhi
        2. MAPREDUCE-7298.01.patch
          1 kB
          Aasha Medhi

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            aasha Aasha Medhi
            aasha Aasha Medhi
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment