Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-16256

HTTPFileSystem disallows users from configuring the connectionTimeout and readTimeout of HttpClient.

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      HADOOP-14383 added a Hadoop filesystem API implementation to localize resources from HTTP server for container localization.

      Currently HttpFileSystem uses the default HttpClient connectionTimeout and readTimeout, which is infinity in most JVM.

      When the HTTP server is slow, this prolongs the execution time of the container localization phase and in some cases we've observed that it runs for hours.

      It would be better to choose reasonable defaults for  connectionTimeout and readTimeout of HttpClient and allow users to configure them.

      Attachments

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-14383 Implement FileSystem that reads from HTTP / HTTPS endpoints

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #774

          Activity

            People

              Unassigned Unassigned
              spvenkat Shanthoosh Venkataraman
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: