Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7073

Optimize TokenCache#obtainTokensForNamenodesInternal

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.10.0, 2.8.4, 3.2.0, 3.0.3, 3.1.1, 2.9.2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      FileInputFormat#listStatus is too slow file system cache is disabled.

      TokenCache#obtainTokensForNamenodesInternal for every filesystem instance Master.getMasterPrincipal(conf) is caled which reloads YarnConfiguration .
      For fileInput with 1k file will reload YarnConfiguration 1k times.

      Master.getMasterPrincipal(conf) can be passed for obtainTokensForNamenodesInternal per filesystem call.

        Attachments

        1. MAPREDUCE-7073.002.patch
          5 kB
          Bibin A Chundatt
        2. MAPREDUCE-7073.001.patch
          2 kB
          Bibin A Chundatt

          Activity

            People

            • Assignee:
              bibinchundatt Bibin A Chundatt
              Reporter:
              bibinchundatt Bibin A Chundatt
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: