Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11506

Configuration variable expansion regex expensive for long values

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      Profiling several large Hadoop jobs, we discovered that a surprising amount of time was spent inside Configuration.get, more specifically, in regex matching caused by the substituteVars call.

      Attachments

        1. HADOOP-11506.001.patch
          4 kB
          Gera Shegalov
        2. HADOOP-11506.002.patch
          6 kB
          Gera Shegalov
        3. HADOOP-11506.003.patch
          9 kB
          Gera Shegalov
        4. HADOOP-11506.004.patch
          9 kB
          Gera Shegalov

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            jira.shegalov Gera Shegalov
            dvryaboy Dmitriy V. Ryaboy
            Votes:
            0 Vote for this issue
            Watchers:
            15 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment