Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11506

Configuration variable expansion regex expensive for long values

    XMLWordPrintableJSON

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Profiling several large Hadoop jobs, we discovered that a surprising amount of time was spent inside Configuration.get, more specifically, in regex matching caused by the substituteVars call.

        Attachments

        1. HADOOP-11506.001.patch
          4 kB
          Gera Shegalov
        2. HADOOP-11506.002.patch
          6 kB
          Gera Shegalov
        3. HADOOP-11506.003.patch
          9 kB
          Gera Shegalov
        4. HADOOP-11506.004.patch
          9 kB
          Gera Shegalov

          Issue Links

          breaks

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-13699 Configuration does not substitute multiple references to the same var

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-463 variable expansion in Configuration

          • Major - Major loss of function.
          • Closed

            Activity

              People

              • Assignee:
                jira.shegalov Gera Shegalov
                Reporter:
                dvryaboy Dmitriy V. Ryaboy
              • Votes:
                0 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: