Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1442

StackOverflowError when JobHistory parses a really long line

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: 0.20.1
    • Fix Version/s: 0.20.3
    • Component/s: jobtracker
    • Labels:
      None

      Description

      JobHistory.parseLine() fails with StackOverflowError on a really big COUNTER value, triggered via the web interface. See attached file.

      1. overflow.history
        67 kB
        bc Wong
      2. mr-1442-y20s-v1.patch
        2 kB
        Luke Lu

        Activity

        Hide
        Luke Lu added a comment -

        This problem is fixed by MAPREDUCE-157 in trunk. The old (.20 and older) regex parser is too brittle to handle large values.

        Show
        Luke Lu added a comment - This problem is fixed by MAPREDUCE-157 in trunk. The old (.20 and older) regex parser is too brittle to handle large values.
        Hide
        Luke Lu added a comment -

        Patch for y20s branch (probably works with generic 0.20.x as well): one line change to add possessive qualifiers to mutually exclusive patterns to avoid useless backtracking and hence stack overflow for values with many escaped characters.

        Show
        Luke Lu added a comment - Patch for y20s branch (probably works with generic 0.20.x as well): one line change to add possessive qualifiers to mutually exclusive patterns to avoid useless backtracking and hence stack overflow for values with many escaped characters.
        Hide
        Amareshwari Sriramadasu added a comment -

        Luke, Can you add the test-case for trunk, just to be sure that it works on trunk ?

        Show
        Amareshwari Sriramadasu added a comment - Luke, Can you add the test-case for trunk, just to be sure that it works on trunk ?
        Hide
        Luke Lu added a comment -

        Amarsri, my test case for the old parser exploits details of the old file format and weaknesses of Java's backtracking NFA regex implementation. The new implementation in trunk uses the standard json format and a mature json parser (jackson) with about 700 tests. It'll be counterproductive for me to add any tests to have any material impact to the test coverage of the new parser.

        Show
        Luke Lu added a comment - Amarsri, my test case for the old parser exploits details of the old file format and weaknesses of Java's backtracking NFA regex implementation. The new implementation in trunk uses the standard json format and a mature json parser (jackson) with about 700 tests. It'll be counterproductive for me to add any tests to have any material impact to the test coverage of the new parser.
        Hide
        Dick King added a comment -

        I reviewed Luke's change, and it looks correct to me.

        I agree with Luke that trunk does not have this problem and does not need this patch or any revision of this patch.

        -dk

        Show
        Dick King added a comment - I reviewed Luke's change, and it looks correct to me. I agree with Luke that trunk does not have this problem and does not need this patch or any revision of this patch. -dk
        Hide
        Luke Lu added a comment -

        Committed in y20s branch as well.

        Show
        Luke Lu added a comment - Committed in y20s branch as well.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        This isn't committed on any of the branches on apache svn. Reopening this.

        Show
        Vinod Kumar Vavilapalli added a comment - This isn't committed on any of the branches on apache svn. Reopening this.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        This is valid only for 20.3 as it is fixed on branches 0.21 and above.

        Please close this as WON'T FIX if there is no intention to provide a patch for 0.20 branch.

        Show
        Vinod Kumar Vavilapalli added a comment - This is valid only for 20.3 as it is fixed on branches 0.21 and above. Please close this as WON'T FIX if there is no intention to provide a patch for 0.20 branch.

          People

          • Assignee:
            Luke Lu
            Reporter:
            bc Wong
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development