Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1482

Better handling of task diagnostic information stored in the TaskInProgress

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: jobtracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Limit the size of diagnostics-string and state-string shipped as part of task status. This will help keep the JobTracker's memory usage under control. Diagnostic string and state string are capped to 1024 chars.

      Description

      Task diagnostic information can be very large at times eating up Jobtracker's memory. There should be some way to avoid storing large error strings in JobTracker.

      1. mapreduce-1482-v1.7.patch
        10 kB
        Amar Kamat
      2. mapreduce-1482-v1.3.patch
        10 kB
        Amar Kamat
      3. mapreduce-1482-v1.2.patch
        9 kB
        Amar Kamat
      4. mapreduce-1482-v1.1.patch
        9 kB
        Amar Kamat

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #263 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/263/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #263 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/263/ )
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #278 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/278/)
        . Truncate state string and diagnostic information in TaskStatus. Contributed by Amar Kamat

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #278 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/278/ ) . Truncate state string and diagnostic information in TaskStatus. Contributed by Amar Kamat
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I have committed this. Thanks, Amar!

        Show
        Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Amar!
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12438124/mapreduce-1482-v1.7.patch
        against trunk revision 922047.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/519/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/519/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/519/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/519/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12438124/mapreduce-1482-v1.7.patch against trunk revision 922047. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/519/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/519/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/519/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/519/console This message is automatically generated.
        Hide
        Amareshwari Sriramadasu added a comment -

        Changes look fine to me.

        Show
        Amareshwari Sriramadasu added a comment - Changes look fine to me.
        Hide
        Amar Kamat added a comment -

        Attaching a new patch for review. In this patch, the state-strings and diag-infos are logged only when they overflow and trimming is involved. Note that at the jobtracker, these strings will always be trimmed before sending and hence no info will be logged in heartbeats. test-patch and ant tests passed on my box.

        Show
        Amar Kamat added a comment - Attaching a new patch for review. In this patch, the state-strings and diag-infos are logged only when they overflow and trimming is involved. Note that at the jobtracker, these strings will always be trimmed before sending and hence no info will be logged in heartbeats. test-patch and ant tests passed on my box.
        Hide
        Amar Kamat added a comment -

        Nicholas,
        I am canceling the patch as the latest patch unnecessarily logs the task state-string and diagnostic information which might affect the heartbeat. I will upload a new patch that will log only if the state-string/diag-info reachs the max limit.

        Show
        Amar Kamat added a comment - Nicholas, I am canceling the patch as the latest patch unnecessarily logs the task state-string and diagnostic information which might affect the heartbeat. I will upload a new patch that will log only if the state-string/diag-info reachs the max limit.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Hudson seems back. Submitting ...

        Show
        Tsz Wo Nicholas Sze added a comment - Hudson seems back. Submitting ...
        Hide
        Amar Kamat added a comment -

        There is one nit with the latest patch :

        • logging the newly set diagnostic-information or state-string would essentially mean logging in heartbeat, which is be bad.
        Show
        Amar Kamat added a comment - There is one nit with the latest patch : logging the newly set diagnostic-information or state-string would essentially mean logging in heartbeat, which is be bad.
        Hide
        Amar Kamat added a comment -

        Attaching a modified patch that guards against null values being passed. test-patch passed. All ant tests except TestMiniMRLocalFS (MAPREDUCE-1520) passed on my box.

        Show
        Amar Kamat added a comment - Attaching a modified patch that guards against null values being passed. test-patch passed. All ant tests except TestMiniMRLocalFS ( MAPREDUCE-1520 ) passed on my box.
        Hide
        Amar Kamat added a comment -

        test-patch passed with this patch. Result :
        [exec] +1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include 3 new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

        Running ant-tests.

        Show
        Amar Kamat added a comment - test-patch passed with this patch. Result : [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. Running ant-tests.
        Hide
        Amar Kamat added a comment -

        Attaching a new patch incorporating Amareshwari's comment to do with logging of diagnostic-info and state-string. Prior to this patch, state-string info was already logged in the task-tracker and diag-info was logged at the jobtracker. With this patch, the actual state-string info and diag-info will be logged in task logs while the truncated strings will be logged in task-tracker and jobtracker.

        Show
        Amar Kamat added a comment - Attaching a new patch incorporating Amareshwari's comment to do with logging of diagnostic-info and state-string. Prior to this patch, state-string info was already logged in the task-tracker and diag-info was logged at the jobtracker. With this patch, the actual state-string info and diag-info will be logged in task logs while the truncated strings will be logged in task-tracker and jobtracker.
        Hide
        Amareshwari Sriramadasu added a comment -

        Shall we add a log saying "Received diagnostic info: " +info in setDiagnosticInfo? This would be at least written to log file before truncation.

        Show
        Amareshwari Sriramadasu added a comment - Shall we add a log saying "Received diagnostic info: " +info in setDiagnosticInfo? This would be at least written to log file before truncation.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1 patch looks good.

        Show
        Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
        Hide
        Amar Kamat added a comment -

        Attaching a patch that caps the task-diagnostic-info and state-string to 1024 chars. test-patch passed. All ant tests except the following have passed :

        • TestMiniMRLocalFS
        • TestSubmitJob
        • TestTTResourceReporting

        All these failed on trunk too.

        Show
        Amar Kamat added a comment - Attaching a patch that caps the task-diagnostic-info and state-string to 1024 chars. test-patch passed. All ant tests except the following have passed : TestMiniMRLocalFS TestSubmitJob TestTTResourceReporting All these failed on trunk too.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1 sounds good to me.

        Show
        Tsz Wo Nicholas Sze added a comment - +1 sounds good to me.
        Hide
        Amar Kamat added a comment -

        Does it make sense to keep it simple for now and cap the size of task-diagnostic information and state-string in TaskStatus to 1000 chars each?

        Show
        Amar Kamat added a comment - Does it make sense to keep it simple for now and cap the size of task-diagnostic information and state-string in TaskStatus to 1000 chars each?
        Hide
        Tsz Wo Nicholas Sze added a comment -

        It seems that all RPC api with String parameters have this problem since it does not prevent the client putting a large string into the parameter value.

        Show
        Tsz Wo Nicholas Sze added a comment - It seems that all RPC api with String parameters have this problem since it does not prevent the client putting a large string into the parameter value.
        Hide
        Amar Kamat added a comment -

        Same is true for TaskStatus.stateString.

        Show
        Amar Kamat added a comment - Same is true for TaskStatus.stateString .

          People

          • Assignee:
            Amar Kamat
            Reporter:
            Amar Kamat
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development