Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.3
    • Fix Version/s: 2.0.2-alpha, 0.23.3
    • Component/s: None
    • Labels:
      None

      Description

      Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period.

      It might make sense for job history server to remove them.

      1. MR-4296.txt
        21 kB
        Robert Joseph Evans
      2. MR-4296.txt
        21 kB
        Robert Joseph Evans
      3. MR-4296.txt
        21 kB
        Robert Joseph Evans
      4. MR-4296.txt
        20 kB
        Robert Joseph Evans

        Activity

        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Arun C Murthy made changes -
        Fix Version/s 2.1.0-alpha [ 12322547 ]
        Gavin made changes -
        Workflow jira [ 12721732 ] no-reopen-closed, patch-avail [ 12722164 ]
        Thomas Graves made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Fix Version/s 2.1.0-alpha [ 12322547 ]
        Fix Version/s 2.2.0-alpha [ 12322548 ]
        Fix Version/s 0.23.3 [ 12322841 ]
        Resolution Fixed [ 1 ]
        Thomas Graves made changes -
        Project Hadoop Map/Reduce [ 12310941 ] Hadoop YARN [ 12313722 ]
        Key MAPREDUCE-4296 YARN-25
        Workflow no-reopen-closed, patch-avail [ 12670740 ] jira [ 12721732 ]
        Status Patch Available [ 10002 ] Open [ 1 ]
        Release Note When upgrading if log aggregation is enabled and yarn.log-aggregation.retain-seconds is set so that logs are going to be deleted, a cluster administrator should change the permissions on all user's logs to be 770.

        hadoop fs -chmod -R 770 <log-dir>/*

        Please be sure that <log-dir> itself is not modified.

        Added in new config yarn.log-aggregation.retain-seconds.
        How long to keep aggregation logs before deleting them. -1 disables. Be careful set this too small and you will spam the name node.
        Affects Version/s 0.23.3 [ 12322841 ]
        Affects Version/s 0.23.3 [ 12320060 ]
        Target Version/s 0.23.3, 2.1.0-alpha [ 12320060, 12321442 ]
        Component/s mrv2 [ 12314301 ]
        Robert Joseph Evans made changes -
        Attachment MR-4296.txt [ 12541413 ]
        Robert Joseph Evans made changes -
        Release Note When upgrading if log aggregation is enabled and yarn.log-aggregation.retain-seconds is set so that logs are going to be deleted, a cluster administrator should change the permissions on all user's logs to be 770.

        hadoop fs -chmod -R 770 <log-dir>/*

        Please be sure that <log-dir> itself is not modified.
        When upgrading if log aggregation is enabled and yarn.log-aggregation.retain-seconds is set so that logs are going to be deleted, a cluster administrator should change the permissions on all user's logs to be 770.

        hadoop fs -chmod -R 770 <log-dir>/*

        Please be sure that <log-dir> itself is not modified.

        Added in new config yarn.log-aggregation.retain-seconds.
        How long to keep aggregation logs before deleting them. -1 disables. Be careful set this too small and you will spam the name node.
        Robert Joseph Evans made changes -
        Attachment MR-4296.txt [ 12541087 ]
        Robert Joseph Evans made changes -
        Release Note When upgrading if log aggregation is enabled and yarn.log-aggregation.retain-seconds is so that logs are going to be deleted, a cluster administrator should change the permissions on all user's logs to be 770.

        hadoop fs -chmod -R 770 <log-dir>/*

        Please be sure that <log-dir> itself is not modified.
        When upgrading if log aggregation is enabled and yarn.log-aggregation.retain-seconds is set so that logs are going to be deleted, a cluster administrator should change the permissions on all user's logs to be 770.

        hadoop fs -chmod -R 770 <log-dir>/*

        Please be sure that <log-dir> itself is not modified.
        Robert Joseph Evans made changes -
        Release Note When upgrading if log aggregation is enabled and yarn.log-aggregation.retain-seconds is so that logs are going to be deleted, a cluster administrator should change the permissions on all user's logs to be 770.

        hadoop fs -chmod -R 770 <log-dir>/*

        Please be sure that <log-dir> itself is not modified.
        Target Version/s 0.23.3 [ 12320060 ] 0.23.3, 2.1.0-alpha [ 12320060, 12321442 ]
        Robert Joseph Evans made changes -
        Attachment MR-4296.txt [ 12541061 ]
        Robert Joseph Evans made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Robert Joseph Evans made changes -
        Attachment MR-4296.txt [ 12540952 ]
        Robert Joseph Evans made changes -
        Target Version/s 2.1.0-alpha [ 12321442 ] 0.23.3 [ 12320060 ]
        Robert Joseph Evans made changes -
        Field Original Value New Value
        Assignee Robert Joseph Evans [ revans2 ]
        Thomas Graves created issue -

          People

          • Assignee:
            Robert Joseph Evans
            Reporter:
            Thomas Graves
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development