Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-10975

EntityGroupFSTimelineStore#ActiveLogParser parses already processed files

    XMLWordPrintableJSON

Details

    Description

      EntityGroupFSTimelineStore#ActiveLogParser parses already processed files again and again even though there is no change in the file. This leads to unnecessary load on DFS where summary files resides and Timeline Store where timeline entities present.

      2021-10-10 19:20:43,940 INFO  timeline.LogInfo - Parsed 6 entities from hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_000001/summarylog-appattempt_1631559260564_0009_000001_2331123893 in 275 msec
      2021-10-10 19:21:44,079 INFO  timeline.LogInfo - Parsed 6 entities from hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_000001/summarylog-appattempt_1631559260564_0009_000001_2331123893 in 341 msec
      2021-10-10 19:22:44,065 INFO  timeline.LogInfo - Parsed 6 entities from hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_000001/summarylog-appattempt_1631559260564_0009_000001_2331123893 in 335 msec
      2021-10-10 19:23:44,038 INFO  timeline.LogInfo - Parsed 6 entities from hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_000001/summarylog-appattempt_1631559260564_0009_000001_2331123893 in 370 msec
      2021-10-10 19:24:44,087 INFO  timeline.LogInfo - Parsed 6 entities from hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_000001/summarylog-appattempt_1631559260564_0009_000001_2331123893 in 317 msec
      2021-10-10 19:25:44,092 INFO  timeline.LogInfo - Parsed 6 entities from hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_000001/summarylog-appattempt_1631559260564_0009_000001_2331123893 in 336 msec
      

      Attachments

        1. YARN-10975.001.patch
          22 kB
          Ravuri Sushma sree
        2. YARN-10975.002.patch
          22 kB
          Ravuri Sushma sree
        3. YARN-10975.003.patch
          24 kB
          Ravuri Sushma sree

        Issue Links

          Activity

            People

              Sushma_28 Ravuri Sushma sree
              prabhujoseph Prabhu Joseph
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h