Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-10850

TimelineService v2 lists containers for all attempts when filtering for one

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • timelinereader
    • None

    Description

      When using the command

      yarn container -list <app_attempt_id>
      

      with an application attempt ID based on the help only the containers for that attempt should be listed.

      -list <Application Name or Attempt ID>    List containers for application
                                                attempt when application
                                                attempt ID is provided. When
                                                application name is provided,
                                                then it finds the instances of
                                                the application based on app's
                                                own implementation, and
                                                -appTypes option must be
                                                specified unless it is the
                                                default yarn-service type. With
                                                app name, it supports optional
                                                use of -version to filter
                                                instances based on app version,
                                                -components to filter instances
                                                based on component names,
                                                -states to filter instances
                                                based on instance state.
      

      When TimelineService v2 is enabled all of the containers for the application are returned.

      hrt_qa@ctr-e172-1620330694487-146061-01-000002:/hwqe/hadoopqe$ yarn applicationattempt -list application_1625124233002_0007
      21/07/01 09:32:23 INFO impl.TimelineReaderClientImpl: Initialized TimelineReader URI=http://ctr-e172-1620330694487-146061-01-000004.hwx.site:8198/ws/v2/timeline/, clusterId=yarn-cluster
      21/07/01 09:32:24 INFO client.AHSProxy: Connecting to Application History server at ctr-e172-1620330694487-146061-01-000004.hwx.site/172.27.113.4:10200
      21/07/01 09:32:24 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
      Total number of application attempts :2
               ApplicationAttempt-Id	               State	                    AM-Container-Id	                       Tracking-URL
      appattempt_1625124233002_0007_000001	              FAILED	container_e43_1625124233002_0007_01_000001	http://ctr-e172-1620330694487-146061-01-000003.hwx.site:8088/proxy/application_1625124233002_0007/
      appattempt_1625124233002_0007_000002	              KILLED	container_e43_1625124233002_0007_02_000001	http://ctr-e172-1620330694487-146061-01-000003.hwx.site:8088/proxy/application_1625124233002_0007/
      

      Querying the 2 app attempts produces the same output:

      hrt_qa@ctr-e172-1620330694487-146061-01-000002:/hwqe/hadoopqe$ yarn container -list appattempt_1625124233002_0007_000001
      21/07/01 09:32:35 INFO impl.TimelineReaderClientImpl: Initialized TimelineReader URI=http://ctr-e172-1620330694487-146061-01-000004.hwx.site:8198/ws/v2/timeline/, clusterId=yarn-cluster
      21/07/01 09:32:35 INFO client.AHSProxy: Connecting to Application History server at ctr-e172-1620330694487-146061-01-000004.hwx.site/172.27.113.4:10200
      21/07/01 09:32:35 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
      21/07/01 09:32:36 INFO conf.Configuration: found resource resource-types.xml at file:/etc/hadoop/7.1.7.0-504/0/resource-types.xml
      Total number of containers :12
                        Container-Id	          Start Time	         Finish Time	               State	                Host	   Node Http Address	                            LOG-URL
      container_e43_1625124233002_0007_02_000004	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000002.hwx.site:25454	ctr-e172-1620330694487-146061-01-000002.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000002.hwx.site:25454/container_e43_1625124233002_0007_02_000004/container_e43_1625124233002_0007_02_000004/hrt_qa
      container_e43_1625124233002_0007_02_000005	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000007.hwx.site:25454	ctr-e172-1620330694487-146061-01-000007.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000007.hwx.site:25454/container_e43_1625124233002_0007_02_000005/container_e43_1625124233002_0007_02_000005/hrt_qa
      container_e43_1625124233002_0007_02_000003	                 N/A	                 N/A	                null	ctr-e172-1620330694487-146061-01-000006.hwx.site:25454	ctr-e172-1620330694487-146061-01-000006.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000006.hwx.site:25454/container_e43_1625124233002_0007_02_000003/container_e43_1625124233002_0007_02_000003/hrt_qa
      container_e43_1625124233002_0007_02_000002	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000004.hwx.site:25454	ctr-e172-1620330694487-146061-01-000004.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000004.hwx.site:25454/container_e43_1625124233002_0007_02_000002/container_e43_1625124233002_0007_02_000002/hrt_qa
      container_e43_1625124233002_0007_02_000006	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000005.hwx.site:25454	ctr-e172-1620330694487-146061-01-000005.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000005.hwx.site:25454/container_e43_1625124233002_0007_02_000006/container_e43_1625124233002_0007_02_000006/hrt_qa
      container_e43_1625124233002_0007_02_000001	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000006.hwx.site:25454	ctr-e172-1620330694487-146061-01-000006.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000006.hwx.site:25454/container_e43_1625124233002_0007_02_000001/container_e43_1625124233002_0007_02_000001/hrt_qa
      container_e43_1625124233002_0007_01_000005	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000002.hwx.site:25454	ctr-e172-1620330694487-146061-01-000002.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000002.hwx.site:25454/container_e43_1625124233002_0007_01_000005/container_e43_1625124233002_0007_01_000005/hrt_qa
      container_e43_1625124233002_0007_01_000004	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000004.hwx.site:25454	ctr-e172-1620330694487-146061-01-000004.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000004.hwx.site:25454/container_e43_1625124233002_0007_01_000004/container_e43_1625124233002_0007_01_000004/hrt_qa
      container_e43_1625124233002_0007_01_000002	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000005.hwx.site:25454	ctr-e172-1620330694487-146061-01-000005.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000005.hwx.site:25454/container_e43_1625124233002_0007_01_000002/container_e43_1625124233002_0007_01_000002/hrt_qa
      container_e43_1625124233002_0007_01_000006	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000006.hwx.site:25454	ctr-e172-1620330694487-146061-01-000006.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000006.hwx.site:25454/container_e43_1625124233002_0007_01_000006/container_e43_1625124233002_0007_01_000006/hrt_qa
      container_e43_1625124233002_0007_01_000003	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000007.hwx.site:25454	ctr-e172-1620330694487-146061-01-000007.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000007.hwx.site:25454/container_e43_1625124233002_0007_01_000003/container_e43_1625124233002_0007_01_000003/hrt_qa
      container_e43_1625124233002_0007_01_000001	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000004.hwx.site:25454	ctr-e172-1620330694487-146061-01-000004.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000004.hwx.site:25454/container_e43_1625124233002_0007_01_000001/container_e43_1625124233002_0007_01_000001/hrt_qa
      
      hrt_qa@ctr-e172-1620330694487-146061-01-000002:/hwqe/hadoopqe$ yarn container -list appattempt_1625124233002_0007_000002
      21/07/01 09:32:50 INFO impl.TimelineReaderClientImpl: Initialized TimelineReader URI=http://ctr-e172-1620330694487-146061-01-000004.hwx.site:8198/ws/v2/timeline/, clusterId=yarn-cluster
      21/07/01 09:32:51 INFO client.AHSProxy: Connecting to Application History server at ctr-e172-1620330694487-146061-01-000004.hwx.site/172.27.113.4:10200
      21/07/01 09:32:51 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
      21/07/01 09:32:51 INFO conf.Configuration: found resource resource-types.xml at file:/etc/hadoop/7.1.7.0-504/0/resource-types.xml
      Total number of containers :12
                        Container-Id	          Start Time	         Finish Time	               State	                Host	   Node Http Address	                            LOG-URL
      container_e43_1625124233002_0007_02_000004	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000002.hwx.site:25454	ctr-e172-1620330694487-146061-01-000002.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000002.hwx.site:25454/container_e43_1625124233002_0007_02_000004/container_e43_1625124233002_0007_02_000004/hrt_qa
      container_e43_1625124233002_0007_02_000005	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000007.hwx.site:25454	ctr-e172-1620330694487-146061-01-000007.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000007.hwx.site:25454/container_e43_1625124233002_0007_02_000005/container_e43_1625124233002_0007_02_000005/hrt_qa
      container_e43_1625124233002_0007_02_000003	                 N/A	                 N/A	                null	ctr-e172-1620330694487-146061-01-000006.hwx.site:25454	ctr-e172-1620330694487-146061-01-000006.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000006.hwx.site:25454/container_e43_1625124233002_0007_02_000003/container_e43_1625124233002_0007_02_000003/hrt_qa
      container_e43_1625124233002_0007_02_000002	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000004.hwx.site:25454	ctr-e172-1620330694487-146061-01-000004.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000004.hwx.site:25454/container_e43_1625124233002_0007_02_000002/container_e43_1625124233002_0007_02_000002/hrt_qa
      container_e43_1625124233002_0007_02_000006	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000005.hwx.site:25454	ctr-e172-1620330694487-146061-01-000005.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000005.hwx.site:25454/container_e43_1625124233002_0007_02_000006/container_e43_1625124233002_0007_02_000006/hrt_qa
      container_e43_1625124233002_0007_02_000001	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000006.hwx.site:25454	ctr-e172-1620330694487-146061-01-000006.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000006.hwx.site:25454/container_e43_1625124233002_0007_02_000001/container_e43_1625124233002_0007_02_000001/hrt_qa
      container_e43_1625124233002_0007_01_000005	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000002.hwx.site:25454	ctr-e172-1620330694487-146061-01-000002.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000002.hwx.site:25454/container_e43_1625124233002_0007_01_000005/container_e43_1625124233002_0007_01_000005/hrt_qa
      container_e43_1625124233002_0007_01_000004	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000004.hwx.site:25454	ctr-e172-1620330694487-146061-01-000004.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000004.hwx.site:25454/container_e43_1625124233002_0007_01_000004/container_e43_1625124233002_0007_01_000004/hrt_qa
      container_e43_1625124233002_0007_01_000002	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000005.hwx.site:25454	ctr-e172-1620330694487-146061-01-000005.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000005.hwx.site:25454/container_e43_1625124233002_0007_01_000002/container_e43_1625124233002_0007_01_000002/hrt_qa
      container_e43_1625124233002_0007_01_000006	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000006.hwx.site:25454	ctr-e172-1620330694487-146061-01-000006.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000006.hwx.site:25454/container_e43_1625124233002_0007_01_000006/container_e43_1625124233002_0007_01_000006/hrt_qa
      container_e43_1625124233002_0007_01_000003	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000007.hwx.site:25454	ctr-e172-1620330694487-146061-01-000007.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000007.hwx.site:25454/container_e43_1625124233002_0007_01_000003/container_e43_1625124233002_0007_01_000003/hrt_qa
      container_e43_1625124233002_0007_01_000001	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000004.hwx.site:25454	ctr-e172-1620330694487-146061-01-000004.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000004.hwx.site:25454/container_e43_1625124233002_0007_01_000001/container_e43_1625124233002_0007_01_000001/hrt_qa
      

      The reason for this is that the container collector method in TimelineServiceV2 gets the application ID from the attempt ID and collects the containers for the application.

      The other implementations however are using the application attempt ID: ApplicationHistoryManagerOnTimelineStore and ApplicationHistoryManagerImpl.

      TSv2 should behave similarly to v1.

      Attachments

        Activity

          People

            TiborKovacs Tibor Kovács
            bteke Benjamin Teke
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: