Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-2914

Port mapping isolator should cleanup unknown orphan containers after all known orphan containers are recovered during recovery.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.23.0
    • None
    • None
    • Twitter Mesos Q2 Sprint 6
    • 3

    Description

      Otherwise, the icmp/arp filter on host eth0 might be removed as a result of _cleanup if 'infos' is empty, causing subsequent '_cleanup' to fail on both known/unknown orphan containers.

      I0612 17:46:51.518501 16308 containerizer.cpp:314] Recovering containerizer
I0612 17:46:51.520612 16308 port_mapping.cpp:1567] Discovered network namespace handle symlink ddcb8397-3552-44f9-bc99-b5b69aa72944 -> 31607
I0612 17:46:51.521183 16308 port_mapping.cpp:1567] Discovered network namespace handle symlink d8c48a4a-fdfb-47dd-b8d8-07188c21600d -> 41020
I0612 17:46:51.521883 16308 port_mapping.cpp:1567] Discovered network namespace handle symlink 8953fc7f-9fca-4931-b0cb-2f4959ddee74 -> 3302
I0612 17:46:51.522542 16308 port_mapping.cpp:1567] Discovered network namespace handle symlink 50f9986f-ebbc-440d-86a7-9fa1a7c55a75 -> 19805
I0612 17:46:51.523643 16308 port_mapping.cpp:2597] Removing IP packet filters with ports [33792,34815] for container with pid 52304
I0612 17:46:51.525063 16308 port_mapping.cpp:2616] Freed ephemeral ports [33792,34816) for container with pid 52304
I0612 17:46:51.547696 16308 port_mapping.cpp:2762] Successfully performed cleanup for pid 52304
I0612 17:46:51.550027 16308 port_mapping.cpp:1698] Network isolator recovery complete
I0612 17:46:51.550946 16329 containerizer.cpp:449] Removing orphan container 111ea69c-6184-4da1-a0e9-c34e8c6deb30
I0612 17:46:51.552686 16329 containerizer.cpp:449] Removing orphan container ddcb8397-3552-44f9-bc99-b5b69aa72944
I0612 17:46:51.552734 16309 cgroups.cpp:2377] Freezing cgroup /sys/fs/cgroup/freezer/mesos/111ea69c-6184-4da1-a0e9-c34e8c6deb30
I0612 17:46:51.554932 16329 containerizer.cpp:449] Removing orphan container 8953fc7f-9fca-4931-b0cb-2f4959ddee74
I0612 17:46:51.555032 16309 cgroups.cpp:2377] Freezing cgroup /sys/fs/cgroup/freezer/mesos/ddcb8397-3552-44f9-bc99-b5b69aa72944
I0612 17:46:51.555629 16308 cgroups.cpp:1420] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos/111ea69c-6184-4da1-a0e9-c34e8c6deb30 after 1.730304ms
I0612 17:46:51.557507 16329 containerizer.cpp:449] Removing orphan container 50f9986f-ebbc-440d-86a7-9fa1a7c55a75
I0612 17:46:51.557611 16309 cgroups.cpp:2377] Freezing cgroup /sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74
I0612 17:46:51.557896 16313 cgroups.cpp:1420] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos/ddcb8397-3552-44f9-bc99-b5b69aa72944 after 1.685248ms
I0612 17:46:51.559412 16310 cgroups.cpp:2394] Thawing cgroup /sys/fs/cgroup/freezer/mesos/111ea69c-6184-4da1-a0e9-c34e8c6deb30
I0612 17:46:51.561564 16329 containerizer.cpp:449] Removing orphan container d8c48a4a-fdfb-47dd-b8d8-07188c21600d
I0612 17:46:51.562489 16315 cgroups.cpp:2377] Freezing cgroup /sys/fs/cgroup/freezer/mesos/50f9986f-ebbc-440d-86a7-9fa1a7c55a75
I0612 17:46:51.562988 16313 cgroups.cpp:2394] Thawing cgroup /sys/fs/cgroup/freezer/mesos/ddcb8397-3552-44f9-bc99-b5b69aa72944
I0612 17:46:51.563303 16310 cgroups.cpp:1449] Successfullly thawed cgroup /sys/fs/cgroup/freezer/mesos/111ea69c-6184-4da1-a0e9-c34e8c6deb30 after 2.076928ms
I0612 17:46:51.566052 16308 cgroups.cpp:2377] Freezing cgroup /sys/fs/cgroup/freezer/mesos/d8c48a4a-fdfb-47dd-b8d8-07188c21600d
I0612 17:46:51.566102 16313 slave.cpp:3911] Finished recovery
W0612 17:46:51.566432 16323 disk.cpp:299] Ignoring cleanup for unknown container 111ea69c-6184-4da1-a0e9-c34e8c6deb30
I0612 17:46:51.566651 16317 cgroups.cpp:1449] Successfullly thawed cgroup /sys/fs/cgroup/freezer/mesos/ddcb8397-3552-44f9-bc99-b5b69aa72944 after 2.12096ms
I0612 17:46:51.566987 16313 slave.cpp:3944] Garbage collecting old slave 20150319-213133-2080910346-5050-57551-S3314
I0612 17:46:51.567777 16318 cgroups.cpp:1420] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos/d8c48a4a-fdfb-47dd-b8d8-07188c21600d after 1.323008ms
W0612 17:46:51.568042 16323 port_mapping.cpp:2544] Ignoring cleanup for unknown container 111ea69c-6184-4da1-a0e9-c34e8c6deb30
I0612 17:46:51.569522 16311 gc.cpp:56] Scheduling '/var/lib/mesos/slaves/20150319-213133-2080910346-5050-57551-S3314' for gc 6.99999341503407days in the future
W0612 17:46:51.569725 16329 disk.cpp:299] Ignoring cleanup for unknown container ddcb8397-3552-44f9-bc99-b5b69aa72944
I0612 17:46:51.570911 16325 cgroups.cpp:2394] Thawing cgroup /sys/fs/cgroup/freezer/mesos/d8c48a4a-fdfb-47dd-b8d8-07188c21600d
I0612 17:46:51.573581 16316 port_mapping.cpp:2597] Removing IP packet filters with ports [35840,36863] for container with pid 31607
I0612 17:46:51.575127 16316 port_mapping.cpp:2616] Freed ephemeral ports [35840,36864) for container with pid 31607
I0612 17:46:51.588284 16330 cgroups.cpp:1449] Successfullly thawed cgroup /sys/fs/cgroup/freezer/mesos/d8c48a4a-fdfb-47dd-b8d8-07188c21600d after 14.503936ms
E0612 17:46:51.622140 16310 containerizer.cpp:480] Failed to clean up an isolator when destroying orphan container ddcb8397-3552-44f9-bc99-b5b69aa72944: The ICMP packet filter on host eth0 does not exist, The ARP packet filter on host eth0 does not exist
W0612 17:46:51.773123 16313 disk.cpp:299] Ignoring cleanup for unknown container d8c48a4a-fdfb-47dd-b8d8-07188c21600d
I0612 17:46:51.774153 16325 port_mapping.cpp:2597] Removing IP packet filters with ports [32768,33791] for container with pid 41020
I0612 17:46:51.775167 16325 port_mapping.cpp:2616] Freed ephemeral ports [32768,33792) for container with pid 41020
E0612 17:46:51.817221 16323 containerizer.cpp:480] Failed to clean up an isolator when destroying orphan container d8c48a4a-fdfb-47dd-b8d8-07188c21600d: The ICMP packet filter on host eth0 does not exist, The ARP packet filter on host eth0 does not exist
I0612 17:46:51.872231 16314 cgroups.cpp:1420] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos/50f9986f-ebbc-440d-86a7-9fa1a7c55a75 after 308.33792ms
I0612 17:46:51.874572 16314 cgroups.cpp:2394] Thawing cgroup /sys/fs/cgroup/freezer/mesos/50f9986f-ebbc-440d-86a7-9fa1a7c55a75
I0612 17:46:51.876566 16314 cgroups.cpp:1449] Successfullly thawed cgroup /sys/fs/cgroup/freezer/mesos/50f9986f-ebbc-440d-86a7-9fa1a7c55a75 after 1.593344ms
2015-06-12 17:46:54,833:16307(0x7f172eb07940):ZOO_INFO@auth_completion_func@1286: Authentication scheme digest succeeded
I0612 17:46:54.835737 16321 group.cpp:385] Trying to create path '/home/mesos/prod/master' in ZooKeeper
I0612 17:46:54.839110 16321 detector.cpp:138] Detected a new leader: (id='1')
I0612 17:46:54.840276 16330 group.cpp:659] Trying to get '/home/mesos/prod/master/info_0000000001' in ZooKeeper
I0612 17:46:54.842350 16330 detector.cpp:452] A new leading master (UPID=master@10.44.14.132:5050) is detected
I0612 17:46:54.843297 16330 slave.cpp:653] New master detected at master@10.44.14.132:5050
I0612 17:46:54.843298 16312 status_update_manager.cpp:171] Pausing sending status updates
I0612 17:46:54.844091 16330 slave.cpp:678] No credentials provided. Attempting to register without authentication
I0612 17:46:54.845087 16330 slave.cpp:689] Detecting new master
I0612 17:47:01.561920 16309 cgroups.cpp:2394] Thawing cgroup /sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74
I0612 17:47:01.564687 16309 cgroups.cpp:1449] Successfullly thawed cgroup /sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74 after 1.924096ms
I0612 17:47:01.565467 16309 cgroups.cpp:2377] Freezing cgroup /sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74
W0612 17:47:09.978946 16326 disk.cpp:299] Ignoring cleanup for unknown container 50f9986f-ebbc-440d-86a7-9fa1a7c55a75
I0612 17:47:09.979818 16327 port_mapping.cpp:2597] Removing IP packet filters with ports [34816,35839] for container with pid 19805
I0612 17:47:09.981474 16327 port_mapping.cpp:2616] Freed ephemeral ports [34816,35840) for container with pid 19805
E0612 17:47:10.278715 16325 containerizer.cpp:480] Failed to clean up an isolator when destroying orphan container 50f9986f-ebbc-440d-86a7-9fa1a7c55a75: The ICMP packet filter on host eth0 does not exist, The ARP packet filter on host eth0 does not exist
I0612 17:47:11.568151 16326 cgroups.cpp:2394] Thawing cgroup /sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74
I0612 17:47:11.570915 16326 cgroups.cpp:1449] Successfullly thawed cgroup /sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74 after 1.987072ms
I0612 17:47:11.571728 16326 cgroups.cpp:2377] Freezing cgroup /sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74
I0612 17:47:14.549536 16316 slave.cpp:821] Registered with master master@10.44.14.132:5050; given slave ID 20150602-190100-2215521290-5050-39399-S23257
I0612 17:47:14.550220 16318 status_update_manager.cpp:178] Resuming sending status updates
I0612 17:47:21.574513 16319 cgroups.cpp:2394] Thawing cgroup /sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74
I0612 17:47:21.576817 16319 cgroups.cpp:1449] Successfullly thawed cgroup /sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74 after 1.587968ms
I0612 17:47:21.577466 16319 cgroups.cpp:2377] Freezing cgroup /sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74
I0612 17:47:31.580281 16310 cgroups.cpp:2394] Thawing cgroup /sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74
I0612 17:47:31.582365 16310 cgroups.cpp:1449] Successfullly thawed cgroup /sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74 after 1.410048ms
I0612 17:47:31.582895 16310 cgroups.cpp:2377] Freezing cgroup /sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74
I0612 17:47:41.585619 16322 cgroups.cpp:2394] Thawing cgroup /sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74
I0612 17:47:41.587703 16322 cgroups.cpp:1449] Successfullly thawed cgroup /sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74 after 1.418752ms
I0612 17:47:41.588436 16322 cgroups.cpp:2377] Freezing cgroup /sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74
I0612 17:47:51.515689 16330 slave.cpp:3733] Current disk usage 11.43%. Max allowed age: 5.499938588861215days
E0612 17:47:51.557831 16330 containerizer.cpp:468] Failed to destroy orphan container 8953fc7f-9fca-4931-b0cb-2f4959ddee74: Timed out after 1mins

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MESOS-2867 Orphan Container Clean-up Failure in PortMappingIsolator

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MESOS-2903 Network isolator should not fail when target state already exists

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Activity

            People

              jieyu Jie Yu
              jieyu Jie Yu
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: