[HIVE-10653] LLAP: registry logs strange lines on daemons

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: llap
• Assignee: Sergey Shelukhin
• Reporter: Sergey Shelukhin
• Created: 07/May/15 22:08
• Updated: 17/Feb/16 01:33
• Resolved: 26/May/15 20:23

Description

Discovered while looking at HIVE-10648; sseth mentioned that this should not be happening.
Most of the daemons described as being killed were actually alive. Several/all LLAP daemons in the cluster logged these messages at approximately the same time (while AM was stuck, incidentally; perhaps they were just bored with no work).

2015-05-07 12:14:30,016 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Starting to refresh ServiceInstanceSet 515383300
2015-05-07 12:14:30,016 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Adding new worker f698eaee-bf6c-484d-9b90-a60d9005760c which mapped to DynamicServiceInstance [alive=true, host=cn057-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,016 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Adding new worker 9d1f50d1-f237-43c1-a8c5-32741e82d18b which mapped to DynamicServiceInstance [alive=true, host=cn041-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,016 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Adding new worker b8a22e2f-652a-4fde-be7a-744786bc93c9 which mapped to DynamicServiceInstance [alive=true, host=cn042-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,016 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Adding new worker 8394e271-e0d5-4589-817e-0181db0866b9 which mapped to DynamicServiceInstance [alive=true, host=cn056-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,016 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Adding new worker 1cabdcce-1089-4de6-abdf-315f18a8b4c0 which mapped to DynamicServiceInstance [alive=true, host=cn054-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,016 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Adding new worker 4027ad61-8c61-4173-90e2-d166ceaad74b which mapped to DynamicServiceInstance [alive=true, host=cn051-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,016 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Adding new worker 7f71a05f-f849-43d2-8fdb-09ba144d4b93 which mapped to DynamicServiceInstance [alive=true, host=cn050-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,016 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Adding new worker 41835ca1-69cd-4290-8c8f-8a9583a5d635 which mapped to DynamicServiceInstance [alive=true, host=cn053-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,016 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Adding new worker 54952e48-41be-48e1-922c-a39d0ee48a33 which mapped to DynamicServiceInstance [alive=true, host=cn055-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,016 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Adding new worker 980dfe6c-d03b-462b-bee3-35d183c74aee which mapped to DynamicServiceInstance [alive=true, host=cn052-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,016 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Adding new worker d524212a-6743-4f18-bcf6-525a0d4b1a0a which mapped to DynamicServiceInstance [alive=true, host=cn046-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,016 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Killing service instance: DynamicServiceInstance [alive=true, host=cn048-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,017 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Deleting dead worker a0ba3b54-3f9a-484d-bef4-e88070b32096 which mapped to DynamicServiceInstance [alive=false, host=cn048-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,017 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Killing service instance: DynamicServiceInstance [alive=true, host=cn046-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,017 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Deleting dead worker 46046711-5b62-45e0-a075-a30416303768 which mapped to DynamicServiceInstance [alive=false, host=cn046-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,017 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Killing service instance: DynamicServiceInstance [alive=true, host=cn043-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,017 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Deleting dead worker e615e594-5f13-4200-a0d0-d12df38cafe7 which mapped to DynamicServiceInstance [alive=false, host=cn043-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,017 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Killing service instance: DynamicServiceInstance [alive=true, host=cn050-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,017 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Deleting dead worker 9caa4a9b-d1c3-4920-9dab-cb244255756c which mapped to DynamicServiceInstance [alive=false, host=cn050-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,017 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Killing service instance: DynamicServiceInstance [alive=true, host=cn049-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,017 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Deleting dead worker a5e1ee21-2bb0-43ee-aee0-f08d93b0546a which mapped to DynamicServiceInstance [alive=false, host=cn049-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,017 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Killing service instance: DynamicServiceInstance [alive=true, host=cn058-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,017 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Deleting dead worker 280c43c0-c772-4546-afd3-fae9bb792e68 which mapped to DynamicServiceInstance [alive=false, host=cn058-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,017 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Killing service instance: DynamicServiceInstance [alive=true, host=cn041-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,017 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Deleting dead worker a0923b34-87a5-48cd-8773-21bd9f1ca2b6 which mapped to DynamicServiceInstance [alive=false, host=cn041-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,017 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Killing service instance: DynamicServiceInstance [alive=true, host=cn056-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,017 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Deleting dead worker 445d7f0a-6fc2-4b4f-a6d7-a0e71ae14923 which mapped to DynamicServiceInstance [alive=false, host=cn056-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,017 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Killing service instance: DynamicServiceInstance [alive=true, host=cn060-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,017 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Deleting dead worker c905182c-c13f-4da2-9c53-c52b2dac61c6 which mapped to DynamicServiceInstance [alive=false, host=cn060-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,017 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Killing service instance: DynamicServiceInstance [alive=true, host=cn052-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,017 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Deleting dead worker eef7adab-863b-4c32-ac1e-e6a5d010a867 which mapped to DynamicServiceInstance [alive=false, host=cn052-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,017 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Killing service instance: DynamicServiceInstance [alive=true, host=cn059-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,017 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Deleting dead worker 342f4992-2608-43ab-a119-b50882e35f75 which mapped to DynamicServiceInstance [alive=false, host=cn059-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,017 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Killing service instance: DynamicServiceInstance [alive=true, host=cn057-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,017 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Deleting dead worker 9f5ff291-bb08-40a4-ad54-a06a65a21652 which mapped to DynamicServiceInstance [alive=false, host=cn057-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,017 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Killing service instance: DynamicServiceInstance [alive=true, host=cn051-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,017 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Deleting dead worker fbc83e2e-6f1c-4392-8250-6162afab93f3 which mapped to DynamicServiceInstance [alive=false, host=cn051-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,017 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Killing service instance: DynamicServiceInstance [alive=true, host=cn042-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,017 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Deleting dead worker 0b3a04bb-0b30-4315-984c-58c58f45fd4e which mapped to DynamicServiceInstance [alive=false, host=cn042-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,017 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Killing service instance: DynamicServiceInstance [alive=true, host=cn054-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,017 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Deleting dead worker fc1c3c5c-e935-4b5a-913d-2f99423cbcd4 which mapped to DynamicServiceInstance [alive=false, host=cn054-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,017 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Killing service instance: DynamicServiceInstance [alive=true, host=cn055-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]
2015-05-07 12:14:30,017 [LlapYarnRegistryRefresher()] INFO org.apache.hadoop.hive.llap.daemon.registry.impl.LlapYarnRegistryImpl: Deleting dead worker a7022879-7c89-4b35-a247-d2132f51f31c which mapped to DynamicServiceInstance [alive=false, host=cn055-10.l42scl.hortonworks.com:15001 with resources=<memory:20480, vCores:6>]

Also Killing message is misleading, it's not actually killing anything as far as I can tell from the code