[IMPALA-6786] List of backends doesn't get updated for dedicated coordinators

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: Impala 2.12.0
• Component/s: Backend
• Epic Color: ghx-label-1
• Assignee: Vuk Ercegovac
• Reporter: Mostafa Mokhtar
• Created: 02/Apr/18 03:03
• Updated: 03/Apr/18 20:56
• Resolved: 03/Apr/18 20:56

Description

List of backend for Impala daemons running with "is_executor=false" doesn't get updated when other new daemons go online or old ones go offline

Repro
1) Setup an Impala daemon with dedicated coordinator
2) Run a couple of query // Might not be necessary//
3) Change the state of the Impala cluster by brining more nodes online or taking some offline
4) Check the subscriber page on the statestore, it should match expected state of the cluster
5) Check the backends page for the dedicated coordinator, it won't match the SS subscriber page

Number of backed from each Impala daemon page

va1007.foo.com: 506 <----- dedicated coordinator
vd0403.foo.com: 651
ve1034.foo.com: 651
vb0817.foo.com: 651
ve1009.foo.com: 651
ve0841.foo.com: 651
vd1032.foo.com: 651
vd1031.foo.com: 651
vd0605.foo.com: 651
vd0640.foo.com: 651
vd1030.foo.com: 651
vd0436.foo.com: 651
ve0829.foo.com: 651
vd1005.foo.com: 651
vd1034.foo.com: 651
ve1030.foo.com: 651
vd0604.foo.com: 651
vd1040.foo.com: 651
vd0633.foo.com: 651
ve0404.foo.com: 651
vb0809.foo.com: 651
ve0434.foo.com: 651
vd1008.foo.com: 651
vd0639.foo.com: 651
vb0832.foo.com: 651
vd0823.foo.com: 651
vd0629.foo.com: 651
ve1031.foo.com: 651
vd0439.foo.com: 651
ve0436.foo.com: 651
vd0423.foo.com: 651
vb0816.foo.com: 651
vd0841.foo.com: 651
ve0425.foo.com: 651
vd0636.foo.com: 651
vd0842.foo.com: 651
ve0818.foo.com: 651
vd0406.foo.com: 651
ve0441.foo.com: 651
vd0410.foo.com: 651
vd1035.foo.com: 651
ve0627.foo.com: 651
ve0637.foo.com: 651
vd0432.foo.com: 651
ve0621.foo.com: 651
vd0810.foo.com: 651
ve0632.foo.com: 651
ve0427.foo.com: 651
vb0836.foo.com: 651
ve0407.foo.com: 651
ve0630.foo.com: 651
ve0837.foo.com: 651
ve0638.foo.com: 651
vd0606.foo.com: 651
ve0620.foo.com: 651
ve0409.foo.com: 651
ve0410.foo.com: 651
ve0824.foo.com: 651
vd0642.foo.com: 651
vd0818.foo.com: 651
vd0808.foo.com: 651
ve0442.foo.com: 651
vd0624.foo.com: 651
ve0617.foo.com: 651
vd0806.foo.com: 651
ve1041.foo.com: 651
vd0422.foo.com: 651
vd0440.foo.com: 651
vd0803.foo.com: 651
vb0838.foo.com: 651
vd0838.foo.com: 651
vd0638.foo.com: 651
vd0434.foo.com: 651
ve0429.foo.com: 651
ve0640.foo.com: 651
vd0431.foo.com: 651
vd0418.foo.com: 651
ve0604.foo.com: 651
ve0804.foo.com: 651
ve0609.foo.com: 651
vb0841.foo.com: 651
vd0830.foo.com: 651
vd0419.foo.com: 651
ve1018.foo.com: 651
vb0807.foo.com: 651
vd0420.foo.com: 651
vd0617.foo.com: 651
vd0602.foo.com: 651
ve1039.foo.com: 651
ve0601.foo.com: 651
vb0801.foo.com: 651
vd0616.foo.com: 651
ve0808.foo.com: 651
vd1029.foo.com: 651
va1009.foo.com: 651
vb1409.foo.com: 651
ve1040.foo.com: 651
vd0430.foo.com: 651
vd1210.foo.com: 651
ve0825.foo.com: 651
ve1021.foo.com: 651
vd0620.foo.com: 651
vd1002.foo.com: 651
va1316.foo.com: 651
ve0831.foo.com: 651
ve1219.foo.com: 651
vd1021.foo.com: 651
va1242.foo.com: 651
vb0811.foo.com: 651
va1132.foo.com: 651
vd1015.foo.com: 651
va1442.foo.com: 651
vd1224.foo.com: 651
vd0614.foo.com: 651
ve1007.foo.com: 651
vd0835.foo.com: 651
ve1042.foo.com: 651
ve1215.foo.com: 651
ve1026.foo.com: 651
va1228.foo.com: 651
ve0401.foo.com: 651
ve0805.foo.com: 651
vd0411.foo.com: 651
va1338.foo.com: 651
va1433.foo.com: 651
vd0828.foo.com: 651
va1236.foo.com: 651
ve1032.foo.com: 651
vd0401.foo.com: 651
vd1205.foo.com: 651
vd0408.foo.com: 651
va1136.foo.com: 651
va1206.foo.com: 651
ve1001.foo.com: 651
va1121.foo.com: 651
ve1229.foo.com: 651
ve0642.foo.com: 651
ve1015.foo.com: 651
ve0634.foo.com: 651
va1411.foo.com: 651
vb0804.foo.com: 651
ve0802.foo.com: 651
vd1240.foo.com: 651
va1112.foo.com: 651
va1103.foo.com: 651
va1328.foo.com: 651
va1142.foo.com: 651
va1128.foo.com: 651
ve1028.foo.com: 651
vd1232.foo.com: 651
ve0419.foo.com: 651
va1332.foo.com: 651
vd1019.foo.com: 651
vd0631.foo.com: 651
vd0609.foo.com: 651
vd0802.foo.com: 651
va1326.foo.com: 651
ve0836.foo.com: 651
ve0813.foo.com: 651
va1446.foo.com: 651
vb1442.foo.com: 651
va1207.foo.com: 651
va1017.foo.com: 651
vd1038.foo.com: 651
ve0636.foo.com: 651
vb1432.foo.com: 651
va1420.foo.com: 651
va1201.foo.com: 651
ve0814.foo.com: 651
va1314.foo.com: 651
ve1019.foo.com: 651
va1133.foo.com: 651
vd0409.foo.com: 651
vd1219.foo.com: 651
vd1242.foo.com: 651
vb0814.foo.com: 651
ve0611.foo.com: 651
vb0842.foo.com: 651
va1440.foo.com: 651
ve0641.foo.com: 651
va1404.foo.com: 651
va1214.foo.com: 651
va1401.foo.com: 651
va1435.foo.com: 651
vd0415.foo.com: 651
va1244.foo.com: 651
vd1028.foo.com: 651
vb1418.foo.com: 651
vd0425.foo.com: 651
ve1004.foo.com: 651
ve0418.foo.com: 651
vb0830.foo.com: 651
va1111.foo.com: 651
ve0822.foo.com: 651
ve1201.foo.com: 651
va1213.foo.com: 651
ve0405.foo.com: 651
vd0612.foo.com: 651
vd0814.foo.com: 651
ve0603.foo.com: 651
vd0435.foo.com: 651
vb1425.foo.com: 651
ve1213.foo.com: 651
ve0439.foo.com: 651
ve0629.foo.com: 651
va1422.foo.com: 651
va1427.foo.com: 651
ve0423.foo.com: 651
ve0615.foo.com: 651
vd1039.foo.com: 651
va1233.foo.com: 651
vd1026.foo.com: 651
vd0615.foo.com: 651
va1221.foo.com: 651
va1240.foo.com: 651
ve0428.foo.com: 651
vd0822.foo.com: 651
vd0836.foo.com: 651
vb1412.foo.com: 651
ve1241.foo.com: 651
vd0619.foo.com: 651
ve1234.foo.com: 651
ve1206.foo.com: 651
ve0821.foo.com: 651
va1216.foo.com: 651
ve0614.foo.com: 651
ve0414.foo.com: 651
vd0840.foo.com: 651
ve0828.foo.com: 651
va1405.foo.com: 651
ve0633.foo.com: 651
vd0626.foo.com: 651
va1123.foo.com: 651
vd1007.foo.com: 651
ve1022.foo.com: 651
va1130.foo.com: 651
vd0816.foo.com: 651
ve1225.foo.com: 651
ve1232.foo.com: 651
ve1033.foo.com: 651
ve0435.foo.com: 651
ve0606.foo.com: 651
ve1230.foo.com: 651
va1114.foo.com: 651
ve0830.foo.com: 651
ve0834.foo.com: 651
va1321.foo.com: 651
vb1413.foo.com: 651
ve0602.foo.com: 651
va1116.foo.com: 651
ve1208.foo.com: 651
vb1407.foo.com: 651
vb1433.foo.com: 651
ve0422.foo.com: 651
ve1005.foo.com: 651
ve0612.foo.com: 651
va1443.foo.com: 651
vd0829.foo.com: 651
ve0421.foo.com: 651
vd1220.foo.com: 651
vd1218.foo.com: 651
va1343.foo.com: 651
vb1441.foo.com: 651
vd1221.foo.com: 651
va1337.foo.com: 651
ve0840.foo.com: 651
ve1204.foo.com: 651
ve0811.foo.com: 651
va1424.foo.com: 651
va1344.foo.com: 651
vd1003.foo.com: 651
vb1416.foo.com: 651
va1010.foo.com: 651
vd0815.foo.com: 651
ve0438.foo.com: 651
ve1011.foo.com: 651
vd1011.foo.com: 651
ve0625.foo.com: 651
vd1226.foo.com: 651
va1202.foo.com: 651
va1120.foo.com: 651
vb1417.foo.com: 651
ve1024.foo.com: 651
vb0812.foo.com: 651
va1307.foo.com: 651
va1413.foo.com: 651
vb1437.foo.com: 651
va1436.foo.com: 651
va1012.foo.com: 651
va1431.foo.com: 651
vd1016.foo.com: 651
vd0416.foo.com: 651
va1327.foo.com: 651
ve0803.foo.com: 651
ve1014.foo.com: 651
ve0635.foo.com: 651
va1210.foo.com: 651
vd0426.foo.com: 651
vb1440.foo.com: 651
vd1236.foo.com: 651
va1234.foo.com: 651
vb0806.foo.com: 651
va1331.foo.com: 651
va1241.foo.com: 651
vd0839.foo.com: 651
vb0827.foo.com: 651
va1315.foo.com: 651
ve1239.foo.com: 651
ve0402.foo.com: 651
vd1223.foo.com: 651
va1222.foo.com: 651
va1317.foo.com: 651
vd1213.foo.com: 651
vd1237.foo.com: 651
va1301.foo.com: 651
vd1234.foo.com: 651
va1345.foo.com: 651
ve0616.foo.com: 651
vd1208.foo.com: 651
vd1235.foo.com: 651
vb1402.foo.com: 651
vb1427.foo.com: 651
vb0833.foo.com: 651
ve1222.foo.com: 651
ve0619.foo.com: 651
vd0628.foo.com: 651
ve1211.foo.com: 651
vd1211.foo.com: 651
vb1434.foo.com: 651
va1145.foo.com: 651
va1220.foo.com: 651
vd1036.foo.com: 651
ve0440.foo.com: 651
va1218.foo.com: 651
va1330.foo.com: 651
ve0823.foo.com: 651
vb1430.foo.com: 651
va1113.foo.com: 651
vd1006.foo.com: 651
ve1035.foo.com: 651
va1309.foo.com: 651
ve0817.foo.com: 651
va1333.foo.com: 651
va1318.foo.com: 651
ve1217.foo.com: 651
vb0824.foo.com: 651
va1135.foo.com: 651
vd0630.foo.com: 651
ve0812.foo.com: 651
vd0402.foo.com: 651
ve1236.foo.com: 651
ve0816.foo.com: 651
vd0637.foo.com: 651
ve0408.foo.com: 651
vb1436.foo.com: 651
va1217.foo.com: 651
vd1202.foo.com: 651
va1212.foo.com: 651
vd0832.foo.com: 651
va1403.foo.com: 651
va1239.foo.com: 651
va1226.foo.com: 651
va1441.foo.com: 651
va1412.foo.com: 651
vd1230.foo.com: 651
va1341.foo.com: 651
vd0824.foo.com: 651
va1334.foo.com: 651
va1319.foo.com: 651
ve1025.foo.com: 651
vd0811.foo.com: 651
va1416.foo.com: 651
vd1027.foo.com: 651
vd1033.foo.com: 651
vd0813.foo.com: 651
va1432.foo.com: 651
va1230.foo.com: 651
vd0441.foo.com: 651
ve0623.foo.com: 651
va1225.foo.com: 651
vb0828.foo.com: 651
vd0826.foo.com: 651
vd0404.foo.com: 651
vd1014.foo.com: 651
va1430.foo.com: 651
ve0431.foo.com: 651
vb0815.foo.com: 651
ve1006.foo.com: 651
ve0806.foo.com: 651
va1231.foo.com: 651
va1227.foo.com: 651
va1329.foo.com: 651
vd1225.foo.com: 651
vd0421.foo.com: 651
vd0635.foo.com: 651
ve0607.foo.com: 651
vb1401.foo.com: 651
vd1216.foo.com: 651
vd1228.foo.com: 651
va1409.foo.com: 651
ve1218.foo.com: 651
vb1422.foo.com: 651
va1215.foo.com: 651
va1219.foo.com: 651
vd0607.foo.com: 651
vd1020.foo.com: 651
ve1013.foo.com: 651
va1426.foo.com: 651
va1243.foo.com: 651
vb0837.foo.com: 651
vd1212.foo.com: 651
ve0613.foo.com: 651
vb1423.foo.com: 651
va1421.foo.com: 651
vd0831.foo.com: 651
ve1020.foo.com: 651
va1302.foo.com: 651
ve1010.foo.com: 651
ve0403.foo.com: 651
vd0608.foo.com: 651
ve0413.foo.com: 651
va1208.foo.com: 651
ve1016.foo.com: 651
va1203.foo.com: 651
vd1009.foo.com: 651
vb0826.foo.com: 651
va1303.foo.com: 651
ve1017.foo.com: 651
vb0813.foo.com: 651
vb1424.foo.com: 651
vd0807.foo.com: 651
va1320.foo.com: 651
vb1438.foo.com: 651
ve1205.foo.com: 651
vd1222.foo.com: 651
ve0420.foo.com: 651
vd0413.foo.com: 651
ve0827.foo.com: 651
ve1202.foo.com: 651
vd1233.foo.com: 651
va1125.foo.com: 651
vb0834.foo.com: 651
vb1403.foo.com: 651
vd0603.foo.com: 651
va1415.foo.com: 651
ve0437.foo.com: 651
ve0618.foo.com: 651
va1340.foo.com: 651
va1235.foo.com: 651
va1419.foo.com: 651
vd0438.foo.com: 651
ve1240.foo.com: 651
vb0810.foo.com: 651
vd0442.foo.com: 651
vd0819.foo.com: 651
va1115.foo.com: 651
vd0820.foo.com: 651
vb0802.foo.com: 651
ve1242.foo.com: 651
va1143.foo.com: 651
vb1429.foo.com: 651
vd0641.foo.com: 651
vd1227.foo.com: 651
vb1420.foo.com: 651
ve1223.foo.com: 651
vb1404.foo.com: 651
va1324.foo.com: 651
va1141.foo.com: 651
ve1012.foo.com: 651
vd0827.foo.com: 651
vd0611.foo.com: 651
vd1239.foo.com: 651
va1406.foo.com: 651
va1410.foo.com: 651
ve1002.foo.com: 651
vd0623.foo.com: 651
va1335.foo.com: 651
ve1207.foo.com: 651
ve1235.foo.com: 651
vb0840.foo.com: 651
vd1022.foo.com: 651
ve0832.foo.com: 651
ve0406.foo.com: 651
ve1203.foo.com: 651
ve0624.foo.com: 651
va1325.foo.com: 651
ve1237.foo.com: 651
vd1241.foo.com: 651
va1237.foo.com: 651
vd0437.foo.com: 651
vd0821.foo.com: 651
ve0833.foo.com: 651
ve0815.foo.com: 651
ve1220.foo.com: 651
ve0412.foo.com: 651
va1407.foo.com: 651
va1122.foo.com: 651
va1013.foo.com: 651
vb1408.foo.com: 651
vb1439.foo.com: 651
va1110.foo.com: 651
vb1426.foo.com: 651
vd0834.foo.com: 651
vd1001.foo.com: 651
vd0428.foo.com: 651
vb0835.foo.com: 651
ve1027.foo.com: 651
vd1217.foo.com: 651
ve1231.foo.com: 651
vd1229.foo.com: 651
ve0411.foo.com: 651
va1126.foo.com: 651
vd1024.foo.com: 651
vb1421.foo.com: 651
ve1221.foo.com: 651
ve1210.foo.com: 651
vd1203.foo.com: 651
va1425.foo.com: 651
va1018.foo.com: 651
vb1415.foo.com: 651
va1429.foo.com: 651
ve1238.foo.com: 651
ve0415.foo.com: 651
va1311.foo.com: 651
vd0601.foo.com: 651
va1428.foo.com: 651
vd0613.foo.com: 651
vd0805.foo.com: 651
ve1029.foo.com: 651
va1131.foo.com: 651
vb1435.foo.com: 651
va1238.foo.com: 651
ve1212.foo.com: 651
ve0810.foo.com: 651
vd1012.foo.com: 651
va1402.foo.com: 651
vd1209.foo.com: 651
va1417.foo.com: 651
ve1224.foo.com: 651
va1306.foo.com: 651
ve0605.foo.com: 651
ve1228.foo.com: 651
vb1406.foo.com: 651
vd0622.foo.com: 651
ve0610.foo.com: 651
vb0805.foo.com: 651
vd0812.foo.com: 651
vd0618.foo.com: 651
ve0807.foo.com: 651
va1434.foo.com: 651
va1204.foo.com: 651
vd0801.foo.com: 651
vb0829.foo.com: 651
ve0628.foo.com: 651
vb0818.foo.com: 651
ve1209.foo.com: 651
va1119.foo.com: 651
vb1428.foo.com: 651
va1223.foo.com: 651
ve1036.foo.com: 651
va1418.foo.com: 651
vd1238.foo.com: 651
va1146.foo.com: 651
ve0839.foo.com: 651
vd1010.foo.com: 651
va1138.foo.com: 651
vd0424.foo.com: 651
va1346.foo.com: 651
va1229.foo.com: 651
ve0430.foo.com: 651
va1439.foo.com: 651
vb0823.foo.com: 651
ve0424.foo.com: 651
va1438.foo.com: 651
vb0820.foo.com: 651
vb1431.foo.com: 651
va1224.foo.com: 651
vd0610.foo.com: 651
ve0820.foo.com: 651
vb0808.foo.com: 651
vb1414.foo.com: 651
vd1204.foo.com: 651
vd0625.foo.com: 651
vb1419.foo.com: 651
ve0826.foo.com: 651
va1336.foo.com: 651
vd0429.foo.com: 651
ve1216.foo.com: 651
vb1411.foo.com: 651
vd0837.foo.com: 651
ve0838.foo.com: 651
vb0822.foo.com: 651
vd0407.foo.com: 651
ve0426.foo.com: 651
ve0842.foo.com: 651
va1127.foo.com: 651
va1129.foo.com: 651
vd1201.foo.com: 651
ve0416.foo.com: 651
va1323.foo.com: 651
va1117.foo.com: 651
ve1227.foo.com: 651
vd1206.foo.com: 651
va1313.foo.com: 651
ve1233.foo.com: 651
va1304.foo.com: 651
va1144.foo.com: 651
vd1017.foo.com: 651
vd0632.foo.com: 651
vd0804.foo.com: 651
vd0833.foo.com: 651
vd1041.foo.com: 651
ve1003.foo.com: 651
vd1023.foo.com: 651
vb0839.foo.com: 651
vd0634.foo.com: 651
vd1037.foo.com: 651
vd0414.foo.com: 651
vd0405.foo.com: 651
va1211.foo.com: 651
va1339.foo.com: 651
ve1214.foo.com: 651
vd0627.foo.com: 651
ve0835.foo.com: 651
vd0809.foo.com: 651
va1437.foo.com: 651
va1308.foo.com: 651
ve0608.foo.com: 651
ve0433.foo.com: 651
vd1215.foo.com: 651
ve0432.foo.com: 651
va1124.foo.com: 651
va1014.foo.com: 651
ve0809.foo.com: 651
vb1405.foo.com: 651
ve1023.foo.com: 651
vb0825.foo.com: 651
vd0412.foo.com: 651
ve0626.foo.com: 651
ve0819.foo.com: 651
va1342.foo.com: 651
ve1008.foo.com: 651
vd1025.foo.com: 651
vd0825.foo.com: 651
vb0803.foo.com: 651
va1408.foo.com: 651
va1139.foo.com: 651
va1140.foo.com: 651
vd1013.foo.com: 651
vd0427.foo.com: 651
ve0631.foo.com: 651
vd0621.foo.com: 651

Issue Links

• duplicates: IMPALA-6785 Starting an Impalad on an already running cluster may result in inconsistent cluster subscription