Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Cannot Reproduce
-
0.90.0
-
None
-
None
-
None
-
all
Description
I recently experienced a cluster malfunction which was caused by a change in DNS config for services co-hosted on the machines running region servers.
The RS are specified using IP addresses in the 'regionservers' file. Those machines are 1.example.com to N.example.com (there are A RRs for those names to each of the N IP addresses in 'regionservers').
Until recently, the PTR RRs for the RS IPs were those x.example.com names.
Then a service was deployed on some of the x.example.com machines, and new A RRs were added for svc.example.com which point to each of the IPs used for the service.
Jointly new PTR records were added too for the given IPs. Those PTR records have 'svc.example.com' as their PTRDATA, and this is causing the HBase cluster to get completely confused.
Since it is perfectly legal to have multiple PTR records, it seems important to make the canonicalization of RS more robust to DNS tweaks.
Maybe generating a UUID when a RS is started would help, this UUID could be used to register the RS in ZK and we would not rely on DNS for obtaining a stable canonical name (which may not even exist...).