Critical Description
Currently as part of SENTRY-1929, we reset the CounterWait on the server that does the HMS full snapshot. Ideally this has to be done across all servers otherwise i see this error in the log of other servers.
2017-09-08 19:05:44,697 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A newer full update is found with image number: 2 2017-09-08 19:05:44,894 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:05:45,115 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A newer full update is found with image number: 2 2017-09-08 19:05:45,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:05:45,963 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:05:46,391 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:05:46,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:05:47,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:05:47,929 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:05:48,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:05:48,898 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:05:49,391 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:05:49,897 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:05:50,452 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:05:50,506 INFO org.apache.sentry.hdfs.HMSPathsDumper: Paths Dump created. 101104 total path strings, 579426 duplicate strings found, compacted to 1077 unique strings. 2017-09-08 19:05:50,535 INFO org.apache.sentry.hdfs.HMSPathsDumper: Paths Dump created. 101104 total path strings, 579426 duplicate strings found, compacted to 1077 unique strings. 2017-09-08 19:05:50,947 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:05:51,904 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:05:51,932 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:05:52,420 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:05:52,923 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:05:53,417 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:05:53,922 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:05:54,414 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:05:54,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:05:55,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:05:55,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:05:56,392 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:05:56,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:05:57,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:05:57,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:05:58,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:05:58,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:05:59,393 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:05:59,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:00,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:00,891 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:01,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:01,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:02,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:02,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:03,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:03,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:04,393 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:04,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:05,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:05,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:06,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:06,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:07,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:07,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:08,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:08,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:09,422 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:09,892 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:10,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:10,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:11,391 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:11,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:12,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:12,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:13,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:13,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:14,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:14,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:15,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:15,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:16,392 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:16,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:17,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190 2017-09-08 19:06:17,553 INFO org.apache.sentry.hdfs.DBUpdateForwarder: Newer delta updates are found up to sequence number: 2287 2017-09-08 19:06:17,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 6 is smaller then the previous one 2190 2017-09-08 19:06:18,066 INFO org.apache.sentry.hdfs.DBUpdateForwarder: Newer delta updates are found up to sequence number: 177 2017-09-08 19:06:18,081 INFO org.apache.sentry.hdfs.DBUpdateForwarder: Newer delta updates are found up to sequence number: 2294 2017-09-08 19:06:18,274 INFO org.apache.sentry.hdfs.DBUpdateForwarder: Newer delta updates are found up to sequence number: 177 2017-09-08 19:06:18,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 13 is smaller then the previous one 2190 2017-09-08 19:06:18,594 INFO org.apache.sentry.hdfs.DBUpdateForwarder: Newer delta updates are found up to sequence number: 181 2017-09-08 19:06:18,610 INFO org.apache.sentry.hdfs.DBUpdateForwarder: Newer delta updates are found up to sequence number: 2298 2017-09-08 19:06:18,788 INFO org.apache.sentry.hdfs.DBUpdateForwarder: Newer delta updates are found up to sequence number: 181 2017-09-08 19:06:18,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190 2017-09-08 19:06:19,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190 2017-09-08 19:06:19,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190 2017-09-08 19:06:20,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190 2017-09-08 19:06:20,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190 2017-09-08 19:06:21,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190 2017-09-08 19:06:21,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190 2017-09-08 19:06:22,393 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190 2017-09-08 19:06:22,918 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190 2017-09-08 19:06:23,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190 2017-09-08 19:06:23,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190 2017-09-08 19:06:24,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190 2017-09-08 19:06:24,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190 2017-09-08 19:06:25,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190 2017-09-08 19:06:25,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190 2017-09-08 19:06:26,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190 2017-09-08 19:06:26,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190 2017-09-08 19:06:27,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190 2017-09-08 19:06:27,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190