Uploaded image for project: 'Sentry'
  1. Sentry
  2. SENTRY-1939

Resetting the CounterWait during full snapshot has to be handled across all sentry servers

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.0.0
    • Component/s: Sentry
    • Labels:

      Description

      Currently as part of SENTRY-1929, we reset the CounterWait on the server that does the HMS full snapshot. Ideally this has to be done across all servers otherwise i see this error in the log of other servers.

      2017-09-08 19:05:44,697 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A newer full update is found with image number: 2
      2017-09-08 19:05:44,894 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:05:45,115 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A newer full update is found with image number: 2
      2017-09-08 19:05:45,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:05:45,963 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:05:46,391 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:05:46,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:05:47,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:05:47,929 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:05:48,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:05:48,898 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:05:49,391 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:05:49,897 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:05:50,452 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:05:50,506 INFO org.apache.sentry.hdfs.HMSPathsDumper: Paths Dump created. 101104 total path strings, 579426 duplicate strings found, compacted to 1077 unique strings.
      2017-09-08 19:05:50,535 INFO org.apache.sentry.hdfs.HMSPathsDumper: Paths Dump created. 101104 total path strings, 579426 duplicate strings found, compacted to 1077 unique strings.
      2017-09-08 19:05:50,947 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:05:51,904 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:05:51,932 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:05:52,420 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:05:52,923 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:05:53,417 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:05:53,922 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:05:54,414 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:05:54,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:05:55,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:05:55,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:05:56,392 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:05:56,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:05:57,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:05:57,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:05:58,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:05:58,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:05:59,393 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:05:59,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:00,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:00,891 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:01,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:01,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:02,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:02,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:03,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:03,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:04,393 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:04,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:05,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:05,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:06,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:06,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:07,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:07,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:08,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:08,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:09,422 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:09,892 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:10,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:10,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:11,391 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:11,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:12,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:12,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:13,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:13,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:14,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:14,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:15,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:15,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:16,392 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:16,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:17,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
      2017-09-08 19:06:17,553 INFO org.apache.sentry.hdfs.DBUpdateForwarder: Newer delta updates are found up to sequence number: 2287
      2017-09-08 19:06:17,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 6 is smaller then the previous one 2190
      2017-09-08 19:06:18,066 INFO org.apache.sentry.hdfs.DBUpdateForwarder: Newer delta updates are found up to sequence number: 177
      2017-09-08 19:06:18,081 INFO org.apache.sentry.hdfs.DBUpdateForwarder: Newer delta updates are found up to sequence number: 2294
      2017-09-08 19:06:18,274 INFO org.apache.sentry.hdfs.DBUpdateForwarder: Newer delta updates are found up to sequence number: 177
      2017-09-08 19:06:18,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 13 is smaller then the previous one 2190
      2017-09-08 19:06:18,594 INFO org.apache.sentry.hdfs.DBUpdateForwarder: Newer delta updates are found up to sequence number: 181
      2017-09-08 19:06:18,610 INFO org.apache.sentry.hdfs.DBUpdateForwarder: Newer delta updates are found up to sequence number: 2298
      2017-09-08 19:06:18,788 INFO org.apache.sentry.hdfs.DBUpdateForwarder: Newer delta updates are found up to sequence number: 181
      2017-09-08 19:06:18,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190
      2017-09-08 19:06:19,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190
      2017-09-08 19:06:19,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190
      2017-09-08 19:06:20,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190
      2017-09-08 19:06:20,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190
      2017-09-08 19:06:21,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190
      2017-09-08 19:06:21,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190
      2017-09-08 19:06:22,393 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190
      2017-09-08 19:06:22,918 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190
      2017-09-08 19:06:23,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190
      2017-09-08 19:06:23,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190
      2017-09-08 19:06:24,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190
      2017-09-08 19:06:24,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190
      2017-09-08 19:06:25,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190
      2017-09-08 19:06:25,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190
      2017-09-08 19:06:26,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190
      2017-09-08 19:06:26,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190
      2017-09-08 19:06:27,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190
      2017-09-08 19:06:27,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 16 is smaller then the previous one 2190
      

        Attachments

        1. SENTRY-1939.01-master.patch
          3 kB
          Vamsee K. Yarlagadda
        2. SENTRY-1939.02-master.patch
          4 kB
          Alex Kolbasov

          Activity

            People

            • Assignee:
              vamsee Vamsee K. Yarlagadda
              Reporter:
              vamsee Vamsee K. Yarlagadda
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: