Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
1.0.0
-
None
-
None
Description
With Kafka 1.0.0 we are seeing a problem using the kafka-consumer-groups.sh --reset-offsets option to reset offsets to a specific date/time in our production environment.
We first tried to use the kafka-consumer-groups.sh command with the --reset-offsets option and with option --to-datetime 2018-02-10T00:00:00.000 in our staging environment and it worked correctly. Running the following command changed it to start processing logs from February 12, 2018 (4 days ago) for a topic that had a large lag. We did a dry run to verify before running with the --execute option.
root@mlpstagemon0101a:/# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server NN.NNN.NN.NN:9092 --group logstash-elasticsearch-latest --to-datetime 2018-02-12T00:00:00.000-06:00 --reset-offsets --topic staging-mon01-rg-elasticsearch --execute
We stopped the kafka mirrors that process this topic before resetting the offsets and started the kafka mirrors after rsetting the offsets. We verified that it correctly started processing logs from February 12, 2018.
Then we tried resetting offsets in a production environment for a topic that had a very large lag using option --to-datetime 2018-02-10T00:00:00.000 and it did not work as expected. We stopped the kafka mirrors that process this topic before resetting the offsets and did a dry run to see what the new offsets would be:
root@mlplon0401e:# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server NN.N.NNN.NNN:9092 --group mirror-consumer-ams03-geo-earliest --to-datetime 2018-02-10T00:00:00.000 --reset-offsets --topic prod_in-ams03-geo-ca_access Note: This will not show information about old Zookeeper-based consumers. ^@^@^@^@ TOPIC PARTITION NEW-OFFSET prod_in-ams03-geo-ca_access 52 52084147 prod_in-ams03-geo-ca_access 106 52154199 prod_in-ams03-geo-ca_access 75 52148673 prod_in-ams03-geo-ca_access 61 52130753 prod_in-ams03-geo-ca_access 49 52151667 prod_in-ams03-geo-ca_access 48 52145233 prod_in-ams03-geo-ca_access 27 52092805 prod_in-ams03-geo-ca_access 26 52139644 prod_in-ams03-geo-ca_access 65 52157504 prod_in-ams03-geo-ca_access 105 52166289 prod_in-ams03-geo-ca_access 38 52160464 prod_in-ams03-geo-ca_access 22 52093451 prod_in-ams03-geo-ca_access 4 52151660 prod_in-ams03-geo-ca_access 90 52160296 prod_in-ams03-geo-ca_access 25 52161691 prod_in-ams03-geo-ca_access 13 52145828 prod_in-ams03-geo-ca_access 56 52162867 prod_in-ams03-geo-ca_access 42 52072094 prod_in-ams03-geo-ca_access 7 52069496 prod_in-ams03-geo-ca_access 117 52087078 prod_in-ams03-geo-ca_access 32 52073732 prod_in-ams03-geo-ca_access 102 52082022 prod_in-ams03-geo-ca_access 76 52141018 prod_in-ams03-geo-ca_access 83 52154542 prod_in-ams03-geo-ca_access 72 52095051 prod_in-ams03-geo-ca_access 85 52149907 prod_in-ams03-geo-ca_access 119 52134435 prod_in-ams03-geo-ca_access 113 52159340 prod_in-ams03-geo-ca_access 55 52146597 prod_in-ams03-geo-ca_access 18 52149079 prod_in-ams03-geo-ca_access 35 52149058 prod_in-ams03-geo-ca_access 99 52143277 prod_in-ams03-geo-ca_access 41 52158872 prod_in-ams03-geo-ca_access 112 52083901 prod_in-ams03-geo-ca_access 34 52137932 prod_in-ams03-geo-ca_access 89 52158135 prod_in-ams03-geo-ca_access 40 52144442 prod_in-ams03-geo-ca_access 53 52138400 prod_in-ams03-geo-ca_access 19 52144966 prod_in-ams03-geo-ca_access 44 52166404 prod_in-ams03-geo-ca_access 31 52155685 prod_in-ams03-geo-ca_access 10 52152151 prod_in-ams03-geo-ca_access 98 52145378 prod_in-ams03-geo-ca_access 69 52153436 prod_in-ams03-geo-ca_access 92 52093455 prod_in-ams03-geo-ca_access 5 52156448 prod_in-ams03-geo-ca_access 17 52063697 prod_in-ams03-geo-ca_access 24 52146410 prod_in-ams03-geo-ca_access 28 52156955 prod_in-ams03-geo-ca_access 82 52081194 prod_in-ams03-geo-ca_access 101 52153592 prod_in-ams03-geo-ca_access 58 52153765 prod_in-ams03-geo-ca_access 94 52162827 prod_in-ams03-geo-ca_access 116 52155307 prod_in-ams03-geo-ca_access 77 52072906 prod_in-ams03-geo-ca_access 68 52150283 prod_in-ams03-geo-ca_access 64 52155580 prod_in-ams03-geo-ca_access 109 52159915 prod_in-ams03-geo-ca_access 111 52153547 prod_in-ams03-geo-ca_access 2 52080434 prod_in-ams03-geo-ca_access 74 52148060 prod_in-ams03-geo-ca_access 47 52090187 prod_in-ams03-geo-ca_access 59 52145161 prod_in-ams03-geo-ca_access 1 52151228 prod_in-ams03-geo-ca_access 104 52159932 prod_in-ams03-geo-ca_access 14 52158366 prod_in-ams03-geo-ca_access 86 52153030 prod_in-ams03-geo-ca_access 81 52155507 prod_in-ams03-geo-ca_access 11 52158931 prod_in-ams03-geo-ca_access 100 52154646 prod_in-ams03-geo-ca_access 23 52169713 prod_in-ams03-geo-ca_access 45 52151320 prod_in-ams03-geo-ca_access 6 52157025 prod_in-ams03-geo-ca_access 87 52088913 prod_in-ams03-geo-ca_access 107 52086878 prod_in-ams03-geo-ca_access 16 52146481 prod_in-ams03-geo-ca_access 20 52148093 prod_in-ams03-geo-ca_access 103 52152437 prod_in-ams03-geo-ca_access 9 52158998 prod_in-ams03-geo-ca_access 29 52151112 prod_in-ams03-geo-ca_access 54 52149226 prod_in-ams03-geo-ca_access 36 52139520 prod_in-ams03-geo-ca_access 0 52152902 prod_in-ams03-geo-ca_access 84 52156057 prod_in-ams03-geo-ca_access 78 52155766 prod_in-ams03-geo-ca_access 118 52147043 prod_in-ams03-geo-ca_access 62 52087864 prod_in-ams03-geo-ca_access 108 52154615 prod_in-ams03-geo-ca_access 88 52147651 prod_in-ams03-geo-ca_access 39 52145255 prod_in-ams03-geo-ca_access 115 52166466 prod_in-ams03-geo-ca_access 51 52150136 prod_in-ams03-geo-ca_access 3 52147544 prod_in-ams03-geo-ca_access 73 52157150 prod_in-ams03-geo-ca_access 66 52158808 prod_in-ams03-geo-ca_access 67 52075616 prod_in-ams03-geo-ca_access 97 52086900 prod_in-ams03-geo-ca_access 15 52153225 prod_in-ams03-geo-ca_access 57 52097899 prod_in-ams03-geo-ca_access 8 52159097 prod_in-ams03-geo-ca_access 114 52149412 prod_in-ams03-geo-ca_access 37 52088584 prod_in-ams03-geo-ca_access 30 52153155 prod_in-ams03-geo-ca_access 79 52143773 prod_in-ams03-geo-ca_access 12 52089423 prod_in-ams03-geo-ca_access 21 52148187 prod_in-ams03-geo-ca_access 50 52150414 prod_in-ams03-geo-ca_access 60 52144624 prod_in-ams03-geo-ca_access 33 52149807 prod_in-ams03-geo-ca_access 46 52165008 prod_in-ams03-geo-ca_access 110 52153643 prod_in-ams03-geo-ca_access 95 52154736 prod_in-ams03-geo-ca_access 71 52150981 prod_in-ams03-geo-ca_access 96 52145091 prod_in-ams03-geo-ca_access 93 52149375 prod_in-ams03-geo-ca_access 91 52160207 prod_in-ams03-geo-ca_access 43 52151689 prod_in-ams03-geo-ca_access 63 52164909 prod_in-ams03-geo-ca_access 80 52151810 prod_in-ams03-geo-ca_access 70 52154522
Then we ran the same command with the --execute option to the command to reset the offsets:
root@mlplon0401e:# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server NN.N.NNN.NNN:9092 --group mirror-consumer-ams03-geo-earliest --to-datetime 2018-02-10T00:00:00.000 --reset-offsets --topic prod_in-ams03-geo-ca_access --execute
And started the kafka mirrors for this topic.
However, it didn't work as expected for this topic in our production environment.
The total log lag before resetting the offsets was 1,691,782,349:
root@mlbams0301a:/var/log/kafka# while true; do echo -n `date +'%Y-%m-%d %H:%M:%S'`; echo -n " "; /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server NN.N.NNN.NNN:9092 --describe --group mirror-consumer-ams03-geo-earliest 2>/dev/null | awk '{if ($1 ~ "prod_in-ams03-geo-ca_access") { current_offset+=$3; end_offset+=$4; }} END{print "lag:", end_offset-current_offset;}'; sleep 15; done 2018-02-14 17:31:01 lag: 1691782349
After resetting the offsets, the log lag dropped to 24,330:
2018-02-14 17:31:17 lag: 24330
This was too much of a drop in the log lag. This shows it did not really reset the offsets to start processing logs starting at Feb 10th (4 days ago). We should be seeing a lag of around 37,170,000 if it worked correctly. Instead, it seemed to reset the offsets to start processing logs starting today (Feb 14th).
We also tried resetting the offsets to start processing logs starting at February 8th (after stopping the mirrors) and it also did not work correctly. Again, it seemed like it did a seek to the current date rather than resetting offsets to the date we specified.
root@mlplon0401e:# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server NN.N.NNN.NNN:9092 --group mirror-consumer-ams03-geo-earliest --to-datetime 2018-02-08T00:00:00.000 --reset-offsets --topic prod_in-ams03-geo-ca_access Note: This will not show information about old Zookeeper-based consumers. ^@^@^@ TOPIC PARTITION NEW-OFFSET prod_in-ams03-geo-ca_access 52 52087106 prod_in-ams03-geo-ca_access 106 52156813 prod_in-ams03-geo-ca_access 75 52151755 prod_in-ams03-geo-ca_access 61 52133608 prod_in-ams03-geo-ca_access 49 52154385 prod_in-ams03-geo-ca_access 48 52148247 prod_in-ams03-geo-ca_access 27 52095874 prod_in-ams03-geo-ca_access 26 52142331 prod_in-ams03-geo-ca_access 65 52160181 prod_in-ams03-geo-ca_access 105 52169166 prod_in-ams03-geo-ca_access 38 52163198 prod_in-ams03-geo-ca_access 22 52096231 prod_in-ams03-geo-ca_access 4 52154512 prod_in-ams03-geo-ca_access 90 52163215 prod_in-ams03-geo-ca_access 25 52164527 prod_in-ams03-geo-ca_access 13 52148553 prod_in-ams03-geo-ca_access 56 52165558 prod_in-ams03-geo-ca_access 42 52075104 prod_in-ams03-geo-ca_access 7 52072333 prod_in-ams03-geo-ca_access 117 52090029 prod_in-ams03-geo-ca_access 32 52076469 prod_in-ams03-geo-ca_access 102 52085066 prod_in-ams03-geo-ca_access 76 52143820 prod_in-ams03-geo-ca_access 83 52157217 prod_in-ams03-geo-ca_access 72 52098060 prod_in-ams03-geo-ca_access 85 52152612 prod_in-ams03-geo-ca_access 119 52137225 prod_in-ams03-geo-ca_access 113 52162012 prod_in-ams03-geo-ca_access 55 52149399 prod_in-ams03-geo-ca_access 18 52151998 prod_in-ams03-geo-ca_access 35 52151756 prod_in-ams03-geo-ca_access 99 52146244 prod_in-ams03-geo-ca_access 41 52161567 prod_in-ams03-geo-ca_access 112 52086712 prod_in-ams03-geo-ca_access 34 52140841 prod_in-ams03-geo-ca_access 89 52160809 prod_in-ams03-geo-ca_access 40 52147177 prod_in-ams03-geo-ca_access 53 52141117 prod_in-ams03-geo-ca_access 19 52147752 prod_in-ams03-geo-ca_access 44 52169047 prod_in-ams03-geo-ca_access 31 52158455 prod_in-ams03-geo-ca_access 10 52155007 prod_in-ams03-geo-ca_access 98 52148061 prod_in-ams03-geo-ca_access 69 52156414 prod_in-ams03-geo-ca_access 92 52096106 prod_in-ams03-geo-ca_access 5 52159142 prod_in-ams03-geo-ca_access 17 52066397 prod_in-ams03-geo-ca_access 24 52149433 prod_in-ams03-geo-ca_access 28 52159909 prod_in-ams03-geo-ca_access 82 52083925 prod_in-ams03-geo-ca_access 101 52156350 prod_in-ams03-geo-ca_access 58 52156500 prod_in-ams03-geo-ca_access 94 52165527 prod_in-ams03-geo-ca_access 116 52158005 prod_in-ams03-geo-ca_access 77 52075690 prod_in-ams03-geo-ca_access 68 52152975 prod_in-ams03-geo-ca_access 64 52158277 prod_in-ams03-geo-ca_access 109 52162696 prod_in-ams03-geo-ca_access 111 52156535 prod_in-ams03-geo-ca_access 2 52083131 prod_in-ams03-geo-ca_access 74 52150817 prod_in-ams03-geo-ca_access 47 52092867 prod_in-ams03-geo-ca_access 59 52147851 prod_in-ams03-geo-ca_access 1 52154047 prod_in-ams03-geo-ca_access 104 52162658 prod_in-ams03-geo-ca_access 14 52161087 prod_in-ams03-geo-ca_access 86 52155828 prod_in-ams03-geo-ca_access 81 52158537 prod_in-ams03-geo-ca_access 11 52161597 prod_in-ams03-geo-ca_access 100 52157311 prod_in-ams03-geo-ca_access 23 52172470 prod_in-ams03-geo-ca_access 45 52154257 prod_in-ams03-geo-ca_access 6 52160103 prod_in-ams03-geo-ca_access 87 52091756 prod_in-ams03-geo-ca_access 107 52089608 prod_in-ams03-geo-ca_access 16 52149265 prod_in-ams03-geo-ca_access 20 52150732 prod_in-ams03-geo-ca_access 103 52155276 prod_in-ams03-geo-ca_access 9 52161987 prod_in-ams03-geo-ca_access 29 52153796 prod_in-ams03-geo-ca_access 54 52152245 prod_in-ams03-geo-ca_access 36 52142445 prod_in-ams03-geo-ca_access 0 52155910 prod_in-ams03-geo-ca_access 84 52159052 prod_in-ams03-geo-ca_access 78 52158632 prod_in-ams03-geo-ca_access 118 52149746 prod_in-ams03-geo-ca_access 62 52090560 prod_in-ams03-geo-ca_access 108 52157511 prod_in-ams03-geo-ca_access 88 52150632 prod_in-ams03-geo-ca_access 39 52148150 prod_in-ams03-geo-ca_access 115 52169203 prod_in-ams03-geo-ca_access 51 52153200 prod_in-ams03-geo-ca_access 3 52150514 prod_in-ams03-geo-ca_access 73 52159766 prod_in-ams03-geo-ca_access 66 52161783 prod_in-ams03-geo-ca_access 67 52078371 prod_in-ams03-geo-ca_access 97 52089805 prod_in-ams03-geo-ca_access 15 52156319 prod_in-ams03-geo-ca_access 57 52100913 prod_in-ams03-geo-ca_access 8 52161732 prod_in-ams03-geo-ca_access 114 52152390 prod_in-ams03-geo-ca_access 37 52091579 prod_in-ams03-geo-ca_access 30 52156230 prod_in-ams03-geo-ca_access 79 52146601 prod_in-ams03-geo-ca_access 12 52092444 prod_in-ams03-geo-ca_access 21 52151103 prod_in-ams03-geo-ca_access 50 52153126 prod_in-ams03-geo-ca_access 60 52147546 prod_in-ams03-geo-ca_access 33 52152862 prod_in-ams03-geo-ca_access 46 52167625 prod_in-ams03-geo-ca_access 110 52156369 prod_in-ams03-geo-ca_access 95 52157493 prod_in-ams03-geo-ca_access 71 52153659 prod_in-ams03-geo-ca_access 96 52148024 prod_in-ams03-geo-ca_access 93 52152429 prod_in-ams03-geo-ca_access 91 52162944 prod_in-ams03-geo-ca_access 43 52154651 prod_in-ams03-geo-ca_access 63 52167783 prod_in-ams03-geo-ca_access 80 52154510 prod_in-ams03-geo-ca_access 70 52157348
If there is an invalid committed offset, could that maybe have caused it to not work correctly? Even so, this should not be the result.
We really want to be able to use this tool to reset offsets to a specified date.
Let me know if you need more information.