[ROCKETMQ-329] Some messages was skipped by consumerOffset when rocketMQ consuming a large number of messages.

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 4.1.0-incubating
• Component/s: rocketmq-broker, rocketmq-client
• Environment: centos 7 ; one mqnamesvr, one mqbroker, with default settings.
• Assignee: Xinyu Zhou
• Reporter: sypeng
• Created: 10/Dec/17 02:32
• Updated: 20/Dec/17 00:48

Description

I started one mqnamesrv and one mqbroker process on centos7 with default settings:
nohup bin/mqname &
nohup bin/mqbroker -n localhost:9876 &
and produced a large number of messages with 5 threads continuously(total 1000000 messages)，found that each consumerOffset skipped about 4500 messages when persist (as shown in the attached picture), which cause them lost while consumer consuming. The consumer lost about 16000 messages eventually.

The speed of message production is about 5000 per second, and the speed of message consumption is 5 per second (the consumer's logic is sleep 200 millseconds and return consume success.)

The info "the message queue not be able to consume, because it's dropped" and the warning "the pull request offset illegal" are printed in the mq's log when this condition happened. (as shown in the attached picture)