[HBASE-17871] scan#setBatch(int) call leads wrong result of VerifyReplication

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.4.0, 2.0.0
• Fix Version/s: 1.4.0, 2.0.0
• Hadoop Flags: Reviewed
• Assignee: Tomu Tsuruhara
• Reporter: Tomu Tsuruhara
• Created: 04/Apr/17 02:03
• Updated: 01/Aug/18 06:22
• Resolved: 06/Apr/17 14:01

Description

VerifyReplication tool printed weird logs.

2017-04-03 23:30:50,252 ERROR [main] org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication: CONTENT_DIFFERENT_ROWS, rowkey=a00001001930000
2017-04-03 23:30:50,280 ERROR [main] org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication: ONLY_IN_PEER_TABLE_ROWS, rowkey=a00001001930000
2017-04-03 23:30:50,387 ERROR [main] org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication: CONTENT_DIFFERENT_ROWS, rowkey=a00001003850000
2017-04-03 23:30:50,414 ERROR [main] org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication: ONLY_IN_PEER_TABLE_ROWS, rowkey=a00001003850000
2017-04-03 23:30:50,480 ERROR [main] org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication: CONTENT_DIFFERENT_ROWS, rowkey=a00001005320000
2017-04-03 23:30:50,508 ERROR [main] org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication: ONLY_IN_PEER_TABLE_ROWS, rowkey=a00001005320000

Here, each bad rows were marked as both CONTENT_DIFFERENT_ROWS and ONLY_IN_PEER_TABLE_ROWS.
This should never happen so I took a look at code and found scan.setBatch call.

    @Override
    public void map(ImmutableBytesWritable row, final Result value,
                    Context context)
        throws IOException {
      if (replicatedScanner == null) {
	    ...
        final Scan scan = new Scan();
        scan.setBatch(batch);

As stated in HBASE-16376, scan#setBatch(int) call implicitly allows scan results to be partial.

Since VerifyReplication is assuming each scanner.next() call returns entire row,
partial results break compare logic.

We should avoid setBatch call here.
Thanks to RPC chunking (explained in this blog https://blogs.apache.org/hbase/entry/scan_improvements_in_hbase_1),
it's safe and acceptable I think.