[HDFS-13392] Incorrect length in Truncate CloseEvents

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0
• Component/s: namenode
• Assignee: Unassigned
• Reporter: David Tucker
• Created: 03/Apr/18 18:26
• Updated: 03/Apr/18 18:26

Description

Under stress (multiple clients truncating separate non-empty files in half simultaneously), the CloseEvent triggered by a Truncate RPC may contain an incorrect length. We're able to reproduce this reliably ~20% of the time (our tests are somewhat randomized/fuzzy).

For example, given this Truncate request:

Request:
  truncate {
    src: "/chai_test65c9a2a0-1188-439d-92e2-96a81c14a266-\357\254\200\357\272\217\357\255\217\343\203\276\324\262\342\204\200\342\213\251/chai_testbd968366-0016-4462-ac12-e48e0487bebd-\340\270\215\334\200\311\226\342\202\242\343\202\236\340\256\205\357\272\217/chai_testb5b155e8-b331-4f67-bdfa-546f82128b5d-\312\254\340\272\201\343\202\242\306\220\340\244\205\342\202\242\343\204\270a\334\240\337\213\340\244\240\343\200\243\342\202\243\343\203\276\313\225\346\206\250"
    newLength: 2003855
    clientName: "\341\264\275\327\220\343\203\250\333\263\343\220\205\357\254\227\340\270\201\340\245\251\306\225\341\203\265\334\220\342\202\243\343\204\206!A\343\206\215\357\254\201\340\273\223\347\224\260"
  }
  Block Size: 1048576B
  Old length: 4007711B (3.82205104828 blocks)
  Truncation: 2003856B (1.91102600098 blocks)
  New length: 2003855B (1.9110250473 blocks)
Response:
  result: true

We see these INotify events:

TruncateEvent {
    path: /chai_test65c9a2a0-1188-439d-92e2-96a81c14a266-ffﺏﭏヾԲ℀⋩/chai_testbd968366-0016-4462-ac12-e48e0487bebd-ญ܀ɖ₢ゞஅﺏ/chai_testb5b155e8-b331-4f67-bdfa-546f82128b5d-ʬກアƐअ₢ㄸaܠߋठ〣₣ヾ˕憨
    length: 2003855
    timestamp: 1522716573143
}

CloseEvent {
    path: /chai_test65c9a2a0-1188-439d-92e2-96a81c14a266-ffﺏﭏヾԲ℀⋩/chai_testbd968366-0016-4462-ac12-e48e0487bebd-ญ܀ɖ₢ゞஅﺏ/chai_testb5b155e8-b331-4f67-bdfa-546f82128b5d-ʬກアƐअ₢ㄸaܠߋठ〣₣ヾ˕憨
    length: -2
    timestamp: 1522716575723
}

-2 is not the only number that shows up as the length, 9223372036854775807 is common too. These are detected by Python 2 tests, and the latter is sys.maxint.