[SVN-4819] svnadmin fails to create empty repository on CSV storage (Windows Cluster Shared Volume)

• Type: Bug
• Status: Reopened
• Priority: Blocker
• Resolution: Unresolved
• Affects Version/s: 1.12.x
• Component/s: svnadmin

Environment

Windows Server 2019 Standard

Windows Failover Cluster

Cluster Shared Volume on top of NTFS

• Assignee: Unassigned
• Reporter: Daniel Watzinger
• Created: 26/Jun/19 10:50
• Updated: 27/Sep/19 14:27

Description

svnadmin reproducibly fails to initialize an empty repository on a cluster shared volume in a Windows Failover Cluster environment on Windows Server 2019.  The initial call seems to be successful.

svnadmin create foobar

A subsequent 

svnadmin verify foobar

fails with

* Verifying metadata at revision 0 ...
* Error verifying repository metadata.
svnadmin: E160004: P2L offset 6b must be larger than L2P offset 6b in r0 footer

Further investigation revealed that the L2P index never gets written to the revision file. I have attached the relevant file of revision 0. 

Index data seems to be appended to the revision file in subversion/libsvn_fs_fs/transaction.c by a call to svn_fs_fs__add_index_data:

...

/* Append the actual index data to the pack file. */
l2p_offset = 0;
SVN_ERR(svn_io_file_seek(file, APR_END, &l2p_offset, pool));
SVN_ERR(svn_fs_fs__l2p_index_append(&l2p_checksum, fs, file, l2p_proto_index, revision, pool, pool));
	
p2l_offset = 0;
SVN_ERR(svn_io_file_seek(file, APR_END, &p2l_offset, pool));
SVN_ERR(svn_fs_fs__p2l_index_append(&p2l_checksum, fs, file, p2l_proto_index, revision, pool, pool));

...

svn_io_file_seek queries file metadata to acquire the required offset to write the next chunk of data. However, both calls yield an identical offset equal to the initial length of the revision file. Thus, the L2P index is overwritten by writing the P2L index data. I was able to confirm this by monitoring svnadmin using Procmon (see attached Procmon.csv) on both local and cluster shared volumes. I strongly suspect that file metadata is not flushed to the disk fast enough before the second call to svn_io_file_seek. As far as i recall (in the case of Windows), filesystem metadata is always cached. To ensure storage of metadata to disk, the file must either be flushed or be opened with FILE_FLAG_WRITE_THROUGH.