[SVN-1899] svndumpfilter holds an entire revision of dumpstream in memory

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: all
• Fix Version/s: 1.2.0
• Component/s: svndumpfilter
• Assignee: Unassigned
• Reporter: Subversion Importer
• Created: 27/May/04 07:00
• Updated: 27/Oct/18 16:41
• Resolved: 10/Mar/09 15:22

Description

Hi

I'm trying to use svndumpfilter (which doesn't seem to be listed as a 
subcomponent btw.) to extract parts of my repository to load into a
separate repository on another machine. It is dying however with the following
strace:

read(0, "\377\377\377\377\377\377\377\377\377\377\377\377\377\377"..., 69632) = 
69632
mmap2(NULL, 1073750016, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) 
= -1 ENOMEM (Cannot allocate memory)
brk(0)                                  = 0x8729000
brk(0x4872a000)                         = 0x8729000
mmap2(NULL, 1073881088, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) 
= -1 ENOMEM (Cannot allocate memory)
mmap2(NULL, 2097152, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) 
= 0x26bd7000
munmap(0x26bd7000, 167936)              = 0
munmap(0x26d00000, 880640)              = 0
mprotect(0x26c00000, 135168, PROT_READ|PROT_WRITE) = 0
mmap2(NULL, 1073750016, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) 
= -1 ENOMEM (Cannot allocate memory)
rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0
tgkill(1278, 1278, SIGABRT)             = 0
--- SIGABRT (Aborted) @ 0 (0) ---
+++ killed by SIGABRT +++

It would appear to be attempting to mmap just over 1Gig of memory on stdin. I'm
running it as follows:

[root@svn root]# strace svndumpfilter include projects/2g include jonathan/2g 
include livia/bcu include livia/vb < 2gstuff > 2gstuff.filtered 2>run.out

2gstuff is a dump of the original repository.

[root@svn root]# ls -l /proc/1306/fd/
total 3
lr-x------  1 root root 64 May 27 08:46 0 -> /root/2gstuff
l-wx------  1 root root 64 May 27 08:46 1 -> /root/2gstuff.filtered
l-wx------  1 root root 64 May 27 08:46 2 -> /root/run.out

My system is running Fedora core 2 using 1.04 rpms supplied in the dev/testing
repositories. It is dumping the 511th revision of the repository when it
dies which contained three 256M files and one 128M file in the commit. Yes
that really is 896Megs in a single commit which is probably why the memory
usage spikes up over the 1Gig mark ... the machine has about a gig of memory
and a fair amount of swap:
[root@svn root]# free
             total       used       free     shared    buffers     cached
Mem:       1034900     166308     868592          0       9804     118976
-/+ buffers/cache:      37528     997372
Swap:      1052248      24528    1027720

I'd be happy to supply any additional information someone may need on
request.

Thank you!

--Craig

Original issue reported by schlenterc