Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: all
    • Fix Version/s: 1.8.0
    • Component/s: svnrdump
    • Labels:
      None

      Description

      svnrdump expects the dump editor drive to happen in a particular order, it
      stores data in the edit baton rather than in file/directory batons.  When using
      serf the editor can be driven in an order that causes svnrdump to produce
      invalid output.
      
      This has been observed in the regression tests, but is easier to reproduce on
      larger data sets, see the thread:
      
      http://mail-archives.apache.org/mod_mbox/subversion-dev/201202.mbox/%3CCAB84uBUryfDpoKdROEWQUbVeVWu=QRURjYsS=+-auSFEMt-fNw@mail.gmail.com%3E
      

        Activity

        Hide
        cmpilato C. Michael Pilato added a comment -

        I did some work on an 'issue-4116-dev' branch to make svnrdump more tolerant of
        the crazy ways in which ra_serf could drive the editor.  I just retintegrated
        that branch back into trunk.
           Sending        .
           Sending        subversion/svnrdump/dump_editor.c
           Transmitting file data .
           Committed revision 1425042.
        
        I'm going to bravely call this FIXED.
        

        Show
        cmpilato C. Michael Pilato added a comment - I did some work on an 'issue-4116-dev' branch to make svnrdump more tolerant of the crazy ways in which ra_serf could drive the editor. I just retintegrated that branch back into trunk. Sending . Sending subversion/svnrdump/dump_editor.c Transmitting file data . Committed revision 1425042. I'm going to bravely call this FIXED.
        Hide
        cmpilato C. Michael Pilato added a comment -

        Taught svnrdump to tell ra_serf to prefer bulk-update mode and to limit its
        number of auxiliary connections to just a single one:
           Sending        subversion/libsvn_ra_serf/update.c
           Sending        subversion/svnrdump/svnrdump.c
           Transmitting file data ..
           Committed revision 1423162.
        
        The first change guarantees a valid editor v1 drive, which fixes this problem
        for svnrdump.
        
        But servers *can* be configured to disallow bulk updates.  So the second change
        is supposed to ensure that at least all the GETs and PROPFINDs happen is a
        reasonably well-ordered fashion.  But at the moment, it appears to be not quite
        enough to make svnrdump succeed in its task.  Still debugging.
        

        Show
        cmpilato C. Michael Pilato added a comment - Taught svnrdump to tell ra_serf to prefer bulk-update mode and to limit its number of auxiliary connections to just a single one: Sending subversion/libsvn_ra_serf/update.c Sending subversion/svnrdump/svnrdump.c Transmitting file data .. Committed revision 1423162. The first change guarantees a valid editor v1 drive, which fixes this problem for svnrdump. But servers *can* be configured to disallow bulk updates. So the second change is supposed to ensure that at least all the GETs and PROPFINDs happen is a reasonably well-ordered fashion. But at the moment, it appears to be not quite enough to make svnrdump succeed in its task. Still debugging.
        Hide
        cmpilato C. Michael Pilato added a comment -

        I recently taught libsvn_ra_serf how to request and handle the "send-all"
        (non-skelta) style of update REPORT response.  A solution to this problem could
        involve simply finding a way to teach libsvn_ra_serf to use that approach for
        remote dumps (while still preferring the less-syncronous approach for other
        update-like operations).
        

        Show
        cmpilato C. Michael Pilato added a comment - I recently taught libsvn_ra_serf how to request and handle the "send-all" (non-skelta) style of update REPORT response. A solution to this problem could involve simply finding a way to teach libsvn_ra_serf to use that approach for remote dumps (while still preferring the less-syncronous approach for other update-like operations).
        Hide
        cmpilato C. Michael Pilato added a comment -

        If I understand correctly, this problem can occur only during the transmission
        of the first revision in a non-incremental 'svnrdump dump' from an
        HTTP(s)-accessible source repository.  That's the only revision that uses the RA
        layer's "update" mechanics, which for libsvn_ra_serf is known to drive editors
        in violation of their documented ordering.  Subsequent revisions use the
        "replay" mechanics which are not currently susceptible to editor drive ordering
        violations.
        

        Show
        cmpilato C. Michael Pilato added a comment - If I understand correctly, this problem can occur only during the transmission of the first revision in a non-incremental 'svnrdump dump' from an HTTP(s)-accessible source repository. That's the only revision that uses the RA layer's "update" mechanics, which for libsvn_ra_serf is known to drive editors in violation of their documented ordering. Subsequent revisions use the "replay" mechanics which are not currently susceptible to editor drive ordering violations.
        Hide
        hwright Hyrum Wright added a comment -

        I attempted to do some work along these lines in the fix-rdump-editor branch, but was unsuccessful.  
        
        That experience in attempting to rewrite svnrdump to use proper file and directory batons is that it 
        probably won't happen with Ev1.  The dump file format is much less amenable to streamy generation from 
        the piecemeal callback paradigm that Ev1 provides, and the right (and probably easiest) thing to do at this 
        point is just implement svnrdump using Ev2.
        
        Hopefully replay grows support for Ev2 soon-ish, but I am concerned that this will become a blocking issue 
        for 1.8, as additional work will need to happen for the RA layers to support Ev2.
        

        Show
        hwright Hyrum Wright added a comment - I attempted to do some work along these lines in the fix-rdump-editor branch, but was unsuccessful. That experience in attempting to rewrite svnrdump to use proper file and directory batons is that it probably won't happen with Ev1. The dump file format is much less amenable to streamy generation from the piecemeal callback paradigm that Ev1 provides, and the right (and probably easiest) thing to do at this point is just implement svnrdump using Ev2. Hopefully replay grows support for Ev2 soon-ish, but I am concerned that this will become a blocking issue for 1.8, as additional work will need to happen for the RA layers to support Ev2.

          People

          • Assignee:
            cmpilato C. Michael Pilato
            Reporter:
            philipm Philip Martin
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development