Uploaded image for project: 'Subversion'
  1. Subversion
  2. SVN-1582

need more options for dealing with large files

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: all
    • Fix Version/s: 1.4-consider
    • Component/s: libsvn_delta
    • Labels:
      None

      Description

      Right now, we're not performing so well with large files.
      
      When the client sends a large file to the server, it's generating svndiff data
      by comparing the file against either a text-base or an empty-file.  On a 1GB
      file, this takes a long time;  after the data is fully generated, it's *then*
      marshalled over the network to the server.  (And once the commit succeeds, the
      server then deltifies the *old* version of the file, causing the client to time
      out!  But that's a separate problem, see issue 1573.)
      
      There are a number of problems here:
      
      * First, we're talking about a classic tradeoff between CPU and network
      optimization.  Lots of CPU-time to discover a possibly small commit diff, with
      the reward of a tiny network transmission...  or near-zero-CPU-time to just
      shove the whole fulltext over the network, which is a huge transmission.  Which
      will it be?  Cmpilato points out that this exactly the sort of choice cvs users
      face when they pass the optional -zN flag... perhaps we should grow a similar
      option.
      
      * Second, we're taking a performance hit from neon's API, which is still a
      "pull" interface.  That is, neon insists on pulling data from the caller when
      performing a PUT request.  As a result, we're first spewing all the svndiff
      datat into a tmpfile, *then* invoking the PUT request.  It would be much faster
      if we could start the PUT request and then "push" svndiff data over the network
      as we generate it.  Maybe it's worth adding this feature to neon?
      
      
      * As an aside:  note that there are at least two different ways to "shove
      fulltext" over the network.  One method involves adding an apply_text() function
      to the editor API, to complement our apply_txdelta().  (This is what svn_fs.h
      has currently.)  Another method is to just rapidly fabricate 'add' vdelta windows.
      

        Attachments

        1. 1_apply-text.txt
          32 kB
          C. Michael Pilato
        2. 2_copy-with-digests.txt
          38 kB
          C. Michael Pilato

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              sussman Ben Collins-Sussman
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: