Uploaded image for project: 'Subversion'
  1. Subversion
  2. SVN-901

show progress information during commit

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: all
    • Fix Version/s: unscheduled
    • Component/s: unknown
    • Labels:

      Description

      large commits are showing a rather dull serie of dots (one dot per
      file I heard), but are not giving any information about percentage
      being done, or amount of data to transmit.
      
      It would be great if there was a more precise progress indicator, as a
      percentage of the total size to transmit, or the total number of files
      to transmit
      

      Original issue reported by timot

        Issue Links

          Activity

          Hide
          subversion-importer Subversion Importer added a comment -

          Posted by Stefan Küng on Fri Aug 8 10:06:21 -0700 2003,
          
          The message he is referring to is:
          
          http://subversion.tigris.org/ds/viewMessage.do?dsMessageId=330910&dsForumId=462
          
          
          

          Original comment by e_wong

          Show
          subversion-importer Subversion Importer added a comment - Posted by Stefan Küng on Fri Aug 8 10:06:21 -0700 2003, The message he is referring to is: http://subversion.tigris.org/ds/viewMessage.do?dsMessageId=330910&dsForumId=462 Original comment by e_wong
          Hide
          brane Branko Čibej added a comment -

          *** Issue 1264 has been marked as a duplicate of this issue. ***
          

          Show
          brane Branko Čibej added a comment - *** Issue 1264 has been marked as a duplicate of this issue. ***
          Hide
          steveking Stefan Küng added a comment -

          Even though *some* progress information is available, it's not enough.
          We now have information about how many bytes are transferred, and sometimes even
          the progress of on data chunk transferred.
          But we still don't have the information on e.g. how many files/chunks have to be
          transferred and how many already are transferred.
          
          For example: if you do an 'svn up' and there's a lot that has changed, it can
          take a while. But there is no way of telling the user how far along the update
          is - it could be seconds, minutes or hours until the update will complete - and
          no way of telling the user that.
          

          Show
          steveking Stefan Küng added a comment - Even though *some* progress information is available, it's not enough. We now have information about how many bytes are transferred, and sometimes even the progress of on data chunk transferred. But we still don't have the information on e.g. how many files/chunks have to be transferred and how many already are transferred. For example: if you do an 'svn up' and there's a lot that has changed, it can take a while. But there is no way of telling the user how far along the update is - it could be seconds, minutes or hours until the update will complete - and no way of telling the user that.
          Hide
          ehuelsmann Erik Huelsmann added a comment -

          Progress information has been available for some time now through ra_dav (now
          renamed ra_neon).
          
          I choose not to do ra_local, because it was (1) really hard and (2) we recommend
          not to use ra_local in production environments.
          
          

          Show
          ehuelsmann Erik Huelsmann added a comment - Progress information has been available for some time now through ra_dav (now renamed ra_neon). I choose not to do ra_local, because it was (1) really hard and (2) we recommend not to use ra_local in production environments.
          Hide
          ehuelsmann Erik Huelsmann added a comment -

          r22340 adds progress information to ra_svn. 2 down, 2 to go.
          

          Show
          ehuelsmann Erik Huelsmann added a comment - r22340 adds progress information to ra_svn. 2 down, 2 to go.
          Hide
          subversion-importer Subversion Importer added a comment -

          r15948 added a progress callback mechanism to the RA layer which is being 
          called by ra_dav.
          

          Original comment by lundblad

          Show
          subversion-importer Subversion Importer added a comment - r15948 added a progress callback mechanism to the RA layer which is being called by ra_dav. Original comment by lundblad
          Hide
          steveking Stefan Küng added a comment -

          Add tortoisesvn keyword. Reason:
          Showing a progress indicator during operations that take some time is expected
          by users using UI's.
          

          Show
          steveking Stefan Küng added a comment - Add tortoisesvn keyword. Reason: Showing a progress indicator during operations that take some time is expected by users using UI's.
          Hide
          steveking Stefan Küng added a comment -

          Since I don't have any luck on the mailing list 
          (http://subversion.tigris.org/servlets/ReadMsg?list=dev&msgNo=42729 - 
          not one single answer yet) I'll add my comments directly to the issue 
          here:
          
          Both issues 1004 and 901 are about the same
          request: more detailed progress information for
          svn commands. Issue 901 got closed, but I think
          it was closed too early. The initial request wasn't
          for a cancellation function (to which it was later
          changed to).
          
          Since this feature is very important especially
          for GUI clients I'd like to make a suggestion:
          
          add a new callback function to svn_client_ctx_t
          
          svn_progress_notify_func_t progress_func;
          void *progress_baton;
          
          the progess function would look like this:
          
          typedef void (*svn_progress_notify_func_t) 
              (void *baton,
              const char * url1,
              const char * url2,
              svn_progress_val_t item_progress,
              svn_progress_val_t item_total,
              svn_progress_unit,
              svn_progress_val_t overall_progress,
              svn_progress_val_t overall_total,
              svn_progress_unit);
          
          with svn_progress_val_t being an unsigned long and
          svn_progress_unit being an enum with members
          like "bytes", "nothing" (for just displaying percent values),
          "files", etc.
          
          A little explanation for the callback parameters:
          url1 and url2 would be either filepaths or urls, while
          url1 is the source and url2 the destination. Clients would
          use that information just for showing the user on what
          items the function is currently working on. If not used,
          they have to be NULL so the client knows that there's
          no information.
          
          item_progress and item_total would be the progress
          of a single item (for operations which work on
          multiple files/urls). item_progress indicates e.g.
          how many bytes of a file are already transferred, and
          item_total would be the total amount of bytes the file
          has. If for example the total isn't known then item_total
          would be null. That way a client still can display 
          some information (like "14kB transferred").
          
          overall_progress and overall_total is the progress pair
          for the whole operation. It doesn't need to be related
          to the values used in item_progress/item_total but can
          be completely independent values. E.g. if the item values
          specify bytes of a file, the overall values could be just
          the number of files - it doesn't need to be the total bytes
          of all files. Here too, if the values are not know by
          subversion keep them null.
          
          The first step would be to extend the svn_client_ctx_t
          structure. No change in the main functions need to be
          done in this first step.
          
          Then step by step the different svn functions
          could start using that callback if it's usefull.
          
          That way the feature could be implemented in small
          steps, without the risk of breaking too much.
          Also, the API change would be done before beta.
          
          I would like to implement the first step (extending svn_client_ctx_t) and 
          then starting on the easier functions. But I really like some feedback 
          first from all developers here - I'm not keen on working just to get my 
          patches rejected again.
          

          Show
          steveking Stefan Küng added a comment - Since I don't have any luck on the mailing list (http://subversion.tigris.org/servlets/ReadMsg?list=dev&msgNo=42729 - not one single answer yet) I'll add my comments directly to the issue here: Both issues 1004 and 901 are about the same request: more detailed progress information for svn commands. Issue 901 got closed, but I think it was closed too early. The initial request wasn't for a cancellation function (to which it was later changed to). Since this feature is very important especially for GUI clients I'd like to make a suggestion: add a new callback function to svn_client_ctx_t svn_progress_notify_func_t progress_func; void *progress_baton; the progess function would look like this: typedef void (*svn_progress_notify_func_t) (void *baton, const char * url1, const char * url2, svn_progress_val_t item_progress, svn_progress_val_t item_total, svn_progress_unit, svn_progress_val_t overall_progress, svn_progress_val_t overall_total, svn_progress_unit); with svn_progress_val_t being an unsigned long and svn_progress_unit being an enum with members like "bytes", "nothing" (for just displaying percent values), "files", etc. A little explanation for the callback parameters: url1 and url2 would be either filepaths or urls, while url1 is the source and url2 the destination. Clients would use that information just for showing the user on what items the function is currently working on. If not used, they have to be NULL so the client knows that there's no information. item_progress and item_total would be the progress of a single item (for operations which work on multiple files/urls). item_progress indicates e.g. how many bytes of a file are already transferred, and item_total would be the total amount of bytes the file has. If for example the total isn't known then item_total would be null. That way a client still can display some information (like "14kB transferred"). overall_progress and overall_total is the progress pair for the whole operation. It doesn't need to be related to the values used in item_progress/item_total but can be completely independent values. E.g. if the item values specify bytes of a file, the overall values could be just the number of files - it doesn't need to be the total bytes of all files. Here too, if the values are not know by subversion keep them null. The first step would be to extend the svn_client_ctx_t structure. No change in the main functions need to be done in this first step. Then step by step the different svn functions could start using that callback if it's usefull. That way the feature could be implemented in small steps, without the risk of breaking too much. Also, the API change would be done before beta. I would like to implement the first step (extending svn_client_ctx_t) and then starting on the easier functions. But I really like some feedback first from all developers here - I'm not keen on working just to get my patches rejected again.
          Hide
          gstein Greg Stein added a comment -

          Well, that temporary file is a workaround. Ideally, we'd push data
          into Neon (as it comes out of the delta), and it would deliver that to
          the server using the chunked transfer coding. But Neon doesn't support
          that model (yet).
          
          It is easy to imagine that in the future, we wouldn't have the file
          length. That isn't to say the progress API should avoid having that
          concept, but that the API should be able to deal with unknown file
          lengths regardless.
          

          Show
          gstein Greg Stein added a comment - Well, that temporary file is a workaround. Ideally, we'd push data into Neon (as it comes out of the delta), and it would deliver that to the server using the chunked transfer coding. But Neon doesn't support that model (yet). It is easy to imagine that in the future, we wouldn't have the file length. That isn't to say the progress API should avoid having that concept, but that the API should be able to deal with unknown file lengths regardless.
          Hide
          jorton Joe Orton added a comment -

          Actually ra_dav can know exactly how much data is about to be sent
          per-file in a commit, since it's spooled to a local temp file before
          being sent.
          

          Show
          jorton Joe Orton added a comment - Actually ra_dav can know exactly how much data is about to be sent per-file in a commit, since it's spooled to a local temp file before being sent.
          Hide
          subversion-importer Subversion Importer added a comment -

          well .. we don't know how much data we have to transmit for each file
          .. but we know how many files we transmit, right? So instead of dots,
          we could count over the total number of files to transmit and show a
          progress there.
          
          DeltaV knows the data to transmit file by file, so we could also have
          a sub-progress bar for each file. (That kind of feature is probably
          more interesting for the GUI clients)
          

          Original comment by timot

          Show
          subversion-importer Subversion Importer added a comment - well .. we don't know how much data we have to transmit for each file .. but we know how many files we transmit, right? So instead of dots, we could count over the total number of files to transmit and show a progress there. DeltaV knows the data to transmit file by file, so we could also have a sub-progress bar for each file. (That kind of feature is probably more interesting for the GUI clients) Original comment by timot
          Hide
          cmpilato C. Michael Pilato added a comment -

          NOTE: the previous patch is an example only.  We do NOT printf() from
          our core libraries!
          

          Show
          cmpilato C. Michael Pilato added a comment - NOTE: the previous patch is an example only. We do NOT printf() from our core libraries!
          Hide
          sussman Ben Collins-Sussman added a comment -

          A guy I met at apachecon actually had a local mod to svn, which
          increased the granularity of update feedback.  His little patch
          (below) hooks into a neon callback, so that you can see some kind of
          progress when receiving a huge file.  We could also use this technique
          when transmitting a huge file.  It's true what Mike says:  we don't
          know exactly how much data will be sent or received.  But we can do
          better than only print a dot or filename after each complete file.  It
          would be nice to see feedback every 50K in either direction, or something.
          
          -----------------
          
          Hi,
          
          here is the patch we just talked about (ignore the #ifndef nogr, they are
          just markers for me)
          
          It's in subversion/libsvn_ra_dav and against revision 3578
          
          Gerald
          
          
          --- fetch.c.orig Wed Nov 20 06:41:29 2002
          +++ fetch.c Fri Nov  8 09:24:45 2002
          @@ -600,6 +600,7 @@
                 return;
               }
          
          +
             if (!cgc->checked_type)
               {
          
          @@ -671,6 +672,16 @@
               }
           }
          
          +
          +#ifndef nogr
          +static void fetch_file_progress(void *userdata, off_t progress, off_t
          total)
          +{
          +printf ("%d/%d\r", progress, total) ;
          +fflush(stdout) ;
          +}
          +
          +#endif
          +
           static svn_error_t *simple_fetch_file(ne_session *sess,
                                                 const char *url,
                                                 const char *relpath,
          @@ -684,6 +695,7 @@
           {
             file_read_ctx_t frc = { 0 };
          
          +
             SVN_ERR_W( (*editor->apply_textdelta)(file_baton,
                                                   &frc.handler,
                                                   &frc.handler_baton),
          @@ -702,12 +714,21 @@
          
             frc.pool = pool;
          
          +#ifndef nogr
          +  ne_set_progress (sess, fetch_file_progress, NULL) ;
          +#endif
          +
          +
             SVN_ERR( custom_get_request(sess, url, relpath,
                                         fetch_file_reader, &frc,
                                         get_rev, get_wc_prop, cb_baton, pool) );
          
             /* close the handler, since the file reading completed successfully. */
             SVN_ERR( (*frc.handler)(NULL, frc.handler_baton) );
          +
          +#ifndef nogr
          +  ne_set_progress (sess, NULL, NULL) ;
          +#endif
          
             return SVN_NO_ERROR;
           }
          
          -------------------------------------------------------------
          Gerald Richter    ecos electronic communication services gmbh
          Internetconnect * Webserver/-design/-datenbanken * Consulting
          
          Post:       Tulpenstrasse 5         D-55276 Dienheim b. Mainz
          E-Mail:     richter@ecos.de         Voice:    +49 6133 925131
          WWW:        http://www.ecos.de      Fax:      +49 6133 925152
          -------------------------------------------------------------
          

          Show
          sussman Ben Collins-Sussman added a comment - A guy I met at apachecon actually had a local mod to svn, which increased the granularity of update feedback. His little patch (below) hooks into a neon callback, so that you can see some kind of progress when receiving a huge file. We could also use this technique when transmitting a huge file. It's true what Mike says: we don't know exactly how much data will be sent or received. But we can do better than only print a dot or filename after each complete file. It would be nice to see feedback every 50K in either direction, or something. ----------------- Hi, here is the patch we just talked about (ignore the #ifndef nogr, they are just markers for me) It's in subversion/libsvn_ra_dav and against revision 3578 Gerald --- fetch.c.orig Wed Nov 20 06:41:29 2002 +++ fetch.c Fri Nov 8 09:24:45 2002 @@ -600,6 +600,7 @@ return; } + if (!cgc->checked_type) { @@ -671,6 +672,16 @@ } } + +#ifndef nogr +static void fetch_file_progress(void *userdata, off_t progress, off_t total) +{ +printf ("%d/%d\r", progress, total) ; +fflush(stdout) ; +} + +#endif + static svn_error_t *simple_fetch_file(ne_session *sess, const char *url, const char *relpath, @@ -684,6 +695,7 @@ { file_read_ctx_t frc = { 0 }; + SVN_ERR_W( (*editor->apply_textdelta)(file_baton, &frc.handler, &frc.handler_baton), @@ -702,12 +714,21 @@ frc.pool = pool; +#ifndef nogr + ne_set_progress (sess, fetch_file_progress, NULL) ; +#endif + + SVN_ERR( custom_get_request(sess, url, relpath, fetch_file_reader, &frc, get_rev, get_wc_prop, cb_baton, pool) ); /* close the handler, since the file reading completed successfully. */ SVN_ERR( (*frc.handler)(NULL, frc.handler_baton) ); + +#ifndef nogr + ne_set_progress (sess, NULL, NULL) ; +#endif return SVN_NO_ERROR; } ------------------------------------------------------------- Gerald Richter ecos electronic communication services gmbh Internetconnect * Webserver/-design/-datenbanken * Consulting Post: Tulpenstrasse 5 D-55276 Dienheim b. Mainz E-Mail: richter@ecos.de Voice: +49 6133 925131 WWW: http://www.ecos.de Fax: +49 6133 925152 -------------------------------------------------------------
          Hide
          cmpilato C. Michael Pilato added a comment -

          Note that because we send deltas across the wire for commits, and
          because each file's delta isn't calculated until just before it is
          transmitted, there's no way to really know the total size of
          to-be-transmitted data for a commit.
          

          Show
          cmpilato C. Michael Pilato added a comment - Note that because we send deltas across the wire for commits, and because each file's delta isn't calculated until just before it is transmitted, there's no way to really know the total size of to-be-transmitted data for a commit.
          Hide
          kfogel Karl Fogel added a comment -

          Marking as post-1.0; although technically an interface change, it can
          probably be done in such a way as not to disturb existing parsers of
          'svn commit'  output, and though it would be nice, there are a lot of
          other higher priority things on the table.
          
          A patch earlier than 1.0 will be most welcome, of course!
          
          

          Show
          kfogel Karl Fogel added a comment - Marking as post-1.0; although technically an interface change, it can probably be done in such a way as not to disturb existing parsers of 'svn commit' output, and though it would be nice, there are a lot of other higher priority things on the table. A patch earlier than 1.0 will be most welcome, of course!

            People

            • Assignee:
              Unassigned
              Reporter:
              subversion-importer Subversion Importer
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:

                Development