Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-2583

Add Grunt command to list the statements in cache

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: 0.11
    • Component/s: None
    • Labels:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Add new grunt command:
      history [-n]: Display the list statements in cache. -n means hiding line numbers.
      Show
      Add new grunt command: history [-n]: Display the list statements in cache. -n means hiding line numbers.

      Description

      It is convenient to list statements in cache:

      grunt> a = load '1.txt';
      grunt> b = foreach a generate $0, $1;
      grunt> list
      a = load '1.txt';
      b = foreach a generate $0, $1;

      1. gruntHistory.patch
        4 kB
        Allan Avendaño
      2. gruntHistory1.patch
        5 kB
        Allan Avendaño
      3. gruntHistory2.patch
        4 kB
        Allan Avendaño
      4. gruntHistory3.patch
        5 kB
        Allan Avendaño
      5. gruntHistory4.patch
        0.7 kB
        Allan Avendaño

        Activity

        Hide
        billgraham Bill Graham added a comment -

        Great idea. What about history instead of list? It's similar to bash and groovy shell. I couldn't find an irb equivalent.

        Show
        billgraham Bill Graham added a comment - Great idea. What about history instead of list ? It's similar to bash and groovy shell. I couldn't find an irb equivalent.
        Hide
        daijy Daniel Dai added a comment -

        Sounds good.

        Show
        daijy Daniel Dai added a comment - Sounds good.
        Hide
        xalan Allan Avendaño added a comment -

        I added an arraylist that works as cache of statements executed so far.

        Show
        xalan Allan Avendaño added a comment - I added an arraylist that works as cache of statements executed so far.
        Hide
        daijy Daniel Dai added a comment -

        Thanks, I will take a look.

        Show
        daijy Daniel Dai added a comment - Thanks, I will take a look.
        Hide
        daijy Daniel Dai added a comment -

        It is right to put in command cache only if it is a Pig statement not a command. However, if an alias is get reused, we should show the last definition, not all. I think we should make use of Graph.scriptCache to do it.

        Show
        daijy Daniel Dai added a comment - It is right to put in command cache only if it is a Pig statement not a command. However, if an alias is get reused, we should show the last definition, not all. I think we should make use of Graph.scriptCache to do it.
        Hide
        xalan Allan Avendaño added a comment -

        I was working on this, but I found something particular if we show the last definition for the alias, for example on this sequence:

        A = load 'data/abs' using PigStorage('|') as (name:chararray, id1:int, id2:int, id3:int);
        D = group A by id1;
        A = load 'data/abs' using PigStorage('|') as (name:chararray, x:int, y:int, z:int);

        With modifications on the patch that I attach, the following history is shown (according to historical modifications on aliases):

        D = group A by id1;
        A = load 'data/abs' using PigStorage('|') as (name:chararray, x:int, y:int, z:int);

        But, D will work on previous schema of A, and not with the last definition. And maybe, we are missing this other historical point of view.

        Show
        xalan Allan Avendaño added a comment - I was working on this, but I found something particular if we show the last definition for the alias, for example on this sequence: A = load 'data/abs' using PigStorage('|') as (name:chararray, id1:int, id2:int, id3:int); D = group A by id1; A = load 'data/abs' using PigStorage('|') as (name:chararray, x:int, y:int, z:int); With modifications on the patch that I attach, the following history is shown (according to historical modifications on aliases): D = group A by id1; A = load 'data/abs' using PigStorage('|') as (name:chararray, x:int, y:int, z:int); But, D will work on previous schema of A, and not with the last definition. And maybe, we are missing this other historical point of view.
        Hide
        azaroth Gianmarco De Francisci Morales added a comment -

        Probably it is fine anyway.
        The order in which they are defined (and printed) will tell you that D got defined before A.

        One use case I see for this history command is to build a Pig script interactively step by step.
        The output of the history can be then copy-pasted to a file to save the script.
        In this case you would need to repeat the D = ... statement anyway to get the correct order.
        To do this for more than one statement you could simply use the output of history, and copy-paste it in grunt up to (excluding) the last A = ... statement.

        Show
        azaroth Gianmarco De Francisci Morales added a comment - Probably it is fine anyway. The order in which they are defined (and printed) will tell you that D got defined before A. One use case I see for this history command is to build a Pig script interactively step by step. The output of the history can be then copy-pasted to a file to save the script. In this case you would need to repeat the D = ... statement anyway to get the correct order. To do this for more than one statement you could simply use the output of history, and copy-paste it in grunt up to (excluding) the last A = ... statement.
        Hide
        prkommireddi Prashant Kommireddi added a comment -

        Should we let the behavior be similar to standard "history" command on UNIX shell?

        localhost:~ pkommireddi$ pwd
        /Users/pkommireddi
        localhost:~ pkommireddi$ date
        Tue May  8 15:26:26 PDT 2012
        localhost:~ pkommireddi$ whoami
        pkommireddi
        localhost:~ pkommireddi$ pwd
        /Users/pkommireddi
        
        localhost:~ pkommireddi$ history
          541  pwd
          542  date
          543  whoami
          544  pwd
          545  history
        

        It might be useful for users while developing/debugging to be able to look at all declarations of an alias.

        Show
        prkommireddi Prashant Kommireddi added a comment - Should we let the behavior be similar to standard "history" command on UNIX shell? localhost:~ pkommireddi$ pwd /Users/pkommireddi localhost:~ pkommireddi$ date Tue May 8 15:26:26 PDT 2012 localhost:~ pkommireddi$ whoami pkommireddi localhost:~ pkommireddi$ pwd /Users/pkommireddi localhost:~ pkommireddi$ history 541 pwd 542 date 543 whoami 544 pwd 545 history It might be useful for users while developing/debugging to be able to look at all declarations of an alias.
        Hide
        daijy Daniel Dai added a comment -

        I take back the "last definition" part. If people keep reusing the same alias, we shall keep the complete history:
        A = load ...
        A = foreach A ...
        A = filter A by ...

        It is possible to detect complete dead statement, but I don't want to complicate the implementation.

        However, we shall reuse scriptCache instead of introducing a new data structure.

        Show
        daijy Daniel Dai added a comment - I take back the "last definition" part. If people keep reusing the same alias, we shall keep the complete history: A = load ... A = foreach A ... A = filter A by ... It is possible to detect complete dead statement, but I don't want to complicate the implementation. However, we shall reuse scriptCache instead of introducing a new data structure.
        Hide
        xalan Allan Avendaño added a comment -

        History of operators used so far. Is used Graph.scriptCache

        Show
        xalan Allan Avendaño added a comment - History of operators used so far. Is used Graph.scriptCache
        Hide
        xalan Allan Avendaño added a comment -

        History of operators used so far. Is used Graph.scriptCache

        Show
        xalan Allan Avendaño added a comment - History of operators used so far. Is used Graph.scriptCache
        Hide
        daijy Daniel Dai added a comment -

        Looks good. Another request however, we shall print line number, but we can optionally turn it off (history -n sounds good?)

        Show
        daijy Daniel Dai added a comment - Looks good. Another request however, we shall print line number, but we can optionally turn it off (history -n sounds good?)
        Hide
        xalan Allan Avendaño added a comment -

        Something like this?

        grunt> history -n
        (1) A = load 'data/abs' using PigStorage('|') as (name:chararray, id1:int, id2:int, id3:int);
        (2) B = group A by id1;
        (3) C = order A by id2;

        grunt> history
        A = load 'data/abs' using PigStorage('|') as (name:chararray, id1:int, id2:int, id3:int);
        B = rank A by id1;
        C = order A by id2;

        Show
        xalan Allan Avendaño added a comment - Something like this? grunt> history -n (1) A = load 'data/abs' using PigStorage('|') as (name:chararray, id1:int, id2:int, id3:int); (2) B = group A by id1; (3) C = order A by id2; grunt> history A = load 'data/abs' using PigStorage('|') as (name:chararray, id1:int, id2:int, id3:int); B = rank A by id1; C = order A by id2;
        Hide
        daijy Daniel Dai added a comment -

        Fine for me.

        Show
        daijy Daniel Dai added a comment - Fine for me.
        Hide
        prkommireddi Prashant Kommireddi added a comment -

        Not a huge deal, but again replicating Unix 'history' behavior (no parenthesis) would be ideal in my opinion.

        pkommireddi@pkommireddi-wsl:~$ pwd
        /home/pkommireddi
        pkommireddi@pkommireddi-wsl:~$ date
        Wed May  9 17:01:37 PDT 2012
        pkommireddi@pkommireddi-wsl:~$ pwd
        /home/pkommireddi
        pkommireddi@pkommireddi-wsl:~$ whoami
        pkommireddi
        pkommireddi@pkommireddi-wsl:~$ history
        
         2989  pwd
         2990  date
         2991  pwd
         2992  whoami
        
        Show
        prkommireddi Prashant Kommireddi added a comment - Not a huge deal, but again replicating Unix 'history' behavior (no parenthesis) would be ideal in my opinion. pkommireddi@pkommireddi-wsl:~$ pwd /home/pkommireddi pkommireddi@pkommireddi-wsl:~$ date Wed May 9 17:01:37 PDT 2012 pkommireddi@pkommireddi-wsl:~$ pwd /home/pkommireddi pkommireddi@pkommireddi-wsl:~$ whoami pkommireddi pkommireddi@pkommireddi-wsl:~$ history 2989 pwd 2990 date 2991 pwd 2992 whoami
        Hide
        daijy Daniel Dai added a comment -

        Yes, that's better, and let's make show line number the default option.

        Show
        daijy Daniel Dai added a comment - Yes, that's better, and let's make show line number the default option.
        Hide
        xalan Allan Avendaño added a comment -

        Last patch shows by default an enumerated history, this can be omitted with "-n"

        Show
        xalan Allan Avendaño added a comment - Last patch shows by default an enumerated history, this can be omitted with "-n"
        Hide
        daijy Daniel Dai added a comment -

        +1. Patch committed to trunk. Thanks Allan!

        Show
        daijy Daniel Dai added a comment - +1. Patch committed to trunk. Thanks Allan!
        Hide
        prkommireddi Prashant Kommireddi added a comment -

        Thanks Allan for the contribution!

        Show
        prkommireddi Prashant Kommireddi added a comment - Thanks Allan for the contribution!
        Hide
        xalan Allan Avendaño added a comment -

        I would like to make some changes on the format to show the list of statements. For example, I make it to display the statements with an indentation like history command does on Unix.

        Show
        xalan Allan Avendaño added a comment - I would like to make some changes on the format to show the list of statements. For example, I make it to display the statements with an indentation like history command does on Unix.
        Hide
        xalan Allan Avendaño added a comment -

        Changes on the displaying format of list of statements used so far.

        Show
        xalan Allan Avendaño added a comment - Changes on the displaying format of list of statements used so far.
        Hide
        julienledem Julien Le Dem added a comment -

        Please open a new ticket for this and resolve this one

        Show
        julienledem Julien Le Dem added a comment - Please open a new ticket for this and resolve this one
        Hide
        julienledem Julien Le Dem added a comment -

        Allan Avendaño I'm closing this ticket as it has been committed.
        Please open a new ticket to further improve your contribution.
        Thanks again

        Show
        julienledem Julien Le Dem added a comment - Allan Avendaño I'm closing this ticket as it has been committed. Please open a new ticket to further improve your contribution. Thanks again

          People

          • Assignee:
            xalan Allan Avendaño
            Reporter:
            daijy Daniel Dai
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development