Derby
  1. Derby
  2. DERBY-4415

Make it easy to plug custom AST printers into the compiler

    Details

    • Type: Improvement Improvement
    • Status: Reopened
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 10.6.1.0
    • Component/s: SQL
    • Labels:
      None

      Description

      It would be nice to be able to tell the compiler to use user-written formatters for pretty-printing Derby's abstract syntax trees. For instance, it would be nice to be able to plug in the AST printers attached to DERBY-791, including the Eclipse-based tree grapher written by Kay Röpke.

      1. derby-4415-01-aa-customASTprinters.diff
        13 kB
        Rick Hillegas
      2. ASTInspector.java
        3 kB
        Rick Hillegas
      3. XmlASTPrinter.java
        11 kB
        Rick Hillegas
      4. derby-4415-01-ab-customASTprinters.diff
        8 kB
        Rick Hillegas
      5. XmlASTPrinter.java
        11 kB
        Rick Hillegas
      6. ASTInspector.java
        3 kB
        Rick Hillegas
      7. XmlASTPrinter.java
        11 kB
        Rick Hillegas
      8. XmlASTPrinter.java
        11 kB
        Rick Hillegas
      9. ASTInspector.java
        4 kB
        Rick Hillegas

        Issue Links

          Activity

          Hide
          Rick Hillegas added a comment -

          Attaching a new version of the ASTInspector. This version was tweaked to function as a custom, optional tool. If other people find this tool to be useful, we could productize it and document it. To load and unload the XmlASTPrinter, you now do the following:

          call syscs_util.syscs_register_tool( 'customTool', true, 'ASTInspector', 'XmlASTPrinter' );
          call syscs_util.syscs_register_tool( 'customTool', false, 'ASTInspector' );

          Here's a little script which shows how to use this tool:

          connect 'jdbc:derby:memory:dummy;create=true';

          create table t( a int );

          call syscs_util.syscs_register_tool( 'customTool', true, 'ASTInspector', 'XmlASTPrinter' );

          select * from t;

          call syscs_util.syscs_register_tool( 'customTool', false, 'ASTInspector' );

          – now look for the output in derbyAST.xml

          Show
          Rick Hillegas added a comment - Attaching a new version of the ASTInspector. This version was tweaked to function as a custom, optional tool. If other people find this tool to be useful, we could productize it and document it. To load and unload the XmlASTPrinter, you now do the following: call syscs_util.syscs_register_tool( 'customTool', true, 'ASTInspector', 'XmlASTPrinter' ); call syscs_util.syscs_register_tool( 'customTool', false, 'ASTInspector' ); Here's a little script which shows how to use this tool: connect 'jdbc:derby:memory:dummy;create=true'; create table t( a int ); call syscs_util.syscs_register_tool( 'customTool', true, 'ASTInspector', 'XmlASTPrinter' ); select * from t; call syscs_util.syscs_register_tool( 'customTool', false, 'ASTInspector' ); – now look for the output in derbyAST.xml
          Hide
          Rick Hillegas added a comment -

          Attaching a new version of XmlASTPrinter, which implements the visitChildrenFirst() method which was added to the Visitor interface recently.

          Show
          Rick Hillegas added a comment - Attaching a new version of XmlASTPrinter, which implements the visitChildrenFirst() method which was added to the Visitor interface recently.
          Hide
          Rick Hillegas added a comment -

          Attaching a new version of the XML tree printer. This makes the following improvements:

          1) Moves the phase marker (parse/bind/optimize) into an attribute so that, when a tree is collapsed, you can see at a glance what compilation phase it belongs to. This helps you quickly browse to the right tree.

          2) Replace angle brackets in text strings with the corresponding xml entities so that the browser doesn't choke on these strings.

          Show
          Rick Hillegas added a comment - Attaching a new version of the XML tree printer. This makes the following improvements: 1) Moves the phase marker (parse/bind/optimize) into an attribute so that, when a tree is collapsed, you can see at a glance what compilation phase it belongs to. This helps you quickly browse to the right tree. 2) Replace angle brackets in text strings with the corresponding xml entities so that the browser doesn't choke on these strings.
          Hide
          Rick Hillegas added a comment -

          Thanks, Knut. Committed derby-4415-01-ab-customASTprinters.diff at subversion revision 827760.

          Show
          Rick Hillegas added a comment - Thanks, Knut. Committed derby-4415-01-ab-customASTprinters.diff at subversion revision 827760.
          Hide
          Knut Anders Hatlen added a comment -

          Thanks, Rick. The latest patch looks good to me (perhaps except some lines exceeding 80 chars), so +1 from me.

          Show
          Knut Anders Hatlen added a comment - Thanks, Rick. The latest patch looks good to me (perhaps except some lines exceeding 80 chars), so +1 from me.
          Hide
          Rick Hillegas added a comment -

          Attaching a new version of the database procedure which pokes user-written Visitors into the compiler: ASTInspector

          Show
          Rick Hillegas added a comment - Attaching a new version of the database procedure which pokes user-written Visitors into the compiler: ASTInspector
          Hide
          Rick Hillegas added a comment -

          Attaching a new version of the example Visitor, which implements the changes in the patch: XmlASTPrinter

          Show
          Rick Hillegas added a comment - Attaching a new version of the example Visitor, which implements the changes in the patch: XmlASTPrinter
          Hide
          Rick Hillegas added a comment -

          Thanks for the quick feedback, Knut. I agree that a separate ASTVisitor interface is a great idea. Attaching a new, simpler version of this patch: derby-4415-01-ab-customASTprinters.diff.

          Touches the following files:

          A java/engine/org/apache/derby/iapi/sql/compile/ASTVisitor.java

          Interface for user-written node Visitors. Provides initialization and teardown methods.

          M java/engine/org/apache/derby/iapi/sql/conn/LanguageConnectionContext.java
          M java/engine/org/apache/derby/impl/sql/conn/GenericLanguageConnectionContext.java

          Adds methods for poking a user-written node Visitor into the compiler's state variable.

          M java/engine/org/apache/derby/impl/sql/GenericStatement.java

          Adds calls to the user-written node Visitor if one has been supplied. The Visitor is called for each processing phase: parse, bind, and optimize.

          Show
          Rick Hillegas added a comment - Thanks for the quick feedback, Knut. I agree that a separate ASTVisitor interface is a great idea. Attaching a new, simpler version of this patch: derby-4415-01-ab-customASTprinters.diff. Touches the following files: A java/engine/org/apache/derby/iapi/sql/compile/ASTVisitor.java Interface for user-written node Visitors. Provides initialization and teardown methods. M java/engine/org/apache/derby/iapi/sql/conn/LanguageConnectionContext.java M java/engine/org/apache/derby/impl/sql/conn/GenericLanguageConnectionContext.java Adds methods for poking a user-written node Visitor into the compiler's state variable. M java/engine/org/apache/derby/impl/sql/GenericStatement.java Adds calls to the user-written node Visitor if one has been supplied. The Visitor is called for each processing phase: parse, bind, and optimize.
          Hide
          Knut Anders Hatlen added a comment -

          Hi Rick,

          This looks like a useful addition. Just a thought, but would it be better to create a new interface (say ASTVisitor) that extended Visitor so that only those visitors that actually use the begin() and end() methods need to implement them?

          Show
          Knut Anders Hatlen added a comment - Hi Rick, This looks like a useful addition. Just a thought, but would it be better to create a new interface (say ASTVisitor) that extended Visitor so that only those visitors that actually use the begin() and end() methods need to implement them?
          Hide
          Rick Hillegas added a comment - - edited

          Here's a sample script which pokes a custom AST printer into the compiler:

          connect 'jdbc:derby:memory:dummy;create=true';

          create table t( a int );

          create procedure setInspector( visitorClassName varchar( 32672 ) )
          language java
          parameter style java
          modifies sql data
          external name 'ASTInspector.setInspector'
          ;

          call setInspector( 'XmlASTPrinter' );

          select * from t;

          The custom printer writes the parsed, bound, and optimized trees as xml into derbyAST.xml. You can then pull that printout into your favorite tool for browsing xml. I use Firefox for this. Firefox lets me collapse and expands elements in this printout. Since each node in an AST is printed as a separate element, this is very handy for revealing the structure of a plan and for hiding the details you are not interested in.

          call setInspector( null );

          Show
          Rick Hillegas added a comment - - edited Here's a sample script which pokes a custom AST printer into the compiler: connect 'jdbc:derby:memory:dummy;create=true'; create table t( a int ); create procedure setInspector( visitorClassName varchar( 32672 ) ) language java parameter style java modifies sql data external name 'ASTInspector.setInspector' ; call setInspector( 'XmlASTPrinter' ); select * from t; The custom printer writes the parsed, bound, and optimized trees as xml into derbyAST.xml. You can then pull that printout into your favorite tool for browsing xml. I use Firefox for this. Firefox lets me collapse and expands elements in this printout. Since each node in an AST is printed as a separate element, this is very handy for revealing the structure of a plan and for hiding the details you are not interested in. call setInspector( null );
          Hide
          Rick Hillegas added a comment -

          Attaching a Visitor which prints the parsed, bound, and optimized trees as xml. This tree printer can be poked into the compiler using the previously attached database procedure.

          Show
          Rick Hillegas added a comment - Attaching a Visitor which prints the parsed, bound, and optimized trees as xml. This tree printer can be poked into the compiler using the previously attached database procedure.
          Hide
          Rick Hillegas added a comment -

          Attaching a database procedure which can poke a custom AST printer into the compiler after the patch is applied.

          Show
          Rick Hillegas added a comment - Attaching a database procedure which can poke a custom AST printer into the compiler after the patch is applied.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-4415-01-aa-customASTprinters. This makes it possible to plug custom AST printers into the compiler. I am running regression tests now.

          Touches the following files:

          M java/engine/org/apache/derby/iapi/sql/conn/LanguageConnectionContext.java
          M java/engine/org/apache/derby/impl/sql/conn/GenericLanguageConnectionContext.java

          Add accessor methods for storing a custom AST printer in the session's compiler state.

          M java/engine/org/apache/derby/iapi/sql/compile/Visitor.java

          Add methods to mark the beginning and end of a printing session.

          M java/engine/org/apache/derby/impl/sql/GenericStatement.java

          If the custom AST printer is set, then use it after each transformation of the AST: 1) after parsing, 2) after binding, and 3) after optimization.

          M java/engine/org/apache/derby/impl/sql/compile/HasVariantValueNodeVisitor.java
          M java/engine/org/apache/derby/impl/sql/compile/VerifyAggregateExpressionsVisitor.java
          M java/engine/org/apache/derby/impl/sql/compile/ReplaceAggregatesWithCRVisitor.java
          M java/engine/org/apache/derby/impl/sql/compile/HasCorrelatedCRsVisitor.java
          M java/engine/org/apache/derby/impl/sql/compile/VTIDeferModPolicy.java
          M java/engine/org/apache/derby/impl/sql/compile/CollectNodesVisitor.java
          M java/engine/org/apache/derby/impl/sql/compile/SubstituteExpressionVisitor.java
          M java/engine/org/apache/derby/impl/sql/compile/HasNodeVisitor.java
          M java/engine/org/apache/derby/impl/sql/compile/BaseTableNumbersVisitor.java
          M java/engine/org/apache/derby/impl/sql/compile/ReferencedTablesVisitor.java
          M java/engine/org/apache/derby/impl/sql/compile/RemapCRsVisitor.java

          Add vacuous implementations of the new begin() and end() brackets to our existing Visitors.

          Show
          Rick Hillegas added a comment - Attaching derby-4415-01-aa-customASTprinters. This makes it possible to plug custom AST printers into the compiler. I am running regression tests now. Touches the following files: M java/engine/org/apache/derby/iapi/sql/conn/LanguageConnectionContext.java M java/engine/org/apache/derby/impl/sql/conn/GenericLanguageConnectionContext.java Add accessor methods for storing a custom AST printer in the session's compiler state. M java/engine/org/apache/derby/iapi/sql/compile/Visitor.java Add methods to mark the beginning and end of a printing session. M java/engine/org/apache/derby/impl/sql/GenericStatement.java If the custom AST printer is set, then use it after each transformation of the AST: 1) after parsing, 2) after binding, and 3) after optimization. M java/engine/org/apache/derby/impl/sql/compile/HasVariantValueNodeVisitor.java M java/engine/org/apache/derby/impl/sql/compile/VerifyAggregateExpressionsVisitor.java M java/engine/org/apache/derby/impl/sql/compile/ReplaceAggregatesWithCRVisitor.java M java/engine/org/apache/derby/impl/sql/compile/HasCorrelatedCRsVisitor.java M java/engine/org/apache/derby/impl/sql/compile/VTIDeferModPolicy.java M java/engine/org/apache/derby/impl/sql/compile/CollectNodesVisitor.java M java/engine/org/apache/derby/impl/sql/compile/SubstituteExpressionVisitor.java M java/engine/org/apache/derby/impl/sql/compile/HasNodeVisitor.java M java/engine/org/apache/derby/impl/sql/compile/BaseTableNumbersVisitor.java M java/engine/org/apache/derby/impl/sql/compile/ReferencedTablesVisitor.java M java/engine/org/apache/derby/impl/sql/compile/RemapCRsVisitor.java Add vacuous implementations of the new begin() and end() brackets to our existing Visitors.

            People

            • Assignee:
              Rick Hillegas
              Reporter:
              Rick Hillegas
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:

                Development