Derby
  1. Derby
  2. DERBY-791

Expose api for printing Abstract Syntax Trees in production (non-debug) servers

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: SQL
    • Urgency:
      Normal

      Description

      Currently you can print Abstract Syntax Trees to derby.log by setting the DumpParseTree tracepoint at server startup. E.g.:

      java -cp $CLASSPATH -Dderby.debug.true=DumpParseTree -Dderby.stream.error.logSeverityLevel=0 org.apache.derby.tools.ij z.sql

      This can be a useful debugging tool. However, it only works on debug servers built with the following flags:

      sanity=true
      debug=true

      We should provide some mechanism for printing these trees in production (non-debug) servers.

      1. derby-791-02-aa-printTargetColumnList.diff
        2 kB
        Rick Hillegas
      2. XmlTreeWalker.java
        12 kB
        Rick Hillegas
      3. ASTGrapher.java
        3 kB
        Kay Röpke
      4. XmlTreeWalker.java
        12 kB
        Kay Röpke
      5. XmlTreeWalker.java
        12 kB
        Rick Hillegas
      6. z.xml
        8 kB
        Rick Hillegas
      7. XmlTreeWalker.java
        12 kB
        Rick Hillegas
      8. derby-791-01-aa-fromListAndResultColumnList.diff
        2 kB
        Rick Hillegas
      9. XmlTreeWalker.java
        9 kB
        Rick Hillegas

        Issue Links

          Activity

          Hide
          Daniel John Debrunner added a comment -

          I was thinking about something similar over the holiday period. I don't think the current mechanism should be used, but instead the tree could be represented as an XML document. It would be great to tie this into the runtime statistics output, so that there was a common XML format for displaying query information.
          It would be good to also make this optional in a production server, rather than increasing the footprint for every deployment. If the node factory was not removed (see DERBY-673) then a debugging node factory option could return nodes that provide the tree printing. This debugging code could be in a separate jar file to the base engine, thus only incurring a footprint cost when runtime debugging is required.

          Show
          Daniel John Debrunner added a comment - I was thinking about something similar over the holiday period. I don't think the current mechanism should be used, but instead the tree could be represented as an XML document. It would be great to tie this into the runtime statistics output, so that there was a common XML format for displaying query information. It would be good to also make this optional in a production server, rather than increasing the footprint for every deployment. If the node factory was not removed (see DERBY-673 ) then a debugging node factory option could return nodes that provide the tree printing. This debugging code could be in a separate jar file to the base engine, thus only incurring a footprint cost when runtime debugging is required.
          Hide
          Daniel John Debrunner added a comment -

          See comments in DERBY-791 for a use of the node factory. It could be that a NodeFactory could still exist with strongly typed init methods instead of constructors.

          Show
          Daniel John Debrunner added a comment - See comments in DERBY-791 for a use of the node factory. It could be that a NodeFactory could still exist with strongly typed init methods instead of constructors.
          Show
          Rick Hillegas added a comment - For more context on this issue, see the following email threads: http://www.nabble.com/Optimizer-work-issues-discussions....-td6573100.html#a6605675 http://www.nabble.com/Having-trouble-finding-a-JIRA-issue-td21733256.html#a21733256
          Hide
          Rick Hillegas added a comment -

          Attaching XmlTreeWalker.java. This is a simple Visitor which prints out the parsed AST in xml. You invoke it like so:

          java XmlTreeWalker "select r.a from r, s where r.b = s.c"

          The program prints out an xml representation of the AST to standard out. Here's the output from the above command:

          <?xml version="1.0" encoding="UTF-8"?>
          <queryTree>
          <queryText>select r.a from r, s where r.b = s.c</queryText>
          <node>
          <nodeType>org.apache.derby.impl.sql.compile.CursorNode</nodeType>
          <contents>
          <member>name: null</member>
          <member>updateMode: UNSPECIFIED (0)</member>
          <member>statementType: SELECT</member>
          </contents>
          <node>
          <nodeType>org.apache.derby.impl.sql.compile.SelectNode</nodeType>
          <contents>
          <member>isDistinct: false</member>
          <member>groupByList: null</member>
          <member>orderByList: null</member>
          <member>resultSetNumber: 0</member>
          <member>referencedTableMap: null</member>
          <member>statementResultSet: false</member>
          </contents>
          <node>
          <nodeType>org.apache.derby.impl.sql.compile.ResultColumnList</nodeType>
          <contents>
          <member>exposedName: A</member>
          <member>name: A</member>
          <member>tableName: null</member>
          <member>isDefaultColumn: false</member>
          <member>wasDefaultColumn: false</member>
          <member>isNameGenerated: false</member>
          <member>sourceTableName: null</member>
          <member>type: null</member>
          <member>columnDescriptor: null</member>
          <member>isGenerated: false</member>
          <member>isGeneratedForUnmatchedColumnInInsert: false</member>
          <member>isGroupingColumn: false</member>
          <member>isReferenced: false</member>
          <member>isRedundant: false</member>
          <member>virtualColumnId: 1</member>
          <member>resultSetNumber: -1</member>
          <member>dataTypeServices: null</member>
          <member>; </member>
          </contents>
          <node>
          <nodeType>org.apache.derby.impl.sql.compile.ResultColumn</nodeType>
          <contents>
          <member>exposedName: A</member>
          <member>name: A</member>
          <member>tableName: null</member>
          <member>isDefaultColumn: false</member>
          <member>wasDefaultColumn: false</member>
          <member>isNameGenerated: false</member>
          <member>sourceTableName: null</member>
          <member>type: null</member>
          <member>columnDescriptor: null</member>
          <member>isGenerated: false</member>
          <member>isGeneratedForUnmatchedColumnInInsert: false</member>
          <member>isGroupingColumn: false</member>
          <member>isReferenced: false</member>
          <member>isRedundant: false</member>
          <member>virtualColumnId: 1</member>
          <member>resultSetNumber: -1</member>
          <member>dataTypeServices: null</member>
          </contents>
          <node>
          <nodeType>org.apache.derby.impl.sql.compile.ColumnReference</nodeType>
          <contents>
          <member>columnName: A</member>
          <member>tableNumber: -1</member>
          <member>columnNumber: 0</member>
          <member>replacesAggregate: false</member>
          <member>tableName: R</member>
          <member>nestingLevel: -1</member>
          <member>sourceLevel: -1</member>
          <member>dataTypeServices: null</member>
          </contents>
          </node>
          </node>
          </node>
          <node>
          <nodeType>org.apache.derby.impl.sql.compile.FromBaseTable</nodeType>
          <contents>
          <member>tableName: R</member>
          <member>tableDescriptor: null</member>
          <member>updateOrDelete: 0</member>
          <member>null</member>
          <member>existsBaseTable: false</member>
          <member>dependencyMap: null</member>
          <member>correlation Name: null</member>
          <member>null</member>
          <member>tableNumber -1</member>
          <member>level 0</member>
          <member>resultSetNumber: 0</member>
          <member>referencedTableMap: null</member>
          <member>statementResultSet: false</member>
          </contents>
          </node>
          <node>
          <nodeType>org.apache.derby.impl.sql.compile.FromBaseTable</nodeType>
          <contents>
          <member>tableName: S</member>
          <member>tableDescriptor: null</member>
          <member>updateOrDelete: 0</member>
          <member>null</member>
          <member>existsBaseTable: false</member>
          <member>dependencyMap: null</member>
          <member>correlation Name: null</member>
          <member>null</member>
          <member>tableNumber -1</member>
          <member>level 0</member>
          <member>resultSetNumber: 0</member>
          <member>referencedTableMap: null</member>
          <member>statementResultSet: false</member>
          </contents>
          </node>
          <node>
          <nodeType>org.apache.derby.impl.sql.compile.BinaryRelationalOperatorNode</nodeType>
          <contents>
          <member>operator: =</member>
          <member>methodName: equals</member>
          <member>dataTypeServices: null</member>
          </contents>
          <node>
          <nodeType>org.apache.derby.impl.sql.compile.ColumnReference</nodeType>
          <contents>
          <member>columnName: B</member>
          <member>tableNumber: -1</member>
          <member>columnNumber: 0</member>
          <member>replacesAggregate: false</member>
          <member>tableName: R</member>
          <member>nestingLevel: -1</member>
          <member>sourceLevel: -1</member>
          <member>dataTypeServices: null</member>
          </contents>
          </node>
          <node>
          <nodeType>org.apache.derby.impl.sql.compile.ColumnReference</nodeType>
          <contents>
          <member>columnName: C</member>
          <member>tableNumber: -1</member>
          <member>columnNumber: 0</member>
          <member>replacesAggregate: false</member>
          <member>tableName: S</member>
          <member>nestingLevel: -1</member>
          <member>sourceLevel: -1</member>
          <member>dataTypeServices: null</member>
          </contents>
          </node>
          </node>
          </node>
          </node>
          </queryTree>

          Show
          Rick Hillegas added a comment - Attaching XmlTreeWalker.java. This is a simple Visitor which prints out the parsed AST in xml. You invoke it like so: java XmlTreeWalker "select r.a from r, s where r.b = s.c" The program prints out an xml representation of the AST to standard out. Here's the output from the above command: <?xml version="1.0" encoding="UTF-8"?> <queryTree> <queryText>select r.a from r, s where r.b = s.c</queryText> <node> <nodeType>org.apache.derby.impl.sql.compile.CursorNode</nodeType> <contents> <member>name: null</member> <member>updateMode: UNSPECIFIED (0)</member> <member>statementType: SELECT</member> </contents> <node> <nodeType>org.apache.derby.impl.sql.compile.SelectNode</nodeType> <contents> <member>isDistinct: false</member> <member>groupByList: null</member> <member>orderByList: null</member> <member>resultSetNumber: 0</member> <member>referencedTableMap: null</member> <member>statementResultSet: false</member> </contents> <node> <nodeType>org.apache.derby.impl.sql.compile.ResultColumnList</nodeType> <contents> <member>exposedName: A</member> <member>name: A</member> <member>tableName: null</member> <member>isDefaultColumn: false</member> <member>wasDefaultColumn: false</member> <member>isNameGenerated: false</member> <member>sourceTableName: null</member> <member>type: null</member> <member>columnDescriptor: null</member> <member>isGenerated: false</member> <member>isGeneratedForUnmatchedColumnInInsert: false</member> <member>isGroupingColumn: false</member> <member>isReferenced: false</member> <member>isRedundant: false</member> <member>virtualColumnId: 1</member> <member>resultSetNumber: -1</member> <member>dataTypeServices: null</member> <member>; </member> </contents> <node> <nodeType>org.apache.derby.impl.sql.compile.ResultColumn</nodeType> <contents> <member>exposedName: A</member> <member>name: A</member> <member>tableName: null</member> <member>isDefaultColumn: false</member> <member>wasDefaultColumn: false</member> <member>isNameGenerated: false</member> <member>sourceTableName: null</member> <member>type: null</member> <member>columnDescriptor: null</member> <member>isGenerated: false</member> <member>isGeneratedForUnmatchedColumnInInsert: false</member> <member>isGroupingColumn: false</member> <member>isReferenced: false</member> <member>isRedundant: false</member> <member>virtualColumnId: 1</member> <member>resultSetNumber: -1</member> <member>dataTypeServices: null</member> </contents> <node> <nodeType>org.apache.derby.impl.sql.compile.ColumnReference</nodeType> <contents> <member>columnName: A</member> <member>tableNumber: -1</member> <member>columnNumber: 0</member> <member>replacesAggregate: false</member> <member>tableName: R</member> <member>nestingLevel: -1</member> <member>sourceLevel: -1</member> <member>dataTypeServices: null</member> </contents> </node> </node> </node> <node> <nodeType>org.apache.derby.impl.sql.compile.FromBaseTable</nodeType> <contents> <member>tableName: R</member> <member>tableDescriptor: null</member> <member>updateOrDelete: 0</member> <member>null</member> <member>existsBaseTable: false</member> <member>dependencyMap: null</member> <member>correlation Name: null</member> <member>null</member> <member>tableNumber -1</member> <member>level 0</member> <member>resultSetNumber: 0</member> <member>referencedTableMap: null</member> <member>statementResultSet: false</member> </contents> </node> <node> <nodeType>org.apache.derby.impl.sql.compile.FromBaseTable</nodeType> <contents> <member>tableName: S</member> <member>tableDescriptor: null</member> <member>updateOrDelete: 0</member> <member>null</member> <member>existsBaseTable: false</member> <member>dependencyMap: null</member> <member>correlation Name: null</member> <member>null</member> <member>tableNumber -1</member> <member>level 0</member> <member>resultSetNumber: 0</member> <member>referencedTableMap: null</member> <member>statementResultSet: false</member> </contents> </node> <node> <nodeType>org.apache.derby.impl.sql.compile.BinaryRelationalOperatorNode</nodeType> <contents> <member>operator: =</member> <member>methodName: equals</member> <member>dataTypeServices: null</member> </contents> <node> <nodeType>org.apache.derby.impl.sql.compile.ColumnReference</nodeType> <contents> <member>columnName: B</member> <member>tableNumber: -1</member> <member>columnNumber: 0</member> <member>replacesAggregate: false</member> <member>tableName: R</member> <member>nestingLevel: -1</member> <member>sourceLevel: -1</member> <member>dataTypeServices: null</member> </contents> </node> <node> <nodeType>org.apache.derby.impl.sql.compile.ColumnReference</nodeType> <contents> <member>columnName: C</member> <member>tableNumber: -1</member> <member>columnNumber: 0</member> <member>replacesAggregate: false</member> <member>tableName: S</member> <member>nestingLevel: -1</member> <member>sourceLevel: -1</member> <member>dataTypeServices: null</member> </contents> </node> </node> </node> </node> </queryTree>
          Hide
          Rick Hillegas added a comment -

          Looking at the output of XmlTreeWalker, the following jumps out at me: The subclasses of QueryTreeNodeVector could use a little upgrading of their accept() and toString() methods. In particular:

          1) FromList.accept() should start out by calling v.visit( this ) as the other accept() implementations do.

          2) FromList and ResultColumnList should have their own overrides for toString()--right now they rely on the toString() method in their superclass, which is too generic to be useful for tree printing.

          Show
          Rick Hillegas added a comment - Looking at the output of XmlTreeWalker, the following jumps out at me: The subclasses of QueryTreeNodeVector could use a little upgrading of their accept() and toString() methods. In particular: 1) FromList.accept() should start out by calling v.visit( this ) as the other accept() implementations do. 2) FromList and ResultColumnList should have their own overrides for toString()--right now they rely on the toString() method in their superclass, which is too generic to be useful for tree printing.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-791-01-aa-fromListAndResultColumnList.diff. This experimental patch fixes up FromList.accept(), FromList.toString(), and ResultColumnList.toString() as mentioned above. This makes the output of XmlTreeWalker more useful.

          Show
          Rick Hillegas added a comment - Attaching derby-791-01-aa-fromListAndResultColumnList.diff. This experimental patch fixes up FromList.accept(), FromList.toString(), and ResultColumnList.toString() as mentioned above. This makes the output of XmlTreeWalker more useful.
          Hide
          Knut Anders Hatlen added a comment -

          This looks very useful, and it's even relatively easy to read. Perhaps it would be even easier to process the output (at least for a machine) if we split the name:value string into two separate tags. That is, instead of

          <member>replacesAggregate: false</member>

          something like

          <member><name>replacesAggregate</name><value>false</value></member>

          or perhaps even better

          <member name="replacesAggregate">false</member>

          Show
          Knut Anders Hatlen added a comment - This looks very useful, and it's even relatively easy to read. Perhaps it would be even easier to process the output (at least for a machine) if we split the name:value string into two separate tags. That is, instead of <member>replacesAggregate: false</member> something like <member><name>replacesAggregate</name><value>false</value></member> or perhaps even better <member name="replacesAggregate">false</member>
          Hide
          Rick Hillegas added a comment -

          Thanks for the feedback, Knut. Attaching a new version of XmlTreeWalker together with the corresponding output: z.xml. This new version formats the node contents as member elements with name and value attributes, much as Knut suggested. The node type is also formatted as an attribute. This makes the xml a little more compact and readable.

          If you are using Firefox, you make be interested in this feature of the browser: By default, mysterious xml documents are presented as a directory tree, with each element being a separately collapsible level. This is a very handy way to view our ASTs. I think that Internet Explorer may do something similar. Safari, however, doesn't present xml files in this useful way.

          Show
          Rick Hillegas added a comment - Thanks for the feedback, Knut. Attaching a new version of XmlTreeWalker together with the corresponding output: z.xml. This new version formats the node contents as member elements with name and value attributes, much as Knut suggested. The node type is also formatted as an attribute. This makes the xml a little more compact and readable. If you are using Firefox, you make be interested in this feature of the browser: By default, mysterious xml documents are presented as a directory tree, with each element being a separately collapsible level. This is a very handy way to view our ASTs. I think that Internet Explorer may do something similar. Safari, however, doesn't present xml files in this useful way.
          Hide
          Rick Hillegas added a comment - - edited

          Attaching another rev of the XmlTreeWalker. This version uses an in-memory database.

          Show
          Rick Hillegas added a comment - - edited Attaching another rev of the XmlTreeWalker. This version uses an in-memory database.
          Hide
          Kay Röpke added a comment -

          Attached are an updated XmlTreeWalker and a Eclipse Zest visualization of the short class names of the AST.
          They are laid out in a tree fashion.
          Even though this mini application is far from perfect, it might be useful to someone. It should be trivial to add more information to the graph (like operator names, literal values, etc).

          For XmlTreeWalker the only difference to Rick's version is that its execute method will accept a visitor object instead of always using this (well, and the method is now public so ASTGrapher can access it).

          To compile this you need Zest from the Eclipse GEF project (and the RCP features). The easiest way to set up the Eclipse project is to create a new Eclipse RCP plugin project and then dropping in these two Java files. Run ASTGrapher.java as an Application (you do not need to start an entire RCP instance).
          The RCP project only serves to properly set up the build path, nothing more.

          Show
          Kay Röpke added a comment - Attached are an updated XmlTreeWalker and a Eclipse Zest visualization of the short class names of the AST. They are laid out in a tree fashion. Even though this mini application is far from perfect, it might be useful to someone. It should be trivial to add more information to the graph (like operator names, literal values, etc). For XmlTreeWalker the only difference to Rick's version is that its execute method will accept a visitor object instead of always using this (well, and the method is now public so ASTGrapher can access it). To compile this you need Zest from the Eclipse GEF project (and the RCP features). The easiest way to set up the Eclipse project is to create a new Eclipse RCP plugin project and then dropping in these two Java files. Run ASTGrapher.java as an Application (you do not need to start an entire RCP instance). The RCP project only serves to properly set up the build path, nothing more.
          Hide
          Rick Hillegas added a comment -

          Attaching a new version of XmlTreeWalker, which implements the visitChildrenFirst() method which was added to the Visitor interface recently.

          Show
          Rick Hillegas added a comment - Attaching a new version of XmlTreeWalker, which implements the visitChildrenFirst() method which was added to the Visitor interface recently.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-791-02-aa-printTargetColumnList.diff. This patch adds some more accept methods so that InsertNodes walk their targetColumnLists in addition to the children they share with their superclasses. Touches the following files:

          M java/engine/org/apache/derby/impl/sql/compile/DMLModStatementNode.java
          M java/engine/org/apache/derby/impl/sql/compile/InsertNode.java

          Show
          Rick Hillegas added a comment - Attaching derby-791-02-aa-printTargetColumnList.diff. This patch adds some more accept methods so that InsertNodes walk their targetColumnLists in addition to the children they share with their superclasses. Touches the following files: M java/engine/org/apache/derby/impl/sql/compile/DMLModStatementNode.java M java/engine/org/apache/derby/impl/sql/compile/InsertNode.java
          Hide
          Rick Hillegas added a comment -

          Committed derby-791-02-aa-printTargetColumnList.diff at subversion revision 835417.

          Show
          Rick Hillegas added a comment - Committed derby-791-02-aa-printTargetColumnList.diff at subversion revision 835417.
          Hide
          Mamta A. Satoor added a comment -

          Rick, can this jira be closed or is there further work to be done? thanks

          Show
          Mamta A. Satoor added a comment - Rick, can this jira be closed or is there further work to be done? thanks
          Hide
          Rick Hillegas added a comment -

          Hi Mamta,

          I think that further product changes related to this issue will probably happen under DERBY-6263. However, XmlTreeWalker may need to be tweaked. I think I need to leave this issue open in order to attach new versions of XmlTreeWalker. Thanks.

          Show
          Rick Hillegas added a comment - Hi Mamta, I think that further product changes related to this issue will probably happen under DERBY-6263 . However, XmlTreeWalker may need to be tweaked. I think I need to leave this issue open in order to attach new versions of XmlTreeWalker. Thanks.

            People

            • Assignee:
              Unassigned
              Reporter:
              Rick Hillegas
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:

                Development