Accumulo
  1. Accumulo
  2. ACCUMULO-303

Implement per-table persistent formatters

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4.0
    • Component/s: client
    • Labels:

      Description

      In the Accumulo shell, Formatters can only be set on a per-shell-session basis. Setting a custom Formatter in a shell session designed for a specific table must be removed when scanning a table of a different format.

      Implement a method to persist a formatter class on a table across shell sessions that encourages the creation and usage of Formatters

        Activity

        Hide
        Josh Elser added a comment -
        • Add a table property that defaults to the DefaultFormatter class.
        • Update the Shell to support per-table Formatters
        • Update the 'formatter' and 'config' shell commands to properly set/unset the table properties and update the current formatter to use for scans.
        Show
        Josh Elser added a comment - Add a table property that defaults to the DefaultFormatter class. Update the Shell to support per-table Formatters Update the 'formatter' and 'config' shell commands to properly set/unset the table properties and update the current formatter to use for scans.
        Hide
        Josh Elser added a comment -

        Attach patch

        Show
        Josh Elser added a comment - Attach patch
        Hide
        Josh Elser added a comment -

        You can test this out by running the following:

        $ bin/accumulo shell -u root

        root@accumulo> table trace
        root@accumulo trace> formatter -f org.apache.accumulo.core.trace.TraceFormatter
        root@accumulo trace> scan
        – formatted trace entries –
        root@accumulo trace> table !METADATA
        root@accumulo !METADATA> scan
        – "normal" !METADATA entries –
        root@accumulo !METADATA> table trace
        root@accumulo trace> scan
        – formatted trace entries –
        root@accumulo trace> exit

        $ bin/accumulo shell -u root

        root@accumulo> table trace
        root@accumulo trace> scan
        – formatted trace entries –
        root@accumulo trace> formatter -r
        root@accumulo trace> scan
        – "normal" trace entries –

        Show
        Josh Elser added a comment - You can test this out by running the following: $ bin/accumulo shell -u root root@accumulo> table trace root@accumulo trace> formatter -f org.apache.accumulo.core.trace.TraceFormatter root@accumulo trace> scan – formatted trace entries – root@accumulo trace> table !METADATA root@accumulo !METADATA> scan – "normal" !METADATA entries – root@accumulo !METADATA> table trace root@accumulo trace> scan – formatted trace entries – root@accumulo trace> exit $ bin/accumulo shell -u root root@accumulo> table trace root@accumulo trace> scan – formatted trace entries – root@accumulo trace> formatter -r root@accumulo trace> scan – "normal" trace entries –
        Hide
        Keith Turner added a comment -

        This is neat.

        I was thinking about the situation where different users want different formatters, or one user does not want the formatter. Could make the option per table per user, but I am not sure I like that option it introduces a lot of complication upfront that may not be wanted/needed. For the situation where someone wants to see what data looks like w/o the formatter we could provide an option to the scanner to prevent formatting. This allows someone to quickly see the raw data w/o having to reconfigure the table.

        It seems like the hasArg argument for the formatter Option constructor in the create table command should be true?

        Could use a functional test.

        Show
        Keith Turner added a comment - This is neat. I was thinking about the situation where different users want different formatters, or one user does not want the formatter. Could make the option per table per user, but I am not sure I like that option it introduces a lot of complication upfront that may not be wanted/needed. For the situation where someone wants to see what data looks like w/o the formatter we could provide an option to the scanner to prevent formatting. This allows someone to quickly see the raw data w/o having to reconfigure the table. It seems like the hasArg argument for the formatter Option constructor in the create table command should be true? Could use a functional test.
        Hide
        Josh Elser added a comment -

        I was initially poking around with the idea of being able to set multiple different formatters on a table, but eventually backed it out for the simplicity of a single formatter per table.

        I also have some concern regarding formatter "logic" being in the createtable, formatter, and config commands. I'd be interested in any feedback as to what you think a good strategy is moving forward. Does the formatter command still need to exist if all it's really doing is acting as a wrapper around a config call? Is it beneficial to preserve some backwards compatibility of the formatter command? Does anyone actually use the formatter command now??

        I can look into adding a functional test as well.

        Show
        Josh Elser added a comment - I was initially poking around with the idea of being able to set multiple different formatters on a table, but eventually backed it out for the simplicity of a single formatter per table. I also have some concern regarding formatter "logic" being in the createtable, formatter, and config commands. I'd be interested in any feedback as to what you think a good strategy is moving forward. Does the formatter command still need to exist if all it's really doing is acting as a wrapper around a config call? Is it beneficial to preserve some backwards compatibility of the formatter command? Does anyone actually use the formatter command now?? I can look into adding a functional test as well.
        Hide
        Eric Newton added a comment -

        If you are up to it, could you create a MockShell that would accept lines of input and provide lines of output to test this? It would make testing the shell a unit test, which would be much faster/frequent.

        Show
        Eric Newton added a comment - If you are up to it, could you create a MockShell that would accept lines of input and provide lines of output to test this? It would make testing the shell a unit test, which would be much faster/frequent.
        Hide
        Josh Elser added a comment -

        I plan on working on a MockShell this weekend.

        I'll also double check all of the commands/options that deal with the formatters given Keith's comment.

        Additionally, I'm going to look into adding an option to scan (and the [e]grep) command to re-enable a user to apply a short-lived formatter (not persisted to a table config). A user could create a formatter that is only useful/relevant to him/herself and it would be nice for there to be some isolation among shell sessions.

        Show
        Josh Elser added a comment - I plan on working on a MockShell this weekend. I'll also double check all of the commands/options that deal with the formatters given Keith's comment. Additionally, I'm going to look into adding an option to scan (and the [e] grep) command to re-enable a user to apply a short-lived formatter (not persisted to a table config). A user could create a formatter that is only useful/relevant to him/herself and it would be nice for there to be some isolation among shell sessions.
        Hide
        Dave Marion added a comment -

        Initially the formatter option only applied to the session. This allowed a developer to create a formatter specific to his/her table and drop the jar into the lib/ext directory. I like the idea of making it part of the table configuration. But, what happens if you make it part of the table configuration and the jar does not exist for some reason? I assume that it only affects the shell and either the jar would have to placed back into the classpath or the configuration option removed.

        Show
        Dave Marion added a comment - Initially the formatter option only applied to the session. This allowed a developer to create a formatter specific to his/her table and drop the jar into the lib/ext directory. I like the idea of making it part of the table configuration. But, what happens if you make it part of the table configuration and the jar does not exist for some reason? I assume that it only affects the shell and either the jar would have to placed back into the classpath or the configuration option removed.
        Hide
        Josh Elser added a comment -

        Yes, you would get a ClassNotFoundException if you tried to set a formatter that doesn't exist on the Accumulo classpath. Additionally, if there was one previously set on the table, you would, I think, see the same ClassNotFoundException when you try to switch to that table in the shell. This could very easily happen if someone upgrades Accumulo and forgets to copy the jars from the old Accumulo installation's lib/.

        Is there a desired benefit to falling back onto the DefaultFormatter if the specified formatter cannot be loaded (ClassNotFoundException, ClassCastException, etc)? Having a missing formatter class set will mess up all the scans until removed or the classpath being fixed; however, (Batch)Scanners won't be affected.

        Show
        Josh Elser added a comment - Yes, you would get a ClassNotFoundException if you tried to set a formatter that doesn't exist on the Accumulo classpath. Additionally, if there was one previously set on the table, you would, I think, see the same ClassNotFoundException when you try to switch to that table in the shell. This could very easily happen if someone upgrades Accumulo and forgets to copy the jars from the old Accumulo installation's lib/. Is there a desired benefit to falling back onto the DefaultFormatter if the specified formatter cannot be loaded (ClassNotFoundException, ClassCastException, etc)? Having a missing formatter class set will mess up all the scans until removed or the classpath being fixed; however, (Batch)Scanners won't be affected.
        Hide
        Keith Turner added a comment -

        It might be nice to print a warning if the class can not be found and continue the scan w/ the default.

        Show
        Keith Turner added a comment - It might be nice to print a warning if the class can not be found and continue the scan w/ the default.
        Hide
        Josh Elser added a comment -

        Sorry for the delay on this. Please poke around on this and let me know if I missed anything we discussed above.

        Things covered in ACCUMULO-303-bugfixes-with-mock-shell.patch:

        • Added argument to formatter option in CreateTableCommand
        • Removed any caching of formatter inside of Shell (will always check ZK)
        • Will fall back to DefaultFormatter if the configured Formatter cannot be loaded
        • Added option to ScanCommand to allows users to provide a formatter of their choice to scan with
        • Reworked Shell, created MockShell, and created FormatterCommandTest which uses MockShell
        Show
        Josh Elser added a comment - Sorry for the delay on this. Please poke around on this and let me know if I missed anything we discussed above. Things covered in ACCUMULO-303 -bugfixes-with-mock-shell.patch: Added argument to formatter option in CreateTableCommand Removed any caching of formatter inside of Shell (will always check ZK) Will fall back to DefaultFormatter if the configured Formatter cannot be loaded Added option to ScanCommand to allows users to provide a formatter of their choice to scan with Reworked Shell, created MockShell, and created FormatterCommandTest which uses MockShell
        Hide
        Josh Elser added a comment -

        ACCUMULO-303-bugfixes-with-mock-shell.patch

        Targets version1.4

        Show
        Josh Elser added a comment - ACCUMULO-303 -bugfixes-with-mock-shell.patch Targets version1.4

          People

          • Assignee:
            Keith Turner
            Reporter:
            Josh Elser
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development