Accumulo
  1. Accumulo
  2. ACCUMULO-1849

Migrate tfile buffer configurations to bcfile configuration paramter

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 1.8.0
    • Component/s: tserver
    • Labels:

      Description

      In 1.6.0 we purged the TFile code, because we only use 2 static methods which utilized 2 configurations with tfile in the name. We should migrate those configurations over into a bcfile named convention (as well as document those configurations).

        Issue Links

          Activity

          John Vines created issue -
          Bill Havanki made changes -
          Field Original Value New Value
          Link This issue relates to ACCUMULO-1679 [ ACCUMULO-1679 ]
          Hide
          Matt Dailey added a comment -

          I took a look at this problem, and I was thinking of solving it like this:

          Before:

            private static int getFSOutputBufferSize(Configuration conf) {
              return conf.getInt(FS_OUTPUT_BUF_SIZE_ATTR, 256 * 1024);
            }
          
            private static int getFSInputBufferSize(Configuration conf) {
              return conf.getInt(FS_INPUT_BUF_SIZE_ATTR, 32 * 1024);
            }
          

          After:

            public static final String TFILE_FS_OUTPUT_BUF_SIZE_ATTR = "tfile.fs.output.buffer.size";
            public static final String TFILE_FS_INPUT_BUF_SIZE_ATTR = "tfile.fs.input.buffer.size";
          
            public static final String BCFILE_FS_OUTPUT_BUF_SIZE_ATTR = "bcfile.fs.output.buffer.size";
            public static final String BCFILE_FS_INPUT_BUF_SIZE_ATTR = "bcfile.fs.input.buffer.size";
          
            public static final int DEFAULT_FS_OUTPUT_BUFFER_SIZE = 256 * 1024;
            public static final int DEFAULT_FS_INPUT_BUFFER_SIZE = 32 * 1024;
          
            public static int getFSOutputBufferSize(Configuration conf) {
              return conf.getInt(BCFILE_FS_OUTPUT_BUF_SIZE_ATTR,
                      conf.getInt(TFILE_FS_OUTPUT_BUF_SIZE_ATTR, DEFAULT_FS_OUTPUT_BUFFER_SIZE));
            }
          
            public static int getFSInputBufferSize(Configuration conf) {
              return conf.getInt(BCFILE_FS_INPUT_BUF_SIZE_ATTR,
                      conf.getInt(TFILE_FS_INPUT_BUF_SIZE_ATTR, DEFAULT_FS_INPUT_BUFFER_SIZE));
            }
          

          Changes:

          • Adds bcfile.x.y.z parameters, which are read from the Configuration with higher precedence than the tfile.x.y.z parameters.
          • All of the constants and the static methods are public.

          I really made things public to assist in unit testing, but wasn't sure about side-effects of that change. This class isn't part of the public API, so I was unsure of how much churn is acceptable. Thoughts?

          Show
          Matt Dailey added a comment - I took a look at this problem, and I was thinking of solving it like this: Before: private static int getFSOutputBufferSize(Configuration conf) { return conf.getInt(FS_OUTPUT_BUF_SIZE_ATTR, 256 * 1024); } private static int getFSInputBufferSize(Configuration conf) { return conf.getInt(FS_INPUT_BUF_SIZE_ATTR, 32 * 1024); } After: public static final String TFILE_FS_OUTPUT_BUF_SIZE_ATTR = "tfile.fs.output.buffer.size" ; public static final String TFILE_FS_INPUT_BUF_SIZE_ATTR = "tfile.fs.input.buffer.size" ; public static final String BCFILE_FS_OUTPUT_BUF_SIZE_ATTR = "bcfile.fs.output.buffer.size" ; public static final String BCFILE_FS_INPUT_BUF_SIZE_ATTR = "bcfile.fs.input.buffer.size" ; public static final int DEFAULT_FS_OUTPUT_BUFFER_SIZE = 256 * 1024; public static final int DEFAULT_FS_INPUT_BUFFER_SIZE = 32 * 1024; public static int getFSOutputBufferSize(Configuration conf) { return conf.getInt(BCFILE_FS_OUTPUT_BUF_SIZE_ATTR, conf.getInt(TFILE_FS_OUTPUT_BUF_SIZE_ATTR, DEFAULT_FS_OUTPUT_BUFFER_SIZE)); } public static int getFSInputBufferSize(Configuration conf) { return conf.getInt(BCFILE_FS_INPUT_BUF_SIZE_ATTR, conf.getInt(TFILE_FS_INPUT_BUF_SIZE_ATTR, DEFAULT_FS_INPUT_BUFFER_SIZE)); } Changes: Adds bcfile.x.y.z parameters, which are read from the Configuration with higher precedence than the tfile.x.y.z parameters. All of the constants and the static methods are public. I really made things public to assist in unit testing, but wasn't sure about side-effects of that change. This class isn't part of the public API, so I was unsure of how much churn is acceptable. Thoughts?
          Josh Elser made changes -
          Fix Version/s 1.8.0 [ 12329879 ]
          Fix Version/s 1.7.0 [ 12324607 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              John Vines
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:

                Development