Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3086

Supporting range scan using TFile, TotalOrderPartitioner and partition index


    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s:
    • Fix Version/s: None
    • Component/s: None
    • Labels:


      Hive/HBase already has similar or more powerful functionality, but using hive/hbase is overkill or inconvenient for some cases, so add some lightweight utility classes to only support range scan should be reasonable. The utility classes include:

      1. InputFormat supporting range scan: Indexed(Text|Binary)InputFormat
        The input directory for IndexInputFormat should contain one partition index and many tfiles, each tfile store a certain range of keys, not overlapping with other tfiles, the boundaries are stored in partition index.
        Add 4 jobconfs: mapred.indexed(text|binary)inputformat.key.(start|end), indicate range scan parameters.
        For a mapreduce job using IndexedInputFormat, IndexedInputFormat.getSplits filter out tfiles which are not in the scan range using partition index
        IndexedInputFormat do not support multi directory & splitting in single file, these can be added in future.
      2. Tool to convert data of other format into IndexedInputForamt: TotalOrderIndexBuilder
        If the input data is already total order partitioned and is tfile format, just add partition index to input directory
        Or run InputSampler to generate partiton index, then run mapreduce job with TotalOrder partitioner to generate tfile backed data, finally move partition index to output directory.
      3. Client tool to scan/search indexed data directory




            • Assignee:
              decster Binglin Chang
              decster Binglin Chang
            • Votes:
              0 Vote for this issue
              7 Start watching this issue


              • Created: