[TAJO-1614] Configuration format proposal for generic storage support

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Fix Version/s: 0.11.0
• Component/s: Storage
• Assignee: Hyunsik Choi
• Reporter: Hyunsik Choi
• Created: 19/May/15 00:39
• Updated: 04/Sep/15 02:14
• Resolved: 04/Sep/15 02:14

Description

This is an example of the first draft of new storage configuration file for generic storage. It allows users to specify:

• table spaces
• space handlers
• data formats
• relationships among them

More detailed explanation is as follows:

• /storages specifies all storage types.
• /storages/* specifies which handler deals with which uri scheme.
• /formats can specify each data format or file format
• /formats/* specifies which format is available in which storage types.
• /formats/*/handler specifies a handler class to allows clients to get scanner or appender.

{
  "spaces": {
    "default": {
      "uri": "hdfs://localhost:8020/tajo/warehouse",
      "configs": [
        {"x": "y"},
        {"x": "y"}
      ]
    },
    "ssd1": {
      "uri": "hdfs://localhost:8020/....",
      "configs": [
        {"x": "y"},
        {"x": "y"}
      ]
    },
    "mysql": {
      "uri": "jdbc://....",
      "configs": [
        {"x": "y"},
        {"x": "y"}
      ]
    }
  },
  
  "storages": {
    "hdfs": {
      "handler": "org.apache.tajo.storage.HdfsTablespace",
      "default-format": "text"
    },
    "file": {
      "handler": "org.apache.tajo.storage.FileTablespace",
      "default-format": "text"
    },
    "hbase": {
      "handler": "org.apache.tajo.storage.hbase.HBaseTablespace",
      "default-format": "hbase"
    },
    "s3": {
      "handler": "org.apache.tajo.storage.S3Tablespace",
      "default-format": "text"
    },
    "jdbc": {
      "handler": "org.apache.tajo.storage.JdbcTablespace",
      "default-format": "rowstore"
    }
  },
  
  "formats": {
    
    "avro": {
      "storage-support": ["hdfs", "file", "s3"],
      "handler": "org.apache.tajo.storage.AvroHandler"
    },
    "csv": {
      "storage-support": ["hdfs", "file", "s3"],
      "handler": "org.apache.tajo.storage.CSVHandler"
    },
    "text": {
      "storage-support": ["hdfs", "file", "s3"],
      "handler": "org.apache.tajo.storage.TextHandler"
    },
    "parquet": {
      "storage-support": ["hdfs", "file", "s3"],
      "handler": "org.apache.tajo.storage.ParquetHandler"
    },
    "raw": {
      "storage-support": ["hdfs", "file", "s3"],
      "handler": "org.apache.tajo.storage.RawHandler"
    },
    "rcfile": {
      "storage-support": ["hdfs", "file", "s3"],
      "handler": "org.apache.tajo.storage.RCFileHandler"
    },
    "row": {
      "storage-support": ["hdfs", "file", "s3"],
      "handler": "org.apache.tajo.storage.RowHandler"
    },
    "sequencefile": {
      "storage-support": ["hdfs", "file", "s3"],
      "handler": "org.apache.tajo.storage.SeqFileHandler"
    },
    
    
    "hbase": {
      "storage-support": ["hbase"],
      "handler": "org.apache.tajo.storage.HbaseHandler"
    },
    "kyro": {
      "storage-support": ["hbase"],
      "handler": "org.apache.tajo.storage.KyroHandler"
    },
    
    "rowstore": {
      "storage-support": ["jdbc"],
      "handler": "org.apache.tajo.storage.RowStoreHandler"
    }
  }
}