Major Description
NOTE: This issue is a proposal. I assigned it to myself to make it clear that it's not ready to implement, I just start a discussion about the proposed change.
org.apache.hadoop.utils.db.DBStore (from HDDS-356) is a new generation MetadataStore to store all persistent state of hdds/ozone scm/om/datanodes.
It supports column families with via the Table interface which supports methods like:
byte[] get(byte[] key) throws IOException; void put(byte[] key, byte[] value)
In our current code we usually use static helpers to do the byte[] -> object and object -> byte[] conversion with protobuf.
For example in KeyManagerImpl the OmKeyInfo.getFromProtobuf is used multiple times to deserialize the OmKeyInfo project.
I propose to create a type-safe table with using:
public interface Table<KEY_TYPE, VALUE_TYPE> extends AutoCloseable
The put and get could be modified to:
VALUE_TYPE[] get(KEY_TYPE[] key) throws IOException;
void put(KEY_TYPE[] key, VALUE_TYPE value)
For example for the key table it could be:
OmKeyInfo get(String key) throws IOException; void put(String key, OmKeyInfo value)
It requires to register internal codec (marshaller/unmarshaller) implementations during the creation of (..)Table.
The registration of the codecs would be optional. Without it the Table could work as now (using byte[],byte[])
Advantages:
- More simplified code (Don't need to repeat the serialization everywhere) less error-prone.
- Clear separation of the layers (As of now I can't see the serialization overhead with OpenTracing) and measurablity). Easy to test different serialization in the future.
- Easier to create additional developer tools to investigate the current state of the rocksdb metadata stores. We had SQLCLI to export all the data to sql, but with registering the format in the rocksdb table we can easily create a calcite based SQL console.
Additional info:
I would modify the interface of the DBStoreBuilder and DBStore:
this.store = DBStoreBuilder.newBuilder(conf) .setName(OM_DB_NAME) .setPath(Paths.get(metaDir.getPath())) .addTable(KEY_TABLE, DBUtil.STRING_KEY_CODEC, new OmKeyInfoCoder()) //... .build();
And using it from the DBStore:
//default, without codec Table<byte[],byte[]> getTable(String name) throws IOException; //advanced with codec from the codec registry Table<String,OmKeyInfo> getTable(String name, Class keyType, Class valueType); //for example table.getTable(KEY_TABLE,String.class,OmKeyInfo.class); //or table.getTable(KEY_TABLE,String.class,UserInfo.class) //exception is thrown: No codec is registered for KEY_TABLE with type UserInfo.
Priority:
I think it's a very useful and valuable step forward but the real priority is lower. Ideal for new contributors especially as it's independent, standalone part of ozone code.