Details
Description
This patch add a MetadataProvider interface and LinkedMetadataManager;
- MetadataProvider provides a way to access linked meta data store through an unified API.
- LinkedMetadataManager manages all MetadataProvider instances.
Issue Links
- is related to
-
TAJO-1652
Automatic metadata registeration from underlying storages
-
- Resolved
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Github user blrunner commented on the pull request:
https://github.com/apache/tajo/pull/673#issuecomment-127482793
Hi @hyunsik
Could you rebase it? 
Github user blrunner commented on the pull request:
https://github.com/apache/tajo/pull/673#issuecomment-127507990
Overall, it looks good to me. And if you add more unit test cases for InsufficientPrivilegeException, it would be more nice.
Github user blrunner commented on a diff in the pull request:
https://github.com/apache/tajo/pull/673#discussion_r36164948
— Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java —
@@ -97,19 +103,17 @@
public CatalogServer() throws IOException
- public CatalogServer(Collection<FunctionDesc> sqlFuncs) throws IOException {
- this();
+ public CatalogServer(Set<MetadataProvider> metadataProviders, Collection<FunctionDesc> sqlFuncs) throws IOException {-
- End diff –
-
Why don't you implement a constructor which needs only MetadataProvider?
Github user blrunner commented on a diff in the pull request:
https://github.com/apache/tajo/pull/673#discussion_r36164997
— Diff: tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java —
@@ -24,6 +24,7 @@
import org.apache.tajo.catalog.proto.CatalogProtos.*;
import org.apache.tajo.common.TajoDataTypes.DataType;
+import javax.annotation.Nullable;
— End diff –
It looks like unused package.
Github user blrunner commented on a diff in the pull request:
https://github.com/apache/tajo/pull/673#discussion_r36165105
— Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java —
@@ -304,9 +295,25 @@ public GetTablespaceListResponse getAllTablespaces(RpcController controller, Nul
throws ServiceException {
rlock.lock();
try {
+
+ // retrieves tablespaces from catalog store
+ final List<TablespaceProto> tableSpaces = Lists.newArrayList(store.getTablespaces());
+
+ // retrieves tablespaces from linked meta data
+ tableSpaces.addAll(Collections2.transform(linkedMetadataManager.getTablespaces(),
— End diff –
Could you explain why you use Collections2?
Github user blrunner commented on a diff in the pull request:
https://github.com/apache/tajo/pull/673#discussion_r36165634
— Diff: tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestLinkedMetadataManager.java —
@@ -0,0 +1,270 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tajo.catalog;
+
+import com.google.common.base.Function;
+import com.google.common.collect.Collections2;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import org.apache.hadoop.fs.Path;
+import org.apache.tajo.TajoConstants;
+import org.apache.tajo.catalog.exception.UndefinedTablespaceException;
+import org.apache.tajo.catalog.proto.CatalogProtos;
+import org.apache.tajo.common.TajoDataTypes.Type;
+import org.apache.tajo.conf.TajoConf;
+import org.apache.tajo.error.Errors;
+import org.apache.tajo.exception.TajoInternalError;
+import org.apache.tajo.util.CommonTestingUtil;
+import org.apache.tajo.util.KeyValueSet;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Collections;
+
+import static org.apache.tajo.TajoConstants.DEFAULT_DATABASE_NAME;
+import static org.junit.Assert.*;
+
+public class TestLinkedMetadataManager {
+
+ static TableDesc TABLE1 = new TableDesc(
+ "table1",
+ new Schema(new Column[]
+ "TEXT", new KeyValueSet(), URI.create("http://space1/x/table1")
+ );
+
+ static TableDesc TABLE2 = new TableDesc(
+ "table2",
+ new Schema(new Column[]{new Column("c1", Type.INT8)}
),
+ "TEXT", new KeyValueSet(), URI.create("http://space1/x/table2")
+ );
+
+ static TableDesc TABLE3 = new TableDesc(
+ "table3",
+ new Schema(new Column[]
+ "TEXT", new KeyValueSet(), URI.create("http://space1/x/table3")
+ );
+
+ static TableDesc TABLE4 = new TableDesc(
+ "table4",
+ new Schema(new Column[]{new Column("c1", Type.INT8)}
),
+ "TEXT", new KeyValueSet(), URI.create("http://space1/x/table4")
+ );
+
+ static class MockupMetadataProvider1 implements MetadataProvider {
+
+ @Override
+ public String getTablespaceName()
+
+ @Override
+ public URI getTablespaceUri() { + return URI.create("http://space1/x"); + }
+
+ @Override
+ public String getDatabaseName() { + return "space1"; + }
+
+ @Override
+ public Collection<String> getCatalogs()
+
+ @Override
+ public Collection<String> getTables(@Nullable String catalog)
+
+ @Override
+ public TableDesc getTableDescriptor(String catalogName, String tableName) {
+ if (tableName.equals("table1"))
else if (tableName.equals("table2"))
{ + return TABLE2; + } +
+ throw new UndefinedTablespaceException(tableName);
+ }
+ }
+
+ static class MockupMetadataProvider2 implements MetadataProvider {
+
+ @Override
+ public String getTablespaceName()
+
+ @Override
+ public URI getTablespaceUri() { + return URI.create("http://space2/y"); + }
+
+ @Override
+ public String getDatabaseName() { + return "space2"; + }
+
+ @Override
+ public Collection<String> getCatalogs()
+
+ @Override
+ public Collection<String> getTables(@Nullable String catalog)
+
+ @Override
+ public TableDesc getTableDescriptor(String catalogName, String tableName) {
+ if (tableName.equals("table3"))
else if (tableName.equals("table4"))
{ + return TABLE4; + } +
+ throw new UndefinedTablespaceException(tableName);
+ }
+ }
+
+ static CatalogServer server;
+ static CatalogService catalog;
+
+ @BeforeClass
+ public static void setUp() throws IOException {
+ TajoConf conf = new TajoConf();
+ conf.setVar(TajoConf.ConfVars.CATALOG_ADDRESS, "127.0.0.1:0");
+
+ server = new CatalogServer(
+ Sets.newHashSet(new MockupMetadataProvider1(), new MockupMetadataProvider2()), Collections.EMPTY_LIST);
+ server.init(TestCatalog.newTajoConfForCatalogTest());
+ server.start();
+ catalog = new LocalCatalogWrapper(server);
+
+ Path defaultTableSpace = CommonTestingUtil.getTestDir();
+
+ if (!catalog.existTablespace(TajoConstants.DEFAULT_TABLESPACE_NAME))
+ if (!catalog.existDatabase(DEFAULT_DATABASE_NAME))
{ + catalog.createDatabase(DEFAULT_DATABASE_NAME, TajoConstants.DEFAULT_TABLESPACE_NAME); + } + }
+
+ @AfterClass
+ public static void tearDown() throws IOException
+
+ @Test
+ public void testGetTablespaceNames() throws Exception
+
+ @Test
+ public void testGetTablespace() throws Exception
+
+ @Test
+ public void testGetTablespaces() throws Exception {
+ Collection<String> names = Collections2.transform(catalog.getAllTablespaces(), new Function<CatalogProtos.TablespaceProto, String>() {
— End diff –
Exceeded character limit for one line.
Github user hyunsik commented on a diff in the pull request:
https://github.com/apache/tajo/pull/673#discussion_r36168885
— Diff: tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestLinkedMetadataManager.java —
@@ -0,0 +1,270 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tajo.catalog;
+
+import com.google.common.base.Function;
+import com.google.common.collect.Collections2;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import org.apache.hadoop.fs.Path;
+import org.apache.tajo.TajoConstants;
+import org.apache.tajo.catalog.exception.UndefinedTablespaceException;
+import org.apache.tajo.catalog.proto.CatalogProtos;
+import org.apache.tajo.common.TajoDataTypes.Type;
+import org.apache.tajo.conf.TajoConf;
+import org.apache.tajo.error.Errors;
+import org.apache.tajo.exception.TajoInternalError;
+import org.apache.tajo.util.CommonTestingUtil;
+import org.apache.tajo.util.KeyValueSet;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Collections;
+
+import static org.apache.tajo.TajoConstants.DEFAULT_DATABASE_NAME;
+import static org.junit.Assert.*;
+
+public class TestLinkedMetadataManager {
+
+ static TableDesc TABLE1 = new TableDesc(
+ "table1",
+ new Schema(new Column[]
+ "TEXT", new KeyValueSet(), URI.create("http://space1/x/table1")
+ );
+
+ static TableDesc TABLE2 = new TableDesc(
+ "table2",
+ new Schema(new Column[]{new Column("c1", Type.INT8)}
),
+ "TEXT", new KeyValueSet(), URI.create("http://space1/x/table2")
+ );
+
+ static TableDesc TABLE3 = new TableDesc(
+ "table3",
+ new Schema(new Column[]
+ "TEXT", new KeyValueSet(), URI.create("http://space1/x/table3")
+ );
+
+ static TableDesc TABLE4 = new TableDesc(
+ "table4",
+ new Schema(new Column[]{new Column("c1", Type.INT8)}
),
+ "TEXT", new KeyValueSet(), URI.create("http://space1/x/table4")
+ );
+
+ static class MockupMetadataProvider1 implements MetadataProvider {
+
+ @Override
+ public String getTablespaceName()
+
+ @Override
+ public URI getTablespaceUri() { + return URI.create("http://space1/x"); + }
+
+ @Override
+ public String getDatabaseName() { + return "space1"; + }
+
+ @Override
+ public Collection<String> getCatalogs()
+
+ @Override
+ public Collection<String> getTables(@Nullable String catalog)
+
+ @Override
+ public TableDesc getTableDescriptor(String catalogName, String tableName) {
+ if (tableName.equals("table1"))
else if (tableName.equals("table2"))
{ + return TABLE2; + } +
+ throw new UndefinedTablespaceException(tableName);
+ }
+ }
+
+ static class MockupMetadataProvider2 implements MetadataProvider {
+
+ @Override
+ public String getTablespaceName()
+
+ @Override
+ public URI getTablespaceUri() { + return URI.create("http://space2/y"); + }
+
+ @Override
+ public String getDatabaseName() { + return "space2"; + }
+
+ @Override
+ public Collection<String> getCatalogs()
+
+ @Override
+ public Collection<String> getTables(@Nullable String catalog)
+
+ @Override
+ public TableDesc getTableDescriptor(String catalogName, String tableName) {
+ if (tableName.equals("table3"))
else if (tableName.equals("table4"))
{ + return TABLE4; + } +
+ throw new UndefinedTablespaceException(tableName);
+ }
+ }
+
+ static CatalogServer server;
+ static CatalogService catalog;
+
+ @BeforeClass
+ public static void setUp() throws IOException {
+ TajoConf conf = new TajoConf();
+ conf.setVar(TajoConf.ConfVars.CATALOG_ADDRESS, "127.0.0.1:0");
+
+ server = new CatalogServer(
+ Sets.newHashSet(new MockupMetadataProvider1(), new MockupMetadataProvider2()), Collections.EMPTY_LIST);
+ server.init(TestCatalog.newTajoConfForCatalogTest());
+ server.start();
+ catalog = new LocalCatalogWrapper(server);
+
+ Path defaultTableSpace = CommonTestingUtil.getTestDir();
+
+ if (!catalog.existTablespace(TajoConstants.DEFAULT_TABLESPACE_NAME))
+ if (!catalog.existDatabase(DEFAULT_DATABASE_NAME))
{ + catalog.createDatabase(DEFAULT_DATABASE_NAME, TajoConstants.DEFAULT_TABLESPACE_NAME); + } + }
+
+ @AfterClass
+ public static void tearDown() throws IOException
+
+ @Test
+ public void testGetTablespaceNames() throws Exception
+
+ @Test
+ public void testGetTablespace() throws Exception
+
+ @Test
+ public void testGetTablespaces() throws Exception {
+ Collection<String> names = Collections2.transform(catalog.getAllTablespaces(), new Function<CatalogProtos.TablespaceProto, String>() {
— End diff –
fixed
Github user hyunsik commented on a diff in the pull request:
https://github.com/apache/tajo/pull/673#discussion_r36168924
— Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java —
@@ -304,9 +295,25 @@ public GetTablespaceListResponse getAllTablespaces(RpcController controller, Nul
throws ServiceException {
rlock.lock();
try {
+
+ // retrieves tablespaces from catalog store
+ final List<TablespaceProto> tableSpaces = Lists.newArrayList(store.getTablespaces());
+
+ // retrieves tablespaces from linked meta data
+ tableSpaces.addAll(Collections2.transform(linkedMetadataManager.getTablespaces(),
— End diff –
Some functional methods like transform and filter are in guava Collections2 class.
Github user hyunsik commented on a diff in the pull request:
https://github.com/apache/tajo/pull/673#discussion_r36169035
— Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java —
@@ -97,19 +103,17 @@
public CatalogServer() throws IOException
- public CatalogServer(Collection<FunctionDesc> sqlFuncs) throws IOException {
- this();
+ public CatalogServer(Set<MetadataProvider> metadataProviders, Collection<FunctionDesc> sqlFuncs) throws IOException {-
- End diff –
-
It is because the constructor is unnecessary.
Github user hyunsik commented on the pull request:
https://github.com/apache/tajo/pull/673#issuecomment-127533418
rebased and reflected the comments.
Github user blrunner commented on the pull request:
https://github.com/apache/tajo/pull/673#issuecomment-127791640
+1
LGTM.
Github user asfgit closed the pull request at:
FAILURE: Integrated in Tajo-master-CODEGEN-build #431 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/431/)
TAJO-1735: Implement MetadataProvider and LinkedMetadataManager. (hyunsik: rev f868c0e237f7523f429d0196e67652e980ca5907)
- tajo-core/src/main/java/org/apache/tajo/master/TajoMaster.java
- tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java
- tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestLinkedMetadataManager.java
- tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/MetadataProvider.java
- tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/InsufficientPrivilegeException.java
- CHANGES
- tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java
- tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/dictionary/InfoSchemaMetadataDictionary.java
- tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java
- tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/LinkedMetadataManager.java
committed.
FAILURE: Integrated in Tajo-master-build #793 (See https://builds.apache.org/job/Tajo-master-build/793/)
TAJO-1735: Implement MetadataProvider and LinkedMetadataManager. (hyunsik: rev f868c0e237f7523f429d0196e67652e980ca5907)
- tajo-core/src/main/java/org/apache/tajo/master/TajoMaster.java
- tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/LinkedMetadataManager.java
- tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java
- tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestLinkedMetadataManager.java
- CHANGES
- tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/MetadataProvider.java
- tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/InsufficientPrivilegeException.java
- tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java
- tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/dictionary/InfoSchemaMetadataDictionary.java
- tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java
GitHub user hyunsik opened a pull request:
https://github.com/apache/tajo/pull/673
TAJO-1735: Implement MetadataProvider and LinkedMetadataManager.You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hyunsik/tajo TAJO-1734
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/tajo/pull/673.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #673
commit 1cc4fb87fd218fd592934112e6709c2e81c0c0e6
Author: Hyunsik Choi <hyunsik@apache.org>
Date: 2015-07-31T09:25:46Z
TAJO-1735: Implement MetadataProvider and LinkedMetadataManager.