Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8
    • Component/s: gora-orientdb
    • Labels:
      None

      Description

      OrientDB [1] client [2] query builder [3] comes under Apache license.
      [1] http://orientdb.com/orientdb/
      [2] https://github.com/orientechnologies/orientdb
      [3] https://github.com/raymanrt/orientqb

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user djkevincr opened a pull request:

          https://github.com/apache/gora/pull/112

          GORA-513 Add initial OrientDB datastore impl.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/djkevincr/gora GORA-513

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/gora/pull/112.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #112


          commit 71a95b98f37d268eeafd2fb653ee1b2158416eaf
          Author: Kevin Ratnasekera <djkevincr@yahoo.com>
          Date: 2017-07-11T21:38:24Z

          GORA-513 Add initial OrientDB datastore impl


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user djkevincr opened a pull request: https://github.com/apache/gora/pull/112 GORA-513 Add initial OrientDB datastore impl. You can merge this pull request into a Git repository by running: $ git pull https://github.com/djkevincr/gora GORA-513 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/gora/pull/112.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #112 commit 71a95b98f37d268eeafd2fb653ee1b2158416eaf Author: Kevin Ratnasekera <djkevincr@yahoo.com> Date: 2017-07-11T21:38:24Z GORA-513 Add initial OrientDB datastore impl
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user lewismc commented on the issue:

          https://github.com/apache/gora/pull/112

          #winning

          Show
          githubbot ASF GitHub Bot added a comment - Github user lewismc commented on the issue: https://github.com/apache/gora/pull/112 #winning
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user djkevincr commented on the issue:

          https://github.com/apache/gora/pull/112

          @lewismc Can you please have look? I was able to pass all the base datastore tests with this PR. I will further improve datastore test cases , mapping test cases and some error handling stuff.

          Show
          githubbot ASF GitHub Bot added a comment - Github user djkevincr commented on the issue: https://github.com/apache/gora/pull/112 @lewismc Can you please have look? I was able to pass all the base datastore tests with this PR. I will further improve datastore test cases , mapping test cases and some error handling stuff.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user renato2099 commented on a diff in the pull request:

          https://github.com/apache/gora/pull/112#discussion_r127613910

          — Diff: gora-orientdb/src/main/java/org/apache/gora/orientdb/store/OrientDBStore.java —
          @@ -0,0 +1,922 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.gora.orientdb.store;
          +
          +import java.io.IOException;
          +import java.nio.ByteBuffer;
          +import java.util.Map;
          +import java.util.Properties;
          +import java.util.List;
          +import java.util.HashMap;
          +import java.util.HashSet;
          +import java.util.ArrayList;
          +import java.util.Date;
          +import java.util.Calendar;
          +import java.util.Collection;
          +import java.util.TimeZone;
          +import java.util.Locale;
          +
          +import com.github.raymanrt.orientqb.query.Parameter;
          +import com.gitub.raymanrt.orientqb.delete.Delete;
          +import com.orientechnologies.orient.client.remote.OServerAdmin;
          +import com.orientechnologies.orient.core.db.OPartitionedDatabasePool;
          +import com.orientechnologies.orient.core.db.OPartitionedDatabasePoolFactory;
          +import com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx;
          +import com.orientechnologies.orient.core.db.record.OTrackedList;
          +import com.orientechnologies.orient.core.db.record.OTrackedMap;
          +import com.orientechnologies.orient.core.db.record.OTrackedSet;
          +import com.orientechnologies.orient.core.metadata.schema.OClass;
          +import com.orientechnologies.orient.core.metadata.schema.OType;
          +import com.orientechnologies.orient.core.record.impl.ODocument;
          +import com.orientechnologies.orient.core.sql.OCommandSQL;
          +import com.orientechnologies.orient.core.sql.query.OConcurrentResultSet;
          +import com.orientechnologies.orient.core.sql.query.OSQLSynchQuery;
          +import org.apache.avro.Schema;
          +import org.apache.avro.util.Utf8;
          +import org.apache.gora.orientdb.query.OrientDBQuery;
          +import org.apache.gora.orientdb.query.OrientDBResult;
          +import org.apache.gora.persistency.impl.BeanFactoryImpl;
          +import org.apache.gora.persistency.impl.DirtyListWrapper;
          +import org.apache.gora.persistency.impl.DirtyMapWrapper;
          +import org.apache.gora.persistency.impl.PersistentBase;
          +import org.apache.gora.query.PartitionQuery;
          +import org.apache.gora.query.Query;
          +import org.apache.gora.query.Result;
          +import org.apache.gora.query.impl.PartitionQueryImpl;
          +import org.apache.gora.store.impl.DataStoreBase;
          +import org.apache.gora.util.AvroUtils;
          +import org.apache.gora.util.ClassLoadingUtils;
          +
          +import javax.xml.bind.DatatypeConverter;
          +
          +import static com.github.raymanrt.orientqb.query.Projection.projection;
          +
          +/**
          + *

          {@link org.apache.gora.orientdb.store.OrientDBStore}

          is the primary class
          + * responsible for facilitating GORA CRUD operations on OrientDB documents.
          + */
          +public class OrientDBStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
          +
          + public static final String DEFAULT_MAPPING_FILE = "/gora-orientdb-mapping.xml";
          + private String ROOT_URL;
          + private String ROOT_DATABASE_URL;
          + private OrientDBStoreParameters orientDbStoreParams;
          + private OrientDBMapping orientDBMapping;
          + private OServerAdmin remoteServerAdmin;
          + private OPartitionedDatabasePool connectionPool;
          + private List<ODocument> docBatch = new ArrayList<>();
          — End diff –

          QQ: would there be a reason to make this docBatch non-thread-safe? I am thinking if we ever use the same GoraConnection shared by multiple threads, this would have to change right? @lewismc do you know if this would affect Nutch?

          Show
          githubbot ASF GitHub Bot added a comment - Github user renato2099 commented on a diff in the pull request: https://github.com/apache/gora/pull/112#discussion_r127613910 — Diff: gora-orientdb/src/main/java/org/apache/gora/orientdb/store/OrientDBStore.java — @@ -0,0 +1,922 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.gora.orientdb.store; + +import java.io.IOException; +import java.nio.ByteBuffer; +import java.util.Map; +import java.util.Properties; +import java.util.List; +import java.util.HashMap; +import java.util.HashSet; +import java.util.ArrayList; +import java.util.Date; +import java.util.Calendar; +import java.util.Collection; +import java.util.TimeZone; +import java.util.Locale; + +import com.github.raymanrt.orientqb.query.Parameter; +import com.gitub.raymanrt.orientqb.delete.Delete; +import com.orientechnologies.orient.client.remote.OServerAdmin; +import com.orientechnologies.orient.core.db.OPartitionedDatabasePool; +import com.orientechnologies.orient.core.db.OPartitionedDatabasePoolFactory; +import com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx; +import com.orientechnologies.orient.core.db.record.OTrackedList; +import com.orientechnologies.orient.core.db.record.OTrackedMap; +import com.orientechnologies.orient.core.db.record.OTrackedSet; +import com.orientechnologies.orient.core.metadata.schema.OClass; +import com.orientechnologies.orient.core.metadata.schema.OType; +import com.orientechnologies.orient.core.record.impl.ODocument; +import com.orientechnologies.orient.core.sql.OCommandSQL; +import com.orientechnologies.orient.core.sql.query.OConcurrentResultSet; +import com.orientechnologies.orient.core.sql.query.OSQLSynchQuery; +import org.apache.avro.Schema; +import org.apache.avro.util.Utf8; +import org.apache.gora.orientdb.query.OrientDBQuery; +import org.apache.gora.orientdb.query.OrientDBResult; +import org.apache.gora.persistency.impl.BeanFactoryImpl; +import org.apache.gora.persistency.impl.DirtyListWrapper; +import org.apache.gora.persistency.impl.DirtyMapWrapper; +import org.apache.gora.persistency.impl.PersistentBase; +import org.apache.gora.query.PartitionQuery; +import org.apache.gora.query.Query; +import org.apache.gora.query.Result; +import org.apache.gora.query.impl.PartitionQueryImpl; +import org.apache.gora.store.impl.DataStoreBase; +import org.apache.gora.util.AvroUtils; +import org.apache.gora.util.ClassLoadingUtils; + +import javax.xml.bind.DatatypeConverter; + +import static com.github.raymanrt.orientqb.query.Projection.projection; + +/** + * {@link org.apache.gora.orientdb.store.OrientDBStore} is the primary class + * responsible for facilitating GORA CRUD operations on OrientDB documents. + */ +public class OrientDBStore<K, T extends PersistentBase> extends DataStoreBase<K, T> { + + public static final String DEFAULT_MAPPING_FILE = "/gora-orientdb-mapping.xml"; + private String ROOT_URL; + private String ROOT_DATABASE_URL; + private OrientDBStoreParameters orientDbStoreParams; + private OrientDBMapping orientDBMapping; + private OServerAdmin remoteServerAdmin; + private OPartitionedDatabasePool connectionPool; + private List<ODocument> docBatch = new ArrayList<>(); — End diff – QQ: would there be a reason to make this docBatch non-thread-safe? I am thinking if we ever use the same GoraConnection shared by multiple threads, this would have to change right? @lewismc do you know if this would affect Nutch?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user renato2099 commented on a diff in the pull request:

          https://github.com/apache/gora/pull/112#discussion_r127613861

          — Diff: gora-orientdb/src/main/java/org/apache/gora/orientdb/store/OrientDBStore.java —
          @@ -0,0 +1,922 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.gora.orientdb.store;
          +
          +import java.io.IOException;
          +import java.nio.ByteBuffer;
          +import java.util.Map;
          +import java.util.Properties;
          +import java.util.List;
          +import java.util.HashMap;
          +import java.util.HashSet;
          +import java.util.ArrayList;
          +import java.util.Date;
          +import java.util.Calendar;
          +import java.util.Collection;
          +import java.util.TimeZone;
          +import java.util.Locale;
          +
          +import com.github.raymanrt.orientqb.query.Parameter;
          +import com.gitub.raymanrt.orientqb.delete.Delete;
          +import com.orientechnologies.orient.client.remote.OServerAdmin;
          +import com.orientechnologies.orient.core.db.OPartitionedDatabasePool;
          +import com.orientechnologies.orient.core.db.OPartitionedDatabasePoolFactory;
          +import com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx;
          +import com.orientechnologies.orient.core.db.record.OTrackedList;
          +import com.orientechnologies.orient.core.db.record.OTrackedMap;
          +import com.orientechnologies.orient.core.db.record.OTrackedSet;
          +import com.orientechnologies.orient.core.metadata.schema.OClass;
          +import com.orientechnologies.orient.core.metadata.schema.OType;
          +import com.orientechnologies.orient.core.record.impl.ODocument;
          +import com.orientechnologies.orient.core.sql.OCommandSQL;
          +import com.orientechnologies.orient.core.sql.query.OConcurrentResultSet;
          +import com.orientechnologies.orient.core.sql.query.OSQLSynchQuery;
          +import org.apache.avro.Schema;
          +import org.apache.avro.util.Utf8;
          +import org.apache.gora.orientdb.query.OrientDBQuery;
          +import org.apache.gora.orientdb.query.OrientDBResult;
          +import org.apache.gora.persistency.impl.BeanFactoryImpl;
          +import org.apache.gora.persistency.impl.DirtyListWrapper;
          +import org.apache.gora.persistency.impl.DirtyMapWrapper;
          +import org.apache.gora.persistency.impl.PersistentBase;
          +import org.apache.gora.query.PartitionQuery;
          +import org.apache.gora.query.Query;
          +import org.apache.gora.query.Result;
          +import org.apache.gora.query.impl.PartitionQueryImpl;
          +import org.apache.gora.store.impl.DataStoreBase;
          +import org.apache.gora.util.AvroUtils;
          +import org.apache.gora.util.ClassLoadingUtils;
          +
          +import javax.xml.bind.DatatypeConverter;
          +
          +import static com.github.raymanrt.orientqb.query.Projection.projection;
          +
          +/**
          + *

          {@link org.apache.gora.orientdb.store.OrientDBStore}

          is the primary class
          + * responsible for facilitating GORA CRUD operations on OrientDB documents.
          + */
          +public class OrientDBStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
          +
          + public static final String DEFAULT_MAPPING_FILE = "/gora-orientdb-mapping.xml";
          + private String ROOT_URL;
          + private String ROOT_DATABASE_URL;
          + private OrientDBStoreParameters orientDbStoreParams;
          + private OrientDBMapping orientDBMapping;
          + private OServerAdmin remoteServerAdmin;
          + private OPartitionedDatabasePool connectionPool;
          + private List<ODocument> docBatch = new ArrayList<>();
          +
          + /**
          + * Initialize the OrientDB dataStore by

          {@link Properties}

          parameters.
          + *
          + * @param keyClass key class type for dataStore.
          + * @param persistentClass persistent class type for dataStore.
          + * @param properties OrientDB dataStore properties EG:- OrientDB client credentials.
          + */
          + @Override
          + public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) {
          + super.initialize(keyClass, persistentClass, properties);
          + try {
          + orientDbStoreParams = OrientDBStoreParameters.load(properties);
          + ROOT_URL = "remote:".concat(orientDbStoreParams.getServerHost()).concat(":")
          + .concat(orientDbStoreParams.getServerPort());
          + ROOT_DATABASE_URL = ROOT_URL.concat("/").concat(orientDbStoreParams.getDatabaseName());
          + remoteServerAdmin = new OServerAdmin(ROOT_URL).connect(orientDbStoreParams.getUserName(),
          + orientDbStoreParams.getUserPassword());
          + if (!remoteServerAdmin.existsDatabase(orientDbStoreParams.getDatabaseName(), "memory"))

          { + remoteServerAdmin.createDatabase(orientDbStoreParams.getDatabaseName(), "document", "memory"); + }

          +
          + if (orientDbStoreParams.getConnectionPoolSize() != null)

          { + int connPoolSize = Integer.valueOf(orientDbStoreParams.getConnectionPoolSize()); + connectionPool = new OPartitionedDatabasePoolFactory(connPoolSize) + .get(ROOT_DATABASE_URL, orientDbStoreParams.getUserName(), + orientDbStoreParams.getUserPassword()); + }

          else

          { + connectionPool = new OPartitionedDatabasePoolFactory().get(ROOT_DATABASE_URL, + orientDbStoreParams.getUserName(), orientDbStoreParams.getUserPassword()); + }

          +
          + OrientDBMappingBuilder<K, T> builder = new OrientDBMappingBuilder<>(this);
          + orientDBMapping = builder.fromFile(orientDbStoreParams.getMappingFile()).build();
          +
          + if (!schemaExists())

          { + createSchema(); + }

          + } catch (Exception e) {
          + LOG.error("Error while initializing OrientDB dataStore: {}",
          + new Object[]

          {e.getMessage()}

          );
          + throw new RuntimeException(e);
          + }
          + }
          +
          + @Override
          + public String getSchemaName(final String mappingSchemaName,
          + final Class<?> persistentClass)

          { + return super.getSchemaName(mappingSchemaName, persistentClass); + }

          +
          + @Override
          + public String getSchemaName()

          { + return orientDBMapping.getDocumentClass(); + }

          +
          + /**
          + * Create a new class of OrientDB documents if necessary. Enforce specified schema over the document class.
          + *
          + */
          + @Override
          + public void createSchema() {
          + if (schemaExists())

          { + return; + }

          +
          + ODatabaseDocumentTx schemaTx = connectionPool.acquire();
          + schemaTx.activateOnCurrentThread();
          + try {
          +
          + OClass documentClass = schemaTx.getMetadata().getSchema().createClass(orientDBMapping.getDocumentClass());
          + documentClass.createProperty("_id",
          + OType.getTypeByClass(super.getKeyClass())).createIndex(OClass.INDEX_TYPE.UNIQUE);
          + for (String docField : orientDBMapping.getDocumentFields())

          { + documentClass.createProperty(docField, + OType.valueOf(orientDBMapping.getDocumentFieldType(docField).name())); + }

          + schemaTx.getMetadata().getSchema().reload();
          + } finally

          { + schemaTx.close(); + }
          + }
          +
          + /**
          + * Deletes enforced schema over OrientDB Document class.
          + *
          + */
          + @Override
          + public void deleteSchema() {
          + ODatabaseDocumentTx schemaTx = connectionPool.acquire();
          + schemaTx.activateOnCurrentThread();
          + try { + schemaTx.getMetadata().getSchema().dropClass(orientDBMapping.getDocumentClass()); + } finally { + schemaTx.close(); + }

          + }
          +
          + /**
          + * Check whether there exist a schema enforced over OrientDB document class.
          + *
          + */
          + @Override
          + public boolean schemaExists() {
          + ODatabaseDocumentTx schemaTx = connectionPool.acquire();
          + schemaTx.activateOnCurrentThread();
          + try

          { + return schemaTx.getMetadata().getSchema() + .existsClass(orientDBMapping.getDocumentClass()); + }

          finally

          { + schemaTx.close(); + }

          + }
          +
          + @Override
          + public T get(K key, String[] fields) {
          + String[] dbFields = getFieldsToQuery(fields);
          + com.github.raymanrt.orientqb.query.Query selectQuery = new com.github.raymanrt.orientqb.query.Query();
          + for (String k : dbFields) {
          + String dbFieldName = orientDBMapping.getDocumentField(k);
          + if (dbFieldName != null && dbFieldName.length() > 0)

          { + selectQuery.select(dbFieldName); + }

          + }
          + selectQuery.from(orientDBMapping.getDocumentClass())
          + .where(projection("_id").eq(Parameter.parameter("key")));
          + Map<String, Object> params = new HashMap<String, Object>();
          + params.put("key", key);
          + OSQLSynchQuery<ODocument> query = new OSQLSynchQuery<ODocument>(selectQuery.toString());
          + ODatabaseDocumentTx selectTx = connectionPool.acquire();
          + selectTx.activateOnCurrentThread();
          + try {
          + List<ODocument> result = selectTx.command(query).execute(params);
          + if (result.size() == 1)

          { + return convertOrientDocToAvroBean(result.get(0), dbFields); + }

          else

          { + return null; + }

          + } finally

          { + selectTx.close(); + }

          + }
          +
          + @Override
          + public void put(K key, T val) {
          + if (val.isDirty()) {
          + OrientDBQuery<K, T> dataStoreQuery = new OrientDBQuery<>(this);
          + dataStoreQuery.setStartKey(key);
          + dataStoreQuery.setEndKey(key);
          + dataStoreQuery.populateOrientDBQuery(orientDBMapping, getFieldsToQuery(null), getFields());
          +
          + ODatabaseDocumentTx selectTx = connectionPool.acquire();
          + selectTx.activateOnCurrentThread();
          + try {
          + List<ODocument> result = selectTx.command(dataStoreQuery.getOrientDBQuery())
          + .execute(dataStoreQuery.getParams());
          + if (result.size() == 1)

          { + ODocument document = updateOrientDocFromAvroBean(key, val, result.get(0)); + docBatch.add(document); + }

          else

          { + ODocument document = convertAvroBeanToOrientDoc(key, val); + docBatch.add(document); + }

          + } finally

          { + selectTx.close(); + }

          + } else {
          + LOG.info("Ignored putting persistent bean {} in the store as it is neither "
          — End diff –

          Maybe this should a debug instead of info so we don't get overflowed with logging

          Show
          githubbot ASF GitHub Bot added a comment - Github user renato2099 commented on a diff in the pull request: https://github.com/apache/gora/pull/112#discussion_r127613861 — Diff: gora-orientdb/src/main/java/org/apache/gora/orientdb/store/OrientDBStore.java — @@ -0,0 +1,922 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.gora.orientdb.store; + +import java.io.IOException; +import java.nio.ByteBuffer; +import java.util.Map; +import java.util.Properties; +import java.util.List; +import java.util.HashMap; +import java.util.HashSet; +import java.util.ArrayList; +import java.util.Date; +import java.util.Calendar; +import java.util.Collection; +import java.util.TimeZone; +import java.util.Locale; + +import com.github.raymanrt.orientqb.query.Parameter; +import com.gitub.raymanrt.orientqb.delete.Delete; +import com.orientechnologies.orient.client.remote.OServerAdmin; +import com.orientechnologies.orient.core.db.OPartitionedDatabasePool; +import com.orientechnologies.orient.core.db.OPartitionedDatabasePoolFactory; +import com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx; +import com.orientechnologies.orient.core.db.record.OTrackedList; +import com.orientechnologies.orient.core.db.record.OTrackedMap; +import com.orientechnologies.orient.core.db.record.OTrackedSet; +import com.orientechnologies.orient.core.metadata.schema.OClass; +import com.orientechnologies.orient.core.metadata.schema.OType; +import com.orientechnologies.orient.core.record.impl.ODocument; +import com.orientechnologies.orient.core.sql.OCommandSQL; +import com.orientechnologies.orient.core.sql.query.OConcurrentResultSet; +import com.orientechnologies.orient.core.sql.query.OSQLSynchQuery; +import org.apache.avro.Schema; +import org.apache.avro.util.Utf8; +import org.apache.gora.orientdb.query.OrientDBQuery; +import org.apache.gora.orientdb.query.OrientDBResult; +import org.apache.gora.persistency.impl.BeanFactoryImpl; +import org.apache.gora.persistency.impl.DirtyListWrapper; +import org.apache.gora.persistency.impl.DirtyMapWrapper; +import org.apache.gora.persistency.impl.PersistentBase; +import org.apache.gora.query.PartitionQuery; +import org.apache.gora.query.Query; +import org.apache.gora.query.Result; +import org.apache.gora.query.impl.PartitionQueryImpl; +import org.apache.gora.store.impl.DataStoreBase; +import org.apache.gora.util.AvroUtils; +import org.apache.gora.util.ClassLoadingUtils; + +import javax.xml.bind.DatatypeConverter; + +import static com.github.raymanrt.orientqb.query.Projection.projection; + +/** + * {@link org.apache.gora.orientdb.store.OrientDBStore} is the primary class + * responsible for facilitating GORA CRUD operations on OrientDB documents. + */ +public class OrientDBStore<K, T extends PersistentBase> extends DataStoreBase<K, T> { + + public static final String DEFAULT_MAPPING_FILE = "/gora-orientdb-mapping.xml"; + private String ROOT_URL; + private String ROOT_DATABASE_URL; + private OrientDBStoreParameters orientDbStoreParams; + private OrientDBMapping orientDBMapping; + private OServerAdmin remoteServerAdmin; + private OPartitionedDatabasePool connectionPool; + private List<ODocument> docBatch = new ArrayList<>(); + + /** + * Initialize the OrientDB dataStore by {@link Properties} parameters. + * + * @param keyClass key class type for dataStore. + * @param persistentClass persistent class type for dataStore. + * @param properties OrientDB dataStore properties EG:- OrientDB client credentials. + */ + @Override + public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) { + super.initialize(keyClass, persistentClass, properties); + try { + orientDbStoreParams = OrientDBStoreParameters.load(properties); + ROOT_URL = "remote:".concat(orientDbStoreParams.getServerHost()).concat(":") + .concat(orientDbStoreParams.getServerPort()); + ROOT_DATABASE_URL = ROOT_URL.concat("/").concat(orientDbStoreParams.getDatabaseName()); + remoteServerAdmin = new OServerAdmin(ROOT_URL).connect(orientDbStoreParams.getUserName(), + orientDbStoreParams.getUserPassword()); + if (!remoteServerAdmin.existsDatabase(orientDbStoreParams.getDatabaseName(), "memory")) { + remoteServerAdmin.createDatabase(orientDbStoreParams.getDatabaseName(), "document", "memory"); + } + + if (orientDbStoreParams.getConnectionPoolSize() != null) { + int connPoolSize = Integer.valueOf(orientDbStoreParams.getConnectionPoolSize()); + connectionPool = new OPartitionedDatabasePoolFactory(connPoolSize) + .get(ROOT_DATABASE_URL, orientDbStoreParams.getUserName(), + orientDbStoreParams.getUserPassword()); + } else { + connectionPool = new OPartitionedDatabasePoolFactory().get(ROOT_DATABASE_URL, + orientDbStoreParams.getUserName(), orientDbStoreParams.getUserPassword()); + } + + OrientDBMappingBuilder<K, T> builder = new OrientDBMappingBuilder<>(this); + orientDBMapping = builder.fromFile(orientDbStoreParams.getMappingFile()).build(); + + if (!schemaExists()) { + createSchema(); + } + } catch (Exception e) { + LOG.error("Error while initializing OrientDB dataStore: {}", + new Object[] {e.getMessage()} ); + throw new RuntimeException(e); + } + } + + @Override + public String getSchemaName(final String mappingSchemaName, + final Class<?> persistentClass) { + return super.getSchemaName(mappingSchemaName, persistentClass); + } + + @Override + public String getSchemaName() { + return orientDBMapping.getDocumentClass(); + } + + /** + * Create a new class of OrientDB documents if necessary. Enforce specified schema over the document class. + * + */ + @Override + public void createSchema() { + if (schemaExists()) { + return; + } + + ODatabaseDocumentTx schemaTx = connectionPool.acquire(); + schemaTx.activateOnCurrentThread(); + try { + + OClass documentClass = schemaTx.getMetadata().getSchema().createClass(orientDBMapping.getDocumentClass()); + documentClass.createProperty("_id", + OType.getTypeByClass(super.getKeyClass())).createIndex(OClass.INDEX_TYPE.UNIQUE); + for (String docField : orientDBMapping.getDocumentFields()) { + documentClass.createProperty(docField, + OType.valueOf(orientDBMapping.getDocumentFieldType(docField).name())); + } + schemaTx.getMetadata().getSchema().reload(); + } finally { + schemaTx.close(); + } + } + + /** + * Deletes enforced schema over OrientDB Document class. + * + */ + @Override + public void deleteSchema() { + ODatabaseDocumentTx schemaTx = connectionPool.acquire(); + schemaTx.activateOnCurrentThread(); + try { + schemaTx.getMetadata().getSchema().dropClass(orientDBMapping.getDocumentClass()); + } finally { + schemaTx.close(); + } + } + + /** + * Check whether there exist a schema enforced over OrientDB document class. + * + */ + @Override + public boolean schemaExists() { + ODatabaseDocumentTx schemaTx = connectionPool.acquire(); + schemaTx.activateOnCurrentThread(); + try { + return schemaTx.getMetadata().getSchema() + .existsClass(orientDBMapping.getDocumentClass()); + } finally { + schemaTx.close(); + } + } + + @Override + public T get(K key, String[] fields) { + String[] dbFields = getFieldsToQuery(fields); + com.github.raymanrt.orientqb.query.Query selectQuery = new com.github.raymanrt.orientqb.query.Query(); + for (String k : dbFields) { + String dbFieldName = orientDBMapping.getDocumentField(k); + if (dbFieldName != null && dbFieldName.length() > 0) { + selectQuery.select(dbFieldName); + } + } + selectQuery.from(orientDBMapping.getDocumentClass()) + .where(projection("_id").eq(Parameter.parameter("key"))); + Map<String, Object> params = new HashMap<String, Object>(); + params.put("key", key); + OSQLSynchQuery<ODocument> query = new OSQLSynchQuery<ODocument>(selectQuery.toString()); + ODatabaseDocumentTx selectTx = connectionPool.acquire(); + selectTx.activateOnCurrentThread(); + try { + List<ODocument> result = selectTx.command(query).execute(params); + if (result.size() == 1) { + return convertOrientDocToAvroBean(result.get(0), dbFields); + } else { + return null; + } + } finally { + selectTx.close(); + } + } + + @Override + public void put(K key, T val) { + if (val.isDirty()) { + OrientDBQuery<K, T> dataStoreQuery = new OrientDBQuery<>(this); + dataStoreQuery.setStartKey(key); + dataStoreQuery.setEndKey(key); + dataStoreQuery.populateOrientDBQuery(orientDBMapping, getFieldsToQuery(null), getFields()); + + ODatabaseDocumentTx selectTx = connectionPool.acquire(); + selectTx.activateOnCurrentThread(); + try { + List<ODocument> result = selectTx.command(dataStoreQuery.getOrientDBQuery()) + .execute(dataStoreQuery.getParams()); + if (result.size() == 1) { + ODocument document = updateOrientDocFromAvroBean(key, val, result.get(0)); + docBatch.add(document); + } else { + ODocument document = convertAvroBeanToOrientDoc(key, val); + docBatch.add(document); + } + } finally { + selectTx.close(); + } + } else { + LOG.info("Ignored putting persistent bean {} in the store as it is neither " — End diff – Maybe this should a debug instead of info so we don't get overflowed with logging
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user djkevincr commented on a diff in the pull request:

          https://github.com/apache/gora/pull/112#discussion_r127641686

          — Diff: gora-orientdb/src/main/java/org/apache/gora/orientdb/store/OrientDBStore.java —
          @@ -0,0 +1,922 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.gora.orientdb.store;
          +
          +import java.io.IOException;
          +import java.nio.ByteBuffer;
          +import java.util.Map;
          +import java.util.Properties;
          +import java.util.List;
          +import java.util.HashMap;
          +import java.util.HashSet;
          +import java.util.ArrayList;
          +import java.util.Date;
          +import java.util.Calendar;
          +import java.util.Collection;
          +import java.util.TimeZone;
          +import java.util.Locale;
          +
          +import com.github.raymanrt.orientqb.query.Parameter;
          +import com.gitub.raymanrt.orientqb.delete.Delete;
          +import com.orientechnologies.orient.client.remote.OServerAdmin;
          +import com.orientechnologies.orient.core.db.OPartitionedDatabasePool;
          +import com.orientechnologies.orient.core.db.OPartitionedDatabasePoolFactory;
          +import com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx;
          +import com.orientechnologies.orient.core.db.record.OTrackedList;
          +import com.orientechnologies.orient.core.db.record.OTrackedMap;
          +import com.orientechnologies.orient.core.db.record.OTrackedSet;
          +import com.orientechnologies.orient.core.metadata.schema.OClass;
          +import com.orientechnologies.orient.core.metadata.schema.OType;
          +import com.orientechnologies.orient.core.record.impl.ODocument;
          +import com.orientechnologies.orient.core.sql.OCommandSQL;
          +import com.orientechnologies.orient.core.sql.query.OConcurrentResultSet;
          +import com.orientechnologies.orient.core.sql.query.OSQLSynchQuery;
          +import org.apache.avro.Schema;
          +import org.apache.avro.util.Utf8;
          +import org.apache.gora.orientdb.query.OrientDBQuery;
          +import org.apache.gora.orientdb.query.OrientDBResult;
          +import org.apache.gora.persistency.impl.BeanFactoryImpl;
          +import org.apache.gora.persistency.impl.DirtyListWrapper;
          +import org.apache.gora.persistency.impl.DirtyMapWrapper;
          +import org.apache.gora.persistency.impl.PersistentBase;
          +import org.apache.gora.query.PartitionQuery;
          +import org.apache.gora.query.Query;
          +import org.apache.gora.query.Result;
          +import org.apache.gora.query.impl.PartitionQueryImpl;
          +import org.apache.gora.store.impl.DataStoreBase;
          +import org.apache.gora.util.AvroUtils;
          +import org.apache.gora.util.ClassLoadingUtils;
          +
          +import javax.xml.bind.DatatypeConverter;
          +
          +import static com.github.raymanrt.orientqb.query.Projection.projection;
          +
          +/**
          + *

          {@link org.apache.gora.orientdb.store.OrientDBStore}

          is the primary class
          + * responsible for facilitating GORA CRUD operations on OrientDB documents.
          + */
          +public class OrientDBStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
          +
          + public static final String DEFAULT_MAPPING_FILE = "/gora-orientdb-mapping.xml";
          + private String ROOT_URL;
          + private String ROOT_DATABASE_URL;
          + private OrientDBStoreParameters orientDbStoreParams;
          + private OrientDBMapping orientDBMapping;
          + private OServerAdmin remoteServerAdmin;
          + private OPartitionedDatabasePool connectionPool;
          + private List<ODocument> docBatch = new ArrayList<>();
          — End diff –

          @renato2099 Yes, you are right, this is something I have missed out. Will correct it.

          Show
          githubbot ASF GitHub Bot added a comment - Github user djkevincr commented on a diff in the pull request: https://github.com/apache/gora/pull/112#discussion_r127641686 — Diff: gora-orientdb/src/main/java/org/apache/gora/orientdb/store/OrientDBStore.java — @@ -0,0 +1,922 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.gora.orientdb.store; + +import java.io.IOException; +import java.nio.ByteBuffer; +import java.util.Map; +import java.util.Properties; +import java.util.List; +import java.util.HashMap; +import java.util.HashSet; +import java.util.ArrayList; +import java.util.Date; +import java.util.Calendar; +import java.util.Collection; +import java.util.TimeZone; +import java.util.Locale; + +import com.github.raymanrt.orientqb.query.Parameter; +import com.gitub.raymanrt.orientqb.delete.Delete; +import com.orientechnologies.orient.client.remote.OServerAdmin; +import com.orientechnologies.orient.core.db.OPartitionedDatabasePool; +import com.orientechnologies.orient.core.db.OPartitionedDatabasePoolFactory; +import com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx; +import com.orientechnologies.orient.core.db.record.OTrackedList; +import com.orientechnologies.orient.core.db.record.OTrackedMap; +import com.orientechnologies.orient.core.db.record.OTrackedSet; +import com.orientechnologies.orient.core.metadata.schema.OClass; +import com.orientechnologies.orient.core.metadata.schema.OType; +import com.orientechnologies.orient.core.record.impl.ODocument; +import com.orientechnologies.orient.core.sql.OCommandSQL; +import com.orientechnologies.orient.core.sql.query.OConcurrentResultSet; +import com.orientechnologies.orient.core.sql.query.OSQLSynchQuery; +import org.apache.avro.Schema; +import org.apache.avro.util.Utf8; +import org.apache.gora.orientdb.query.OrientDBQuery; +import org.apache.gora.orientdb.query.OrientDBResult; +import org.apache.gora.persistency.impl.BeanFactoryImpl; +import org.apache.gora.persistency.impl.DirtyListWrapper; +import org.apache.gora.persistency.impl.DirtyMapWrapper; +import org.apache.gora.persistency.impl.PersistentBase; +import org.apache.gora.query.PartitionQuery; +import org.apache.gora.query.Query; +import org.apache.gora.query.Result; +import org.apache.gora.query.impl.PartitionQueryImpl; +import org.apache.gora.store.impl.DataStoreBase; +import org.apache.gora.util.AvroUtils; +import org.apache.gora.util.ClassLoadingUtils; + +import javax.xml.bind.DatatypeConverter; + +import static com.github.raymanrt.orientqb.query.Projection.projection; + +/** + * {@link org.apache.gora.orientdb.store.OrientDBStore} is the primary class + * responsible for facilitating GORA CRUD operations on OrientDB documents. + */ +public class OrientDBStore<K, T extends PersistentBase> extends DataStoreBase<K, T> { + + public static final String DEFAULT_MAPPING_FILE = "/gora-orientdb-mapping.xml"; + private String ROOT_URL; + private String ROOT_DATABASE_URL; + private OrientDBStoreParameters orientDbStoreParams; + private OrientDBMapping orientDBMapping; + private OServerAdmin remoteServerAdmin; + private OPartitionedDatabasePool connectionPool; + private List<ODocument> docBatch = new ArrayList<>(); — End diff – @renato2099 Yes, you are right, this is something I have missed out. Will correct it.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user djkevincr commented on a diff in the pull request:

          https://github.com/apache/gora/pull/112#discussion_r127641711

          — Diff: gora-orientdb/src/main/java/org/apache/gora/orientdb/store/OrientDBStore.java —
          @@ -0,0 +1,922 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.gora.orientdb.store;
          +
          +import java.io.IOException;
          +import java.nio.ByteBuffer;
          +import java.util.Map;
          +import java.util.Properties;
          +import java.util.List;
          +import java.util.HashMap;
          +import java.util.HashSet;
          +import java.util.ArrayList;
          +import java.util.Date;
          +import java.util.Calendar;
          +import java.util.Collection;
          +import java.util.TimeZone;
          +import java.util.Locale;
          +
          +import com.github.raymanrt.orientqb.query.Parameter;
          +import com.gitub.raymanrt.orientqb.delete.Delete;
          +import com.orientechnologies.orient.client.remote.OServerAdmin;
          +import com.orientechnologies.orient.core.db.OPartitionedDatabasePool;
          +import com.orientechnologies.orient.core.db.OPartitionedDatabasePoolFactory;
          +import com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx;
          +import com.orientechnologies.orient.core.db.record.OTrackedList;
          +import com.orientechnologies.orient.core.db.record.OTrackedMap;
          +import com.orientechnologies.orient.core.db.record.OTrackedSet;
          +import com.orientechnologies.orient.core.metadata.schema.OClass;
          +import com.orientechnologies.orient.core.metadata.schema.OType;
          +import com.orientechnologies.orient.core.record.impl.ODocument;
          +import com.orientechnologies.orient.core.sql.OCommandSQL;
          +import com.orientechnologies.orient.core.sql.query.OConcurrentResultSet;
          +import com.orientechnologies.orient.core.sql.query.OSQLSynchQuery;
          +import org.apache.avro.Schema;
          +import org.apache.avro.util.Utf8;
          +import org.apache.gora.orientdb.query.OrientDBQuery;
          +import org.apache.gora.orientdb.query.OrientDBResult;
          +import org.apache.gora.persistency.impl.BeanFactoryImpl;
          +import org.apache.gora.persistency.impl.DirtyListWrapper;
          +import org.apache.gora.persistency.impl.DirtyMapWrapper;
          +import org.apache.gora.persistency.impl.PersistentBase;
          +import org.apache.gora.query.PartitionQuery;
          +import org.apache.gora.query.Query;
          +import org.apache.gora.query.Result;
          +import org.apache.gora.query.impl.PartitionQueryImpl;
          +import org.apache.gora.store.impl.DataStoreBase;
          +import org.apache.gora.util.AvroUtils;
          +import org.apache.gora.util.ClassLoadingUtils;
          +
          +import javax.xml.bind.DatatypeConverter;
          +
          +import static com.github.raymanrt.orientqb.query.Projection.projection;
          +
          +/**
          + *

          {@link org.apache.gora.orientdb.store.OrientDBStore}

          is the primary class
          + * responsible for facilitating GORA CRUD operations on OrientDB documents.
          + */
          +public class OrientDBStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
          +
          + public static final String DEFAULT_MAPPING_FILE = "/gora-orientdb-mapping.xml";
          + private String ROOT_URL;
          + private String ROOT_DATABASE_URL;
          + private OrientDBStoreParameters orientDbStoreParams;
          + private OrientDBMapping orientDBMapping;
          + private OServerAdmin remoteServerAdmin;
          + private OPartitionedDatabasePool connectionPool;
          + private List<ODocument> docBatch = new ArrayList<>();
          +
          + /**
          + * Initialize the OrientDB dataStore by

          {@link Properties}

          parameters.
          + *
          + * @param keyClass key class type for dataStore.
          + * @param persistentClass persistent class type for dataStore.
          + * @param properties OrientDB dataStore properties EG:- OrientDB client credentials.
          + */
          + @Override
          + public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) {
          + super.initialize(keyClass, persistentClass, properties);
          + try {
          + orientDbStoreParams = OrientDBStoreParameters.load(properties);
          + ROOT_URL = "remote:".concat(orientDbStoreParams.getServerHost()).concat(":")
          + .concat(orientDbStoreParams.getServerPort());
          + ROOT_DATABASE_URL = ROOT_URL.concat("/").concat(orientDbStoreParams.getDatabaseName());
          + remoteServerAdmin = new OServerAdmin(ROOT_URL).connect(orientDbStoreParams.getUserName(),
          + orientDbStoreParams.getUserPassword());
          + if (!remoteServerAdmin.existsDatabase(orientDbStoreParams.getDatabaseName(), "memory"))

          { + remoteServerAdmin.createDatabase(orientDbStoreParams.getDatabaseName(), "document", "memory"); + }

          +
          + if (orientDbStoreParams.getConnectionPoolSize() != null)

          { + int connPoolSize = Integer.valueOf(orientDbStoreParams.getConnectionPoolSize()); + connectionPool = new OPartitionedDatabasePoolFactory(connPoolSize) + .get(ROOT_DATABASE_URL, orientDbStoreParams.getUserName(), + orientDbStoreParams.getUserPassword()); + }

          else

          { + connectionPool = new OPartitionedDatabasePoolFactory().get(ROOT_DATABASE_URL, + orientDbStoreParams.getUserName(), orientDbStoreParams.getUserPassword()); + }

          +
          + OrientDBMappingBuilder<K, T> builder = new OrientDBMappingBuilder<>(this);
          + orientDBMapping = builder.fromFile(orientDbStoreParams.getMappingFile()).build();
          +
          + if (!schemaExists())

          { + createSchema(); + }

          + } catch (Exception e) {
          + LOG.error("Error while initializing OrientDB dataStore: {}",
          + new Object[]

          {e.getMessage()}

          );
          + throw new RuntimeException(e);
          + }
          + }
          +
          + @Override
          + public String getSchemaName(final String mappingSchemaName,
          + final Class<?> persistentClass)

          { + return super.getSchemaName(mappingSchemaName, persistentClass); + }

          +
          + @Override
          + public String getSchemaName()

          { + return orientDBMapping.getDocumentClass(); + }

          +
          + /**
          + * Create a new class of OrientDB documents if necessary. Enforce specified schema over the document class.
          + *
          + */
          + @Override
          + public void createSchema() {
          + if (schemaExists())

          { + return; + }

          +
          + ODatabaseDocumentTx schemaTx = connectionPool.acquire();
          + schemaTx.activateOnCurrentThread();
          + try {
          +
          + OClass documentClass = schemaTx.getMetadata().getSchema().createClass(orientDBMapping.getDocumentClass());
          + documentClass.createProperty("_id",
          + OType.getTypeByClass(super.getKeyClass())).createIndex(OClass.INDEX_TYPE.UNIQUE);
          + for (String docField : orientDBMapping.getDocumentFields())

          { + documentClass.createProperty(docField, + OType.valueOf(orientDBMapping.getDocumentFieldType(docField).name())); + }

          + schemaTx.getMetadata().getSchema().reload();
          + } finally

          { + schemaTx.close(); + }
          + }
          +
          + /**
          + * Deletes enforced schema over OrientDB Document class.
          + *
          + */
          + @Override
          + public void deleteSchema() {
          + ODatabaseDocumentTx schemaTx = connectionPool.acquire();
          + schemaTx.activateOnCurrentThread();
          + try { + schemaTx.getMetadata().getSchema().dropClass(orientDBMapping.getDocumentClass()); + } finally { + schemaTx.close(); + }

          + }
          +
          + /**
          + * Check whether there exist a schema enforced over OrientDB document class.
          + *
          + */
          + @Override
          + public boolean schemaExists() {
          + ODatabaseDocumentTx schemaTx = connectionPool.acquire();
          + schemaTx.activateOnCurrentThread();
          + try

          { + return schemaTx.getMetadata().getSchema() + .existsClass(orientDBMapping.getDocumentClass()); + }

          finally

          { + schemaTx.close(); + }

          + }
          +
          + @Override
          + public T get(K key, String[] fields) {
          + String[] dbFields = getFieldsToQuery(fields);
          + com.github.raymanrt.orientqb.query.Query selectQuery = new com.github.raymanrt.orientqb.query.Query();
          + for (String k : dbFields) {
          + String dbFieldName = orientDBMapping.getDocumentField(k);
          + if (dbFieldName != null && dbFieldName.length() > 0)

          { + selectQuery.select(dbFieldName); + }

          + }
          + selectQuery.from(orientDBMapping.getDocumentClass())
          + .where(projection("_id").eq(Parameter.parameter("key")));
          + Map<String, Object> params = new HashMap<String, Object>();
          + params.put("key", key);
          + OSQLSynchQuery<ODocument> query = new OSQLSynchQuery<ODocument>(selectQuery.toString());
          + ODatabaseDocumentTx selectTx = connectionPool.acquire();
          + selectTx.activateOnCurrentThread();
          + try {
          + List<ODocument> result = selectTx.command(query).execute(params);
          + if (result.size() == 1)

          { + return convertOrientDocToAvroBean(result.get(0), dbFields); + }

          else

          { + return null; + }

          + } finally

          { + selectTx.close(); + }

          + }
          +
          + @Override
          + public void put(K key, T val) {
          + if (val.isDirty()) {
          + OrientDBQuery<K, T> dataStoreQuery = new OrientDBQuery<>(this);
          + dataStoreQuery.setStartKey(key);
          + dataStoreQuery.setEndKey(key);
          + dataStoreQuery.populateOrientDBQuery(orientDBMapping, getFieldsToQuery(null), getFields());
          +
          + ODatabaseDocumentTx selectTx = connectionPool.acquire();
          + selectTx.activateOnCurrentThread();
          + try {
          + List<ODocument> result = selectTx.command(dataStoreQuery.getOrientDBQuery())
          + .execute(dataStoreQuery.getParams());
          + if (result.size() == 1)

          { + ODocument document = updateOrientDocFromAvroBean(key, val, result.get(0)); + docBatch.add(document); + }

          else

          { + ODocument document = convertAvroBeanToOrientDoc(key, val); + docBatch.add(document); + }

          + } finally

          { + selectTx.close(); + }

          + } else {
          + LOG.info("Ignored putting persistent bean {} in the store as it is neither "
          — End diff –

          +1, will do.

          Show
          githubbot ASF GitHub Bot added a comment - Github user djkevincr commented on a diff in the pull request: https://github.com/apache/gora/pull/112#discussion_r127641711 — Diff: gora-orientdb/src/main/java/org/apache/gora/orientdb/store/OrientDBStore.java — @@ -0,0 +1,922 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.gora.orientdb.store; + +import java.io.IOException; +import java.nio.ByteBuffer; +import java.util.Map; +import java.util.Properties; +import java.util.List; +import java.util.HashMap; +import java.util.HashSet; +import java.util.ArrayList; +import java.util.Date; +import java.util.Calendar; +import java.util.Collection; +import java.util.TimeZone; +import java.util.Locale; + +import com.github.raymanrt.orientqb.query.Parameter; +import com.gitub.raymanrt.orientqb.delete.Delete; +import com.orientechnologies.orient.client.remote.OServerAdmin; +import com.orientechnologies.orient.core.db.OPartitionedDatabasePool; +import com.orientechnologies.orient.core.db.OPartitionedDatabasePoolFactory; +import com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx; +import com.orientechnologies.orient.core.db.record.OTrackedList; +import com.orientechnologies.orient.core.db.record.OTrackedMap; +import com.orientechnologies.orient.core.db.record.OTrackedSet; +import com.orientechnologies.orient.core.metadata.schema.OClass; +import com.orientechnologies.orient.core.metadata.schema.OType; +import com.orientechnologies.orient.core.record.impl.ODocument; +import com.orientechnologies.orient.core.sql.OCommandSQL; +import com.orientechnologies.orient.core.sql.query.OConcurrentResultSet; +import com.orientechnologies.orient.core.sql.query.OSQLSynchQuery; +import org.apache.avro.Schema; +import org.apache.avro.util.Utf8; +import org.apache.gora.orientdb.query.OrientDBQuery; +import org.apache.gora.orientdb.query.OrientDBResult; +import org.apache.gora.persistency.impl.BeanFactoryImpl; +import org.apache.gora.persistency.impl.DirtyListWrapper; +import org.apache.gora.persistency.impl.DirtyMapWrapper; +import org.apache.gora.persistency.impl.PersistentBase; +import org.apache.gora.query.PartitionQuery; +import org.apache.gora.query.Query; +import org.apache.gora.query.Result; +import org.apache.gora.query.impl.PartitionQueryImpl; +import org.apache.gora.store.impl.DataStoreBase; +import org.apache.gora.util.AvroUtils; +import org.apache.gora.util.ClassLoadingUtils; + +import javax.xml.bind.DatatypeConverter; + +import static com.github.raymanrt.orientqb.query.Projection.projection; + +/** + * {@link org.apache.gora.orientdb.store.OrientDBStore} is the primary class + * responsible for facilitating GORA CRUD operations on OrientDB documents. + */ +public class OrientDBStore<K, T extends PersistentBase> extends DataStoreBase<K, T> { + + public static final String DEFAULT_MAPPING_FILE = "/gora-orientdb-mapping.xml"; + private String ROOT_URL; + private String ROOT_DATABASE_URL; + private OrientDBStoreParameters orientDbStoreParams; + private OrientDBMapping orientDBMapping; + private OServerAdmin remoteServerAdmin; + private OPartitionedDatabasePool connectionPool; + private List<ODocument> docBatch = new ArrayList<>(); + + /** + * Initialize the OrientDB dataStore by {@link Properties} parameters. + * + * @param keyClass key class type for dataStore. + * @param persistentClass persistent class type for dataStore. + * @param properties OrientDB dataStore properties EG:- OrientDB client credentials. + */ + @Override + public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) { + super.initialize(keyClass, persistentClass, properties); + try { + orientDbStoreParams = OrientDBStoreParameters.load(properties); + ROOT_URL = "remote:".concat(orientDbStoreParams.getServerHost()).concat(":") + .concat(orientDbStoreParams.getServerPort()); + ROOT_DATABASE_URL = ROOT_URL.concat("/").concat(orientDbStoreParams.getDatabaseName()); + remoteServerAdmin = new OServerAdmin(ROOT_URL).connect(orientDbStoreParams.getUserName(), + orientDbStoreParams.getUserPassword()); + if (!remoteServerAdmin.existsDatabase(orientDbStoreParams.getDatabaseName(), "memory")) { + remoteServerAdmin.createDatabase(orientDbStoreParams.getDatabaseName(), "document", "memory"); + } + + if (orientDbStoreParams.getConnectionPoolSize() != null) { + int connPoolSize = Integer.valueOf(orientDbStoreParams.getConnectionPoolSize()); + connectionPool = new OPartitionedDatabasePoolFactory(connPoolSize) + .get(ROOT_DATABASE_URL, orientDbStoreParams.getUserName(), + orientDbStoreParams.getUserPassword()); + } else { + connectionPool = new OPartitionedDatabasePoolFactory().get(ROOT_DATABASE_URL, + orientDbStoreParams.getUserName(), orientDbStoreParams.getUserPassword()); + } + + OrientDBMappingBuilder<K, T> builder = new OrientDBMappingBuilder<>(this); + orientDBMapping = builder.fromFile(orientDbStoreParams.getMappingFile()).build(); + + if (!schemaExists()) { + createSchema(); + } + } catch (Exception e) { + LOG.error("Error while initializing OrientDB dataStore: {}", + new Object[] {e.getMessage()} ); + throw new RuntimeException(e); + } + } + + @Override + public String getSchemaName(final String mappingSchemaName, + final Class<?> persistentClass) { + return super.getSchemaName(mappingSchemaName, persistentClass); + } + + @Override + public String getSchemaName() { + return orientDBMapping.getDocumentClass(); + } + + /** + * Create a new class of OrientDB documents if necessary. Enforce specified schema over the document class. + * + */ + @Override + public void createSchema() { + if (schemaExists()) { + return; + } + + ODatabaseDocumentTx schemaTx = connectionPool.acquire(); + schemaTx.activateOnCurrentThread(); + try { + + OClass documentClass = schemaTx.getMetadata().getSchema().createClass(orientDBMapping.getDocumentClass()); + documentClass.createProperty("_id", + OType.getTypeByClass(super.getKeyClass())).createIndex(OClass.INDEX_TYPE.UNIQUE); + for (String docField : orientDBMapping.getDocumentFields()) { + documentClass.createProperty(docField, + OType.valueOf(orientDBMapping.getDocumentFieldType(docField).name())); + } + schemaTx.getMetadata().getSchema().reload(); + } finally { + schemaTx.close(); + } + } + + /** + * Deletes enforced schema over OrientDB Document class. + * + */ + @Override + public void deleteSchema() { + ODatabaseDocumentTx schemaTx = connectionPool.acquire(); + schemaTx.activateOnCurrentThread(); + try { + schemaTx.getMetadata().getSchema().dropClass(orientDBMapping.getDocumentClass()); + } finally { + schemaTx.close(); + } + } + + /** + * Check whether there exist a schema enforced over OrientDB document class. + * + */ + @Override + public boolean schemaExists() { + ODatabaseDocumentTx schemaTx = connectionPool.acquire(); + schemaTx.activateOnCurrentThread(); + try { + return schemaTx.getMetadata().getSchema() + .existsClass(orientDBMapping.getDocumentClass()); + } finally { + schemaTx.close(); + } + } + + @Override + public T get(K key, String[] fields) { + String[] dbFields = getFieldsToQuery(fields); + com.github.raymanrt.orientqb.query.Query selectQuery = new com.github.raymanrt.orientqb.query.Query(); + for (String k : dbFields) { + String dbFieldName = orientDBMapping.getDocumentField(k); + if (dbFieldName != null && dbFieldName.length() > 0) { + selectQuery.select(dbFieldName); + } + } + selectQuery.from(orientDBMapping.getDocumentClass()) + .where(projection("_id").eq(Parameter.parameter("key"))); + Map<String, Object> params = new HashMap<String, Object>(); + params.put("key", key); + OSQLSynchQuery<ODocument> query = new OSQLSynchQuery<ODocument>(selectQuery.toString()); + ODatabaseDocumentTx selectTx = connectionPool.acquire(); + selectTx.activateOnCurrentThread(); + try { + List<ODocument> result = selectTx.command(query).execute(params); + if (result.size() == 1) { + return convertOrientDocToAvroBean(result.get(0), dbFields); + } else { + return null; + } + } finally { + selectTx.close(); + } + } + + @Override + public void put(K key, T val) { + if (val.isDirty()) { + OrientDBQuery<K, T> dataStoreQuery = new OrientDBQuery<>(this); + dataStoreQuery.setStartKey(key); + dataStoreQuery.setEndKey(key); + dataStoreQuery.populateOrientDBQuery(orientDBMapping, getFieldsToQuery(null), getFields()); + + ODatabaseDocumentTx selectTx = connectionPool.acquire(); + selectTx.activateOnCurrentThread(); + try { + List<ODocument> result = selectTx.command(dataStoreQuery.getOrientDBQuery()) + .execute(dataStoreQuery.getParams()); + if (result.size() == 1) { + ODocument document = updateOrientDocFromAvroBean(key, val, result.get(0)); + docBatch.add(document); + } else { + ODocument document = convertAvroBeanToOrientDoc(key, val); + docBatch.add(document); + } + } finally { + selectTx.close(); + } + } else { + LOG.info("Ignored putting persistent bean {} in the store as it is neither " — End diff – +1, will do.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user lewismc commented on the issue:

          https://github.com/apache/gora/pull/112

          Would be nice to get this into Gora @djkevincr
          We could potentially make a release in the near future.

          Show
          githubbot ASF GitHub Bot added a comment - Github user lewismc commented on the issue: https://github.com/apache/gora/pull/112 Would be nice to get this into Gora @djkevincr We could potentially make a release in the near future.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user djkevincr commented on the issue:

          https://github.com/apache/gora/pull/112

          @lewismc Yes Lewis. I am working on this, will finish it off very soon.

          Show
          githubbot ASF GitHub Bot added a comment - Github user djkevincr commented on the issue: https://github.com/apache/gora/pull/112 @lewismc Yes Lewis. I am working on this, will finish it off very soon.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user lewismc commented on the issue:

          https://github.com/apache/gora/pull/112

          Hi @djkevincr I would like to move towards pushing a release candidate for Gora 0.8 soon. Having this module included in the release would be excellent.

          Show
          githubbot ASF GitHub Bot added a comment - Github user lewismc commented on the issue: https://github.com/apache/gora/pull/112 Hi @djkevincr I would like to move towards pushing a release candidate for Gora 0.8 soon. Having this module included in the release would be excellent.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user djkevincr commented on the issue:

          https://github.com/apache/gora/pull/112

          @lewismc I will merge this changes within today best case or it will be end of tomorrow. Sorry about the delays. Hope that's fine with you. We need to get this thing to the release and that would be great.

          Show
          githubbot ASF GitHub Bot added a comment - Github user djkevincr commented on the issue: https://github.com/apache/gora/pull/112 @lewismc I will merge this changes within today best case or it will be end of tomorrow. Sorry about the delays. Hope that's fine with you. We need to get this thing to the release and that would be great.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user lewismc commented on the issue:

          https://github.com/apache/gora/pull/112

          ack

          > Sorry about the delays.

          absolutely no problems at all... the patch looks very good.

          Show
          githubbot ASF GitHub Bot added a comment - Github user lewismc commented on the issue: https://github.com/apache/gora/pull/112 ack > Sorry about the delays. absolutely no problems at all... the patch looks very good.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user djkevincr commented on the issue:

          https://github.com/apache/gora/pull/112

          @lewismc please give me bit of time working on this.

          Show
          githubbot ASF GitHub Bot added a comment - Github user djkevincr commented on the issue: https://github.com/apache/gora/pull/112 @lewismc please give me bit of time working on this.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/gora/pull/112

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/gora/pull/112
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user djkevincr commented on the issue:

          https://github.com/apache/gora/pull/112

          @lewismc this is merged now. You may proceed pushing a release candidate for Gora 0.8.

          Show
          githubbot ASF GitHub Bot added a comment - Github user djkevincr commented on the issue: https://github.com/apache/gora/pull/112 @lewismc this is merged now. You may proceed pushing a release candidate for Gora 0.8.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build gora-trunk #1732 (See https://builds.apache.org/job/gora-trunk/1732/)
          GORA-513 Add initial OrientDB datastore impl (djkevincr: rev 71a95b98f37d268eeafd2fb653ee1b2158416eaf)

          • (add) gora-orientdb/src/test/java/org/apache/gora/orientdb/mapreduce/OrientDBStoreMapReduceTest.java
          • (add) gora-orientdb/src/main/java/org/apache/gora/orientdb/store/OrientDBMapping.java
          • (add) gora-orientdb/src/main/java/org/apache/gora/orientdb/store/OrientDBMappingBuilder.java
          • (add) gora-orientdb/src/main/java/org/apache/gora/orientdb/query/package-info.java
          • (add) gora-orientdb/src/main/java/org/apache/gora/orientdb/store/OrientDBStoreParameters.java
          • (add) gora-orientdb/src/test/resources/orientdb-server-config.xml
          • (add) gora-orientdb/src/main/java/org/apache/gora/orientdb/store/package-info.java
          • (add) gora-orientdb/pom.xml
          • (add) gora-orientdb/src/test/java/org/apache/gora/orientdb/GoraOrientDBTestDriver.java
          • (add) gora-orientdb/src/main/java/org/apache/gora/orientdb/query/OrientDBResult.java
          • (add) gora-orientdb/src/test/java/org/apache/gora/orientdb/package-info.java
          • (add) gora-orientdb/src/test/java/org/apache/gora/orientdb/store/OrientDBGoraDataStoreTest.java
          • (add) gora-orientdb/src/main/java/org/apache/gora/orientdb/package-info.java
          • (add) gora-orientdb/src/test/resources/gora-orientdb-mapping.xml
          • (edit) pom.xml
          • (add) gora-orientdb/src/test/java/org/apache/gora/orientdb/store/package-info.java
          • (add) gora-orientdb/src/test/java/org/apache/gora/orientdb/mapreduce/package-info.java
          • (add) gora-orientdb/src/test/resources/gora.properties
          • (add) gora-orientdb/src/main/java/org/apache/gora/orientdb/query/OrientDBQuery.java
          • (add) gora-orientdb/src/main/java/org/apache/gora/orientdb/store/OrientDBStore.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build gora-trunk #1732 (See https://builds.apache.org/job/gora-trunk/1732/ ) GORA-513 Add initial OrientDB datastore impl (djkevincr: rev 71a95b98f37d268eeafd2fb653ee1b2158416eaf) (add) gora-orientdb/src/test/java/org/apache/gora/orientdb/mapreduce/OrientDBStoreMapReduceTest.java (add) gora-orientdb/src/main/java/org/apache/gora/orientdb/store/OrientDBMapping.java (add) gora-orientdb/src/main/java/org/apache/gora/orientdb/store/OrientDBMappingBuilder.java (add) gora-orientdb/src/main/java/org/apache/gora/orientdb/query/package-info.java (add) gora-orientdb/src/main/java/org/apache/gora/orientdb/store/OrientDBStoreParameters.java (add) gora-orientdb/src/test/resources/orientdb-server-config.xml (add) gora-orientdb/src/main/java/org/apache/gora/orientdb/store/package-info.java (add) gora-orientdb/pom.xml (add) gora-orientdb/src/test/java/org/apache/gora/orientdb/GoraOrientDBTestDriver.java (add) gora-orientdb/src/main/java/org/apache/gora/orientdb/query/OrientDBResult.java (add) gora-orientdb/src/test/java/org/apache/gora/orientdb/package-info.java (add) gora-orientdb/src/test/java/org/apache/gora/orientdb/store/OrientDBGoraDataStoreTest.java (add) gora-orientdb/src/main/java/org/apache/gora/orientdb/package-info.java (add) gora-orientdb/src/test/resources/gora-orientdb-mapping.xml (edit) pom.xml (add) gora-orientdb/src/test/java/org/apache/gora/orientdb/store/package-info.java (add) gora-orientdb/src/test/java/org/apache/gora/orientdb/mapreduce/package-info.java (add) gora-orientdb/src/test/resources/gora.properties (add) gora-orientdb/src/main/java/org/apache/gora/orientdb/query/OrientDBQuery.java (add) gora-orientdb/src/main/java/org/apache/gora/orientdb/store/OrientDBStore.java

            People

            • Assignee:
              djkevincr Kevin Ratnasekera
              Reporter:
              djkevincr Kevin Ratnasekera
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development