IGNITE-1192 Implementation (Integration with Spring Data)

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

IGNITE-1192 Implementation (Integration with Spring Data)

Eduard Shangareev
Guys, greetings!

I am working on Spring Data integration. There are two ways how we can use
Spring Data:
1. generating layer for Ignite usage;
2. using existing Spring Data configuration for persistence (Spring Data
JPA, for example, or Mongo).

I am sure that we should concentrate on  first part because Ignite already
has no problem with persistence.

Main feature Spring Data provides - Repository concept, a user just defines
some interface for entity access and spring data framework automatically
generates CRUD/query methods.
And I am going to use Ignite via SQL-layer.

Documentation of spring-data-commons (which I will use to refer):
http://docs.spring.io/spring-data/data-commons/docs/current/reference/html/


The list of features and problems:
1. Support Java8 (Streams, CompletableFuture);
2. Async query results (Future, ListenableFuture, CompletableFuture as
return type);
3. List of supported query return types (Appendix D);
4. List of supported query keywords (Appendix C);
5. Support of Pageable, Sort as method parameters;
6. Transaction support;
7. Lock support;
8. Querydsl support;
9. Query by example (QbE) support;
10. Enabling auto-cache configuration/creation by repository interface;
11. Support entities with keys.
12. Explicit query support (user-defined).
13. Support other Ignite capabilities.

Well, we should decide what we want to see in first version of spring-data
support.

Below I will try to explain paragraphs in more detail.

1. Nothing to explain. I think it can wait.

2. We already have async API, so there should be no problem to support this.

3. It is connected with 1 and 5. I think we will not
support GeoResult, GeoResults, GeoPage, guava's Optional.

4. I think it should match our SQL-capabilities

5. As I know we support already LIMIT, OFFSET, ORDER BY. I am wondered
by Page return type (should we support it?).

6. There is no support in sql now.

7. We can do it by adding extra methods to IgniteRepository.

8. I haven't seen what it is but looks interesting.
9. See p.9.

10. At least we should determine in some way which cache we should use to
execute queries. In the prototype I am using own annotation for it.
But maybe we should be able to create cache configuration from scratch only
by repository interface (with the usage of own annotations).


11. By default, spring data assumes that entity contains keys. But it is
not our case. We can throw UnsupportedException when methods with an only
entity are used as method-parameter, or check that there is a field with
@Id annotation.
Also, there is separate module 'Spring Data Key Value' which provides some
infrastructure for key/value storage integration (
https://github.com/spring-projects/spring-data-keyvalue). Need check if it
makes sense to use it for Ignite integration.

12. In the prototype I have defined annotation @Query. The user can
annotate query-method with it and we will use it to query cache.

13. There are many features in Ignite. Which should we support in this
integration?

Also, you can look on Semen's comment in ticket for repository example:
https://issues.apache.org/jira/browse/IGNITE-1192?focusedCommentId=15398164&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15398164

Any thoughts, opinions?
Reply | Threaded
Open this post in threaded view
|

Re: IGNITE-1192 Implementation (Integration with Spring Data)

Eduard Shangareev
I have looked at Querydsl and QbE and I think their support should be
postponed.
Reply | Threaded
Open this post in threaded view
|

Re: IGNITE-1192 Implementation (Integration with Spring Data)

Eduard Shangareev
So there is no async support for query, therefore we can't support futures
as return type (paragraph 2).

On Thu, Aug 4, 2016 at 6:08 PM, Eduard Shangareev <
[hidden email]> wrote:

> I have looked at Querydsl and QbE and I think their support should be
> postponed.
>
Reply | Threaded
Open this post in threaded view
|

Re: IGNITE-1192 Implementation (Integration with Spring Data)

Valentin Kulichenko
Hi Eduard,

This is too much information :)

I think you should create separate tickets for different aspects of Spring
Data integration and use IGNITE-1192 only as an umbrella, because otherwise
it will be flooded by comments. This will allow to implement this
functionality step by step and track the progress better.

Agree?

-Val

On Fri, Aug 5, 2016 at 4:17 AM, Eduard Shangareev <
[hidden email]> wrote:

> So there is no async support for query, therefore we can't support futures
> as return type (paragraph 2).
>
> On Thu, Aug 4, 2016 at 6:08 PM, Eduard Shangareev <
> [hidden email]> wrote:
>
> > I have looked at Querydsl and QbE and I think their support should be
> > postponed.
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: IGNITE-1192 Implementation (Integration with Spring Data)

Eduard Shangareev
Hi, Valentin.
I am going to close these ticket with some prototype and create new ones
(maybe, with umbrella one).

Will it work?

On Sat, Aug 6, 2016 at 8:03 AM, Valentin Kulichenko <
[hidden email]> wrote:

> Hi Eduard,
>
> This is too much information :)
>
> I think you should create separate tickets for different aspects of Spring
> Data integration and use IGNITE-1192 only as an umbrella, because otherwise
> it will be flooded by comments. This will allow to implement this
> functionality step by step and track the progress better.
>
> Agree?
>
> -Val
>
> On Fri, Aug 5, 2016 at 4:17 AM, Eduard Shangareev <
> [hidden email]> wrote:
>
> > So there is no async support for query, therefore we can't support
> futures
> > as return type (paragraph 2).
> >
> > On Thu, Aug 4, 2016 at 6:08 PM, Eduard Shangareev <
> > [hidden email]> wrote:
> >
> > > I have looked at Querydsl and QbE and I think their support should be
> > > postponed.
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: IGNITE-1192 Implementation (Integration with Spring Data)

Valentin Kulichenko
Eduard,

It's up to you :) I'm just saying that we should split the big task into
smaller ones.

-Val

On Mon, Aug 8, 2016 at 3:57 AM, Eduard Shangareev <
[hidden email]> wrote:

> Hi, Valentin.
> I am going to close these ticket with some prototype and create new ones
> (maybe, with umbrella one).
>
> Will it work?
>
> On Sat, Aug 6, 2016 at 8:03 AM, Valentin Kulichenko <
> [hidden email]> wrote:
>
> > Hi Eduard,
> >
> > This is too much information :)
> >
> > I think you should create separate tickets for different aspects of
> Spring
> > Data integration and use IGNITE-1192 only as an umbrella, because
> otherwise
> > it will be flooded by comments. This will allow to implement this
> > functionality step by step and track the progress better.
> >
> > Agree?
> >
> > -Val
> >
> > On Fri, Aug 5, 2016 at 4:17 AM, Eduard Shangareev <
> > [hidden email]> wrote:
> >
> > > So there is no async support for query, therefore we can't support
> > futures
> > > as return type (paragraph 2).
> > >
> > > On Thu, Aug 4, 2016 at 6:08 PM, Eduard Shangareev <
> > > [hidden email]> wrote:
> > >
> > > > I have looked at Querydsl and QbE and I think their support should be
> > > > postponed.
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: IGNITE-1192 Implementation (Integration with Spring Data)

Eduard Shangareev
Guys, I have made a pull request.

https://github.com/apache/ignite/pull/931
<https://issues.apache.org/jira/browse/IGNITE-1192#>

New module spring-data was added.

The example of repository:

@RepositoryConfig(cacheName = "cache")
public interface FirstRepository extends IgniteRepository<Person, Integer> {
    /** */
    public List<Person> findByFirstName(String val);

    /** */
    public List<Person> findByFirstNameContaining(String val);

    /** */
    public List<Person> findByFirstNameRegex(String val, Pageable pageable);

    /** */
    public Collection<Person> findTopByFirstNameContaining(String val);

    /** */
    public Iterable<Person> findFirst10ByFirstNameLike(String val);

    /** */
    public int countByFirstNameLike(String val);

    /** */
    public int countByFirstNameLikeAndSecondNameLike(String like1,
String like2);

    /** */
    public int countByFirstNameStartingWithOrSecondNameStartingWith(String
like1, String like2);

    /** */
    public List<Cache.Entry<Integer, Person>> findBySecondNameLike(String val);

    /** */
    public Cache.Entry<Integer, Person> findTopBySecondNameLike(String val);

    /** */
    public Person findTopBySecondNameStartingWith(String val);

    /** */
    @Query("firstName = ?")
    public List<Person> simpleQuery(String val);

    /** */
    @Query("firstName REGEXP ?")
    public List<Person> queryWithSort(String val, Sort sort);

    /** */
    @Query("SELECT * FROM Person WHERE firstName REGEXP ?")
    public List<Person> queryWithPageable(String val, Pageable pageable);

    /** */
    @Query("SELECT secondName FROM Person WHERE firstName REGEXP ?")
    public List<String> selectField(String val, Pageable pageable);

    /** */
    @Query("SELECT _key, secondName FROM Person WHERE firstName REGEXP ?")
    public List<List> selectSeveralField(String val, Pageable pageable);

    /** */
    @Query("SELECT count(1) FROM (SELECT DISTINCT secondName FROM
Person WHERE firstName REGEXP ?)")
    public int countQuery(String val);
}


A user should extend IgniteRepository and provide cache name via
@RepositoryConfig.

IgniteRepository extends CrudRepository and deprecates unsupported
operation (without id).





On Tue, Aug 9, 2016 at 12:06 AM, Valentin Kulichenko <
[hidden email]> wrote:

> Eduard,
>
> It's up to you :) I'm just saying that we should split the big task into
> smaller ones.
>
> -Val
>
> On Mon, Aug 8, 2016 at 3:57 AM, Eduard Shangareev <
> [hidden email]> wrote:
>
> > Hi, Valentin.
> > I am going to close these ticket with some prototype and create new ones
> > (maybe, with umbrella one).
> >
> > Will it work?
> >
> > On Sat, Aug 6, 2016 at 8:03 AM, Valentin Kulichenko <
> > [hidden email]> wrote:
> >
> > > Hi Eduard,
> > >
> > > This is too much information :)
> > >
> > > I think you should create separate tickets for different aspects of
> > Spring
> > > Data integration and use IGNITE-1192 only as an umbrella, because
> > otherwise
> > > it will be flooded by comments. This will allow to implement this
> > > functionality step by step and track the progress better.
> > >
> > > Agree?
> > >
> > > -Val
> > >
> > > On Fri, Aug 5, 2016 at 4:17 AM, Eduard Shangareev <
> > > [hidden email]> wrote:
> > >
> > > > So there is no async support for query, therefore we can't support
> > > futures
> > > > as return type (paragraph 2).
> > > >
> > > > On Thu, Aug 4, 2016 at 6:08 PM, Eduard Shangareev <
> > > > [hidden email]> wrote:
> > > >
> > > > > I have looked at Querydsl and QbE and I think their support should
> be
> > > > > postponed.
> > > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: IGNITE-1192 Implementation (Integration with Spring Data)

Eduard Shangareev
Greetings, guys.

So, the current implementation was found adequate for the first version.
But there are two issues which Semen found.

1) now cache name is bound to a repository at compile time, could you
please try to investigate if a more flexible binding is possible?
2) as I see now all spring-data integrations are hosted here -
https://github.com/spring-projects, Could you please investigated how
ignite integration can be added there?

First one, I don't see possibilities to provide cache names from Spring
configuration to repository factory (except system properties).
Second, I haven't found how to publish our spring-data implementation to
spring-projects repository.

I will continue the investigation.
But maybe someone already knows the answers.


On Tue, Aug 9, 2016 at 2:46 PM, Eduard Shangareev <
[hidden email]> wrote:

> Guys, I have made a pull request.
>
> https://github.com/apache/ignite/pull/931
> <https://issues.apache.org/jira/browse/IGNITE-1192#>
>
> New module spring-data was added.
>
> The example of repository:
>
> @RepositoryConfig(cacheName = "cache")
> public interface FirstRepository extends IgniteRepository<Person, Integer> {
>     /** */
>     public List<Person> findByFirstName(String val);
>
>     /** */
>     public List<Person> findByFirstNameContaining(String val);
>
>     /** */
>     public List<Person> findByFirstNameRegex(String val, Pageable pageable);
>
>     /** */
>     public Collection<Person> findTopByFirstNameContaining(String val);
>
>     /** */
>     public Iterable<Person> findFirst10ByFirstNameLike(String val);
>
>     /** */
>     public int countByFirstNameLike(String val);
>
>     /** */
>     public int countByFirstNameLikeAndSecondNameLike(String like1, String like2);
>
>     /** */
>     public int countByFirstNameStartingWithOrSecondNameStartingWith(String like1, String like2);
>
>     /** */
>     public List<Cache.Entry<Integer, Person>> findBySecondNameLike(String val);
>
>     /** */
>     public Cache.Entry<Integer, Person> findTopBySecondNameLike(String val);
>
>     /** */
>     public Person findTopBySecondNameStartingWith(String val);
>
>     /** */
>     @Query("firstName = ?")
>     public List<Person> simpleQuery(String val);
>
>     /** */
>     @Query("firstName REGEXP ?")
>     public List<Person> queryWithSort(String val, Sort sort);
>
>     /** */
>     @Query("SELECT * FROM Person WHERE firstName REGEXP ?")
>     public List<Person> queryWithPageable(String val, Pageable pageable);
>
>     /** */
>     @Query("SELECT secondName FROM Person WHERE firstName REGEXP ?")
>     public List<String> selectField(String val, Pageable pageable);
>
>     /** */
>     @Query("SELECT _key, secondName FROM Person WHERE firstName REGEXP ?")
>     public List<List> selectSeveralField(String val, Pageable pageable);
>
>     /** */
>     @Query("SELECT count(1) FROM (SELECT DISTINCT secondName FROM Person WHERE firstName REGEXP ?)")
>     public int countQuery(String val);
> }
>
>
> A user should extend IgniteRepository and provide cache name via
> @RepositoryConfig.
>
> IgniteRepository extends CrudRepository and deprecates unsupported
> operation (without id).
>
>
>
>
>
> On Tue, Aug 9, 2016 at 12:06 AM, Valentin Kulichenko <
> [hidden email]> wrote:
>
>> Eduard,
>>
>> It's up to you :) I'm just saying that we should split the big task into
>> smaller ones.
>>
>> -Val
>>
>> On Mon, Aug 8, 2016 at 3:57 AM, Eduard Shangareev <
>> [hidden email]> wrote:
>>
>> > Hi, Valentin.
>> > I am going to close these ticket with some prototype and create new ones
>> > (maybe, with umbrella one).
>> >
>> > Will it work?
>> >
>> > On Sat, Aug 6, 2016 at 8:03 AM, Valentin Kulichenko <
>> > [hidden email]> wrote:
>> >
>> > > Hi Eduard,
>> > >
>> > > This is too much information :)
>> > >
>> > > I think you should create separate tickets for different aspects of
>> > Spring
>> > > Data integration and use IGNITE-1192 only as an umbrella, because
>> > otherwise
>> > > it will be flooded by comments. This will allow to implement this
>> > > functionality step by step and track the progress better.
>> > >
>> > > Agree?
>> > >
>> > > -Val
>> > >
>> > > On Fri, Aug 5, 2016 at 4:17 AM, Eduard Shangareev <
>> > > [hidden email]> wrote:
>> > >
>> > > > So there is no async support for query, therefore we can't support
>> > > futures
>> > > > as return type (paragraph 2).
>> > > >
>> > > > On Thu, Aug 4, 2016 at 6:08 PM, Eduard Shangareev <
>> > > > [hidden email]> wrote:
>> > > >
>> > > > > I have looked at Querydsl and QbE and I think their support
>> should be
>> > > > > postponed.
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: IGNITE-1192 Implementation (Integration with Spring Data)

dmagda
Folks, good news,

The integration has been just merged into the master branch and will be released in Apache Ignite 2.0.

Eduard, thanks for tremendous work done by you especially on the side of query generator module! In general, I’ve just polished your pull request, added more tests and introduced an example:
https://github.com/apache/ignite/tree/master/examples/src/main/java/org/apache/ignite/examples/springdata

A readme.io <http://readme.io/> documentation will be ready throughout 2 nearest weeks.


Denis

> On Sep 1, 2016, at 1:43 AM, Eduard Shangareev <[hidden email]> wrote:
>
> Greetings, guys.
>
> So, the current implementation was found adequate for the first version.
> But there are two issues which Semen found.
>
> 1) now cache name is bound to a repository at compile time, could you
> please try to investigate if a more flexible binding is possible?
> 2) as I see now all spring-data integrations are hosted here -
> https://github.com/spring-projects, Could you please investigated how
> ignite integration can be added there?
>
> First one, I don't see possibilities to provide cache names from Spring
> configuration to repository factory (except system properties).
> Second, I haven't found how to publish our spring-data implementation to
> spring-projects repository.
>
> I will continue the investigation.
> But maybe someone already knows the answers.
>
>
> On Tue, Aug 9, 2016 at 2:46 PM, Eduard Shangareev <
> [hidden email]> wrote:
>
>> Guys, I have made a pull request.
>>
>> https://github.com/apache/ignite/pull/931
>> <https://issues.apache.org/jira/browse/IGNITE-1192#>
>>
>> New module spring-data was added.
>>
>> The example of repository:
>>
>> @RepositoryConfig(cacheName = "cache")
>> public interface FirstRepository extends IgniteRepository<Person, Integer> {
>>    /** */
>>    public List<Person> findByFirstName(String val);
>>
>>    /** */
>>    public List<Person> findByFirstNameContaining(String val);
>>
>>    /** */
>>    public List<Person> findByFirstNameRegex(String val, Pageable pageable);
>>
>>    /** */
>>    public Collection<Person> findTopByFirstNameContaining(String val);
>>
>>    /** */
>>    public Iterable<Person> findFirst10ByFirstNameLike(String val);
>>
>>    /** */
>>    public int countByFirstNameLike(String val);
>>
>>    /** */
>>    public int countByFirstNameLikeAndSecondNameLike(String like1, String like2);
>>
>>    /** */
>>    public int countByFirstNameStartingWithOrSecondNameStartingWith(String like1, String like2);
>>
>>    /** */
>>    public List<Cache.Entry<Integer, Person>> findBySecondNameLike(String val);
>>
>>    /** */
>>    public Cache.Entry<Integer, Person> findTopBySecondNameLike(String val);
>>
>>    /** */
>>    public Person findTopBySecondNameStartingWith(String val);
>>
>>    /** */
>>    @Query("firstName = ?")
>>    public List<Person> simpleQuery(String val);
>>
>>    /** */
>>    @Query("firstName REGEXP ?")
>>    public List<Person> queryWithSort(String val, Sort sort);
>>
>>    /** */
>>    @Query("SELECT * FROM Person WHERE firstName REGEXP ?")
>>    public List<Person> queryWithPageable(String val, Pageable pageable);
>>
>>    /** */
>>    @Query("SELECT secondName FROM Person WHERE firstName REGEXP ?")
>>    public List<String> selectField(String val, Pageable pageable);
>>
>>    /** */
>>    @Query("SELECT _key, secondName FROM Person WHERE firstName REGEXP ?")
>>    public List<List> selectSeveralField(String val, Pageable pageable);
>>
>>    /** */
>>    @Query("SELECT count(1) FROM (SELECT DISTINCT secondName FROM Person WHERE firstName REGEXP ?)")
>>    public int countQuery(String val);
>> }
>>
>>
>> A user should extend IgniteRepository and provide cache name via
>> @RepositoryConfig.
>>
>> IgniteRepository extends CrudRepository and deprecates unsupported
>> operation (without id).
>>
>>
>>
>>
>>
>> On Tue, Aug 9, 2016 at 12:06 AM, Valentin Kulichenko <
>> [hidden email]> wrote:
>>
>>> Eduard,
>>>
>>> It's up to you :) I'm just saying that we should split the big task into
>>> smaller ones.
>>>
>>> -Val
>>>
>>> On Mon, Aug 8, 2016 at 3:57 AM, Eduard Shangareev <
>>> [hidden email]> wrote:
>>>
>>>> Hi, Valentin.
>>>> I am going to close these ticket with some prototype and create new ones
>>>> (maybe, with umbrella one).
>>>>
>>>> Will it work?
>>>>
>>>> On Sat, Aug 6, 2016 at 8:03 AM, Valentin Kulichenko <
>>>> [hidden email]> wrote:
>>>>
>>>>> Hi Eduard,
>>>>>
>>>>> This is too much information :)
>>>>>
>>>>> I think you should create separate tickets for different aspects of
>>>> Spring
>>>>> Data integration and use IGNITE-1192 only as an umbrella, because
>>>> otherwise
>>>>> it will be flooded by comments. This will allow to implement this
>>>>> functionality step by step and track the progress better.
>>>>>
>>>>> Agree?
>>>>>
>>>>> -Val
>>>>>
>>>>> On Fri, Aug 5, 2016 at 4:17 AM, Eduard Shangareev <
>>>>> [hidden email]> wrote:
>>>>>
>>>>>> So there is no async support for query, therefore we can't support
>>>>> futures
>>>>>> as return type (paragraph 2).
>>>>>>
>>>>>> On Thu, Aug 4, 2016 at 6:08 PM, Eduard Shangareev <
>>>>>> [hidden email]> wrote:
>>>>>>
>>>>>>> I have looked at Querydsl and QbE and I think their support
>>> should be
>>>>>>> postponed.
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>