Transformers in SCAN queries
Locked
Transformers in SCAN queries
 
|
Igniters,
I keep getting requests from our users to add optional transformers to SCAN queries. This will allow to iterate through cache, but do not transfer whole key-value pairs across networks (e.g., get only keys). The feature looks useful and I created a ticket [1]. I am struggling with the design now. The problem is that I wanted to extend existing ScanQuery object for this, but this seems to be impossible because it already extends Query<Cache.Entry<K, V>> and thus can iterate only through entries. The only option I see now is to create a separate query type, copy-paste everything from ScanQuery and add *mandatory* transformer. Something like this: ScanTransformQuery<K, V, R> extends Query<R> { IgniteBiPredicate<K, V> filter; IgniteClosure<Cache.Entry<K, V>, R> transformer; int part; ... } Thoughts? Does anyone has other ideas? [1] https://issues.apache.org/jira/browse/IGNITE-2546 -Val |
|
Valentin,
Wouldn’t the same effect be achieved by broadcasting a closure to the cluster and executing scan-query on every node locally? D. On Wed, Feb 3, 2016 at 9:17 PM, Valentin Kulichenko < [hidden email]> wrote: > Igniters, > > I keep getting requests from our users to add optional transformers to SCAN > queries. This will allow to iterate through cache, but do not transfer > whole key-value pairs across networks (e.g., get only keys). The feature > looks useful and I created a ticket [1]. > > I am struggling with the design now. The problem is that I wanted to extend > existing ScanQuery object for this, but this seems to be impossible because > it already extends Query<Cache.Entry<K, V>> and thus can iterate only > through entries. > > The only option I see now is to create a separate query type, copy-paste > everything from ScanQuery and add *mandatory* transformer. Something like > this: > > ScanTransformQuery<K, V, R> extends Query<R> { > IgniteBiPredicate<K, V> filter; > IgniteClosure<Cache.Entry<K, V>, R> transformer; > int part; > ... > } > > Thoughts? Does anyone has other ideas? > > [1] https://issues.apache.org/jira/browse/IGNITE-2546 > > -Val > |
Re: Transformers in SCAN queries
|
|
Dmitry,
The main difference in my view is that you lose pagination when sending results from servers to client. What if one wants to iterate through all entries in cache? On Wed, Feb 3, 2016 at 9:47 PM, Dmitriy Setrakyan <[hidden email]> wrote: > Valentin, > > Wouldn’t the same effect be achieved by broadcasting a closure to the > cluster and executing scan-query on every node locally? > > D. > > On Wed, Feb 3, 2016 at 9:17 PM, Valentin Kulichenko < > [hidden email]> wrote: > > > Igniters, > > > > I keep getting requests from our users to add optional transformers to > SCAN > > queries. This will allow to iterate through cache, but do not transfer > > whole key-value pairs across networks (e.g., get only keys). The feature > > looks useful and I created a ticket [1]. > > > > I am struggling with the design now. The problem is that I wanted to > extend > > existing ScanQuery object for this, but this seems to be impossible > because > > it already extends Query<Cache.Entry<K, V>> and thus can iterate only > > through entries. > > > > The only option I see now is to create a separate query type, copy-paste > > everything from ScanQuery and add *mandatory* transformer. Something like > > this: > > > > ScanTransformQuery<K, V, R> extends Query<R> { > > IgniteBiPredicate<K, V> filter; > > IgniteClosure<Cache.Entry<K, V>, R> transformer; > > int part; > > ... > > } > > > > Thoughts? Does anyone has other ideas? > > > > [1] https://issues.apache.org/jira/browse/IGNITE-2546 > > > > -Val > > > |
|
|
On Wed, Feb 3, 2016 at 10:28 PM, Valentin Kulichenko <
[hidden email]> wrote: > Dmitry, > > The main difference in my view is that you lose pagination when sending > results from servers to client. What if one wants to iterate through all > entries in cache? > I see. Perhaps we should fix the pagination for compute instead of adding transformers for queries? > > On Wed, Feb 3, 2016 at 9:47 PM, Dmitriy Setrakyan <[hidden email]> > wrote: > > > Valentin, > > > > Wouldn’t the same effect be achieved by broadcasting a closure to the > > cluster and executing scan-query on every node locally? > > > > D. > > > > On Wed, Feb 3, 2016 at 9:17 PM, Valentin Kulichenko < > > [hidden email]> wrote: > > > > > Igniters, > > > > > > I keep getting requests from our users to add optional transformers to > > SCAN > > > queries. This will allow to iterate through cache, but do not transfer > > > whole key-value pairs across networks (e.g., get only keys). The > feature > > > looks useful and I created a ticket [1]. > > > > > > I am struggling with the design now. The problem is that I wanted to > > extend > > > existing ScanQuery object for this, but this seems to be impossible > > because > > > it already extends Query<Cache.Entry<K, V>> and thus can iterate only > > > through entries. > > > > > > The only option I see now is to create a separate query type, > copy-paste > > > everything from ScanQuery and add *mandatory* transformer. Something > like > > > this: > > > > > > ScanTransformQuery<K, V, R> extends Query<R> { > > > IgniteBiPredicate<K, V> filter; > > > IgniteClosure<Cache.Entry<K, V>, R> transformer; > > > int part; > > > ... > > > } > > > > > > Thoughts? Does anyone has other ideas? > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-2546 > > > > > > -Val > > > > > > |
Re: Transformers in SCAN queries
|
|
I think scan query implementation can be more complex than just sending
closures to all nodes. e.g. it should handle topology changes. IMO it is not good idea to use compute instead of queries. On Thu, Feb 4, 2016 at 10:55 AM, Dmitriy Setrakyan <[hidden email]> wrote: > On Wed, Feb 3, 2016 at 10:28 PM, Valentin Kulichenko < > [hidden email]> wrote: > > > Dmitry, > > > > The main difference in my view is that you lose pagination when sending > > results from servers to client. What if one wants to iterate through all > > entries in cache? > > > > I see. Perhaps we should fix the pagination for compute instead of adding > transformers for queries? > > > > > > On Wed, Feb 3, 2016 at 9:47 PM, Dmitriy Setrakyan <[hidden email] > > > > wrote: > > > > > Valentin, > > > > > > Wouldn’t the same effect be achieved by broadcasting a closure to the > > > cluster and executing scan-query on every node locally? > > > > > > D. > > > > > > On Wed, Feb 3, 2016 at 9:17 PM, Valentin Kulichenko < > > > [hidden email]> wrote: > > > > > > > Igniters, > > > > > > > > I keep getting requests from our users to add optional transformers > to > > > SCAN > > > > queries. This will allow to iterate through cache, but do not > transfer > > > > whole key-value pairs across networks (e.g., get only keys). The > > feature > > > > looks useful and I created a ticket [1]. > > > > > > > > I am struggling with the design now. The problem is that I wanted to > > > extend > > > > existing ScanQuery object for this, but this seems to be impossible > > > because > > > > it already extends Query<Cache.Entry<K, V>> and thus can iterate only > > > > through entries. > > > > > > > > The only option I see now is to create a separate query type, > > copy-paste > > > > everything from ScanQuery and add *mandatory* transformer. Something > > like > > > > this: > > > > > > > > ScanTransformQuery<K, V, R> extends Query<R> { > > > > IgniteBiPredicate<K, V> filter; > > > > IgniteClosure<Cache.Entry<K, V>, R> transformer; > > > > int part; > > > > ... > > > > } > > > > > > > > Thoughts? Does anyone has other ideas? > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-2546 > > > > > > > > -Val > > > > > > > > > > |
Re: Transformers in SCAN queries
|
|
In reply to this post by Valentin Kulichenko
Another perhaps bigger problem with running queries (including scan queries) using closures was discussed at length on the @dev not so long ago. It has to do with partitions migration due to cluster topology changes which may result in the query returning incomplete result. And while it is possible to solve this problem for the scan queries by using some clever tricks, all bets are off with the SQL queries.Andrey
_____________________________ From: Valentin Kulichenko <[hidden email]> Sent: Thursday, February 4, 2016 6:29 AM Subject: Re: Transformers in SCAN queries To: <[hidden email]> Dmitry, The main difference in my view is that you lose pagination when sending results from servers to client. What if one wants to iterate through all entries in cache? On Wed, Feb 3, 2016 at 9:47 PM, Dmitriy Setrakyan <[hidden email]> wrote: > Valentin, > > Wouldn’t the same effect be achieved by broadcasting a closure to the > cluster and executing scan-query on every node locally? > > D. > > On Wed, Feb 3, 2016 at 9:17 PM, Valentin Kulichenko < > [hidden email]> wrote: > > > Igniters, > > > > I keep getting requests from our users to add optional transformers to > SCAN > > queries. This will allow to iterate through cache, but do not transfer > > whole key-value pairs across networks (e.g., get only keys). The feature > > looks useful and I created a ticket [1]. > > > > I am struggling with the design now. The problem is that I wanted to > extend > > existing ScanQuery object for this, but this seems to be impossible > because > > it already extends Query<Cache.Entry<K, V>> and thus can iterate only > > through entries. > > > > The only option I see now is to create a separate query type, copy-paste > > everything from ScanQuery and add *mandatory* transformer. Something like > > this: > > > > ScanTransformQuery<K, V, R> extends Query<R> { > > IgniteBiPredicate<K, V> filter; > > IgniteClosure<Cache.Entry<K, V>, R> transformer; > > int part; > > ... > > } > > > > Thoughts? Does anyone has other ideas? > > > > [1] https://issues.apache.org/jira/browse/IGNITE-2546 > > > > -Val > > > |
|
|
Val,
can we introduce new method into IgnteCache API? Now we have method: public <R> QueryCursor<R> query(Query<R> qry); New method will be something like this: <R> QueryCursor<R> query(Query<E> qry, Transformer<E, R> transfomer); It allows provide transformers for all query types and chnages will be related only with query cursor functionality. Will it work? On Thu, Feb 4, 2016 at 11:13 AM, Andrey Kornev <[hidden email]> wrote: > Another perhaps bigger problem with running queries (including scan > queries) using closures was discussed at length on the @dev not so long > ago. It has to do with partitions migration due to cluster topology changes > which may result in the query returning incomplete result. And while it is > possible to solve this problem for the scan queries by using some clever > tricks, all bets are off with the SQL queries.Andrey > _____________________________ > From: Valentin Kulichenko <[hidden email]> > Sent: Thursday, February 4, 2016 6:29 AM > Subject: Re: Transformers in SCAN queries > To: <[hidden email]> > > > Dmitry, > > The main difference in my view is that you lose pagination when sending > results from servers to client. What if one wants to iterate through all > entries in cache? > > On Wed, Feb 3, 2016 at 9:47 PM, Dmitriy Setrakyan <[hidden email]> > wrote: > > > Valentin, > > > > Wouldn’t the same effect be achieved by broadcasting a closure to the > > cluster and executing scan-query on every node locally? > > > > D. > > > > On Wed, Feb 3, 2016 at 9:17 PM, Valentin Kulichenko < > > [hidden email]> wrote: > > > > > Igniters, > > > > > > I keep getting requests from our users to add optional transformers to > > SCAN > > > queries. This will allow to iterate through cache, but do not transfer > > > whole key-value pairs across networks (e.g., get only keys). The > feature > > > looks useful and I created a ticket [1]. > > > > > > I am struggling with the design now. The problem is that I wanted to > > extend > > > existing ScanQuery object for this, but this seems to be impossible > > because > > > it already extends Query<Cache.Entry<K, V>> and thus can iterate only > > > through entries. > > > > > > The only option I see now is to create a separate query type, > copy-paste > > > everything from ScanQuery and add *mandatory* transformer. Something > like > > > this: > > > > > > ScanTransformQuery<K, V, R> extends Query<R> { > > > IgniteBiPredicate<K, V> filter; > > > IgniteClosure<Cache.Entry<K, V>, R> transformer; > > > int part; > > > ... > > > } > > > > > > Thoughts? Does anyone has other ideas? > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-2546 > > > > > > -Val > > > > > > > > > > -- Andrey Gura GridGain Systems, Inc. www.gridgain.com |
|
|
I don't like the idea of having additional method *query(Query<E> qry,
Transformer<E, R> transfomer); *because I don't see how these transformers will work for example with SQL, but this API makes you think that transformers are supported for all the query types. Sergi 2016-02-04 16:46 GMT+03:00 Andrey Gura <[hidden email]>: > Val, > > can we introduce new method into IgnteCache API? > > Now we have method: public <R> QueryCursor<R> query(Query<R> qry); > > New method will be something like this: <R> QueryCursor<R> query(Query<E> > qry, Transformer<E, R> transfomer); > > It allows provide transformers for all query types and chnages will be > related only with query cursor functionality. > > Will it work? > > On Thu, Feb 4, 2016 at 11:13 AM, Andrey Kornev <[hidden email]> > wrote: > > > Another perhaps bigger problem with running queries (including scan > > queries) using closures was discussed at length on the @dev not so long > > ago. It has to do with partitions migration due to cluster topology > changes > > which may result in the query returning incomplete result. And while it > is > > possible to solve this problem for the scan queries by using some clever > > tricks, all bets are off with the SQL queries.Andrey > > _____________________________ > > From: Valentin Kulichenko <[hidden email]> > > Sent: Thursday, February 4, 2016 6:29 AM > > Subject: Re: Transformers in SCAN queries > > To: <[hidden email]> > > > > > > Dmitry, > > > > The main difference in my view is that you lose pagination when sending > > results from servers to client. What if one wants to iterate through all > > entries in cache? > > > > On Wed, Feb 3, 2016 at 9:47 PM, Dmitriy Setrakyan < > [hidden email]> > > wrote: > > > > > Valentin, > > > > > > Wouldn’t the same effect be achieved by broadcasting a closure to the > > > cluster and executing scan-query on every node locally? > > > > > > D. > > > > > > On Wed, Feb 3, 2016 at 9:17 PM, Valentin Kulichenko < > > > [hidden email]> wrote: > > > > > > > Igniters, > > > > > > > > I keep getting requests from our users to add optional transformers > to > > > SCAN > > > > queries. This will allow to iterate through cache, but do not > transfer > > > > whole key-value pairs across networks (e.g., get only keys). The > > feature > > > > looks useful and I created a ticket [1]. > > > > > > > > I am struggling with the design now. The problem is that I wanted to > > > extend > > > > existing ScanQuery object for this, but this seems to be impossible > > > because > > > > it already extends Query<Cache.Entry<K, V>> and thus can iterate > only > > > > through entries. > > > > > > > > The only option I see now is to create a separate query type, > > copy-paste > > > > everything from ScanQuery and add *mandatory* transformer. Something > > like > > > > this: > > > > > > > > ScanTransformQuery<K, V, R> extends Query<R> { > > > > IgniteBiPredicate<K, V> filter; > > > > IgniteClosure<Cache.Entry<K, V>, R> transformer; > > > > int part; > > > > ... > > > > } > > > > > > > > Thoughts? Does anyone has other ideas? > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-2546 > > > > > > > > -Val > > > > > > > > > > > > > > > > > > > > > -- > Andrey Gura > GridGain Systems, Inc. > www.gridgain.com > |
|
|
SqlQuery, TextQuery and SpiQuery are similar to ScanQuery because all of
them also defined as Query<Cache.Entry<>>. It can be usefull to have one query SqlQuery that can return different result that will be produced from cache entry by transformer. Actually only SqlFieldsQuery has different definition. So transformers can be applied to any type of query (including SqlFieldsQuery, I believe). On Thu, Feb 4, 2016 at 7:42 PM, Sergi Vladykin <[hidden email]> wrote: > I don't like the idea of having additional method *query(Query<E> qry, > Transformer<E, R> transfomer); *because I don't see how these transformers > will work for example with SQL, but this API makes you think that > transformers are supported for all the query types. > > Sergi > > 2016-02-04 16:46 GMT+03:00 Andrey Gura <[hidden email]>: > > > Val, > > > > can we introduce new method into IgnteCache API? > > > > Now we have method: public <R> QueryCursor<R> query(Query<R> qry); > > > > New method will be something like this: <R> QueryCursor<R> query(Query<E> > > qry, Transformer<E, R> transfomer); > > > > It allows provide transformers for all query types and chnages will be > > related only with query cursor functionality. > > > > Will it work? > > > > On Thu, Feb 4, 2016 at 11:13 AM, Andrey Kornev <[hidden email] > > > > wrote: > > > > > Another perhaps bigger problem with running queries (including scan > > > queries) using closures was discussed at length on the @dev not so long > > > ago. It has to do with partitions migration due to cluster topology > > changes > > > which may result in the query returning incomplete result. And while it > > is > > > possible to solve this problem for the scan queries by using some > clever > > > tricks, all bets are off with the SQL queries.Andrey > > > _____________________________ > > > From: Valentin Kulichenko <[hidden email]> > > > Sent: Thursday, February 4, 2016 6:29 AM > > > Subject: Re: Transformers in SCAN queries > > > To: <[hidden email]> > > > > > > > > > Dmitry, > > > > > > The main difference in my view is that you lose pagination when > sending > > > results from servers to client. What if one wants to iterate through > all > > > entries in cache? > > > > > > On Wed, Feb 3, 2016 at 9:47 PM, Dmitriy Setrakyan < > > [hidden email]> > > > wrote: > > > > > > > Valentin, > > > > > > > > Wouldn’t the same effect be achieved by broadcasting a closure to > the > > > > cluster and executing scan-query on every node locally? > > > > > > > > D. > > > > > > > > On Wed, Feb 3, 2016 at 9:17 PM, Valentin Kulichenko < > > > > [hidden email]> wrote: > > > > > > > > > Igniters, > > > > > > > > > > I keep getting requests from our users to add optional > transformers > > to > > > > SCAN > > > > > queries. This will allow to iterate through cache, but do not > > transfer > > > > > whole key-value pairs across networks (e.g., get only keys). The > > > feature > > > > > looks useful and I created a ticket [1]. > > > > > > > > > > I am struggling with the design now. The problem is that I wanted > to > > > > extend > > > > > existing ScanQuery object for this, but this seems to be > impossible > > > > because > > > > > it already extends Query<Cache.Entry<K, V>> and thus can iterate > > only > > > > > through entries. > > > > > > > > > > The only option I see now is to create a separate query type, > > > copy-paste > > > > > everything from ScanQuery and add *mandatory* transformer. > Something > > > like > > > > > this: > > > > > > > > > > ScanTransformQuery<K, V, R> extends Query<R> { > > > > > IgniteBiPredicate<K, V> filter; > > > > > IgniteClosure<Cache.Entry<K, V>, R> transformer; > > > > > int part; > > > > > ... > > > > > } > > > > > > > > > > Thoughts? Does anyone has other ideas? > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-2546 > > > > > > > > > > -Val > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > Andrey Gura > > GridGain Systems, Inc. > > www.gridgain.com > > > -- Andrey Gura GridGain Systems, Inc. www.gridgain.com |
|
|
The whole point of Transformer is to do remote transform, how will this
work with SqlFieldsQuery? What you are going to transform remotely here? I believe all the transformations must happen at SQL level here. Sergi 2016-02-04 20:10 GMT+03:00 Andrey Gura <[hidden email]>: > SqlQuery, TextQuery and SpiQuery are similar to ScanQuery because all of > them also defined as Query<Cache.Entry<>>. > > It can be usefull to have one query SqlQuery that can return different > result that will be produced from cache entry by transformer. > > Actually only SqlFieldsQuery has different definition. So transformers can > be applied to any type of query (including SqlFieldsQuery, I believe). > > On Thu, Feb 4, 2016 at 7:42 PM, Sergi Vladykin <[hidden email]> > wrote: > > > I don't like the idea of having additional method *query(Query<E> qry, > > Transformer<E, R> transfomer); *because I don't see how these > transformers > > will work for example with SQL, but this API makes you think that > > transformers are supported for all the query types. > > > > Sergi > > > > 2016-02-04 16:46 GMT+03:00 Andrey Gura <[hidden email]>: > > > > > Val, > > > > > > can we introduce new method into IgnteCache API? > > > > > > Now we have method: public <R> QueryCursor<R> query(Query<R> qry); > > > > > > New method will be something like this: <R> QueryCursor<R> > query(Query<E> > > > qry, Transformer<E, R> transfomer); > > > > > > It allows provide transformers for all query types and chnages will be > > > related only with query cursor functionality. > > > > > > Will it work? > > > > > > On Thu, Feb 4, 2016 at 11:13 AM, Andrey Kornev < > [hidden email] > > > > > > wrote: > > > > > > > Another perhaps bigger problem with running queries (including scan > > > > queries) using closures was discussed at length on the @dev not so > long > > > > ago. It has to do with partitions migration due to cluster topology > > > changes > > > > which may result in the query returning incomplete result. And while > it > > > is > > > > possible to solve this problem for the scan queries by using some > > clever > > > > tricks, all bets are off with the SQL queries.Andrey > > > > _____________________________ > > > > From: Valentin Kulichenko <[hidden email]> > > > > Sent: Thursday, February 4, 2016 6:29 AM > > > > Subject: Re: Transformers in SCAN queries > > > > To: <[hidden email]> > > > > > > > > > > > > Dmitry, > > > > > > > > The main difference in my view is that you lose pagination when > > sending > > > > results from servers to client. What if one wants to iterate through > > all > > > > entries in cache? > > > > > > > > On Wed, Feb 3, 2016 at 9:47 PM, Dmitriy Setrakyan < > > > [hidden email]> > > > > wrote: > > > > > > > > > Valentin, > > > > > > > > > > Wouldn’t the same effect be achieved by broadcasting a closure to > > the > > > > > cluster and executing scan-query on every node locally? > > > > > > > > > > D. > > > > > > > > > > On Wed, Feb 3, 2016 at 9:17 PM, Valentin Kulichenko < > > > > > [hidden email]> wrote: > > > > > > > > > > > Igniters, > > > > > > > > > > > > I keep getting requests from our users to add optional > > transformers > > > to > > > > > SCAN > > > > > > queries. This will allow to iterate through cache, but do not > > > transfer > > > > > > whole key-value pairs across networks (e.g., get only keys). The > > > > feature > > > > > > looks useful and I created a ticket [1]. > > > > > > > > > > > > I am struggling with the design now. The problem is that I > wanted > > to > > > > > extend > > > > > > existing ScanQuery object for this, but this seems to be > > impossible > > > > > because > > > > > > it already extends Query<Cache.Entry<K, V>> and thus can iterate > > > only > > > > > > through entries. > > > > > > > > > > > > The only option I see now is to create a separate query type, > > > > copy-paste > > > > > > everything from ScanQuery and add *mandatory* transformer. > > Something > > > > like > > > > > > this: > > > > > > > > > > > > ScanTransformQuery<K, V, R> extends Query<R> { > > > > > > IgniteBiPredicate<K, V> filter; > > > > > > IgniteClosure<Cache.Entry<K, V>, R> transformer; > > > > > > int part; > > > > > > ... > > > > > > } > > > > > > > > > > > > Thoughts? Does anyone has other ideas? > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-2546 > > > > > > > > > > > > -Val > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Andrey Gura > > > GridGain Systems, Inc. > > > www.gridgain.com > > > > > > > > > -- > Andrey Gura > GridGain Systems, Inc. > www.gridgain.com > |
|
|
Sergi,
> What you are going to transform remotely here? I'm not going. I believe :) Just hypothetical use case: You have one SqlFieldsQuery but different requirements for returned values. For one case you have to return some string fields as is and for another case you have to trim string to 32 characters. Of course you still can trim strings locally but transformers allow you do it remotely. Anyway, this solution can be usefull for rest query types. On Thu, Feb 4, 2016 at 8:54 PM, Sergi Vladykin <[hidden email]> wrote: > The whole point of Transformer is to do remote transform, how will this > work with SqlFieldsQuery? What you are going to transform remotely here? I > believe all the transformations must happen at SQL level here. > > Sergi > > > > 2016-02-04 20:10 GMT+03:00 Andrey Gura <[hidden email]>: > > > SqlQuery, TextQuery and SpiQuery are similar to ScanQuery because all of > > them also defined as Query<Cache.Entry<>>. > > > > It can be usefull to have one query SqlQuery that can return different > > result that will be produced from cache entry by transformer. > > > > Actually only SqlFieldsQuery has different definition. So transformers > can > > be applied to any type of query (including SqlFieldsQuery, I believe). > > > > On Thu, Feb 4, 2016 at 7:42 PM, Sergi Vladykin <[hidden email] > > > > wrote: > > > > > I don't like the idea of having additional method *query(Query<E> qry, > > > Transformer<E, R> transfomer); *because I don't see how these > > transformers > > > will work for example with SQL, but this API makes you think that > > > transformers are supported for all the query types. > > > > > > Sergi > > > > > > 2016-02-04 16:46 GMT+03:00 Andrey Gura <[hidden email]>: > > > > > > > Val, > > > > > > > > can we introduce new method into IgnteCache API? > > > > > > > > Now we have method: public <R> QueryCursor<R> query(Query<R> qry); > > > > > > > > New method will be something like this: <R> QueryCursor<R> > > query(Query<E> > > > > qry, Transformer<E, R> transfomer); > > > > > > > > It allows provide transformers for all query types and chnages will > be > > > > related only with query cursor functionality. > > > > > > > > Will it work? > > > > > > > > On Thu, Feb 4, 2016 at 11:13 AM, Andrey Kornev < > > [hidden email] > > > > > > > > wrote: > > > > > > > > > Another perhaps bigger problem with running queries (including scan > > > > > queries) using closures was discussed at length on the @dev not so > > long > > > > > ago. It has to do with partitions migration due to cluster topology > > > > changes > > > > > which may result in the query returning incomplete result. And > while > > it > > > > is > > > > > possible to solve this problem for the scan queries by using some > > > clever > > > > > tricks, all bets are off with the SQL queries.Andrey > > > > > _____________________________ > > > > > From: Valentin Kulichenko <[hidden email]> > > > > > Sent: Thursday, February 4, 2016 6:29 AM > > > > > Subject: Re: Transformers in SCAN queries > > > > > To: <[hidden email]> > > > > > > > > > > > > > > > Dmitry, > > > > > > > > > > The main difference in my view is that you lose pagination when > > > sending > > > > > results from servers to client. What if one wants to iterate > through > > > all > > > > > entries in cache? > > > > > > > > > > On Wed, Feb 3, 2016 at 9:47 PM, Dmitriy Setrakyan < > > > > [hidden email]> > > > > > wrote: > > > > > > > > > > > Valentin, > > > > > > > > > > > > Wouldn’t the same effect be achieved by broadcasting a closure > to > > > the > > > > > > cluster and executing scan-query on every node locally? > > > > > > > > > > > > D. > > > > > > > > > > > > On Wed, Feb 3, 2016 at 9:17 PM, Valentin Kulichenko < > > > > > > [hidden email]> wrote: > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > I keep getting requests from our users to add optional > > > transformers > > > > to > > > > > > SCAN > > > > > > > queries. This will allow to iterate through cache, but do not > > > > transfer > > > > > > > whole key-value pairs across networks (e.g., get only keys). > The > > > > > feature > > > > > > > looks useful and I created a ticket [1]. > > > > > > > > > > > > > > I am struggling with the design now. The problem is that I > > wanted > > > to > > > > > > extend > > > > > > > existing ScanQuery object for this, but this seems to be > > > impossible > > > > > > because > > > > > > > it already extends Query<Cache.Entry<K, V>> and thus can > iterate > > > > only > > > > > > > through entries. > > > > > > > > > > > > > > The only option I see now is to create a separate query type, > > > > > copy-paste > > > > > > > everything from ScanQuery and add *mandatory* transformer. > > > Something > > > > > like > > > > > > > this: > > > > > > > > > > > > > > ScanTransformQuery<K, V, R> extends Query<R> { > > > > > > > IgniteBiPredicate<K, V> filter; > > > > > > > IgniteClosure<Cache.Entry<K, V>, R> transformer; > > > > > > > int part; > > > > > > > ... > > > > > > > } > > > > > > > > > > > > > > Thoughts? Does anyone has other ideas? > > > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-2546 > > > > > > > > > > > > > > -Val > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Andrey Gura > > > > GridGain Systems, Inc. > > > > www.gridgain.com > > > > > > > > > > > > > > > -- > > Andrey Gura > > GridGain Systems, Inc. > > www.gridgain.com > > > -- Andrey Gura GridGain Systems, Inc. www.gridgain.com |
Re: Transformers in SCAN queries
|
|
Agree with Sergi. Mixing SQL with Java code transformers is confusing. In
rare case when it's really required, user can implement a custom function. I copy-pasted the API to the ticket [1]. Please provide any additional comments there. [1] https://issues.apache.org/jira/browse/IGNITE-2546 -Val On Thu, Feb 4, 2016 at 10:08 AM, Andrey Gura <[hidden email]> wrote: > Sergi, > > > > What you are going to transform remotely here? > > > I'm not going. I believe :) > > Just hypothetical use case: You have one SqlFieldsQuery but different > requirements for returned values. For one case you have to return some > string fields as is and for another case you have to trim string to 32 > characters. Of course you still can trim strings locally but transformers > allow you do it remotely. > > Anyway, this solution can be usefull for rest query types. > > On Thu, Feb 4, 2016 at 8:54 PM, Sergi Vladykin <[hidden email]> > wrote: > > > The whole point of Transformer is to do remote transform, how will this > > work with SqlFieldsQuery? What you are going to transform remotely here? > I > > believe all the transformations must happen at SQL level here. > > > > Sergi > > > > > > > > 2016-02-04 20:10 GMT+03:00 Andrey Gura <[hidden email]>: > > > > > SqlQuery, TextQuery and SpiQuery are similar to ScanQuery because all > of > > > them also defined as Query<Cache.Entry<>>. > > > > > > It can be usefull to have one query SqlQuery that can return different > > > result that will be produced from cache entry by transformer. > > > > > > Actually only SqlFieldsQuery has different definition. So transformers > > can > > > be applied to any type of query (including SqlFieldsQuery, I believe). > > > > > > On Thu, Feb 4, 2016 at 7:42 PM, Sergi Vladykin < > [hidden email] > > > > > > wrote: > > > > > > > I don't like the idea of having additional method *query(Query<E> > qry, > > > > Transformer<E, R> transfomer); *because I don't see how these > > > transformers > > > > will work for example with SQL, but this API makes you think that > > > > transformers are supported for all the query types. > > > > > > > > Sergi > > > > > > > > 2016-02-04 16:46 GMT+03:00 Andrey Gura <[hidden email]>: > > > > > > > > > Val, > > > > > > > > > > can we introduce new method into IgnteCache API? > > > > > > > > > > Now we have method: public <R> QueryCursor<R> query(Query<R> qry); > > > > > > > > > > New method will be something like this: <R> QueryCursor<R> > > > query(Query<E> > > > > > qry, Transformer<E, R> transfomer); > > > > > > > > > > It allows provide transformers for all query types and chnages will > > be > > > > > related only with query cursor functionality. > > > > > > > > > > Will it work? > > > > > > > > > > On Thu, Feb 4, 2016 at 11:13 AM, Andrey Kornev < > > > [hidden email] > > > > > > > > > > wrote: > > > > > > > > > > > Another perhaps bigger problem with running queries (including > scan > > > > > > queries) using closures was discussed at length on the @dev not > so > > > long > > > > > > ago. It has to do with partitions migration due to cluster > topology > > > > > changes > > > > > > which may result in the query returning incomplete result. And > > while > > > it > > > > > is > > > > > > possible to solve this problem for the scan queries by using some > > > > clever > > > > > > tricks, all bets are off with the SQL queries.Andrey > > > > > > _____________________________ > > > > > > From: Valentin Kulichenko <[hidden email]> > > > > > > Sent: Thursday, February 4, 2016 6:29 AM > > > > > > Subject: Re: Transformers in SCAN queries > > > > > > To: <[hidden email]> > > > > > > > > > > > > > > > > > > Dmitry, > > > > > > > > > > > > The main difference in my view is that you lose pagination when > > > > sending > > > > > > results from servers to client. What if one wants to iterate > > through > > > > all > > > > > > entries in cache? > > > > > > > > > > > > On Wed, Feb 3, 2016 at 9:47 PM, Dmitriy Setrakyan < > > > > > [hidden email]> > > > > > > wrote: > > > > > > > > > > > > > Valentin, > > > > > > > > > > > > > > Wouldn’t the same effect be achieved by broadcasting a closure > > to > > > > the > > > > > > > cluster and executing scan-query on every node locally? > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > On Wed, Feb 3, 2016 at 9:17 PM, Valentin Kulichenko < > > > > > > > [hidden email]> wrote: > > > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > > > I keep getting requests from our users to add optional > > > > transformers > > > > > to > > > > > > > SCAN > > > > > > > > queries. This will allow to iterate through cache, but do > not > > > > > transfer > > > > > > > > whole key-value pairs across networks (e.g., get only keys). > > The > > > > > > feature > > > > > > > > looks useful and I created a ticket [1]. > > > > > > > > > > > > > > > > I am struggling with the design now. The problem is that I > > > wanted > > > > to > > > > > > > extend > > > > > > > > existing ScanQuery object for this, but this seems to be > > > > impossible > > > > > > > because > > > > > > > > it already extends Query<Cache.Entry<K, V>> and thus can > > iterate > > > > > only > > > > > > > > through entries. > > > > > > > > > > > > > > > > The only option I see now is to create a separate query > type, > > > > > > copy-paste > > > > > > > > everything from ScanQuery and add *mandatory* transformer. > > > > Something > > > > > > like > > > > > > > > this: > > > > > > > > > > > > > > > > ScanTransformQuery<K, V, R> extends Query<R> { > > > > > > > > IgniteBiPredicate<K, V> filter; > > > > > > > > IgniteClosure<Cache.Entry<K, V>, R> transformer; > > > > > > > > int part; > > > > > > > > ... > > > > > > > > } > > > > > > > > > > > > > > > > Thoughts? Does anyone has other ideas? > > > > > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-2546 > > > > > > > > > > > > > > > > -Val > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Andrey Gura > > > > > GridGain Systems, Inc. > > > > > www.gridgain.com > > > > > > > > > > > > > > > > > > > > > -- > > > Andrey Gura > > > GridGain Systems, Inc. > > > www.gridgain.com > > > > > > > > > -- > Andrey Gura > GridGain Systems, Inc. > www.gridgain.com > |
|
|
Val,
From my point of view, special query class that allows transforming is confusing. Two points about it: 1. ScanTransformQuery will duplicate ScanQuery code with all drawbacks that we have. Moreover, any fix for ScanQuery should be repeated for ScanTransformQuery. It will lead to bugs. DRY. 2. If some users want to do transformations for SqlQuery we will introduce SqlTransformQuery. Right? At this point see previous item. Transformation logic is some kind of strategy. IMHO, the most convenient API should get transformation logic as some function provided by user. On Thu, Feb 4, 2016 at 11:05 PM, Valentin Kulichenko < [hidden email]> wrote: > Agree with Sergi. Mixing SQL with Java code transformers is confusing. In > rare case when it's really required, user can implement a custom function. > > I copy-pasted the API to the ticket [1]. Please provide any additional > comments there. > > [1] https://issues.apache.org/jira/browse/IGNITE-2546 > > -Val > > On Thu, Feb 4, 2016 at 10:08 AM, Andrey Gura <[hidden email]> wrote: > > > Sergi, > > > > > > > What you are going to transform remotely here? > > > > > > I'm not going. I believe :) > > > > Just hypothetical use case: You have one SqlFieldsQuery but different > > requirements for returned values. For one case you have to return some > > string fields as is and for another case you have to trim string to 32 > > characters. Of course you still can trim strings locally but transformers > > allow you do it remotely. > > > > Anyway, this solution can be usefull for rest query types. > > > > On Thu, Feb 4, 2016 at 8:54 PM, Sergi Vladykin <[hidden email] > > > > wrote: > > > > > The whole point of Transformer is to do remote transform, how will this > > > work with SqlFieldsQuery? What you are going to transform remotely > here? > > I > > > believe all the transformations must happen at SQL level here. > > > > > > Sergi > > > > > > > > > > > > 2016-02-04 20:10 GMT+03:00 Andrey Gura <[hidden email]>: > > > > > > > SqlQuery, TextQuery and SpiQuery are similar to ScanQuery because all > > of > > > > them also defined as Query<Cache.Entry<>>. > > > > > > > > It can be usefull to have one query SqlQuery that can return > different > > > > result that will be produced from cache entry by transformer. > > > > > > > > Actually only SqlFieldsQuery has different definition. So > transformers > > > can > > > > be applied to any type of query (including SqlFieldsQuery, I > believe). > > > > > > > > On Thu, Feb 4, 2016 at 7:42 PM, Sergi Vladykin < > > [hidden email] > > > > > > > > wrote: > > > > > > > > > I don't like the idea of having additional method *query(Query<E> > > qry, > > > > > Transformer<E, R> transfomer); *because I don't see how these > > > > transformers > > > > > will work for example with SQL, but this API makes you think that > > > > > transformers are supported for all the query types. > > > > > > > > > > Sergi > > > > > > > > > > 2016-02-04 16:46 GMT+03:00 Andrey Gura <[hidden email]>: > > > > > > > > > > > Val, > > > > > > > > > > > > can we introduce new method into IgnteCache API? > > > > > > > > > > > > Now we have method: public <R> QueryCursor<R> query(Query<R> > qry); > > > > > > > > > > > > New method will be something like this: <R> QueryCursor<R> > > > > query(Query<E> > > > > > > qry, Transformer<E, R> transfomer); > > > > > > > > > > > > It allows provide transformers for all query types and chnages > will > > > be > > > > > > related only with query cursor functionality. > > > > > > > > > > > > Will it work? > > > > > > > > > > > > On Thu, Feb 4, 2016 at 11:13 AM, Andrey Kornev < > > > > [hidden email] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Another perhaps bigger problem with running queries (including > > scan > > > > > > > queries) using closures was discussed at length on the @dev not > > so > > > > long > > > > > > > ago. It has to do with partitions migration due to cluster > > topology > > > > > > changes > > > > > > > which may result in the query returning incomplete result. And > > > while > > > > it > > > > > > is > > > > > > > possible to solve this problem for the scan queries by using > some > > > > > clever > > > > > > > tricks, all bets are off with the SQL queries.Andrey > > > > > > > _____________________________ > > > > > > > From: Valentin Kulichenko <[hidden email]> > > > > > > > Sent: Thursday, February 4, 2016 6:29 AM > > > > > > > Subject: Re: Transformers in SCAN queries > > > > > > > To: <[hidden email]> > > > > > > > > > > > > > > > > > > > > > Dmitry, > > > > > > > > > > > > > > The main difference in my view is that you lose pagination > when > > > > > sending > > > > > > > results from servers to client. What if one wants to iterate > > > through > > > > > all > > > > > > > entries in cache? > > > > > > > > > > > > > > On Wed, Feb 3, 2016 at 9:47 PM, Dmitriy Setrakyan < > > > > > > [hidden email]> > > > > > > > wrote: > > > > > > > > > > > > > > > Valentin, > > > > > > > > > > > > > > > > Wouldn’t the same effect be achieved by broadcasting a > closure > > > to > > > > > the > > > > > > > > cluster and executing scan-query on every node locally? > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > On Wed, Feb 3, 2016 at 9:17 PM, Valentin Kulichenko < > > > > > > > > [hidden email]> wrote: > > > > > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > > > > > I keep getting requests from our users to add optional > > > > > transformers > > > > > > to > > > > > > > > SCAN > > > > > > > > > queries. This will allow to iterate through cache, but do > > not > > > > > > transfer > > > > > > > > > whole key-value pairs across networks (e.g., get only > keys). > > > The > > > > > > > feature > > > > > > > > > looks useful and I created a ticket [1]. > > > > > > > > > > > > > > > > > > I am struggling with the design now. The problem is that I > > > > wanted > > > > > to > > > > > > > > extend > > > > > > > > > existing ScanQuery object for this, but this seems to be > > > > > impossible > > > > > > > > because > > > > > > > > > it already extends Query<Cache.Entry<K, V>> and thus can > > > iterate > > > > > > only > > > > > > > > > through entries. > > > > > > > > > > > > > > > > > > The only option I see now is to create a separate query > > type, > > > > > > > copy-paste > > > > > > > > > everything from ScanQuery and add *mandatory* transformer. > > > > > Something > > > > > > > like > > > > > > > > > this: > > > > > > > > > > > > > > > > > > ScanTransformQuery<K, V, R> extends Query<R> { > > > > > > > > > IgniteBiPredicate<K, V> filter; > > > > > > > > > IgniteClosure<Cache.Entry<K, V>, R> transformer; > > > > > > > > > int part; > > > > > > > > > ... > > > > > > > > > } > > > > > > > > > > > > > > > > > > Thoughts? Does anyone has other ideas? > > > > > > > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-2546 > > > > > > > > > > > > > > > > > > -Val > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Andrey Gura > > > > > > GridGain Systems, Inc. > > > > > > www.gridgain.com > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Andrey Gura > > > > GridGain Systems, Inc. > > > > www.gridgain.com > > > > > > > > > > > > > > > -- > > Andrey Gura > > GridGain Systems, Inc. > > www.gridgain.com > > > -- Andrey Gura GridGain Systems, Inc. www.gridgain.com |
|
|
I like the new method on IgniteCache API:
<R> QueryCursor<R> query(Query<E> qry, Transformer<E, R> transformer); I do agree that it makes little sense for the SqlFieldsQuery, but we can simply throw an unsupported exception here. For all other queries it does make sense. Val, can you put together all the thoughts expressed in this thread into a ticket? D. On Thu, Feb 4, 2016 at 4:14 PM, Andrey Gura <[hidden email]> wrote: > Val, > > From my point of view, special query class that allows transforming is > confusing. Two points about it: > > 1. ScanTransformQuery will duplicate ScanQuery code with all drawbacks that > we have. Moreover, any fix for ScanQuery should be repeated for > ScanTransformQuery. It will lead to bugs. DRY. > > 2. If some users want to do transformations for SqlQuery we will introduce > SqlTransformQuery. Right? At this point see previous item. > > Transformation logic is some kind of strategy. IMHO, the most convenient > API should get transformation logic as some function provided by user. > > On Thu, Feb 4, 2016 at 11:05 PM, Valentin Kulichenko < > [hidden email]> wrote: > > > Agree with Sergi. Mixing SQL with Java code transformers is confusing. In > > rare case when it's really required, user can implement a custom > function. > > > > I copy-pasted the API to the ticket [1]. Please provide any additional > > comments there. > > > > [1] https://issues.apache.org/jira/browse/IGNITE-2546 > > > > -Val > > > > On Thu, Feb 4, 2016 at 10:08 AM, Andrey Gura <[hidden email]> wrote: > > > > > Sergi, > > > > > > > > > > What you are going to transform remotely here? > > > > > > > > > I'm not going. I believe :) > > > > > > Just hypothetical use case: You have one SqlFieldsQuery but different > > > requirements for returned values. For one case you have to return some > > > string fields as is and for another case you have to trim string to 32 > > > characters. Of course you still can trim strings locally but > transformers > > > allow you do it remotely. > > > > > > Anyway, this solution can be usefull for rest query types. > > > > > > On Thu, Feb 4, 2016 at 8:54 PM, Sergi Vladykin < > [hidden email] > > > > > > wrote: > > > > > > > The whole point of Transformer is to do remote transform, how will > this > > > > work with SqlFieldsQuery? What you are going to transform remotely > > here? > > > I > > > > believe all the transformations must happen at SQL level here. > > > > > > > > Sergi > > > > > > > > > > > > > > > > 2016-02-04 20:10 GMT+03:00 Andrey Gura <[hidden email]>: > > > > > > > > > SqlQuery, TextQuery and SpiQuery are similar to ScanQuery because > all > > > of > > > > > them also defined as Query<Cache.Entry<>>. > > > > > > > > > > It can be usefull to have one query SqlQuery that can return > > different > > > > > result that will be produced from cache entry by transformer. > > > > > > > > > > Actually only SqlFieldsQuery has different definition. So > > transformers > > > > can > > > > > be applied to any type of query (including SqlFieldsQuery, I > > believe). > > > > > > > > > > On Thu, Feb 4, 2016 at 7:42 PM, Sergi Vladykin < > > > [hidden email] > > > > > > > > > > wrote: > > > > > > > > > > > I don't like the idea of having additional method *query(Query<E> > > > qry, > > > > > > Transformer<E, R> transfomer); *because I don't see how these > > > > > transformers > > > > > > will work for example with SQL, but this API makes you think that > > > > > > transformers are supported for all the query types. > > > > > > > > > > > > Sergi > > > > > > > > > > > > 2016-02-04 16:46 GMT+03:00 Andrey Gura <[hidden email]>: > > > > > > > > > > > > > Val, > > > > > > > > > > > > > > can we introduce new method into IgnteCache API? > > > > > > > > > > > > > > Now we have method: public <R> QueryCursor<R> query(Query<R> > > qry); > > > > > > > > > > > > > > New method will be something like this: <R> QueryCursor<R> > > > > > query(Query<E> > > > > > > > qry, Transformer<E, R> transfomer); > > > > > > > > > > > > > > It allows provide transformers for all query types and chnages > > will > > > > be > > > > > > > related only with query cursor functionality. > > > > > > > > > > > > > > Will it work? > > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 11:13 AM, Andrey Kornev < > > > > > [hidden email] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Another perhaps bigger problem with running queries > (including > > > scan > > > > > > > > queries) using closures was discussed at length on the @dev > not > > > so > > > > > long > > > > > > > > ago. It has to do with partitions migration due to cluster > > > topology > > > > > > > changes > > > > > > > > which may result in the query returning incomplete result. > And > > > > while > > > > > it > > > > > > > is > > > > > > > > possible to solve this problem for the scan queries by using > > some > > > > > > clever > > > > > > > > tricks, all bets are off with the SQL queries.Andrey > > > > > > > > _____________________________ > > > > > > > > From: Valentin Kulichenko <[hidden email]> > > > > > > > > Sent: Thursday, February 4, 2016 6:29 AM > > > > > > > > Subject: Re: Transformers in SCAN queries > > > > > > > > To: <[hidden email]> > > > > > > > > > > > > > > > > > > > > > > > > Dmitry, > > > > > > > > > > > > > > > > The main difference in my view is that you lose pagination > > when > > > > > > sending > > > > > > > > results from servers to client. What if one wants to iterate > > > > through > > > > > > all > > > > > > > > entries in cache? > > > > > > > > > > > > > > > > On Wed, Feb 3, 2016 at 9:47 PM, Dmitriy Setrakyan < > > > > > > > [hidden email]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Valentin, > > > > > > > > > > > > > > > > > > Wouldn’t the same effect be achieved by broadcasting a > > closure > > > > to > > > > > > the > > > > > > > > > cluster and executing scan-query on every node locally? > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > On Wed, Feb 3, 2016 at 9:17 PM, Valentin Kulichenko < > > > > > > > > > [hidden email]> wrote: > > > > > > > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > > > > > > > I keep getting requests from our users to add optional > > > > > > transformers > > > > > > > to > > > > > > > > > SCAN > > > > > > > > > > queries. This will allow to iterate through cache, but > do > > > not > > > > > > > transfer > > > > > > > > > > whole key-value pairs across networks (e.g., get only > > keys). > > > > The > > > > > > > > feature > > > > > > > > > > looks useful and I created a ticket [1]. > > > > > > > > > > > > > > > > > > > > I am struggling with the design now. The problem is > that I > > > > > wanted > > > > > > to > > > > > > > > > extend > > > > > > > > > > existing ScanQuery object for this, but this seems to be > > > > > > impossible > > > > > > > > > because > > > > > > > > > > it already extends Query<Cache.Entry<K, V>> and thus can > > > > iterate > > > > > > > only > > > > > > > > > > through entries. > > > > > > > > > > > > > > > > > > > > The only option I see now is to create a separate query > > > type, > > > > > > > > copy-paste > > > > > > > > > > everything from ScanQuery and add *mandatory* > transformer. > > > > > > Something > > > > > > > > like > > > > > > > > > > this: > > > > > > > > > > > > > > > > > > > > ScanTransformQuery<K, V, R> extends Query<R> { > > > > > > > > > > IgniteBiPredicate<K, V> filter; > > > > > > > > > > IgniteClosure<Cache.Entry<K, V>, R> transformer; > > > > > > > > > > int part; > > > > > > > > > > ... > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > Thoughts? Does anyone has other ideas? > > > > > > > > > > > > > > > > > > > > [1] > https://issues.apache.org/jira/browse/IGNITE-2546 > > > > > > > > > > > > > > > > > > > > -Val > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Andrey Gura > > > > > > > GridGain Systems, Inc. > > > > > > > www.gridgain.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Andrey Gura > > > > > GridGain Systems, Inc. > > > > > www.gridgain.com > > > > > > > > > > > > > > > > > > > > > -- > > > Andrey Gura > > > GridGain Systems, Inc. > > > www.gridgain.com > > > > > > > > > -- > Andrey Gura > GridGain Systems, Inc. > www.gridgain.com > |
Re: Transformers in SCAN queries
|
|
I changed the description in the ticket. If anyone has comments about the
implementation details, feel free to add them there. -Val On Wed, Feb 10, 2016 at 2:30 PM, Dmitriy Setrakyan <[hidden email]> wrote: > I like the new method on IgniteCache API: > > <R> QueryCursor<R> query(Query<E> qry, Transformer<E, R> transformer); > > I do agree that it makes little sense for the SqlFieldsQuery, but we can > simply throw an unsupported exception here. For all other queries it does > make sense. > > Val, can you put together all the thoughts expressed in this thread into a > ticket? > > D. > > On Thu, Feb 4, 2016 at 4:14 PM, Andrey Gura <[hidden email]> wrote: > > > Val, > > > > From my point of view, special query class that allows transforming is > > confusing. Two points about it: > > > > 1. ScanTransformQuery will duplicate ScanQuery code with all drawbacks > that > > we have. Moreover, any fix for ScanQuery should be repeated for > > ScanTransformQuery. It will lead to bugs. DRY. > > > > 2. If some users want to do transformations for SqlQuery we will > introduce > > SqlTransformQuery. Right? At this point see previous item. > > > > Transformation logic is some kind of strategy. IMHO, the most convenient > > API should get transformation logic as some function provided by user. > > > > On Thu, Feb 4, 2016 at 11:05 PM, Valentin Kulichenko < > > [hidden email]> wrote: > > > > > Agree with Sergi. Mixing SQL with Java code transformers is confusing. > In > > > rare case when it's really required, user can implement a custom > > function. > > > > > > I copy-pasted the API to the ticket [1]. Please provide any additional > > > comments there. > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-2546 > > > > > > -Val > > > > > > On Thu, Feb 4, 2016 at 10:08 AM, Andrey Gura <[hidden email]> > wrote: > > > > > > > Sergi, > > > > > > > > > > > > > What you are going to transform remotely here? > > > > > > > > > > > > I'm not going. I believe :) > > > > > > > > Just hypothetical use case: You have one SqlFieldsQuery but different > > > > requirements for returned values. For one case you have to return > some > > > > string fields as is and for another case you have to trim string to > 32 > > > > characters. Of course you still can trim strings locally but > > transformers > > > > allow you do it remotely. > > > > > > > > Anyway, this solution can be usefull for rest query types. > > > > > > > > On Thu, Feb 4, 2016 at 8:54 PM, Sergi Vladykin < > > [hidden email] > > > > > > > > wrote: > > > > > > > > > The whole point of Transformer is to do remote transform, how will > > this > > > > > work with SqlFieldsQuery? What you are going to transform remotely > > > here? > > > > I > > > > > believe all the transformations must happen at SQL level here. > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > 2016-02-04 20:10 GMT+03:00 Andrey Gura <[hidden email]>: > > > > > > > > > > > SqlQuery, TextQuery and SpiQuery are similar to ScanQuery because > > all > > > > of > > > > > > them also defined as Query<Cache.Entry<>>. > > > > > > > > > > > > It can be usefull to have one query SqlQuery that can return > > > different > > > > > > result that will be produced from cache entry by transformer. > > > > > > > > > > > > Actually only SqlFieldsQuery has different definition. So > > > transformers > > > > > can > > > > > > be applied to any type of query (including SqlFieldsQuery, I > > > believe). > > > > > > > > > > > > On Thu, Feb 4, 2016 at 7:42 PM, Sergi Vladykin < > > > > [hidden email] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > I don't like the idea of having additional method > *query(Query<E> > > > > qry, > > > > > > > Transformer<E, R> transfomer); *because I don't see how these > > > > > > transformers > > > > > > > will work for example with SQL, but this API makes you think > that > > > > > > > transformers are supported for all the query types. > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > 2016-02-04 16:46 GMT+03:00 Andrey Gura <[hidden email]>: > > > > > > > > > > > > > > > Val, > > > > > > > > > > > > > > > > can we introduce new method into IgnteCache API? > > > > > > > > > > > > > > > > Now we have method: public <R> QueryCursor<R> query(Query<R> > > > qry); > > > > > > > > > > > > > > > > New method will be something like this: <R> QueryCursor<R> > > > > > > query(Query<E> > > > > > > > > qry, Transformer<E, R> transfomer); > > > > > > > > > > > > > > > > It allows provide transformers for all query types and > chnages > > > will > > > > > be > > > > > > > > related only with query cursor functionality. > > > > > > > > > > > > > > > > Will it work? > > > > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 11:13 AM, Andrey Kornev < > > > > > > [hidden email] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Another perhaps bigger problem with running queries > > (including > > > > scan > > > > > > > > > queries) using closures was discussed at length on the @dev > > not > > > > so > > > > > > long > > > > > > > > > ago. It has to do with partitions migration due to cluster > > > > topology > > > > > > > > changes > > > > > > > > > which may result in the query returning incomplete result. > > And > > > > > while > > > > > > it > > > > > > > > is > > > > > > > > > possible to solve this problem for the scan queries by > using > > > some > > > > > > > clever > > > > > > > > > tricks, all bets are off with the SQL queries.Andrey > > > > > > > > > _____________________________ > > > > > > > > > From: Valentin Kulichenko <[hidden email]> > > > > > > > > > Sent: Thursday, February 4, 2016 6:29 AM > > > > > > > > > Subject: Re: Transformers in SCAN queries > > > > > > > > > To: <[hidden email]> > > > > > > > > > > > > > > > > > > > > > > > > > > > Dmitry, > > > > > > > > > > > > > > > > > > The main difference in my view is that you lose pagination > > > when > > > > > > > sending > > > > > > > > > results from servers to client. What if one wants to > iterate > > > > > through > > > > > > > all > > > > > > > > > entries in cache? > > > > > > > > > > > > > > > > > > On Wed, Feb 3, 2016 at 9:47 PM, Dmitriy Setrakyan < > > > > > > > > [hidden email]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Valentin, > > > > > > > > > > > > > > > > > > > > Wouldn’t the same effect be achieved by broadcasting a > > > closure > > > > > to > > > > > > > the > > > > > > > > > > cluster and executing scan-query on every node locally? > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > On Wed, Feb 3, 2016 at 9:17 PM, Valentin Kulichenko < > > > > > > > > > > [hidden email]> wrote: > > > > > > > > > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > > > > > > > > > I keep getting requests from our users to add optional > > > > > > > transformers > > > > > > > > to > > > > > > > > > > SCAN > > > > > > > > > > > queries. This will allow to iterate through cache, but > > do > > > > not > > > > > > > > transfer > > > > > > > > > > > whole key-value pairs across networks (e.g., get only > > > keys). > > > > > The > > > > > > > > > feature > > > > > > > > > > > looks useful and I created a ticket [1]. > > > > > > > > > > > > > > > > > > > > > > I am struggling with the design now. The problem is > > that I > > > > > > wanted > > > > > > > to > > > > > > > > > > extend > > > > > > > > > > > existing ScanQuery object for this, but this seems to > be > > > > > > > impossible > > > > > > > > > > because > > > > > > > > > > > it already extends Query<Cache.Entry<K, V>> and thus > can > > > > > iterate > > > > > > > > only > > > > > > > > > > > through entries. > > > > > > > > > > > > > > > > > > > > > > The only option I see now is to create a separate > query > > > > type, > > > > > > > > > copy-paste > > > > > > > > > > > everything from ScanQuery and add *mandatory* > > transformer. > > > > > > > Something > > > > > > > > > like > > > > > > > > > > > this: > > > > > > > > > > > > > > > > > > > > > > ScanTransformQuery<K, V, R> extends Query<R> { > > > > > > > > > > > IgniteBiPredicate<K, V> filter; > > > > > > > > > > > IgniteClosure<Cache.Entry<K, V>, R> transformer; > > > > > > > > > > > int part; > > > > > > > > > > > ... > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > Thoughts? Does anyone has other ideas? > > > > > > > > > > > > > > > > > > > > > > [1] > > https://issues.apache.org/jira/browse/IGNITE-2546 > > > > > > > > > > > > > > > > > > > > > > -Val > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Andrey Gura > > > > > > > > GridGain Systems, Inc. > > > > > > > > www.gridgain.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Andrey Gura > > > > > > GridGain Systems, Inc. > > > > > > www.gridgain.com > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Andrey Gura > > > > GridGain Systems, Inc. > > > > www.gridgain.com > > > > > > > > > > > > > > > -- > > Andrey Gura > > GridGain Systems, Inc. > > www.gridgain.com > > > |
|
|
Despite of trasformer makes little sense for the SqlFieldsQuery I don't see
any reason for throwing of UnsupportedOperationException. I think it is stange limitation from user standpoint. On Thu, Feb 11, 2016 at 6:01 AM, Valentin Kulichenko < [hidden email]> wrote: > I changed the description in the ticket. If anyone has comments about the > implementation details, feel free to add them there. > > -Val > > On Wed, Feb 10, 2016 at 2:30 PM, Dmitriy Setrakyan <[hidden email]> > wrote: > > > I like the new method on IgniteCache API: > > > > <R> QueryCursor<R> query(Query<E> qry, Transformer<E, R> transformer); > > > > I do agree that it makes little sense for the SqlFieldsQuery, but we can > > simply throw an unsupported exception here. For all other queries it does > > make sense. > > > > Val, can you put together all the thoughts expressed in this thread into > a > > ticket? > > > > D. > > > > On Thu, Feb 4, 2016 at 4:14 PM, Andrey Gura <[hidden email]> wrote: > > > > > Val, > > > > > > From my point of view, special query class that allows transforming is > > > confusing. Two points about it: > > > > > > 1. ScanTransformQuery will duplicate ScanQuery code with all drawbacks > > that > > > we have. Moreover, any fix for ScanQuery should be repeated for > > > ScanTransformQuery. It will lead to bugs. DRY. > > > > > > 2. If some users want to do transformations for SqlQuery we will > > introduce > > > SqlTransformQuery. Right? At this point see previous item. > > > > > > Transformation logic is some kind of strategy. IMHO, the most > convenient > > > API should get transformation logic as some function provided by user. > > > > > > On Thu, Feb 4, 2016 at 11:05 PM, Valentin Kulichenko < > > > [hidden email]> wrote: > > > > > > > Agree with Sergi. Mixing SQL with Java code transformers is > confusing. > > In > > > > rare case when it's really required, user can implement a custom > > > function. > > > > > > > > I copy-pasted the API to the ticket [1]. Please provide any > additional > > > > comments there. > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-2546 > > > > > > > > -Val > > > > > > > > On Thu, Feb 4, 2016 at 10:08 AM, Andrey Gura <[hidden email]> > > wrote: > > > > > > > > > Sergi, > > > > > > > > > > > > > > > > What you are going to transform remotely here? > > > > > > > > > > > > > > > I'm not going. I believe :) > > > > > > > > > > Just hypothetical use case: You have one SqlFieldsQuery but > different > > > > > requirements for returned values. For one case you have to return > > some > > > > > string fields as is and for another case you have to trim string to > > 32 > > > > > characters. Of course you still can trim strings locally but > > > transformers > > > > > allow you do it remotely. > > > > > > > > > > Anyway, this solution can be usefull for rest query types. > > > > > > > > > > On Thu, Feb 4, 2016 at 8:54 PM, Sergi Vladykin < > > > [hidden email] > > > > > > > > > > wrote: > > > > > > > > > > > The whole point of Transformer is to do remote transform, how > will > > > this > > > > > > work with SqlFieldsQuery? What you are going to transform > remotely > > > > here? > > > > > I > > > > > > believe all the transformations must happen at SQL level here. > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > 2016-02-04 20:10 GMT+03:00 Andrey Gura <[hidden email]>: > > > > > > > > > > > > > SqlQuery, TextQuery and SpiQuery are similar to ScanQuery > because > > > all > > > > > of > > > > > > > them also defined as Query<Cache.Entry<>>. > > > > > > > > > > > > > > It can be usefull to have one query SqlQuery that can return > > > > different > > > > > > > result that will be produced from cache entry by transformer. > > > > > > > > > > > > > > Actually only SqlFieldsQuery has different definition. So > > > > transformers > > > > > > can > > > > > > > be applied to any type of query (including SqlFieldsQuery, I > > > > believe). > > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 7:42 PM, Sergi Vladykin < > > > > > [hidden email] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > I don't like the idea of having additional method > > *query(Query<E> > > > > > qry, > > > > > > > > Transformer<E, R> transfomer); *because I don't see how these > > > > > > > transformers > > > > > > > > will work for example with SQL, but this API makes you think > > that > > > > > > > > transformers are supported for all the query types. > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > 2016-02-04 16:46 GMT+03:00 Andrey Gura <[hidden email]>: > > > > > > > > > > > > > > > > > Val, > > > > > > > > > > > > > > > > > > can we introduce new method into IgnteCache API? > > > > > > > > > > > > > > > > > > Now we have method: public <R> QueryCursor<R> > query(Query<R> > > > > qry); > > > > > > > > > > > > > > > > > > New method will be something like this: <R> QueryCursor<R> > > > > > > > query(Query<E> > > > > > > > > > qry, Transformer<E, R> transfomer); > > > > > > > > > > > > > > > > > > It allows provide transformers for all query types and > > chnages > > > > will > > > > > > be > > > > > > > > > related only with query cursor functionality. > > > > > > > > > > > > > > > > > > Will it work? > > > > > > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 11:13 AM, Andrey Kornev < > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Another perhaps bigger problem with running queries > > > (including > > > > > scan > > > > > > > > > > queries) using closures was discussed at length on the > @dev > > > not > > > > > so > > > > > > > long > > > > > > > > > > ago. It has to do with partitions migration due to > cluster > > > > > topology > > > > > > > > > changes > > > > > > > > > > which may result in the query returning incomplete > result. > > > And > > > > > > while > > > > > > > it > > > > > > > > > is > > > > > > > > > > possible to solve this problem for the scan queries by > > using > > > > some > > > > > > > > clever > > > > > > > > > > tricks, all bets are off with the SQL queries.Andrey > > > > > > > > > > _____________________________ > > > > > > > > > > From: Valentin Kulichenko <[hidden email] > > > > > > > > > > > > Sent: Thursday, February 4, 2016 6:29 AM > > > > > > > > > > Subject: Re: Transformers in SCAN queries > > > > > > > > > > To: <[hidden email]> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Dmitry, > > > > > > > > > > > > > > > > > > > > The main difference in my view is that you lose > pagination > > > > when > > > > > > > > sending > > > > > > > > > > results from servers to client. What if one wants to > > iterate > > > > > > through > > > > > > > > all > > > > > > > > > > entries in cache? > > > > > > > > > > > > > > > > > > > > On Wed, Feb 3, 2016 at 9:47 PM, Dmitriy Setrakyan < > > > > > > > > > [hidden email]> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Valentin, > > > > > > > > > > > > > > > > > > > > > > Wouldn’t the same effect be achieved by broadcasting a > > > > closure > > > > > > to > > > > > > > > the > > > > > > > > > > > cluster and executing scan-query on every node > locally? > > > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > On Wed, Feb 3, 2016 at 9:17 PM, Valentin Kulichenko < > > > > > > > > > > > [hidden email]> wrote: > > > > > > > > > > > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > > > > > > > > > > > I keep getting requests from our users to add > optional > > > > > > > > transformers > > > > > > > > > to > > > > > > > > > > > SCAN > > > > > > > > > > > > queries. This will allow to iterate through cache, > but > > > do > > > > > not > > > > > > > > > transfer > > > > > > > > > > > > whole key-value pairs across networks (e.g., get > only > > > > keys). > > > > > > The > > > > > > > > > > feature > > > > > > > > > > > > looks useful and I created a ticket [1]. > > > > > > > > > > > > > > > > > > > > > > > > I am struggling with the design now. The problem is > > > that I > > > > > > > wanted > > > > > > > > to > > > > > > > > > > > extend > > > > > > > > > > > > existing ScanQuery object for this, but this seems > to > > be > > > > > > > > impossible > > > > > > > > > > > because > > > > > > > > > > > > it already extends Query<Cache.Entry<K, V>> and thus > > can > > > > > > iterate > > > > > > > > > only > > > > > > > > > > > > through entries. > > > > > > > > > > > > > > > > > > > > > > > > The only option I see now is to create a separate > > query > > > > > type, > > > > > > > > > > copy-paste > > > > > > > > > > > > everything from ScanQuery and add *mandatory* > > > transformer. > > > > > > > > Something > > > > > > > > > > like > > > > > > > > > > > > this: > > > > > > > > > > > > > > > > > > > > > > > > ScanTransformQuery<K, V, R> extends Query<R> { > > > > > > > > > > > > IgniteBiPredicate<K, V> filter; > > > > > > > > > > > > IgniteClosure<Cache.Entry<K, V>, R> transformer; > > > > > > > > > > > > int part; > > > > > > > > > > > > ... > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > Thoughts? Does anyone has other ideas? > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > https://issues.apache.org/jira/browse/IGNITE-2546 > > > > > > > > > > > > > > > > > > > > > > > > -Val > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Andrey Gura > > > > > > > > > GridGain Systems, Inc. > > > > > > > > > www.gridgain.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Andrey Gura > > > > > > > GridGain Systems, Inc. > > > > > > > www.gridgain.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Andrey Gura > > > > > GridGain Systems, Inc. > > > > > www.gridgain.com > > > > > > > > > > > > > > > > > > > > > -- > > > Andrey Gura > > > GridGain Systems, Inc. > > > www.gridgain.com > > > > > > -- Andrey Gura GridGain Systems, Inc. www.gridgain.com |
|
|
On Thu, Feb 11, 2016 at 6:18 AM, Andrey Gura <[hidden email]> wrote:
> Despite of trasformer makes little sense for the SqlFieldsQuery I don't see > any reason for throwing of UnsupportedOperationException. > I think it is stange limitation from user standpoint. > I think either supporting it or not supporting looks strange for SqlFieldsQuery. I suggested to throw the exception only to simplify the development - simply one less thing to worry about. > > On Thu, Feb 11, 2016 at 6:01 AM, Valentin Kulichenko < > [hidden email]> wrote: > > > I changed the description in the ticket. If anyone has comments about the > > implementation details, feel free to add them there. > > > > -Val > > > > On Wed, Feb 10, 2016 at 2:30 PM, Dmitriy Setrakyan < > [hidden email]> > > wrote: > > > > > I like the new method on IgniteCache API: > > > > > > <R> QueryCursor<R> query(Query<E> qry, Transformer<E, R> transformer); > > > > > > I do agree that it makes little sense for the SqlFieldsQuery, but we > can > > > simply throw an unsupported exception here. For all other queries it > does > > > make sense. > > > > > > Val, can you put together all the thoughts expressed in this thread > into > > a > > > ticket? > > > > > > D. > > > > > > On Thu, Feb 4, 2016 at 4:14 PM, Andrey Gura <[hidden email]> > wrote: > > > > > > > Val, > > > > > > > > From my point of view, special query class that allows transforming > is > > > > confusing. Two points about it: > > > > > > > > 1. ScanTransformQuery will duplicate ScanQuery code with all > drawbacks > > > that > > > > we have. Moreover, any fix for ScanQuery should be repeated for > > > > ScanTransformQuery. It will lead to bugs. DRY. > > > > > > > > 2. If some users want to do transformations for SqlQuery we will > > > introduce > > > > SqlTransformQuery. Right? At this point see previous item. > > > > > > > > Transformation logic is some kind of strategy. IMHO, the most > > convenient > > > > API should get transformation logic as some function provided by > user. > > > > > > > > On Thu, Feb 4, 2016 at 11:05 PM, Valentin Kulichenko < > > > > [hidden email]> wrote: > > > > > > > > > Agree with Sergi. Mixing SQL with Java code transformers is > > confusing. > > > In > > > > > rare case when it's really required, user can implement a custom > > > > function. > > > > > > > > > > I copy-pasted the API to the ticket [1]. Please provide any > > additional > > > > > comments there. > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-2546 > > > > > > > > > > -Val > > > > > > > > > > On Thu, Feb 4, 2016 at 10:08 AM, Andrey Gura <[hidden email]> > > > wrote: > > > > > > > > > > > Sergi, > > > > > > > > > > > > > > > > > > > What you are going to transform remotely here? > > > > > > > > > > > > > > > > > > I'm not going. I believe :) > > > > > > > > > > > > Just hypothetical use case: You have one SqlFieldsQuery but > > different > > > > > > requirements for returned values. For one case you have to return > > > some > > > > > > string fields as is and for another case you have to trim string > to > > > 32 > > > > > > characters. Of course you still can trim strings locally but > > > > transformers > > > > > > allow you do it remotely. > > > > > > > > > > > > Anyway, this solution can be usefull for rest query types. > > > > > > > > > > > > On Thu, Feb 4, 2016 at 8:54 PM, Sergi Vladykin < > > > > [hidden email] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > The whole point of Transformer is to do remote transform, how > > will > > > > this > > > > > > > work with SqlFieldsQuery? What you are going to transform > > remotely > > > > > here? > > > > > > I > > > > > > > believe all the transformations must happen at SQL level here. > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2016-02-04 20:10 GMT+03:00 Andrey Gura <[hidden email]>: > > > > > > > > > > > > > > > SqlQuery, TextQuery and SpiQuery are similar to ScanQuery > > because > > > > all > > > > > > of > > > > > > > > them also defined as Query<Cache.Entry<>>. > > > > > > > > > > > > > > > > It can be usefull to have one query SqlQuery that can return > > > > > different > > > > > > > > result that will be produced from cache entry by transformer. > > > > > > > > > > > > > > > > Actually only SqlFieldsQuery has different definition. So > > > > > transformers > > > > > > > can > > > > > > > > be applied to any type of query (including SqlFieldsQuery, I > > > > > believe). > > > > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 7:42 PM, Sergi Vladykin < > > > > > > [hidden email] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > I don't like the idea of having additional method > > > *query(Query<E> > > > > > > qry, > > > > > > > > > Transformer<E, R> transfomer); *because I don't see how > these > > > > > > > > transformers > > > > > > > > > will work for example with SQL, but this API makes you > think > > > that > > > > > > > > > transformers are supported for all the query types. > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > 2016-02-04 16:46 GMT+03:00 Andrey Gura <[hidden email] > >: > > > > > > > > > > > > > > > > > > > Val, > > > > > > > > > > > > > > > > > > > > can we introduce new method into IgnteCache API? > > > > > > > > > > > > > > > > > > > > Now we have method: public <R> QueryCursor<R> > > query(Query<R> > > > > > qry); > > > > > > > > > > > > > > > > > > > > New method will be something like this: <R> > QueryCursor<R> > > > > > > > > query(Query<E> > > > > > > > > > > qry, Transformer<E, R> transfomer); > > > > > > > > > > > > > > > > > > > > It allows provide transformers for all query types and > > > chnages > > > > > will > > > > > > > be > > > > > > > > > > related only with query cursor functionality. > > > > > > > > > > > > > > > > > > > > Will it work? > > > > > > > > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 11:13 AM, Andrey Kornev < > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Another perhaps bigger problem with running queries > > > > (including > > > > > > scan > > > > > > > > > > > queries) using closures was discussed at length on the > > @dev > > > > not > > > > > > so > > > > > > > > long > > > > > > > > > > > ago. It has to do with partitions migration due to > > cluster > > > > > > topology > > > > > > > > > > changes > > > > > > > > > > > which may result in the query returning incomplete > > result. > > > > And > > > > > > > while > > > > > > > > it > > > > > > > > > > is > > > > > > > > > > > possible to solve this problem for the scan queries by > > > using > > > > > some > > > > > > > > > clever > > > > > > > > > > > tricks, all bets are off with the SQL queries.Andrey > > > > > > > > > > > _____________________________ > > > > > > > > > > > From: Valentin Kulichenko < > [hidden email] > > > > > > > > > > > > > > Sent: Thursday, February 4, 2016 6:29 AM > > > > > > > > > > > Subject: Re: Transformers in SCAN queries > > > > > > > > > > > To: <[hidden email]> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Dmitry, > > > > > > > > > > > > > > > > > > > > > > The main difference in my view is that you lose > > pagination > > > > > when > > > > > > > > > sending > > > > > > > > > > > results from servers to client. What if one wants to > > > iterate > > > > > > > through > > > > > > > > > all > > > > > > > > > > > entries in cache? > > > > > > > > > > > > > > > > > > > > > > On Wed, Feb 3, 2016 at 9:47 PM, Dmitriy Setrakyan < > > > > > > > > > > [hidden email]> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Valentin, > > > > > > > > > > > > > > > > > > > > > > > > Wouldn’t the same effect be achieved by > broadcasting a > > > > > closure > > > > > > > to > > > > > > > > > the > > > > > > > > > > > > cluster and executing scan-query on every node > > locally? > > > > > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Feb 3, 2016 at 9:17 PM, Valentin Kulichenko > < > > > > > > > > > > > > [hidden email]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > > > > > > > > > > > > > I keep getting requests from our users to add > > optional > > > > > > > > > transformers > > > > > > > > > > to > > > > > > > > > > > > SCAN > > > > > > > > > > > > > queries. This will allow to iterate through cache, > > but > > > > do > > > > > > not > > > > > > > > > > transfer > > > > > > > > > > > > > whole key-value pairs across networks (e.g., get > > only > > > > > keys). > > > > > > > The > > > > > > > > > > > feature > > > > > > > > > > > > > looks useful and I created a ticket [1]. > > > > > > > > > > > > > > > > > > > > > > > > > > I am struggling with the design now. The problem > is > > > > that I > > > > > > > > wanted > > > > > > > > > to > > > > > > > > > > > > extend > > > > > > > > > > > > > existing ScanQuery object for this, but this seems > > to > > > be > > > > > > > > > impossible > > > > > > > > > > > > because > > > > > > > > > > > > > it already extends Query<Cache.Entry<K, V>> and > thus > > > can > > > > > > > iterate > > > > > > > > > > only > > > > > > > > > > > > > through entries. > > > > > > > > > > > > > > > > > > > > > > > > > > The only option I see now is to create a separate > > > query > > > > > > type, > > > > > > > > > > > copy-paste > > > > > > > > > > > > > everything from ScanQuery and add *mandatory* > > > > transformer. > > > > > > > > > Something > > > > > > > > > > > like > > > > > > > > > > > > > this: > > > > > > > > > > > > > > > > > > > > > > > > > > ScanTransformQuery<K, V, R> extends Query<R> { > > > > > > > > > > > > > IgniteBiPredicate<K, V> filter; > > > > > > > > > > > > > IgniteClosure<Cache.Entry<K, V>, R> > transformer; > > > > > > > > > > > > > int part; > > > > > > > > > > > > > ... > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > > > Thoughts? Does anyone has other ideas? > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > https://issues.apache.org/jira/browse/IGNITE-2546 > > > > > > > > > > > > > > > > > > > > > > > > > > -Val > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Andrey Gura > > > > > > > > > > GridGain Systems, Inc. > > > > > > > > > > www.gridgain.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Andrey Gura > > > > > > > > GridGain Systems, Inc. > > > > > > > > www.gridgain.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Andrey Gura > > > > > > GridGain Systems, Inc. > > > > > > www.gridgain.com > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Andrey Gura > > > > GridGain Systems, Inc. > > > > www.gridgain.com > > > > > > > > > > > > > -- > Andrey Gura > GridGain Systems, Inc. > www.gridgain.com > |
«
Return to Apache Ignite Developers - Legacy Mail Archive
|
1 view|%1 views
| Free forum by Nabble | Edit this page |