Batch size parameter at DataStreamerCacheUpdaters.batched()

[Roman Guseinov]

Hello Igniters,

In some cases, batched stream receiver can help us to improve performance:

try (IgniteDataStreamer<Integer, String> streamer =
ignite.dataStreamer(cacheName)) {
    streamer.receiver(DataStreamerCacheUpdaters.batched());

    streamer.addData(getData());
}

Unfortunately, the bad thing is that the receiver internally calls "putAll"
for all data. I think it would be useful to have an option to specify a
batch size like:

DataStreamerCacheUpdaters.batched(256)

What do you think about this? Is it make sense to create a ticket?

Thanks.

Best Regards,
Roman



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[Valentin Kulichenko]

Roman,

DataStreamerCacheUpdaters class is actually not a part of public API, so I
don't see a reason to change it unless there is a need for this internally
in Ignite.

-Val

On Thu, Feb 15, 2018 at 5:52 AM, Roman Guseinov <[hidden email]> wrote:

> Hello Igniters,
>
> In some cases, batched stream receiver can help us to improve performance:
>
> try (IgniteDataStreamer&lt;Integer, String> streamer =
> ignite.dataStreamer(cacheName)) {
>     streamer.receiver(DataStreamerCacheUpdaters.batched());
>
>     streamer.addData(getData());
> }
>
> Unfortunately, the bad thing is that the receiver internally calls "putAll"
> for all data. I think it would be useful to have an option to specify a
> batch size like:
>
> DataStreamerCacheUpdaters.batched(256)
>
> What do you think about this? Is it make sense to create a ticket?
>
> Thanks.
>
> Best Regards,
> Roman
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

[Roman Guseinov]

Hi Val,

Thanks for your response. It seems that is public:
https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/datastreamer/DataStreamerCacheUpdaters.java

Best Regards,
Roman



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[Roman Guseinov]

Val,

I got what do you mean. Everything inside the org.apache.ignite.internal is
considered to be not public API.

Best Regards,
Roman



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[Denis Mekhanikov]

Guys,

I think, it makes sense to move this receiver implementation to public API.
It has much better performance, than the default one, that performs single
puts.

I don't see any point in adding a batch size parameter to it though, since
DataStreamer already has such setting itself.

Denis

пт, 16 февр. 2018 г. в 13:29, Roman Guseinov <[hidden email]>:

> Val,
>
> I got what do you mean. Everything inside the org.apache.ignite.internal is
> considered to be not public API.
>
> Best Regards,
> Roman
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

[Valentin Kulichenko]

As far as I remember, it used to be public and then was moved to internal.
The main issue with these updaters was that batching is dangerous because
you can get deadlocks if keys are not sorted (which is the case for
BATCHED). There is also BATCHED_SORTED, but it requires keys to be
Comparable and I doubt we do any validation, so usability is questionable.

We can think about how to improve this functionality, but in current state
I would definitely not include it on public API. User can always implement
custom receiver if needed.

-Val

On Fri, Feb 16, 2018 at 5:04 AM, Denis Mekhanikov <[hidden email]>
wrote:

> Guys,
>
> I think, it makes sense to move this receiver implementation to public API.
> It has much better performance, than the default one, that performs single
> puts.
>
> I don't see any point in adding a batch size parameter to it though, since
> DataStreamer already has such setting itself.
>
> Denis
>
> пт, 16 февр. 2018 г. в 13:29, Roman Guseinov <[hidden email]>:
>
> > Val,
> >
> > I got what do you mean. Everything inside the org.apache.ignite.internal
> is
> > considered to be not public API.
> >
> > Best Regards,
> > Roman
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> >
>