Ability to disable WAL in runtime to speed up load

[Vladimir Ozerov]

Igniters,

We have a ticket to implement runtime WAL state management [1].It will be
possible to disable it temporarily. This is very useful for data loading
case. Our experiments show that we can improve data loading time by a
factor of 2x-10x depending on configuration, indexes and cluster topology.

We had a private chat with Anton Vinogradov and Alexey Goncharuk, and came
to certain design which you can find inside the ticket. Comments are
welcomed.

In this email I would like to focus on API of this feature. Considerations:
1) It should be possible to disable WAL globally or on per-cache basis.
2) It should be possible to get current state of WAL for the cache.

Quick draft from my side:
void disableWal();
void disableWal(String... cacheNames);
void enableWal();
void enableWal(String... cacheNames);
boolean isWalEnabled(String... cacheNames);

Please help improving it.
- Do we really need methods to disable WAL for all caches? This is not very
intuitive feature.
- Where to place these operations? Ignite interface? Ignite + IgniteCache?
Separate facade?
- Do we need async counterparts?
- Do we need a feature which completes when WAL is enabled back?

Vladimir.

[1] https://issues.apache.org/jira/browse/IGNITE-6411

[Vladimir Ozerov]

One more idea - ability to create a cache with initially disabled WAL.
Might be useful.

On Wed, Oct 25, 2017 at 2:35 PM, Vladimir Ozerov <[hidden email]>
wrote:

> Igniters,
>
> We have a ticket to implement runtime WAL state management [1].It will be
> possible to disable it temporarily. This is very useful for data loading
> case. Our experiments show that we can improve data loading time by a
> factor of 2x-10x depending on configuration, indexes and cluster topology.
>
> We had a private chat with Anton Vinogradov and Alexey Goncharuk, and came
> to certain design which you can find inside the ticket. Comments are
> welcomed.
>
> In this email I would like to focus on API of this feature. Considerations:
> 1) It should be possible to disable WAL globally or on per-cache basis.
> 2) It should be possible to get current state of WAL for the cache.
>
> Quick draft from my side:
> void disableWal();
> void disableWal(String... cacheNames);
> void enableWal();
> void enableWal(String... cacheNames);
> boolean isWalEnabled(String... cacheNames);
>
> Please help improving it.
> - Do we really need methods to disable WAL for all caches? This is not
> very intuitive feature.
> - Where to place these operations? Ignite interface? Ignite + IgniteCache?
> Separate facade?
> - Do we need async counterparts?
> - Do we need a feature which completes when WAL is enabled back?
>
> Vladimir.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-6411
>

[Pavel Tupitsyn]

IMO IgniteCache.disableWal() should be enough.

Also what about an option to disable WAL when IgniteDataStreamer is active?

On Wed, Oct 25, 2017 at 2:38 PM, Vladimir Ozerov <[hidden email]>
wrote:

> One more idea - ability to create a cache with initially disabled WAL.
> Might be useful.
>
> On Wed, Oct 25, 2017 at 2:35 PM, Vladimir Ozerov <[hidden email]>
> wrote:
>
> > Igniters,
> >
> > We have a ticket to implement runtime WAL state management [1].It will be
> > possible to disable it temporarily. This is very useful for data loading
> > case. Our experiments show that we can improve data loading time by a
> > factor of 2x-10x depending on configuration, indexes and cluster
> topology.
> >
> > We had a private chat with Anton Vinogradov and Alexey Goncharuk, and
> came
> > to certain design which you can find inside the ticket. Comments are
> > welcomed.
> >
> > In this email I would like to focus on API of this feature.
> Considerations:
> > 1) It should be possible to disable WAL globally or on per-cache basis.
> > 2) It should be possible to get current state of WAL for the cache.
> >
> > Quick draft from my side:
> > void disableWal();
> > void disableWal(String... cacheNames);
> > void enableWal();
> > void enableWal(String... cacheNames);
> > boolean isWalEnabled(String... cacheNames);
> >
> > Please help improving it.
> > - Do we really need methods to disable WAL for all caches? This is not
> > very intuitive feature.
> > - Where to place these operations? Ignite interface? Ignite +
> IgniteCache?
> > Separate facade?
> > - Do we need async counterparts?
> > - Do we need a feature which completes when WAL is enabled back?
> >
> > Vladimir.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-6411
> >
>

[Vladimir Ozerov]

Pavel,

This feature will work independently of streamer. If you want to load data
with streamer, then you disable WAL first through some API call or SQL
command, and then start loading.

On Wed, Oct 25, 2017 at 2:41 PM, Pavel Tupitsyn <[hidden email]>
wrote:

> IMO IgniteCache.disableWal() should be enough.
>
> Also what about an option to disable WAL when IgniteDataStreamer is active?
>
> On Wed, Oct 25, 2017 at 2:38 PM, Vladimir Ozerov <[hidden email]>
> wrote:
>
> > One more idea - ability to create a cache with initially disabled WAL.
> > Might be useful.
> >
> > On Wed, Oct 25, 2017 at 2:35 PM, Vladimir Ozerov <[hidden email]>
> > wrote:
> >
> > > Igniters,
> > >
> > > We have a ticket to implement runtime WAL state management [1].It will
> be
> > > possible to disable it temporarily. This is very useful for data
> loading
> > > case. Our experiments show that we can improve data loading time by a
> > > factor of 2x-10x depending on configuration, indexes and cluster
> > topology.
> > >
> > > We had a private chat with Anton Vinogradov and Alexey Goncharuk, and
> > came
> > > to certain design which you can find inside the ticket. Comments are
> > > welcomed.
> > >
> > > In this email I would like to focus on API of this feature.
> > Considerations:
> > > 1) It should be possible to disable WAL globally or on per-cache basis.
> > > 2) It should be possible to get current state of WAL for the cache.
> > >
> > > Quick draft from my side:
> > > void disableWal();
> > > void disableWal(String... cacheNames);
> > > void enableWal();
> > > void enableWal(String... cacheNames);
> > > boolean isWalEnabled(String... cacheNames);
> > >
> > > Please help improving it.
> > > - Do we really need methods to disable WAL for all caches? This is not
> > > very intuitive feature.
> > > - Where to place these operations? Ignite interface? Ignite +
> > IgniteCache?
> > > Separate facade?
> > > - Do we need async counterparts?
> > > - Do we need a feature which completes when WAL is enabled back?
> > >
> > > Vladimir.
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-6411
> > >
> >
>

[Alexey Goncharuk]

I do not like boolean isWalEnabled(String... cacheNames) - it's semantics
is confusing. Should it return true if WAL is enabled for all caches or if
WAL is enabled for at least one cache? IMO, since this is a local-read
operation, single cache per call is enough.

As for the API placement, it looks like we need another facade
(IgniteControl ?) where active(bool) should also be moved. I do not feel
it's current placement is right, but creating a new facade for a single
property looked like an overkill, but now this facade becomes more tangible.

2017-10-25 14:55 GMT+03:00 Vladimir Ozerov <[hidden email]>:

> Pavel,
>
> This feature will work independently of streamer. If you want to load data
> with streamer, then you disable WAL first through some API call or SQL
> command, and then start loading.
>
> On Wed, Oct 25, 2017 at 2:41 PM, Pavel Tupitsyn <[hidden email]>
> wrote:
>
> > IMO IgniteCache.disableWal() should be enough.
> >
> > Also what about an option to disable WAL when IgniteDataStreamer is
> active?
> >
> > On Wed, Oct 25, 2017 at 2:38 PM, Vladimir Ozerov <[hidden email]>
> > wrote:
> >
> > > One more idea - ability to create a cache with initially disabled WAL.
> > > Might be useful.
> > >
> > > On Wed, Oct 25, 2017 at 2:35 PM, Vladimir Ozerov <[hidden email]
> >
> > > wrote:
> > >
> > > > Igniters,
> > > >
> > > > We have a ticket to implement runtime WAL state management [1].It
> will
> > be
> > > > possible to disable it temporarily. This is very useful for data
> > loading
> > > > case. Our experiments show that we can improve data loading time by a
> > > > factor of 2x-10x depending on configuration, indexes and cluster
> > > topology.
> > > >
> > > > We had a private chat with Anton Vinogradov and Alexey Goncharuk, and
> > > came
> > > > to certain design which you can find inside the ticket. Comments are
> > > > welcomed.
> > > >
> > > > In this email I would like to focus on API of this feature.
> > > Considerations:
> > > > 1) It should be possible to disable WAL globally or on per-cache
> basis.
> > > > 2) It should be possible to get current state of WAL for the cache.
> > > >
> > > > Quick draft from my side:
> > > > void disableWal();
> > > > void disableWal(String... cacheNames);
> > > > void enableWal();
> > > > void enableWal(String... cacheNames);
> > > > boolean isWalEnabled(String... cacheNames);
> > > >
> > > > Please help improving it.
> > > > - Do we really need methods to disable WAL for all caches? This is
> not
> > > > very intuitive feature.
> > > > - Where to place these operations? Ignite interface? Ignite +
> > > IgniteCache?
> > > > Separate facade?
> > > > - Do we need async counterparts?
> > > > - Do we need a feature which completes when WAL is enabled back?
> > > >
> > > > Vladimir.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-6411
> > > >
> > >
> >
>

[Pavel Tupitsyn]

Vladimir,

It would be useful to be able to automatically disable WAL when streaming
starts
and re-enable after it ends, don't you think so?

Something like IgniteDataStreamer.disableWal property.

This is in addition to other API calls that you suggested.

On Wed, Oct 25, 2017 at 3:25 PM, Alexey Goncharuk <
[hidden email]> wrote:

> I do not like boolean isWalEnabled(String... cacheNames) - it's semantics
> is confusing. Should it return true if WAL is enabled for all caches or if
> WAL is enabled for at least one cache? IMO, since this is a local-read
> operation, single cache per call is enough.
>
> As for the API placement, it looks like we need another facade
> (IgniteControl ?) where active(bool) should also be moved. I do not feel
> it's current placement is right, but creating a new facade for a single
> property looked like an overkill, but now this facade becomes more
> tangible.
>
> 2017-10-25 14:55 GMT+03:00 Vladimir Ozerov <[hidden email]>:
>
> > Pavel,
> >
> > This feature will work independently of streamer. If you want to load
> data
> > with streamer, then you disable WAL first through some API call or SQL
> > command, and then start loading.
> >
> > On Wed, Oct 25, 2017 at 2:41 PM, Pavel Tupitsyn <[hidden email]>
> > wrote:
> >
> > > IMO IgniteCache.disableWal() should be enough.
> > >
> > > Also what about an option to disable WAL when IgniteDataStreamer is
> > active?
> > >
> > > On Wed, Oct 25, 2017 at 2:38 PM, Vladimir Ozerov <[hidden email]
> >
> > > wrote:
> > >
> > > > One more idea - ability to create a cache with initially disabled
> WAL.
> > > > Might be useful.
> > > >
> > > > On Wed, Oct 25, 2017 at 2:35 PM, Vladimir Ozerov <
> [hidden email]
> > >
> > > > wrote:
> > > >
> > > > > Igniters,
> > > > >
> > > > > We have a ticket to implement runtime WAL state management [1].It
> > will
> > > be
> > > > > possible to disable it temporarily. This is very useful for data
> > > loading
> > > > > case. Our experiments show that we can improve data loading time
> by a
> > > > > factor of 2x-10x depending on configuration, indexes and cluster
> > > > topology.
> > > > >
> > > > > We had a private chat with Anton Vinogradov and Alexey Goncharuk,
> and
> > > > came
> > > > > to certain design which you can find inside the ticket. Comments
> are
> > > > > welcomed.
> > > > >
> > > > > In this email I would like to focus on API of this feature.
> > > > Considerations:
> > > > > 1) It should be possible to disable WAL globally or on per-cache
> > basis.
> > > > > 2) It should be possible to get current state of WAL for the cache.
> > > > >
> > > > > Quick draft from my side:
> > > > > void disableWal();
> > > > > void disableWal(String... cacheNames);
> > > > > void enableWal();
> > > > > void enableWal(String... cacheNames);
> > > > > boolean isWalEnabled(String... cacheNames);
> > > > >
> > > > > Please help improving it.
> > > > > - Do we really need methods to disable WAL for all caches? This is
> > not
> > > > > very intuitive feature.
> > > > > - Where to place these operations? Ignite interface? Ignite +
> > > > IgniteCache?
> > > > > Separate facade?
> > > > > - Do we need async counterparts?
> > > > > - Do we need a feature which completes when WAL is enabled back?
> > > > >
> > > > > Vladimir.
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-6411
> > > > >
> > > >
> > >
> >
>

[Anton Vinogradov]

Pavel,

WAL disabling is a very dangerous operation and it seems to be not a good
idea to allow run regular operation with .disabledWal().
Let's think twice how to make new API safe.

On Wed, Oct 25, 2017 at 3:25 PM, Pavel Tupitsyn <[hidden email]>
wrote:

> Vladimir,
>
> It would be useful to be able to automatically disable WAL when streaming
> starts
> and re-enable after it ends, don't you think so?
>
> Something like IgniteDataStreamer.disableWal property.
>
> This is in addition to other API calls that you suggested.
>
> On Wed, Oct 25, 2017 at 3:25 PM, Alexey Goncharuk <
> [hidden email]> wrote:
>
> > I do not like boolean isWalEnabled(String... cacheNames) - it's semantics
> > is confusing. Should it return true if WAL is enabled for all caches or
> if
> > WAL is enabled for at least one cache? IMO, since this is a local-read
> > operation, single cache per call is enough.
> >
> > As for the API placement, it looks like we need another facade
> > (IgniteControl ?) where active(bool) should also be moved. I do not feel
> > it's current placement is right, but creating a new facade for a single
> > property looked like an overkill, but now this facade becomes more
> > tangible.
> >
> > 2017-10-25 14:55 GMT+03:00 Vladimir Ozerov <[hidden email]>:
> >
> > > Pavel,
> > >
> > > This feature will work independently of streamer. If you want to load
> > data
> > > with streamer, then you disable WAL first through some API call or SQL
> > > command, and then start loading.
> > >
> > > On Wed, Oct 25, 2017 at 2:41 PM, Pavel Tupitsyn <[hidden email]>
> > > wrote:
> > >
> > > > IMO IgniteCache.disableWal() should be enough.
> > > >
> > > > Also what about an option to disable WAL when IgniteDataStreamer is
> > > active?
> > > >
> > > > On Wed, Oct 25, 2017 at 2:38 PM, Vladimir Ozerov <
> [hidden email]
> > >
> > > > wrote:
> > > >
> > > > > One more idea - ability to create a cache with initially disabled
> > WAL.
> > > > > Might be useful.
> > > > >
> > > > > On Wed, Oct 25, 2017 at 2:35 PM, Vladimir Ozerov <
> > [hidden email]
> > > >
> > > > > wrote:
> > > > >
> > > > > > Igniters,
> > > > > >
> > > > > > We have a ticket to implement runtime WAL state management [1].It
> > > will
> > > > be
> > > > > > possible to disable it temporarily. This is very useful for data
> > > > loading
> > > > > > case. Our experiments show that we can improve data loading time
> > by a
> > > > > > factor of 2x-10x depending on configuration, indexes and cluster
> > > > > topology.
> > > > > >
> > > > > > We had a private chat with Anton Vinogradov and Alexey Goncharuk,
> > and
> > > > > came
> > > > > > to certain design which you can find inside the ticket. Comments
> > are
> > > > > > welcomed.
> > > > > >
> > > > > > In this email I would like to focus on API of this feature.
> > > > > Considerations:
> > > > > > 1) It should be possible to disable WAL globally or on per-cache
> > > basis.
> > > > > > 2) It should be possible to get current state of WAL for the
> cache.
> > > > > >
> > > > > > Quick draft from my side:
> > > > > > void disableWal();
> > > > > > void disableWal(String... cacheNames);
> > > > > > void enableWal();
> > > > > > void enableWal(String... cacheNames);
> > > > > > boolean isWalEnabled(String... cacheNames);
> > > > > >
> > > > > > Please help improving it.
> > > > > > - Do we really need methods to disable WAL for all caches? This
> is
> > > not
> > > > > > very intuitive feature.
> > > > > > - Where to place these operations? Ignite interface? Ignite +
> > > > > IgniteCache?
> > > > > > Separate facade?
> > > > > > - Do we need async counterparts?
> > > > > > - Do we need a feature which completes when WAL is enabled back?
> > > > > >
> > > > > > Vladimir.
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-6411
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Vladimir Ozerov]

Alex,

My bad, I meant "isWalEnabled(String cacheName)" of course.

On Wed, Oct 25, 2017 at 3:34 PM, Anton Vinogradov <[hidden email]>
wrote:

> Pavel,
>
> WAL disabling is a very dangerous operation and it seems to be not a good
> idea to allow run regular operation with .disabledWal().
> Let's think twice how to make new API safe.
>
> On Wed, Oct 25, 2017 at 3:25 PM, Pavel Tupitsyn <[hidden email]>
> wrote:
>
> > Vladimir,
> >
> > It would be useful to be able to automatically disable WAL when streaming
> > starts
> > and re-enable after it ends, don't you think so?
> >
> > Something like IgniteDataStreamer.disableWal property.
> >
> > This is in addition to other API calls that you suggested.
> >
> > On Wed, Oct 25, 2017 at 3:25 PM, Alexey Goncharuk <
> > [hidden email]> wrote:
> >
> > > I do not like boolean isWalEnabled(String... cacheNames) - it's
> semantics
> > > is confusing. Should it return true if WAL is enabled for all caches or
> > if
> > > WAL is enabled for at least one cache? IMO, since this is a local-read
> > > operation, single cache per call is enough.
> > >
> > > As for the API placement, it looks like we need another facade
> > > (IgniteControl ?) where active(bool) should also be moved. I do not
> feel
> > > it's current placement is right, but creating a new facade for a single
> > > property looked like an overkill, but now this facade becomes more
> > > tangible.
> > >
> > > 2017-10-25 14:55 GMT+03:00 Vladimir Ozerov <[hidden email]>:
> > >
> > > > Pavel,
> > > >
> > > > This feature will work independently of streamer. If you want to load
> > > data
> > > > with streamer, then you disable WAL first through some API call or
> SQL
> > > > command, and then start loading.
> > > >
> > > > On Wed, Oct 25, 2017 at 2:41 PM, Pavel Tupitsyn <
> [hidden email]>
> > > > wrote:
> > > >
> > > > > IMO IgniteCache.disableWal() should be enough.
> > > > >
> > > > > Also what about an option to disable WAL when IgniteDataStreamer is
> > > > active?
> > > > >
> > > > > On Wed, Oct 25, 2017 at 2:38 PM, Vladimir Ozerov <
> > [hidden email]
> > > >
> > > > > wrote:
> > > > >
> > > > > > One more idea - ability to create a cache with initially disabled
> > > WAL.
> > > > > > Might be useful.
> > > > > >
> > > > > > On Wed, Oct 25, 2017 at 2:35 PM, Vladimir Ozerov <
> > > [hidden email]
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Igniters,
> > > > > > >
> > > > > > > We have a ticket to implement runtime WAL state management
> [1].It
> > > > will
> > > > > be
> > > > > > > possible to disable it temporarily. This is very useful for
> data
> > > > > loading
> > > > > > > case. Our experiments show that we can improve data loading
> time
> > > by a
> > > > > > > factor of 2x-10x depending on configuration, indexes and
> cluster
> > > > > > topology.
> > > > > > >
> > > > > > > We had a private chat with Anton Vinogradov and Alexey
> Goncharuk,
> > > and
> > > > > > came
> > > > > > > to certain design which you can find inside the ticket.
> Comments
> > > are
> > > > > > > welcomed.
> > > > > > >
> > > > > > > In this email I would like to focus on API of this feature.
> > > > > > Considerations:
> > > > > > > 1) It should be possible to disable WAL globally or on
> per-cache
> > > > basis.
> > > > > > > 2) It should be possible to get current state of WAL for the
> > cache.
> > > > > > >
> > > > > > > Quick draft from my side:
> > > > > > > void disableWal();
> > > > > > > void disableWal(String... cacheNames);
> > > > > > > void enableWal();
> > > > > > > void enableWal(String... cacheNames);
> > > > > > > boolean isWalEnabled(String... cacheNames);
> > > > > > >
> > > > > > > Please help improving it.
> > > > > > > - Do we really need methods to disable WAL for all caches? This
> > is
> > > > not
> > > > > > > very intuitive feature.
> > > > > > > - Where to place these operations? Ignite interface? Ignite +
> > > > > > IgniteCache?
> > > > > > > Separate facade?
> > > > > > > - Do we need async counterparts?
> > > > > > > - Do we need a feature which completes when WAL is enabled
> back?
> > > > > > >
> > > > > > > Vladimir.
> > > > > > >
> > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-6411
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Vladimir Ozerov]

Pavel,

No, this is not the case we are trying to cover. WAL disabling should be
separate operation, which is not tied to any API, whether this is streamer,
cache put, or DML.

On Wed, Oct 25, 2017 at 5:20 PM, Vladimir Ozerov <[hidden email]>
wrote:

> Alex,
>
> My bad, I meant "isWalEnabled(String cacheName)" of course.
>
> On Wed, Oct 25, 2017 at 3:34 PM, Anton Vinogradov <
> [hidden email]> wrote:
>
>> Pavel,
>>
>> WAL disabling is a very dangerous operation and it seems to be not a good
>> idea to allow run regular operation with .disabledWal().
>> Let's think twice how to make new API safe.
>>
>> On Wed, Oct 25, 2017 at 3:25 PM, Pavel Tupitsyn <[hidden email]>
>> wrote:
>>
>> > Vladimir,
>> >
>> > It would be useful to be able to automatically disable WAL when
>> streaming
>> > starts
>> > and re-enable after it ends, don't you think so?
>> >
>> > Something like IgniteDataStreamer.disableWal property.
>> >
>> > This is in addition to other API calls that you suggested.
>> >
>> > On Wed, Oct 25, 2017 at 3:25 PM, Alexey Goncharuk <
>> > [hidden email]> wrote:
>> >
>> > > I do not like boolean isWalEnabled(String... cacheNames) - it's
>> semantics
>> > > is confusing. Should it return true if WAL is enabled for all caches
>> or
>> > if
>> > > WAL is enabled for at least one cache? IMO, since this is a local-read
>> > > operation, single cache per call is enough.
>> > >
>> > > As for the API placement, it looks like we need another facade
>> > > (IgniteControl ?) where active(bool) should also be moved. I do not
>> feel
>> > > it's current placement is right, but creating a new facade for a
>> single
>> > > property looked like an overkill, but now this facade becomes more
>> > > tangible.
>> > >
>> > > 2017-10-25 14:55 GMT+03:00 Vladimir Ozerov <[hidden email]>:
>> > >
>> > > > Pavel,
>> > > >
>> > > > This feature will work independently of streamer. If you want to
>> load
>> > > data
>> > > > with streamer, then you disable WAL first through some API call or
>> SQL
>> > > > command, and then start loading.
>> > > >
>> > > > On Wed, Oct 25, 2017 at 2:41 PM, Pavel Tupitsyn <
>> [hidden email]>
>> > > > wrote:
>> > > >
>> > > > > IMO IgniteCache.disableWal() should be enough.
>> > > > >
>> > > > > Also what about an option to disable WAL when IgniteDataStreamer
>> is
>> > > > active?
>> > > > >
>> > > > > On Wed, Oct 25, 2017 at 2:38 PM, Vladimir Ozerov <
>> > [hidden email]
>> > > >
>> > > > > wrote:
>> > > > >
>> > > > > > One more idea - ability to create a cache with initially
>> disabled
>> > > WAL.
>> > > > > > Might be useful.
>> > > > > >
>> > > > > > On Wed, Oct 25, 2017 at 2:35 PM, Vladimir Ozerov <
>> > > [hidden email]
>> > > > >
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Igniters,
>> > > > > > >
>> > > > > > > We have a ticket to implement runtime WAL state management
>> [1].It
>> > > > will
>> > > > > be
>> > > > > > > possible to disable it temporarily. This is very useful for
>> data
>> > > > > loading
>> > > > > > > case. Our experiments show that we can improve data loading
>> time
>> > > by a
>> > > > > > > factor of 2x-10x depending on configuration, indexes and
>> cluster
>> > > > > > topology.
>> > > > > > >
>> > > > > > > We had a private chat with Anton Vinogradov and Alexey
>> Goncharuk,
>> > > and
>> > > > > > came
>> > > > > > > to certain design which you can find inside the ticket.
>> Comments
>> > > are
>> > > > > > > welcomed.
>> > > > > > >
>> > > > > > > In this email I would like to focus on API of this feature.
>> > > > > > Considerations:
>> > > > > > > 1) It should be possible to disable WAL globally or on
>> per-cache
>> > > > basis.
>> > > > > > > 2) It should be possible to get current state of WAL for the
>> > cache.
>> > > > > > >
>> > > > > > > Quick draft from my side:
>> > > > > > > void disableWal();
>> > > > > > > void disableWal(String... cacheNames);
>> > > > > > > void enableWal();
>> > > > > > > void enableWal(String... cacheNames);
>> > > > > > > boolean isWalEnabled(String... cacheNames);
>> > > > > > >
>> > > > > > > Please help improving it.
>> > > > > > > - Do we really need methods to disable WAL for all caches?
>> This
>> > is
>> > > > not
>> > > > > > > very intuitive feature.
>> > > > > > > - Where to place these operations? Ignite interface? Ignite +
>> > > > > > IgniteCache?
>> > > > > > > Separate facade?
>> > > > > > > - Do we need async counterparts?
>> > > > > > > - Do we need a feature which completes when WAL is enabled
>> back?
>> > > > > > >
>> > > > > > > Vladimir.
>> > > > > > >
>> > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-6411
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

[dsetrakyan]

Vladimir,

Several questions:

   1. On which interface do you plan to add enable/disableWal()?
   2. Is it possible that when enabling/disabling WAL for multiple caches,
   some fail and some succeed?
   3. is enable/disableWal a synchronous operation?
   4. What happens if a user starts streaming and forgets to disable the
   WAL?

D.

On Wed, Oct 25, 2017 at 7:21 AM, Vladimir Ozerov <[hidden email]>
wrote:

> Pavel,
>
> No, this is not the case we are trying to cover. WAL disabling should be
> separate operation, which is not tied to any API, whether this is streamer,
> cache put, or DML.
>
> On Wed, Oct 25, 2017 at 5:20 PM, Vladimir Ozerov <[hidden email]>
> wrote:
>
> > Alex,
> >
> > My bad, I meant "isWalEnabled(String cacheName)" of course.
> >
> > On Wed, Oct 25, 2017 at 3:34 PM, Anton Vinogradov <
> > [hidden email]> wrote:
> >
> >> Pavel,
> >>
> >> WAL disabling is a very dangerous operation and it seems to be not a
> good
> >> idea to allow run regular operation with .disabledWal().
> >> Let's think twice how to make new API safe.
> >>
> >> On Wed, Oct 25, 2017 at 3:25 PM, Pavel Tupitsyn <[hidden email]>
> >> wrote:
> >>
> >> > Vladimir,
> >> >
> >> > It would be useful to be able to automatically disable WAL when
> >> streaming
> >> > starts
> >> > and re-enable after it ends, don't you think so?
> >> >
> >> > Something like IgniteDataStreamer.disableWal property.
> >> >
> >> > This is in addition to other API calls that you suggested.
> >> >
> >> > On Wed, Oct 25, 2017 at 3:25 PM, Alexey Goncharuk <
> >> > [hidden email]> wrote:
> >> >
> >> > > I do not like boolean isWalEnabled(String... cacheNames) - it's
> >> semantics
> >> > > is confusing. Should it return true if WAL is enabled for all caches
> >> or
> >> > if
> >> > > WAL is enabled for at least one cache? IMO, since this is a
> local-read
> >> > > operation, single cache per call is enough.
> >> > >
> >> > > As for the API placement, it looks like we need another facade
> >> > > (IgniteControl ?) where active(bool) should also be moved. I do not
> >> feel
> >> > > it's current placement is right, but creating a new facade for a
> >> single
> >> > > property looked like an overkill, but now this facade becomes more
> >> > > tangible.
> >> > >
> >> > > 2017-10-25 14:55 GMT+03:00 Vladimir Ozerov <[hidden email]>:
> >> > >
> >> > > > Pavel,
> >> > > >
> >> > > > This feature will work independently of streamer. If you want to
> >> load
> >> > > data
> >> > > > with streamer, then you disable WAL first through some API call or
> >> SQL
> >> > > > command, and then start loading.
> >> > > >
> >> > > > On Wed, Oct 25, 2017 at 2:41 PM, Pavel Tupitsyn <
> >> [hidden email]>
> >> > > > wrote:
> >> > > >
> >> > > > > IMO IgniteCache.disableWal() should be enough.
> >> > > > >
> >> > > > > Also what about an option to disable WAL when IgniteDataStreamer
> >> is
> >> > > > active?
> >> > > > >
> >> > > > > On Wed, Oct 25, 2017 at 2:38 PM, Vladimir Ozerov <
> >> > [hidden email]
> >> > > >
> >> > > > > wrote:
> >> > > > >
> >> > > > > > One more idea - ability to create a cache with initially
> >> disabled
> >> > > WAL.
> >> > > > > > Might be useful.
> >> > > > > >
> >> > > > > > On Wed, Oct 25, 2017 at 2:35 PM, Vladimir Ozerov <
> >> > > [hidden email]
> >> > > > >
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Igniters,
> >> > > > > > >
> >> > > > > > > We have a ticket to implement runtime WAL state management
> >> [1].It
> >> > > > will
> >> > > > > be
> >> > > > > > > possible to disable it temporarily. This is very useful for
> >> data
> >> > > > > loading
> >> > > > > > > case. Our experiments show that we can improve data loading
> >> time
> >> > > by a
> >> > > > > > > factor of 2x-10x depending on configuration, indexes and
> >> cluster
> >> > > > > > topology.
> >> > > > > > >
> >> > > > > > > We had a private chat with Anton Vinogradov and Alexey
> >> Goncharuk,
> >> > > and
> >> > > > > > came
> >> > > > > > > to certain design which you can find inside the ticket.
> >> Comments
> >> > > are
> >> > > > > > > welcomed.
> >> > > > > > >
> >> > > > > > > In this email I would like to focus on API of this feature.
> >> > > > > > Considerations:
> >> > > > > > > 1) It should be possible to disable WAL globally or on
> >> per-cache
> >> > > > basis.
> >> > > > > > > 2) It should be possible to get current state of WAL for the
> >> > cache.
> >> > > > > > >
> >> > > > > > > Quick draft from my side:
> >> > > > > > > void disableWal();
> >> > > > > > > void disableWal(String... cacheNames);
> >> > > > > > > void enableWal();
> >> > > > > > > void enableWal(String... cacheNames);
> >> > > > > > > boolean isWalEnabled(String... cacheNames);
> >> > > > > > >
> >> > > > > > > Please help improving it.
> >> > > > > > > - Do we really need methods to disable WAL for all caches?
> >> This
> >> > is
> >> > > > not
> >> > > > > > > very intuitive feature.
> >> > > > > > > - Where to place these operations? Ignite interface? Ignite
> +
> >> > > > > > IgniteCache?
> >> > > > > > > Separate facade?
> >> > > > > > > - Do we need async counterparts?
> >> > > > > > > - Do we need a feature which completes when WAL is enabled
> >> back?
> >> > > > > > >
> >> > > > > > > Vladimir.
> >> > > > > > >
> >> > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-6411
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>

[dmagda]

Guys,

Sever things to clarify:
- disabled WAL means disabled checkpointing (no data on disk at all)?
- after turning the WAL back when a preloading is over, how can I know a moment in time when the disk holds the whole copy of data so that it’s safe to restart the whole cluster?

—
Denis

> On Oct 25, 2017, at 4:35 AM, Vladimir Ozerov <[hidden email]> wrote:
>
> Igniters,
>
> We have a ticket to implement runtime WAL state management [1].It will be
> possible to disable it temporarily. This is very useful for data loading
> case. Our experiments show that we can improve data loading time by a
> factor of 2x-10x depending on configuration, indexes and cluster topology.
>
> We had a private chat with Anton Vinogradov and Alexey Goncharuk, and came
> to certain design which you can find inside the ticket. Comments are
> welcomed.
>
> In this email I would like to focus on API of this feature. Considerations:
> 1) It should be possible to disable WAL globally or on per-cache basis.
> 2) It should be possible to get current state of WAL for the cache.
>
> Quick draft from my side:
> void disableWal();
> void disableWal(String... cacheNames);
> void enableWal();
> void enableWal(String... cacheNames);
> boolean isWalEnabled(String... cacheNames);
>
> Please help improving it.
> - Do we really need methods to disable WAL for all caches? This is not very
> intuitive feature.
> - Where to place these operations? Ignite interface? Ignite + IgniteCache?
> Separate facade?
> - Do we need async counterparts?
> - Do we need a feature which completes when WAL is enabled back?
>
> Vladimir.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-6411

[Alexey Goncharuk]

Denis,

1) Not necessarily. You may want to stream more data than you have memory.
In this case, Ignite must spill some data to disk, and in the first phase,
this will be done by the same checkpointing mechanics. In the second phase,
we can implement a smarter and faster (no-COW) data spill, but still, there
might be disk writes.
2) As Vladimir noted, the WAL activation process will internally have a
future that will be completed after all nodes flushed their changes. We can
either pass this future to the user or make the switch process synchronous.

2017-10-25 18:32 GMT+03:00 Denis Magda <[hidden email]>:

> Guys,
>
> Sever things to clarify:
> - disabled WAL means disabled checkpointing (no data on disk at all)?
> - after turning the WAL back when a preloading is over, how can I know a
> moment in time when the disk holds the whole copy of data so that it’s safe
> to restart the whole cluster?
>
> —
> Denis
>
> > On Oct 25, 2017, at 4:35 AM, Vladimir Ozerov <[hidden email]>
> wrote:
> >
> > Igniters,
> >
> > We have a ticket to implement runtime WAL state management [1].It will be
> > possible to disable it temporarily. This is very useful for data loading
> > case. Our experiments show that we can improve data loading time by a
> > factor of 2x-10x depending on configuration, indexes and cluster
> topology.
> >
> > We had a private chat with Anton Vinogradov and Alexey Goncharuk, and
> came
> > to certain design which you can find inside the ticket. Comments are
> > welcomed.
> >
> > In this email I would like to focus on API of this feature.
> Considerations:
> > 1) It should be possible to disable WAL globally or on per-cache basis.
> > 2) It should be possible to get current state of WAL for the cache.
> >
> > Quick draft from my side:
> > void disableWal();
> > void disableWal(String... cacheNames);
> > void enableWal();
> > void enableWal(String... cacheNames);
> > boolean isWalEnabled(String... cacheNames);
> >
> > Please help improving it.
> > - Do we really need methods to disable WAL for all caches? This is not
> very
> > intuitive feature.
> > - Where to place these operations? Ignite interface? Ignite +
> IgniteCache?
> > Separate facade?
> > - Do we need async counterparts?
> > - Do we need a feature which completes when WAL is enabled back?
> >
> > Vladimir.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-6411
>
>