Persistent Store Stabilization for release

[dmagda]

Igniters,

The Persistent Store donation was accepted by the community and now we should think of its readiness for a public release.

I know that many of you are involved it the donation stabilization. Could you share the current status of the feature? How far is it from the releasable state?

—
Denis

[dsetrakyan]

Thanks Denis. I also would like to encourage to post all the stabilization
updates for the donated branch here. This will help the community to get
familiar with the new code and functionality.

In addition, everyone in the community should feel free to peruse through
the code in this branch and ask any questions that may come up:

https://github.com/apache/ignite/tree/ignite-5267/modules/pds

D.


On Wed, May 31, 2017 at 2:37 PM, Denis Magda <[hidden email]> wrote:

> Igniters,
>
> The Persistent Store donation was accepted by the community and now we
> should think of its readiness for a public release.
>
> I know that many of you are involved it the donation stabilization. Could
> you share the current status of the feature? How far is it from the
> releasable state?
>
> —
> Denis

[Sergey Chugunov]

Hello Denis,

There are three suites [1] [2] [3] configured on TeamCity with bunch of
tests for PDS functionality.

As you can see most of tests are passing now (only 6 tests are
failing, for *partitionLoss
*tests problem was almost sorted out), it looks like that this
functionality is pretty stable.


[1] Ignite PDS 1
<http://ci.ignite.apache.org/viewType.html?buildTypeId=Ignite20Tests_IgnitePds1&branch_Ignite20Tests=ignite-5267&tab=buildTypeStatusDiv>
[2] Ignite PDS 2
<http://ci.ignite.apache.org/viewType.html?buildTypeId=Ignite20Tests_IgnitePds2&branch_Ignite20Tests=ignite-5267&tab=buildTypeStatusDiv>
[3] Ignite OutOfMemory
<http://ci.ignite.apache.org/viewType.html?buildTypeId=Ignite20Tests_IgnitePdsOutOfMemory&branch_Ignite20Tests=ignite-5267&tab=buildTypeStatusDiv>

On Thu, Jun 1, 2017 at 1:24 AM, Dmitriy Setrakyan <[hidden email]>
wrote:

> Thanks Denis. I also would like to encourage to post all the stabilization
> updates for the donated branch here. This will help the community to get
> familiar with the new code and functionality.
>
> In addition, everyone in the community should feel free to peruse through
> the code in this branch and ask any questions that may come up:
>
> https://github.com/apache/ignite/tree/ignite-5267/modules/pds
>
> D.
>
>
> On Wed, May 31, 2017 at 2:37 PM, Denis Magda <[hidden email]> wrote:
>
> > Igniters,
> >
> > The Persistent Store donation was accepted by the community and now we
> > should think of its readiness for a public release.
> >
> > I know that many of you are involved it the donation stabilization. Could
> > you share the current status of the feature? How far is it from the
> > releasable state?
> >
> > —
> > Denis
>

[dsetrakyan]

BTW, can anyone explain to me why do we keep the new persistence code in a
separate module instead of merging it to core?

On Thu, Jun 1, 2017 at 4:32 AM, Sergey Chugunov <[hidden email]>
wrote:

> Hello Denis,
>
> There are three suites [1] [2] [3] configured on TeamCity with bunch of
> tests for PDS functionality.
>
> As you can see most of tests are passing now (only 6 tests are
> failing, for *partitionLoss
> *tests problem was almost sorted out), it looks like that this
> functionality is pretty stable.
>
>
> [1] Ignite PDS 1
> <http://ci.ignite.apache.org/viewType.html?buildTypeId=
> Ignite20Tests_IgnitePds1&branch_Ignite20Tests=ignite-
> 5267&tab=buildTypeStatusDiv>
> [2] Ignite PDS 2
> <http://ci.ignite.apache.org/viewType.html?buildTypeId=
> Ignite20Tests_IgnitePds2&branch_Ignite20Tests=ignite-
> 5267&tab=buildTypeStatusDiv>
> [3] Ignite OutOfMemory
> <http://ci.ignite.apache.org/viewType.html?buildTypeId=Ignite20Tests_
> IgnitePdsOutOfMemory&branch_Ignite20Tests=ignite-5267&tab=
> buildTypeStatusDiv>
>
> On Thu, Jun 1, 2017 at 1:24 AM, Dmitriy Setrakyan <[hidden email]>
> wrote:
>
> > Thanks Denis. I also would like to encourage to post all the
> stabilization
> > updates for the donated branch here. This will help the community to get
> > familiar with the new code and functionality.
> >
> > In addition, everyone in the community should feel free to peruse through
> > the code in this branch and ask any questions that may come up:
> >
> > https://github.com/apache/ignite/tree/ignite-5267/modules/pds
> >
> > D.
> >
> >
> > On Wed, May 31, 2017 at 2:37 PM, Denis Magda <[hidden email]> wrote:
> >
> > > Igniters,
> > >
> > > The Persistent Store donation was accepted by the community and now we
> > > should think of its readiness for a public release.
> > >
> > > I know that many of you are involved it the donation stabilization.
> Could
> > > you share the current status of the feature? How far is it from the
> > > releasable state?
> > >
> > > —
> > > Denis
> >
>

[Alexey Goncharuk]

Guys,

I merged recent changes from the master branch (SQL-related changes) to the
integration branch, we need another TC run to see if the merge broke
something.

2017-06-01 19:14 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:

> BTW, can anyone explain to me why do we keep the new persistence code in a
> separate module instead of merging it to core?
>
> On Thu, Jun 1, 2017 at 4:32 AM, Sergey Chugunov <[hidden email]
> >
> wrote:
>
> > Hello Denis,
> >
> > There are three suites [1] [2] [3] configured on TeamCity with bunch of
> > tests for PDS functionality.
> >
> > As you can see most of tests are passing now (only 6 tests are
> > failing, for *partitionLoss
> > *tests problem was almost sorted out), it looks like that this
> > functionality is pretty stable.
> >
> >
> > [1] Ignite PDS 1
> > <http://ci.ignite.apache.org/viewType.html?buildTypeId=
> > Ignite20Tests_IgnitePds1&branch_Ignite20Tests=ignite-
> > 5267&tab=buildTypeStatusDiv>
> > [2] Ignite PDS 2
> > <http://ci.ignite.apache.org/viewType.html?buildTypeId=
> > Ignite20Tests_IgnitePds2&branch_Ignite20Tests=ignite-
> > 5267&tab=buildTypeStatusDiv>
> > [3] Ignite OutOfMemory
> > <http://ci.ignite.apache.org/viewType.html?buildTypeId=Ignite20Tests_
> > IgnitePdsOutOfMemory&branch_Ignite20Tests=ignite-5267&tab=
> > buildTypeStatusDiv>
> >
> > On Thu, Jun 1, 2017 at 1:24 AM, Dmitriy Setrakyan <[hidden email]
> >
> > wrote:
> >
> > > Thanks Denis. I also would like to encourage to post all the
> > stabilization
> > > updates for the donated branch here. This will help the community to
> get
> > > familiar with the new code and functionality.
> > >
> > > In addition, everyone in the community should feel free to peruse
> through
> > > the code in this branch and ask any questions that may come up:
> > >
> > > https://github.com/apache/ignite/tree/ignite-5267/modules/pds
> > >
> > > D.
> > >
> > >
> > > On Wed, May 31, 2017 at 2:37 PM, Denis Magda <[hidden email]>
> wrote:
> > >
> > > > Igniters,
> > > >
> > > > The Persistent Store donation was accepted by the community and now
> we
> > > > should think of its readiness for a public release.
> > > >
> > > > I know that many of you are involved it the donation stabilization.
> > Could
> > > > you share the current status of the feature? How far is it from the
> > > > releasable state?
> > > >
> > > > —
> > > > Denis
> > >
> >
>

[Semyon Boikov]

Guys,

I see some tests in PDS suite fail with message
"junit.framework.AssertionFailedError: only for one run, must be removed
soon". We never add suche messages without reference to JIRA issue, please
fix.

Thanks!

On Thu, Jun 1, 2017 at 7:22 PM, Alexey Goncharuk <[hidden email]
> wrote:

> Guys,
>
> I merged recent changes from the master branch (SQL-related changes) to the
> integration branch, we need another TC run to see if the merge broke
> something.
>
> 2017-06-01 19:14 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
>
> > BTW, can anyone explain to me why do we keep the new persistence code in
> a
> > separate module instead of merging it to core?
> >
> > On Thu, Jun 1, 2017 at 4:32 AM, Sergey Chugunov <
> [hidden email]
> > >
> > wrote:
> >
> > > Hello Denis,
> > >
> > > There are three suites [1] [2] [3] configured on TeamCity with bunch of
> > > tests for PDS functionality.
> > >
> > > As you can see most of tests are passing now (only 6 tests are
> > > failing, for *partitionLoss
> > > *tests problem was almost sorted out), it looks like that this
> > > functionality is pretty stable.
> > >
> > >
> > > [1] Ignite PDS 1
> > > <http://ci.ignite.apache.org/viewType.html?buildTypeId=
> > > Ignite20Tests_IgnitePds1&branch_Ignite20Tests=ignite-
> > > 5267&tab=buildTypeStatusDiv>
> > > [2] Ignite PDS 2
> > > <http://ci.ignite.apache.org/viewType.html?buildTypeId=
> > > Ignite20Tests_IgnitePds2&branch_Ignite20Tests=ignite-
> > > 5267&tab=buildTypeStatusDiv>
> > > [3] Ignite OutOfMemory
> > > <http://ci.ignite.apache.org/viewType.html?buildTypeId=Ignite20Tests_
> > > IgnitePdsOutOfMemory&branch_Ignite20Tests=ignite-5267&tab=
> > > buildTypeStatusDiv>
> > >
> > > On Thu, Jun 1, 2017 at 1:24 AM, Dmitriy Setrakyan <
> [hidden email]
> > >
> > > wrote:
> > >
> > > > Thanks Denis. I also would like to encourage to post all the
> > > stabilization
> > > > updates for the donated branch here. This will help the community to
> > get
> > > > familiar with the new code and functionality.
> > > >
> > > > In addition, everyone in the community should feel free to peruse
> > through
> > > > the code in this branch and ask any questions that may come up:
> > > >
> > > > https://github.com/apache/ignite/tree/ignite-5267/modules/pds
> > > >
> > > > D.
> > > >
> > > >
> > > > On Wed, May 31, 2017 at 2:37 PM, Denis Magda <[hidden email]>
> > wrote:
> > > >
> > > > > Igniters,
> > > > >
> > > > > The Persistent Store donation was accepted by the community and now
> > we
> > > > > should think of its readiness for a public release.
> > > > >
> > > > > I know that many of you are involved it the donation stabilization.
> > > Could
> > > > > you share the current status of the feature? How far is it from the
> > > > > releasable state?
> > > > >
> > > > > —
> > > > > Denis
> > > >
> > >
> >
>

[Alexey Goncharuk]

Semyon, thanks for spotting this. I've removed the fails as tests should be
passing now.

2017-06-02 10:20 GMT+03:00 Semyon Boikov <[hidden email]>:

> Guys,
>
> I see some tests in PDS suite fail with message
> "junit.framework.AssertionFailedError: only for one run, must be removed
> soon". We never add suche messages without reference to JIRA issue, please
> fix.
>
> Thanks!
>
> On Thu, Jun 1, 2017 at 7:22 PM, Alexey Goncharuk <
> [hidden email]
> > wrote:
>
> > Guys,
> >
> > I merged recent changes from the master branch (SQL-related changes) to
> the
> > integration branch, we need another TC run to see if the merge broke
> > something.
> >
> > 2017-06-01 19:14 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
> >
> > > BTW, can anyone explain to me why do we keep the new persistence code
> in
> > a
> > > separate module instead of merging it to core?
> > >
> > > On Thu, Jun 1, 2017 at 4:32 AM, Sergey Chugunov <
> > [hidden email]
> > > >
> > > wrote:
> > >
> > > > Hello Denis,
> > > >
> > > > There are three suites [1] [2] [3] configured on TeamCity with bunch
> of
> > > > tests for PDS functionality.
> > > >
> > > > As you can see most of tests are passing now (only 6 tests are
> > > > failing, for *partitionLoss
> > > > *tests problem was almost sorted out), it looks like that this
> > > > functionality is pretty stable.
> > > >
> > > >
> > > > [1] Ignite PDS 1
> > > > <http://ci.ignite.apache.org/viewType.html?buildTypeId=
> > > > Ignite20Tests_IgnitePds1&branch_Ignite20Tests=ignite-
> > > > 5267&tab=buildTypeStatusDiv>
> > > > [2] Ignite PDS 2
> > > > <http://ci.ignite.apache.org/viewType.html?buildTypeId=
> > > > Ignite20Tests_IgnitePds2&branch_Ignite20Tests=ignite-
> > > > 5267&tab=buildTypeStatusDiv>
> > > > [3] Ignite OutOfMemory
> > > > <http://ci.ignite.apache.org/viewType.html?buildTypeId=
> Ignite20Tests_
> > > > IgnitePdsOutOfMemory&branch_Ignite20Tests=ignite-5267&tab=
> > > > buildTypeStatusDiv>
> > > >
> > > > On Thu, Jun 1, 2017 at 1:24 AM, Dmitriy Setrakyan <
> > [hidden email]
> > > >
> > > > wrote:
> > > >
> > > > > Thanks Denis. I also would like to encourage to post all the
> > > > stabilization
> > > > > updates for the donated branch here. This will help the community
> to
> > > get
> > > > > familiar with the new code and functionality.
> > > > >
> > > > > In addition, everyone in the community should feel free to peruse
> > > through
> > > > > the code in this branch and ask any questions that may come up:
> > > > >
> > > > > https://github.com/apache/ignite/tree/ignite-5267/modules/pds
> > > > >
> > > > > D.
> > > > >
> > > > >
> > > > > On Wed, May 31, 2017 at 2:37 PM, Denis Magda <[hidden email]>
> > > wrote:
> > > > >
> > > > > > Igniters,
> > > > > >
> > > > > > The Persistent Store donation was accepted by the community and
> now
> > > we
> > > > > > should think of its readiness for a public release.
> > > > > >
> > > > > > I know that many of you are involved it the donation
> stabilization.
> > > > Could
> > > > > > you share the current status of the feature? How far is it from
> the
> > > > > > releasable state?
> > > > > >
> > > > > > —
> > > > > > Denis
> > > > >
> > > >
> > >
> >
>

[Semyon Boikov]

Also in persistence code I see some TODOs (//TODO GG-), all TODOs should
reference some ingite JIRA issue,

Thanks!

On Fri, Jun 2, 2017 at 11:29 AM, Alexey Goncharuk <
[hidden email]> wrote:

> Semyon, thanks for spotting this. I've removed the fails as tests should be
> passing now.
>
> 2017-06-02 10:20 GMT+03:00 Semyon Boikov <[hidden email]>:
>
> > Guys,
> >
> > I see some tests in PDS suite fail with message
> > "junit.framework.AssertionFailedError: only for one run, must be removed
> > soon". We never add suche messages without reference to JIRA issue,
> please
> > fix.
> >
> > Thanks!
> >
> > On Thu, Jun 1, 2017 at 7:22 PM, Alexey Goncharuk <
> > [hidden email]
> > > wrote:
> >
> > > Guys,
> > >
> > > I merged recent changes from the master branch (SQL-related changes) to
> > the
> > > integration branch, we need another TC run to see if the merge broke
> > > something.
> > >
> > > 2017-06-01 19:14 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
> > >
> > > > BTW, can anyone explain to me why do we keep the new persistence code
> > in
> > > a
> > > > separate module instead of merging it to core?
> > > >
> > > > On Thu, Jun 1, 2017 at 4:32 AM, Sergey Chugunov <
> > > [hidden email]
> > > > >
> > > > wrote:
> > > >
> > > > > Hello Denis,
> > > > >
> > > > > There are three suites [1] [2] [3] configured on TeamCity with
> bunch
> > of
> > > > > tests for PDS functionality.
> > > > >
> > > > > As you can see most of tests are passing now (only 6 tests are
> > > > > failing, for *partitionLoss
> > > > > *tests problem was almost sorted out), it looks like that this
> > > > > functionality is pretty stable.
> > > > >
> > > > >
> > > > > [1] Ignite PDS 1
> > > > > <http://ci.ignite.apache.org/viewType.html?buildTypeId=
> > > > > Ignite20Tests_IgnitePds1&branch_Ignite20Tests=ignite-
> > > > > 5267&tab=buildTypeStatusDiv>
> > > > > [2] Ignite PDS 2
> > > > > <http://ci.ignite.apache.org/viewType.html?buildTypeId=
> > > > > Ignite20Tests_IgnitePds2&branch_Ignite20Tests=ignite-
> > > > > 5267&tab=buildTypeStatusDiv>
> > > > > [3] Ignite OutOfMemory
> > > > > <http://ci.ignite.apache.org/viewType.html?buildTypeId=
> > Ignite20Tests_
> > > > > IgnitePdsOutOfMemory&branch_Ignite20Tests=ignite-5267&tab=
> > > > > buildTypeStatusDiv>
> > > > >
> > > > > On Thu, Jun 1, 2017 at 1:24 AM, Dmitriy Setrakyan <
> > > [hidden email]
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Thanks Denis. I also would like to encourage to post all the
> > > > > stabilization
> > > > > > updates for the donated branch here. This will help the community
> > to
> > > > get
> > > > > > familiar with the new code and functionality.
> > > > > >
> > > > > > In addition, everyone in the community should feel free to peruse
> > > > through
> > > > > > the code in this branch and ask any questions that may come up:
> > > > > >
> > > > > > https://github.com/apache/ignite/tree/ignite-5267/modules/pds
> > > > > >
> > > > > > D.
> > > > > >
> > > > > >
> > > > > > On Wed, May 31, 2017 at 2:37 PM, Denis Magda <[hidden email]>
> > > > wrote:
> > > > > >
> > > > > > > Igniters,
> > > > > > >
> > > > > > > The Persistent Store donation was accepted by the community and
> > now
> > > > we
> > > > > > > should think of its readiness for a public release.
> > > > > > >
> > > > > > > I know that many of you are involved it the donation
> > stabilization.
> > > > > Could
> > > > > > > you share the current status of the feature? How far is it from
> > the
> > > > > > > releasable state?
> > > > > > >
> > > > > > > —
> > > > > > > Denis
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Alexey Goncharuk]

Dmitriy,

The original idea behind moving persistence to a separate module was
easiness of review and the fact that it is likely that in the future it
will have some external dependencies. I guess we can have those
dependencies as a separate module and move persistence to the core module.

2017-06-01 19:14 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:

> BTW, can anyone explain to me why do we keep the new persistence code in a
> separate module instead of merging it to core?
>
> On Thu, Jun 1, 2017 at 4:32 AM, Sergey Chugunov <[hidden email]
> >
> wrote:
>
> > Hello Denis,
> >
> > There are three suites [1] [2] [3] configured on TeamCity with bunch of
> > tests for PDS functionality.
> >
> > As you can see most of tests are passing now (only 6 tests are
> > failing, for *partitionLoss
> > *tests problem was almost sorted out), it looks like that this
> > functionality is pretty stable.
> >
> >
> > [1] Ignite PDS 1
> > <http://ci.ignite.apache.org/viewType.html?buildTypeId=
> > Ignite20Tests_IgnitePds1&branch_Ignite20Tests=ignite-
> > 5267&tab=buildTypeStatusDiv>
> > [2] Ignite PDS 2
> > <http://ci.ignite.apache.org/viewType.html?buildTypeId=
> > Ignite20Tests_IgnitePds2&branch_Ignite20Tests=ignite-
> > 5267&tab=buildTypeStatusDiv>
> > [3] Ignite OutOfMemory
> > <http://ci.ignite.apache.org/viewType.html?buildTypeId=Ignite20Tests_
> > IgnitePdsOutOfMemory&branch_Ignite20Tests=ignite-5267&tab=
> > buildTypeStatusDiv>
> >
> > On Thu, Jun 1, 2017 at 1:24 AM, Dmitriy Setrakyan <[hidden email]
> >
> > wrote:
> >
> > > Thanks Denis. I also would like to encourage to post all the
> > stabilization
> > > updates for the donated branch here. This will help the community to
> get
> > > familiar with the new code and functionality.
> > >
> > > In addition, everyone in the community should feel free to peruse
> through
> > > the code in this branch and ask any questions that may come up:
> > >
> > > https://github.com/apache/ignite/tree/ignite-5267/modules/pds
> > >
> > > D.
> > >
> > >
> > > On Wed, May 31, 2017 at 2:37 PM, Denis Magda <[hidden email]>
> wrote:
> > >
> > > > Igniters,
> > > >
> > > > The Persistent Store donation was accepted by the community and now
> we
> > > > should think of its readiness for a public release.
> > > >
> > > > I know that many of you are involved it the donation stabilization.
> > Could
> > > > you share the current status of the feature? How far is it from the
> > > > releasable state?
> > > >
> > > > —
> > > > Denis
> > >
> >
>

[dsetrakyan]

On Fri, Jun 2, 2017 at 6:46 AM, Alexey Goncharuk <[hidden email]
> wrote:

> Dmitriy,
>
> The original idea behind moving persistence to a separate module was
> easiness of review and the fact that it is likely that in the future it
> will have some external dependencies. I guess we can have those
> dependencies as a separate module and move persistence to the core module.
>

Yup, let's do that once the branch is stable.

[Semyon Boikov]

Does anybody monitor TeamCity for ignite-5267? I see suite 'Ignite PDS 2'
started to hang 100% after recent merges.

Thanks

On Fri, Jun 2, 2017 at 7:19 PM, Dmitriy Setrakyan <[hidden email]>
wrote:

> On Fri, Jun 2, 2017 at 6:46 AM, Alexey Goncharuk <
> [hidden email]
> > wrote:
>
> > Dmitriy,
> >
> > The original idea behind moving persistence to a separate module was
> > easiness of review and the fact that it is likely that in the future it
> > will have some external dependencies. I guess we can have those
> > dependencies as a separate module and move persistence to the core
> module.
> >
>
> Yup, let's do that once the branch is stable.
>

[dsetrakyan]

On Tue, Jun 6, 2017 at 1:21 AM, Semyon Boikov <[hidden email]> wrote:

> Does anybody monitor TeamCity for ignite-5267? I see suite 'Ignite PDS 2'
> started to hang 100% after recent merges.
>

Do we have any progress on this?

[Alexey Goncharuk]

I am currently in the middle of the debugging - the issue does not look
related to the persistent store changes now.

2017-06-07 5:47 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:

> On Tue, Jun 6, 2017 at 1:21 AM, Semyon Boikov <[hidden email]>
> wrote:
>
> > Does anybody monitor TeamCity for ignite-5267? I see suite 'Ignite PDS 2'
> > started to hang 100% after recent merges.
> >
>
> Do we have any progress on this?
>

[dsetrakyan]

On Wed, Jun 7, 2017 at 12:25 AM, Alexey Goncharuk <
[hidden email]> wrote:

> I am currently in the middle of the debugging - the issue does not look
> related to the persistent store changes now.
>

Still would be nice to let the community know whenever the TC is in
stable-enough state to continue.

>
> 2017-06-07 5:47 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
>
> > On Tue, Jun 6, 2017 at 1:21 AM, Semyon Boikov <[hidden email]>
> > wrote:
> >
> > > Does anybody monitor TeamCity for ignite-5267? I see suite 'Ignite PDS
> 2'
> > > started to hang 100% after recent merges.
> > >
> >
> > Do we have any progress on this?
> >
>