Persistent Distributed Store Metrics

[Konstantin Boudnik-2]

On Mon, Jun 05, 2017 at 07:41PM, Dmitriy Setrakyan wrote:

> On Mon, Jun 5, 2017 at 6:46 PM, Konstantin Boudnik <[hidden email]> wrote:
>
> > Wow, hold on - as far as I remember there was a VOTE to accept the
> > contribution of the code into the project _on a branch_. We haven't vetted
> > its
> > inclusion into the next release, We are still at the phase of getting
> > familiar
> > with the code.
> >
>
> Cos, the community has been vetting the inclusion of the new code for over
> 3 weeks already (the process and dates are documented here [2]). To be
> honest, I am not sure what the appropriate time frame should be, but I
> would think that a month would be a good check-in point.

I would think it should be as long as we, as the community, are comfortable
with the massive change coming into the release. We already been through the
discussion on the timing, etc. and I don't want to harp on that.

> There is also an active stabilization thread for the persistence branch
> [3]. I encouraged the community to get involved and post any questions or
> concerns there as well.
>
> There is an upcoming in-memory computing conference that is coming up in
> June in Amsterdam [4], where there are many Ignite talks scheduled. It
> would be great to be able to talk about the persistence features of Ignite
> there as well. I would like to ask the community to mobilize with reviewing
> the donated code, so we could have something concrete to tell the audience
> on the conference.

I am sure that having the code on the branch is good enough to be able to talk
about this. Having this in the release isn't really a contingency to be able
to make a presentation, right?

Cos

> [2]
> http://incubator.apache.org/ip-clearance/persistent-distributed-store-ignite.html
> [3]
> http://apache-ignite-developers.2346864.n4.nabble.com/Persistent-Store-Stabilization-for-release-td18288.html
> [4] https://imcsummit.org/
>
>
>
> > And from what I am seeing in the discussions like this [1], we need to be
> > extra careful.
> >
>
> I would keep the discussion in [1] separate from the persistence store.
> These are 2 unrelated issues. I will respond on [1] either today or
> tomorrow, but I agree in general that it should be fixed ASAP.
>
>
> > BTW, you have sent this email 9 days before the vote had happened! A bit
> > too
> > soon, if you ask me.
> >
>
> Cos, this email was sent 1 week after the initial donated code branch was
> presented to the community (see [2] above). The developers involved were
> eager to make it available to the users as soon as possible, but no code
> has been merged to the master branch yet.
>
> I would like to encourage everyone in the community to participate in the
> persistence branch coding discussions, like the one in this thread, and
> familiarize themselves with the code.
>
>
> >
> > [1] https://is.gd/UQCr51
> >
> > Cos
> >
> > On Wed, May 17, 2017 at 11:16AM, Dmitriy Govorukhin wrote:
> > > Folk,
> > >
> > > As you know, ignite 2.1 will contain new module (pds), it will be
> > > provide ability to store data on disk. Let's discuss what type of
> > > metrics we need for this?
> > > I think it must be metrics per memory policy, per cache, checkpoint,
> > > and global metrics which will be aggregate all metrics.
> > >
> > > I did sketch.
> > >
> > > PersistentStoreMetrics.java
> > >
> > > public interface PersistentStoreMetrics {
> > >
> > >     // Global metrics.
> > >
> > >     public long getMemorySize();
> > >
> > >     public long getDiskSize();
> > >
> > >     public long getPagesInMemory();
> > >
> > >     public long getPagesSizeInMemory();
> > >
> > >     public long getPagesOnDisk();
> > >
> > >     public long getPagesSizeOnDisk();
> > >
> > >     public long getFreePages();
> > >
> > >     public long getFreePagesSize();
> > >
> > >     public long getDirtyPages();
> > >
> > >     public long getDirtyPagesSize();
> > >
> > >     public long walLog();
> > >
> > >     public long walLogSize();
> > >
> > >     // Frequency.
> > >
> > >     public long getPagesRead();
> > >
> > >     public long getPagesWrite();
> > >
> > >     public long getFsync();
> > >
> > >     public long getWal();
> > >
> > >     public long getAverageWalFsyncTime();
> > >
> > >     // Per cache.
> > >
> > >     public PersistentStoreCacheMetrics cache(String name);
> > >
> > >     public PersistentStoreCacheMetrics cache(int cacheId);
> > >
> > >     // For last checkpoint.
> > >
> > >     public PersistentStoreCheckpointMetrics getLastCheckPoint();
> > > }
> > >
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > >
> > > PersistentStoreCacheMetrics.java
> > >
> > > public interface PersistentStoreCacheMetrics {
> > >
> > >     public String name();
> > >
> > >     public double getFillFactor();
> > >
> > >     public double getFillFactor(int part);
> > >
> > >     public long getMemorySize();
> > >
> > >     public long getDiskSize();
> > >
> > >     public long getPagesInMemory();
> > >
> > >     public long getPagesSizeInMemory();
> > >
> > >     public long getPagesOnDisk();
> > >
> > >     public long getPagesSizeOnDisk();
> > >
> > >     public long getFreePages();
> > >
> > >     public long getFreePagesSize();
> > >
> > >     public long getDirtyPages();
> > >
> > >     public long getDirtyPagesSize();
> > >
> > >     public long getPagesRead();
> > >
> > >     public long getPagesWritten();
> > > }
> > >
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > >
> > > PersistentStoreCheckpointMetrics.java
> > >
> > > public interface PersistentStoreCheckpointMetrics {
> > >
> > >     public long getTotalPages();
> > >
> > >     //TODO Page type is internal?
> > >     public long[] pagesType();
> > >
> > >     public long getExecutingTime();
> > >
> > >     public long getFsyncTime();
> > >
> > >     public long getPagesCopyOnWrite();
> > > }
> >

[dmagda]

> On Jun 5, 2017, at 8:02 PM, Konstantin Boudnik <[hidden email]> wrote:
>
> On Mon, Jun 05, 2017 at 07:41PM, Dmitriy Setrakyan wrote:
>> On Mon, Jun 5, 2017 at 6:46 PM, Konstantin Boudnik <[hidden email]> wrote:
>>
>>> Wow, hold on - as far as I remember there was a VOTE to accept the
>>> contribution of the code into the project _on a branch_. We haven't vetted
>>> its
>>> inclusion into the next release, We are still at the phase of getting
>>> familiar
>>> with the code.
>>>
>>
>> Cos, the community has been vetting the inclusion of the new code for over
>> 3 weeks already (the process and dates are documented here [2]). To be
>> honest, I am not sure what the appropriate time frame should be, but I
>> would think that a month would be a good check-in point.
>
> I would think it should be as long as we, as the community, are comfortable
> with the massive change coming into the release. We already been through the
> discussion on the timing, etc. and I don't want to harp on that.

Just in case I’m working on the high-level storage documentation. Here is a draft for your reference:
https://apacheignite.readme.io/docs/distributed-persistent-store <https://apacheignite.readme.io/docs/distributed-persistent-store>

The page is invisible for wide audience at the moment. Only for Ignite committers.

—
Denis

> On Jun 5, 2017, at 8:02 PM, Konstantin Boudnik <[hidden email]> wrote:
>
> On Mon, Jun 05, 2017 at 07:41PM, Dmitriy Setrakyan wrote:
>> On Mon, Jun 5, 2017 at 6:46 PM, Konstantin Boudnik <[hidden email]> wrote:
>>
>>> Wow, hold on - as far as I remember there was a VOTE to accept the
>>> contribution of the code into the project _on a branch_. We haven't vetted
>>> its
>>> inclusion into the next release, We are still at the phase of getting
>>> familiar
>>> with the code.
>>>
>>
>> Cos, the community has been vetting the inclusion of the new code for over
>> 3 weeks already (the process and dates are documented here [2]). To be
>> honest, I am not sure what the appropriate time frame should be, but I
>> would think that a month would be a good check-in point.
>
> I would think it should be as long as we, as the community, are comfortable
> with the massive change coming into the release. We already been through the
> discussion on the timing, etc. and I don't want to harp on that.
>
>> There is also an active stabilization thread for the persistence branch
>> [3]. I encouraged the community to get involved and post any questions or
>> concerns there as well.
>>
>> There is an upcoming in-memory computing conference that is coming up in
>> June in Amsterdam [4], where there are many Ignite talks scheduled. It
>> would be great to be able to talk about the persistence features of Ignite
>> there as well. I would like to ask the community to mobilize with reviewing
>> the donated code, so we could have something concrete to tell the audience
>> on the conference.
>
> I am sure that having the code on the branch is good enough to be able to talk
> about this. Having this in the release isn't really a contingency to be able
> to make a presentation, right?
>
> Cos
>
>> [2]
>> http://incubator.apache.org/ip-clearance/persistent-distributed-store-ignite.html
>> [3]
>> http://apache-ignite-developers.2346864.n4.nabble.com/Persistent-Store-Stabilization-for-release-td18288.html
>> [4] https://imcsummit.org/
>>
>>
>>
>>> And from what I am seeing in the discussions like this [1], we need to be
>>> extra careful.
>>>
>>
>> I would keep the discussion in [1] separate from the persistence store.
>> These are 2 unrelated issues. I will respond on [1] either today or
>> tomorrow, but I agree in general that it should be fixed ASAP.
>>
>>
>>> BTW, you have sent this email 9 days before the vote had happened! A bit
>>> too
>>> soon, if you ask me.
>>>
>>
>> Cos, this email was sent 1 week after the initial donated code branch was
>> presented to the community (see [2] above). The developers involved were
>> eager to make it available to the users as soon as possible, but no code
>> has been merged to the master branch yet.
>>
>> I would like to encourage everyone in the community to participate in the
>> persistence branch coding discussions, like the one in this thread, and
>> familiarize themselves with the code.
>>
>>
>>>
>>> [1] https://is.gd/UQCr51
>>>
>>> Cos
>>>
>>> On Wed, May 17, 2017 at 11:16AM, Dmitriy Govorukhin wrote:
>>>> Folk,
>>>>
>>>> As you know, ignite 2.1 will contain new module (pds), it will be
>>>> provide ability to store data on disk. Let's discuss what type of
>>>> metrics we need for this?
>>>> I think it must be metrics per memory policy, per cache, checkpoint,
>>>> and global metrics which will be aggregate all metrics.
>>>>
>>>> I did sketch.
>>>>
>>>> PersistentStoreMetrics.java
>>>>
>>>> public interface PersistentStoreMetrics {
>>>>
>>>>    // Global metrics.
>>>>
>>>>    public long getMemorySize();
>>>>
>>>>    public long getDiskSize();
>>>>
>>>>    public long getPagesInMemory();
>>>>
>>>>    public long getPagesSizeInMemory();
>>>>
>>>>    public long getPagesOnDisk();
>>>>
>>>>    public long getPagesSizeOnDisk();
>>>>
>>>>    public long getFreePages();
>>>>
>>>>    public long getFreePagesSize();
>>>>
>>>>    public long getDirtyPages();
>>>>
>>>>    public long getDirtyPagesSize();
>>>>
>>>>    public long walLog();
>>>>
>>>>    public long walLogSize();
>>>>
>>>>    // Frequency.
>>>>
>>>>    public long getPagesRead();
>>>>
>>>>    public long getPagesWrite();
>>>>
>>>>    public long getFsync();
>>>>
>>>>    public long getWal();
>>>>
>>>>    public long getAverageWalFsyncTime();
>>>>
>>>>    // Per cache.
>>>>
>>>>    public PersistentStoreCacheMetrics cache(String name);
>>>>
>>>>    public PersistentStoreCacheMetrics cache(int cacheId);
>>>>
>>>>    // For last checkpoint.
>>>>
>>>>    public PersistentStoreCheckpointMetrics getLastCheckPoint();
>>>> }
>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>
>>>> PersistentStoreCacheMetrics.java
>>>>
>>>> public interface PersistentStoreCacheMetrics {
>>>>
>>>>    public String name();
>>>>
>>>>    public double getFillFactor();
>>>>
>>>>    public double getFillFactor(int part);
>>>>
>>>>    public long getMemorySize();
>>>>
>>>>    public long getDiskSize();
>>>>
>>>>    public long getPagesInMemory();
>>>>
>>>>    public long getPagesSizeInMemory();
>>>>
>>>>    public long getPagesOnDisk();
>>>>
>>>>    public long getPagesSizeOnDisk();
>>>>
>>>>    public long getFreePages();
>>>>
>>>>    public long getFreePagesSize();
>>>>
>>>>    public long getDirtyPages();
>>>>
>>>>    public long getDirtyPagesSize();
>>>>
>>>>    public long getPagesRead();
>>>>
>>>>    public long getPagesWritten();
>>>> }
>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>
>>>> PersistentStoreCheckpointMetrics.java
>>>>
>>>> public interface PersistentStoreCheckpointMetrics {
>>>>
>>>>    public long getTotalPages();
>>>>
>>>>    //TODO Page type is internal?
>>>>    public long[] pagesType();
>>>>
>>>>    public long getExecutingTime();
>>>>
>>>>    public long getFsyncTime();
>>>>
>>>>    public long getPagesCopyOnWrite();
>>>> }
>>>

[Alexey Goncharuk]

Denis,

Thanks for the review. I addressed your comments and merged the changes to
ignite-5267 branch.

2017-06-06 22:56 GMT+03:00 Denis Magda <[hidden email]>:

> > On Jun 5, 2017, at 8:02 PM, Konstantin Boudnik <[hidden email]> wrote:
> >
> > On Mon, Jun 05, 2017 at 07:41PM, Dmitriy Setrakyan wrote:
> >> On Mon, Jun 5, 2017 at 6:46 PM, Konstantin Boudnik <[hidden email]>
> wrote:
> >>
> >>> Wow, hold on - as far as I remember there was a VOTE to accept the
> >>> contribution of the code into the project _on a branch_. We haven't
> vetted
> >>> its
> >>> inclusion into the next release, We are still at the phase of getting
> >>> familiar
> >>> with the code.
> >>>
> >>
> >> Cos, the community has been vetting the inclusion of the new code for
> over
> >> 3 weeks already (the process and dates are documented here [2]). To be
> >> honest, I am not sure what the appropriate time frame should be, but I
> >> would think that a month would be a good check-in point.
> >
> > I would think it should be as long as we, as the community, are
> comfortable
> > with the massive change coming into the release. We already been through
> the
> > discussion on the timing, etc. and I don't want to harp on that.
>
> Just in case I’m working on the high-level storage documentation. Here is
> a draft for your reference:
> https://apacheignite.readme.io/docs/distributed-persistent-store <
> https://apacheignite.readme.io/docs/distributed-persistent-store>
>
> The page is invisible for wide audience at the moment. Only for Ignite
> committers.
>
> —
> Denis
>
> > On Jun 5, 2017, at 8:02 PM, Konstantin Boudnik <[hidden email]> wrote:
> >
> > On Mon, Jun 05, 2017 at 07:41PM, Dmitriy Setrakyan wrote:
> >> On Mon, Jun 5, 2017 at 6:46 PM, Konstantin Boudnik <[hidden email]>
> wrote:
> >>
> >>> Wow, hold on - as far as I remember there was a VOTE to accept the
> >>> contribution of the code into the project _on a branch_. We haven't
> vetted
> >>> its
> >>> inclusion into the next release, We are still at the phase of getting
> >>> familiar
> >>> with the code.
> >>>
> >>
> >> Cos, the community has been vetting the inclusion of the new code for
> over
> >> 3 weeks already (the process and dates are documented here [2]). To be
> >> honest, I am not sure what the appropriate time frame should be, but I
> >> would think that a month would be a good check-in point.
> >
> > I would think it should be as long as we, as the community, are
> comfortable
> > with the massive change coming into the release. We already been through
> the
> > discussion on the timing, etc. and I don't want to harp on that.
> >
> >> There is also an active stabilization thread for the persistence branch
> >> [3]. I encouraged the community to get involved and post any questions
> or
> >> concerns there as well.
> >>
> >> There is an upcoming in-memory computing conference that is coming up in
> >> June in Amsterdam [4], where there are many Ignite talks scheduled. It
> >> would be great to be able to talk about the persistence features of
> Ignite
> >> there as well. I would like to ask the community to mobilize with
> reviewing
> >> the donated code, so we could have something concrete to tell the
> audience
> >> on the conference.
> >
> > I am sure that having the code on the branch is good enough to be able
> to talk
> > about this. Having this in the release isn't really a contingency to be
> able
> > to make a presentation, right?
> >
> > Cos
> >
> >> [2]
> >> http://incubator.apache.org/ip-clearance/persistent-
> distributed-store-ignite.html
> >> [3]
> >> http://apache-ignite-developers.2346864.n4.nabble.com/Persistent-Store-
> Stabilization-for-release-td18288.html
> >> [4] https://imcsummit.org/
> >>
> >>
> >>
> >>> And from what I am seeing in the discussions like this [1], we need to
> be
> >>> extra careful.
> >>>
> >>
> >> I would keep the discussion in [1] separate from the persistence store.
> >> These are 2 unrelated issues. I will respond on [1] either today or
> >> tomorrow, but I agree in general that it should be fixed ASAP.
> >>
> >>
> >>> BTW, you have sent this email 9 days before the vote had happened! A
> bit
> >>> too
> >>> soon, if you ask me.
> >>>
> >>
> >> Cos, this email was sent 1 week after the initial donated code branch
> was
> >> presented to the community (see [2] above). The developers involved were
> >> eager to make it available to the users as soon as possible, but no code
> >> has been merged to the master branch yet.
> >>
> >> I would like to encourage everyone in the community to participate in
> the
> >> persistence branch coding discussions, like the one in this thread, and
> >> familiarize themselves with the code.
> >>
> >>
> >>>
> >>> [1] https://is.gd/UQCr51
> >>>
> >>> Cos
> >>>
> >>> On Wed, May 17, 2017 at 11:16AM, Dmitriy Govorukhin wrote:
> >>>> Folk,
> >>>>
> >>>> As you know, ignite 2.1 will contain new module (pds), it will be
> >>>> provide ability to store data on disk. Let's discuss what type of
> >>>> metrics we need for this?
> >>>> I think it must be metrics per memory policy, per cache, checkpoint,
> >>>> and global metrics which will be aggregate all metrics.
> >>>>
> >>>> I did sketch.
> >>>>
> >>>> PersistentStoreMetrics.java
> >>>>
> >>>> public interface PersistentStoreMetrics {
> >>>>
> >>>>    // Global metrics.
> >>>>
> >>>>    public long getMemorySize();
> >>>>
> >>>>    public long getDiskSize();
> >>>>
> >>>>    public long getPagesInMemory();
> >>>>
> >>>>    public long getPagesSizeInMemory();
> >>>>
> >>>>    public long getPagesOnDisk();
> >>>>
> >>>>    public long getPagesSizeOnDisk();
> >>>>
> >>>>    public long getFreePages();
> >>>>
> >>>>    public long getFreePagesSize();
> >>>>
> >>>>    public long getDirtyPages();
> >>>>
> >>>>    public long getDirtyPagesSize();
> >>>>
> >>>>    public long walLog();
> >>>>
> >>>>    public long walLogSize();
> >>>>
> >>>>    // Frequency.
> >>>>
> >>>>    public long getPagesRead();
> >>>>
> >>>>    public long getPagesWrite();
> >>>>
> >>>>    public long getFsync();
> >>>>
> >>>>    public long getWal();
> >>>>
> >>>>    public long getAverageWalFsyncTime();
> >>>>
> >>>>    // Per cache.
> >>>>
> >>>>    public PersistentStoreCacheMetrics cache(String name);
> >>>>
> >>>>    public PersistentStoreCacheMetrics cache(int cacheId);
> >>>>
> >>>>    // For last checkpoint.
> >>>>
> >>>>    public PersistentStoreCheckpointMetrics getLastCheckPoint();
> >>>> }
> >>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>
> >>>> PersistentStoreCacheMetrics.java
> >>>>
> >>>> public interface PersistentStoreCacheMetrics {
> >>>>
> >>>>    public String name();
> >>>>
> >>>>    public double getFillFactor();
> >>>>
> >>>>    public double getFillFactor(int part);
> >>>>
> >>>>    public long getMemorySize();
> >>>>
> >>>>    public long getDiskSize();
> >>>>
> >>>>    public long getPagesInMemory();
> >>>>
> >>>>    public long getPagesSizeInMemory();
> >>>>
> >>>>    public long getPagesOnDisk();
> >>>>
> >>>>    public long getPagesSizeOnDisk();
> >>>>
> >>>>    public long getFreePages();
> >>>>
> >>>>    public long getFreePagesSize();
> >>>>
> >>>>    public long getDirtyPages();
> >>>>
> >>>>    public long getDirtyPagesSize();
> >>>>
> >>>>    public long getPagesRead();
> >>>>
> >>>>    public long getPagesWritten();
> >>>> }
> >>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>
> >>>> PersistentStoreCheckpointMetrics.java
> >>>>
> >>>> public interface PersistentStoreCheckpointMetrics {
> >>>>
> >>>>    public long getTotalPages();
> >>>>
> >>>>    //TODO Page type is internal?
> >>>>    public long[] pagesType();
> >>>>
> >>>>    public long getExecutingTime();
> >>>>
> >>>>    public long getFsyncTime();
> >>>>
> >>>>    public long getPagesCopyOnWrite();
> >>>> }
> >>>
>
>