Capacity Planning - Calculating Memory Usage

[Alexandr Kuramshin]

Hello, Igniters,

I'd like to represent updated article [1] about the subject.

And I'll very appreciate your comments and questions about it.

Please review.

[1] http://apacheignite.gridgain.org/docs/capacity-planning

--
Thanks,
Alexandr Kuramshin

[dmagda]

Alexander, great update, thanks.

Yakov, Alexey Scherbakov, Val,

Please do the review first since you’re as the ones who deal a lot with performance and memory usage related questions.

—
Denis

> On Dec 12, 2016, at 11:37 PM, Alexandr Kuramshin <[hidden email]> wrote:
>
> Hello, Igniters,
>
> I'd like to represent updated article [1] about the subject.
>
> And I'll very appreciate your comments and questions about it.
>
> Please review.
>
> [1] http://apacheignite.gridgain.org/docs/capacity-planning
>
> --
> Thanks,
> Alexandr Kuramshin

[Andrew Mashenkov]

Alexandr,

In addition. If expire policy is configured, there is additional overhead
to entries can be tracked by TtlManager.
This overhead is OnHeap and does not depend on cache MemoryMode (until
Ignite-3840 will be in master).

For now overhead is about 32-40 bytes (EntryWrapper itself) + (40-48) bytes
(ConcurrentSkipList node) per entry.



On Tue, Dec 13, 2016 at 10:37 AM, Alexandr Kuramshin <[hidden email]>
wrote:

> Hello, Igniters,
>
> I'd like to represent updated article [1] about the subject.
>
> And I'll very appreciate your comments and questions about it.
>
> Please review.
>
> [1] http://apacheignite.gridgain.org/docs/capacity-planning
>
> --
> Thanks,
> Alexandr Kuramshin
>

--
С уважением,
Машенков Андрей Владимирович
Тел. [phone]

Best regards,
Andrey V. Mashenkov
Cerr: [phone]

[Alexandr Kuramshin]

Thank you, Andrey,

I'll do additional tests with expire policy and update the article.

2016-12-13 22:10 GMT+07:00 Andrey Mashenkov <[hidden email]>:

> Alexandr,
>
> In addition. If expire policy is configured, there is additional overhead
> to entries can be tracked by TtlManager.
> This overhead is OnHeap and does not depend on cache MemoryMode (until
> Ignite-3840 will be in master).
>
> For now overhead is about 32-40 bytes (EntryWrapper itself) + (40-48) bytes
> (ConcurrentSkipList node) per entry.
>
>
>
> On Tue, Dec 13, 2016 at 10:37 AM, Alexandr Kuramshin <[hidden email]
> >
> wrote:
>
> > Hello, Igniters,
> >
> > I'd like to represent updated article [1] about the subject.
> >
> > And I'll very appreciate your comments and questions about it.
> >
> > Please review.
> >
> > [1] http://apacheignite.gridgain.org/docs/capacity-planning
> >
> > --
> > Thanks,
> > Alexandr Kuramshin
> >
>
>
>
> --
> С уважением,
> Машенков Андрей Владимирович
> Тел. [phone]
>
> Best regards,
> Andrey V. Mashenkov
> Cerr: [phone]
>

--
Thanks,
Alexandr Kuramshin

[Valentin Kulichenko]

Alexandr,

Can you please clarify what is "First index overhead" and "Next index
overhead"? Generally, I think overhead provided by indexes should be
described in more details, now it's not very clear what happens when
indexes are added.

Also the calculation example should be a separate section.

-Val

On Wed, Dec 14, 2016 at 1:07 AM, Alexandr Kuramshin <[hidden email]>
wrote:

> Thank you, Andrey,
>
> I'll do additional tests with expire policy and update the article.
>
> 2016-12-13 22:10 GMT+07:00 Andrey Mashenkov <[hidden email]>:
>
> > Alexandr,
> >
> > In addition. If expire policy is configured, there is additional overhead
> > to entries can be tracked by TtlManager.
> > This overhead is OnHeap and does not depend on cache MemoryMode (until
> > Ignite-3840 will be in master).
> >
> > For now overhead is about 32-40 bytes (EntryWrapper itself) + (40-48)
> bytes
> > (ConcurrentSkipList node) per entry.
> >
> >
> >
> > On Tue, Dec 13, 2016 at 10:37 AM, Alexandr Kuramshin <
> [hidden email]
> > >
> > wrote:
> >
> > > Hello, Igniters,
> > >
> > > I'd like to represent updated article [1] about the subject.
> > >
> > > And I'll very appreciate your comments and questions about it.
> > >
> > > Please review.
> > >
> > > [1] http://apacheignite.gridgain.org/docs/capacity-planning
> > >
> > > --
> > > Thanks,
> > > Alexandr Kuramshin
> > >
> >
> >
> >
> > --
> > С уважением,
> > Машенков Андрей Владимирович
> > Тел. [phone]
> >
> > Best regards,
> > Andrey V. Mashenkov
> > Cerr: [phone]
> >
>
>
>
> --
> Thanks,
> Alexandr Kuramshin
>

What is the relationship of apacheignite.gridgain.org to the open source
Apache Ignite project? Thanks!

On Tue, Dec 13, 2016 at 2:37 AM, Alexandr Kuramshin <[hidden email]>
wrote:

> Hello, Igniters,
>
> I'd like to represent updated article [1] about the subject.
>
> And I'll very appreciate your comments and questions about it.
>
> Please review.
>
> [1] http://apacheignite.gridgain.org/docs/capacity-planning
>
> --
> Thanks,
> Alexandr Kuramshin
>

[dmagda]

This is the documentation domain that has been left historically. In general, Ignite was contributed to Apache by GridGain a couple of years ago and now GridGain provides additional enterprise level features on top of Ignite.

The valid link to the doc is:
https://apacheignite.readme.io/docs/capacity-planning <https://apacheignite.readme.io/docs/capacity-planning>

—
Denis

> On Dec 20, 2016, at 12:47 PM, Oddo Da <[hidden email]> wrote:
>
> What is the relationship of apacheignite.gridgain.org to the open source
> Apache Ignite project? Thanks!
>
> On Tue, Dec 13, 2016 at 2:37 AM, Alexandr Kuramshin <[hidden email]>
> wrote:
>
>> Hello, Igniters,
>>
>> I'd like to represent updated article [1] about the subject.
>>
>> And I'll very appreciate your comments and questions about it.
>>
>> Please review.
>>
>> [1] http://apacheignite.gridgain.org/docs/capacity-planning
>>
>> --
>> Thanks,
>> Alexandr Kuramshin
>>

[Alexandr Kuramshin]

Hi Val,

the understanding is simple.

When you enables the single index on entry class you get "First index
overhead" per entry.

When you enables two indices on entry class you get "First index overhead"
+ "Next index overhead" per entry.

With three indices you get "First index overhead" + 2 * "Next index
overhead", and so on...

Each annotated field with @QuerySqlField is an index, except multiple
fields annotated with @QuerySqlField.Group.

Another way to defining indices is to use property "queryEntities" and it's
subproperty "indexes". See the article [1]

[1] https://apacheignite.readme.io/docs/indexes

2016-12-20 8:38 GMT+07:00 Valentin Kulichenko <[hidden email]
>:

> Alexandr,
>
> Can you please clarify what is "First index overhead" and "Next index
> overhead"? Generally, I think overhead provided by indexes should be
> described in more details, now it's not very clear what happens when
> indexes are added.
>
> Also the calculation example should be a separate section.
>
> -Val
>
> On Wed, Dec 14, 2016 at 1:07 AM, Alexandr Kuramshin <[hidden email]>
> wrote:
>
> > Thank you, Andrey,
> >
> > I'll do additional tests with expire policy and update the article.
> >
> > 2016-12-13 22:10 GMT+07:00 Andrey Mashenkov <[hidden email]
> >:
> >
> > > Alexandr,
> > >
> > > In addition. If expire policy is configured, there is additional
> overhead
> > > to entries can be tracked by TtlManager.
> > > This overhead is OnHeap and does not depend on cache MemoryMode (until
> > > Ignite-3840 will be in master).
> > >
> > > For now overhead is about 32-40 bytes (EntryWrapper itself) + (40-48)
> > bytes
> > > (ConcurrentSkipList node) per entry.
> > >
> > >
> > >
> > > On Tue, Dec 13, 2016 at 10:37 AM, Alexandr Kuramshin <
> > [hidden email]
> > > >
> > > wrote:
> > >
> > > > Hello, Igniters,
> > > >
> > > > I'd like to represent updated article [1] about the subject.
> > > >
> > > > And I'll very appreciate your comments and questions about it.
> > > >
> > > > Please review.
> > > >
> > > > [1] http://apacheignite.gridgain.org/docs/capacity-planning
> > > >
> > > > --
> > > > Thanks,
> > > > Alexandr Kuramshin
> > > >
> > >
> > >
> > >
> > > --
> > > С уважением,
> > > Машенков Андрей Владимирович
> > > Тел. [phone]
> > >
> > > Best regards,
> > > Andrey V. Mashenkov
> > > Cerr: [phone]
> > >
> >
> >
> >
> > --
> > Thanks,
> > Alexandr Kuramshin
> >
>

--
Thanks,
Alexandr Kuramshin

[Valentin Kulichenko]

Alexandr,

See my comments below.

On Wed, Dec 21, 2016 at 7:01 PM, Alexandr Kuramshin <[hidden email]>
wrote:

> Hi Val,
>
> the understanding is simple.
>
> When you enables the single index on entry class you get "First index
> overhead" per entry.
>
> When you enables two indices on entry class you get "First index overhead"
> + "Next index overhead" per entry.
>
> With three indices you get "First index overhead" + 2 * "Next index
> overhead", and so on...
>

This should be explained in more detail, probably with some trivial
example. Currently it's very unclear.


>
> Each annotated field with @QuerySqlField is an index, except multiple
> fields annotated with @QuerySqlField.Group.
>

This actually confuses me a lot, because a field can be created with or
without index? Can you please clarify? How much overhead is introduced by a
field without index? With index? What about group indexes?

>
> Another way to defining indices is to use property "queryEntities" and it's
> subproperty "indexes". See the article [1]
>
> [1] https://apacheignite.readme.io/docs/indexes
>
> 2016-12-20 8:38 GMT+07:00 Valentin Kulichenko <
> [hidden email]
> >:
>
> > Alexandr,
> >
> > Can you please clarify what is "First index overhead" and "Next index
> > overhead"? Generally, I think overhead provided by indexes should be
> > described in more details, now it's not very clear what happens when
> > indexes are added.
> >
> > Also the calculation example should be a separate section.
> >
> > -Val
> >
> > On Wed, Dec 14, 2016 at 1:07 AM, Alexandr Kuramshin <
> [hidden email]>
> > wrote:
> >
> > > Thank you, Andrey,
> > >
> > > I'll do additional tests with expire policy and update the article.
> > >
> > > 2016-12-13 22:10 GMT+07:00 Andrey Mashenkov <
> [hidden email]
> > >:
> > >
> > > > Alexandr,
> > > >
> > > > In addition. If expire policy is configured, there is additional
> > overhead
> > > > to entries can be tracked by TtlManager.
> > > > This overhead is OnHeap and does not depend on cache MemoryMode
> (until
> > > > Ignite-3840 will be in master).
> > > >
> > > > For now overhead is about 32-40 bytes (EntryWrapper itself) + (40-48)
> > > bytes
> > > > (ConcurrentSkipList node) per entry.
> > > >
> > > >
> > > >
> > > > On Tue, Dec 13, 2016 at 10:37 AM, Alexandr Kuramshin <
> > > [hidden email]
> > > > >
> > > > wrote:
> > > >
> > > > > Hello, Igniters,
> > > > >
> > > > > I'd like to represent updated article [1] about the subject.
> > > > >
> > > > > And I'll very appreciate your comments and questions about it.
> > > > >
> > > > > Please review.
> > > > >
> > > > > [1] http://apacheignite.gridgain.org/docs/capacity-planning
> > > > >
> > > > > --
> > > > > Thanks,
> > > > > Alexandr Kuramshin
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > С уважением,
> > > > Машенков Андрей Владимирович
> > > > Тел. [phone]
> > > >
> > > > Best regards,
> > > > Andrey V. Mashenkov
> > > > Cerr: [phone]
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks,
> > > Alexandr Kuramshin
> > >
> >
>
>
>
> --
> Thanks,
> Alexandr Kuramshin
>

[Alexandr Kuramshin]

Hi Val,

I'm sorry, of course only @QuerySqlField(index = true) makes an index on
objects field. Fields without indexes make none additional overhead.

Group index on multiple fields is a one index (isn't it?)

I don't understand what is still unclear.

Entry footprint  = key footprint + value footprint + entry overhead + index
overhead.

Index overhead depends on how many indices are enabled for the entry type.

2016-12-23 2:06 GMT+07:00 Valentin Kulichenko <[hidden email]
>:

> Alexandr,
>
> See my comments below.
>
> On Wed, Dec 21, 2016 at 7:01 PM, Alexandr Kuramshin <[hidden email]>
> wrote:
>
> > Hi Val,
> >
> > the understanding is simple.
> >
> > When you enables the single index on entry class you get "First index
> > overhead" per entry.
> >
> > When you enables two indices on entry class you get "First index
> overhead"
> > + "Next index overhead" per entry.
> >
> > With three indices you get "First index overhead" + 2 * "Next index
> > overhead", and so on...
> >
>
> This should be explained in more detail, probably with some trivial
> example. Currently it's very unclear.
>
>
> >
> > Each annotated field with @QuerySqlField is an index, except multiple
> > fields annotated with @QuerySqlField.Group.
> >
>
> This actually confuses me a lot, because a field can be created with or
> without index? Can you please clarify? How much overhead is introduced by a
> field without index? With index? What about group indexes?
>
>
> >
> > Another way to defining indices is to use property "queryEntities" and
> it's
> > subproperty "indexes". See the article [1]
> >
> > [1] https://apacheignite.readme.io/docs/indexes
> >
> > 2016-12-20 8:38 GMT+07:00 Valentin Kulichenko <
> > [hidden email]
> > >:
> >
> > > Alexandr,
> > >
> > > Can you please clarify what is "First index overhead" and "Next index
> > > overhead"? Generally, I think overhead provided by indexes should be
> > > described in more details, now it's not very clear what happens when
> > > indexes are added.
> > >
> > > Also the calculation example should be a separate section.
> > >
> > > -Val
> > >
> > > On Wed, Dec 14, 2016 at 1:07 AM, Alexandr Kuramshin <
> > [hidden email]>
> > > wrote:
> > >
> > > > Thank you, Andrey,
> > > >
> > > > I'll do additional tests with expire policy and update the article.
> > > >
> > > > 2016-12-13 22:10 GMT+07:00 Andrey Mashenkov <
> > [hidden email]
> > > >:
> > > >
> > > > > Alexandr,
> > > > >
> > > > > In addition. If expire policy is configured, there is additional
> > > overhead
> > > > > to entries can be tracked by TtlManager.
> > > > > This overhead is OnHeap and does not depend on cache MemoryMode
> > (until
> > > > > Ignite-3840 will be in master).
> > > > >
> > > > > For now overhead is about 32-40 bytes (EntryWrapper itself) +
> (40-48)
> > > > bytes
> > > > > (ConcurrentSkipList node) per entry.
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Dec 13, 2016 at 10:37 AM, Alexandr Kuramshin <
> > > > [hidden email]
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hello, Igniters,
> > > > > >
> > > > > > I'd like to represent updated article [1] about the subject.
> > > > > >
> > > > > > And I'll very appreciate your comments and questions about it.
> > > > > >
> > > > > > Please review.
> > > > > >
> > > > > > [1] http://apacheignite.gridgain.org/docs/capacity-planning
> > > > > >
> > > > > > --
> > > > > > Thanks,
> > > > > > Alexandr Kuramshin
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > С уважением,
> > > > > Машенков Андрей Владимирович
> > > > > Тел. [phone]
> > > > >
> > > > > Best regards,
> > > > > Andrey V. Mashenkov
> > > > > Cerr: [phone]
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Thanks,
> > > > Alexandr Kuramshin
> > > >
> > >
> >
> >
> >
> > --
> > Thanks,
> > Alexandr Kuramshin
> >
>

--
Thanks,
Alexandr Kuramshin