Improve logging of Data Region configuration

[Ilya Kasnacheev-2]

Hello!

It came to my attention that we output data regions' configurations twice
when starting node, but we never output list of data regions (including
system, etc) that were actually started.

First we have IgniteConfiguration printed (quiet=false):
2019-07-24 02:33:33.918[INFO ][Thread-139][o.a.i.i.IgniteKernal%GridNodeName
] IgniteConfiguration [... dfltDataRegConf=DataRegionConfiguration [name=
mem_plc, maxSize=635655159808, initSize=268435456, swapPath=null,
pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100,
metricsEnabled=true, metricsSubIntervalCount=5, metricsRateTimeInterval=1000
, persistenceEnabled=true, checkpointPageBufSize=17179869184], storagePath=/
ssd/data, checkpointFreq=30000, lockWaitTime=10000, checkpointThreads=4,
checkpointWriteOrder=SEQUENTIAL, walHistSize=2147483647, walSegments=10,
walSegmentSize=1073741824, walPath=/ssd/data/wal, walArchivePath=/sas/
wal_archive, metricsEnabled=false, walMode=LOG_ONLY, walTlbSize=131072,
walBuffSize=5242880, walFlushFreq=2000, walFsyncDelay=1000,
walRecordIterBuffSize=67108864, alwaysWriteFullPages=false, fileIOFactory=
org.apache.ignite.internal.processors.cache.persistence.file.
AsyncFileIOFactory@3612c49a, metricsSubIntervalCnt=5,
metricsRateTimeInterval=60000, walAutoArchiveAfterInactivity=-1,
writeThrottlingEnabled=false, walCompactionEnabled=true, walCompactionLevel=
1], ...]

Then we have all configured Data Regions printed per IGNITE-8803
(quiet=true):
 [11:30:36] Data Regions Configured:
 [11:30:36]  ^-- plcWithMetrics [initSize=256,0 MiB, maxSize=6,3 GiB,
persistence=false, lazyMemoryAllocation=true]
 [11:30:36]  ^-- plcNoMetrics [initSize=256,0 MiB, maxSize=6,3 GiB,
persistence=false, lazyMemoryAllocation=true]

Then we print number of Data Regions that were initialized as per
IGNITE-7196, but not regions themselves (quiet=false):
Configured data regions initialized successfully [total=4]

I propose to keep the first one (IgniteConfiguration), remove the second
one (Data Regions Configured), and promote the last one to quiet mode while
also outputting the regions themselves like this:
 [11:30:36] Data Regions Initialized Successfully: 4
 [11:30:36]  ^-- plcWithMetrics [initSize=256,0 MiB, maxSize=6,3 GiB,
persistence=true, lazyMemoryAllocation=true]
 [11:30:36]  ^-- plcNoMetrics [initSize=256,0 MiB, maxSize=6,3 GiB,
persistence=true, lazyMemoryAllocation=true]
 [11:30:36]  ^-- sysMemPlc [initSize=40,0 MiB, maxSize=100,0 MiB,
persistence=true, lazyMemoryAllocation=false]
 [11:30:36]  ^-- volatileMemPlc [initSize=40,0 MiB, maxSize=100,0 MiB,
persistence=false, lazyMemoryAllocation=true]

(maybe it will also include information about current usage of region in
line with IGNITE-9305's "Metrics for local node"

WDYT?

Regards,

[Ivan Pavlukhin]

Ilya,

Indeed the matters can be improved.

Is not it useful to mark what region is default? Also some doubts
about internal memory regions. It is not obvious that we should print
an information about them for every user. If we need to have some
determinism about offheap memory than I can think about logging
amounts for internal needs of total ones (a sum for all regions).

вт, 24 дек. 2019 г. в 15:38, Ilya Kasnacheev <[hidden email]>:

>
> Hello!
>
> It came to my attention that we output data regions' configurations twice
> when starting node, but we never output list of data regions (including
> system, etc) that were actually started.
>
> First we have IgniteConfiguration printed (quiet=false):
> 2019-07-24 02:33:33.918[INFO ][Thread-139][o.a.i.i.IgniteKernal%GridNodeName
> ] IgniteConfiguration [... dfltDataRegConf=DataRegionConfiguration [name=
> mem_plc, maxSize=635655159808, initSize=268435456, swapPath=null,
> pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100,
> metricsEnabled=true, metricsSubIntervalCount=5, metricsRateTimeInterval=1000
> , persistenceEnabled=true, checkpointPageBufSize=17179869184], storagePath=/
> ssd/data, checkpointFreq=30000, lockWaitTime=10000, checkpointThreads=4,
> checkpointWriteOrder=SEQUENTIAL, walHistSize=2147483647, walSegments=10,
> walSegmentSize=1073741824, walPath=/ssd/data/wal, walArchivePath=/sas/
> wal_archive, metricsEnabled=false, walMode=LOG_ONLY, walTlbSize=131072,
> walBuffSize=5242880, walFlushFreq=2000, walFsyncDelay=1000,
> walRecordIterBuffSize=67108864, alwaysWriteFullPages=false, fileIOFactory=
> org.apache.ignite.internal.processors.cache.persistence.file.
> AsyncFileIOFactory@3612c49a, metricsSubIntervalCnt=5,
> metricsRateTimeInterval=60000, walAutoArchiveAfterInactivity=-1,
> writeThrottlingEnabled=false, walCompactionEnabled=true, walCompactionLevel=
> 1], ...]
>
> Then we have all configured Data Regions printed per IGNITE-8803
> (quiet=true):
>  [11:30:36] Data Regions Configured:
>  [11:30:36]  ^-- plcWithMetrics [initSize=256,0 MiB, maxSize=6,3 GiB,
> persistence=false, lazyMemoryAllocation=true]
>  [11:30:36]  ^-- plcNoMetrics [initSize=256,0 MiB, maxSize=6,3 GiB,
> persistence=false, lazyMemoryAllocation=true]
>
> Then we print number of Data Regions that were initialized as per
> IGNITE-7196, but not regions themselves (quiet=false):
> Configured data regions initialized successfully [total=4]
>
> I propose to keep the first one (IgniteConfiguration), remove the second
> one (Data Regions Configured), and promote the last one to quiet mode while
> also outputting the regions themselves like this:
>  [11:30:36] Data Regions Initialized Successfully: 4
>  [11:30:36]  ^-- plcWithMetrics [initSize=256,0 MiB, maxSize=6,3 GiB,
> persistence=true, lazyMemoryAllocation=true]
>  [11:30:36]  ^-- plcNoMetrics [initSize=256,0 MiB, maxSize=6,3 GiB,
> persistence=true, lazyMemoryAllocation=true]
>  [11:30:36]  ^-- sysMemPlc [initSize=40,0 MiB, maxSize=100,0 MiB,
> persistence=true, lazyMemoryAllocation=false]
>  [11:30:36]  ^-- volatileMemPlc [initSize=40,0 MiB, maxSize=100,0 MiB,
> persistence=false, lazyMemoryAllocation=true]
>
> (maybe it will also include information about current usage of region in
> line with IGNITE-9305's "Metrics for local node"
>
> WDYT?
>
> Regards,

--
Best regards,
Ivan Pavlukhin

[Ilya Kasnacheev]

Hello!

Okay, I will mark default region.

We already log information about internal memory regions, so removing it
will require a lot of consensus. However, I can clearly map them as system
regions when printing.

Regards,
--
Ilya Kasnacheev


чт, 26 дек. 2019 г. в 12:10, Ivan Pavlukhin <[hidden email]>:

> Ilya,
>
> Indeed the matters can be improved.
>
> Is not it useful to mark what region is default? Also some doubts
> about internal memory regions. It is not obvious that we should print
> an information about them for every user. If we need to have some
> determinism about offheap memory than I can think about logging
> amounts for internal needs of total ones (a sum for all regions).
>
> вт, 24 дек. 2019 г. в 15:38, Ilya Kasnacheev <[hidden email]>:
> >
> > Hello!
> >
> > It came to my attention that we output data regions' configurations twice
> > when starting node, but we never output list of data regions (including
> > system, etc) that were actually started.
> >
> > First we have IgniteConfiguration printed (quiet=false):
> > 2019-07-24 02:33:33.918[INFO
> ][Thread-139][o.a.i.i.IgniteKernal%GridNodeName
> > ] IgniteConfiguration [... dfltDataRegConf=DataRegionConfiguration [name=
> > mem_plc, maxSize=635655159808, initSize=268435456, swapPath=null,
> > pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100,
> > metricsEnabled=true, metricsSubIntervalCount=5,
> metricsRateTimeInterval=1000
> > , persistenceEnabled=true, checkpointPageBufSize=17179869184],
> storagePath=/
> > ssd/data, checkpointFreq=30000, lockWaitTime=10000, checkpointThreads=4,
> > checkpointWriteOrder=SEQUENTIAL, walHistSize=2147483647, walSegments=10,
> > walSegmentSize=1073741824, walPath=/ssd/data/wal, walArchivePath=/sas/
> > wal_archive, metricsEnabled=false, walMode=LOG_ONLY, walTlbSize=131072,
> > walBuffSize=5242880, walFlushFreq=2000, walFsyncDelay=1000,
> > walRecordIterBuffSize=67108864, alwaysWriteFullPages=false,
> fileIOFactory=
> > org.apache.ignite.internal.processors.cache.persistence.file.
> > AsyncFileIOFactory@3612c49a, metricsSubIntervalCnt=5,
> > metricsRateTimeInterval=60000, walAutoArchiveAfterInactivity=-1,
> > writeThrottlingEnabled=false, walCompactionEnabled=true,
> walCompactionLevel=
> > 1], ...]
> >
> > Then we have all configured Data Regions printed per IGNITE-8803
> > (quiet=true):
> >  [11:30:36] Data Regions Configured:
> >  [11:30:36]  ^-- plcWithMetrics [initSize=256,0 MiB, maxSize=6,3 GiB,
> > persistence=false, lazyMemoryAllocation=true]
> >  [11:30:36]  ^-- plcNoMetrics [initSize=256,0 MiB, maxSize=6,3 GiB,
> > persistence=false, lazyMemoryAllocation=true]
> >
> > Then we print number of Data Regions that were initialized as per
> > IGNITE-7196, but not regions themselves (quiet=false):
> > Configured data regions initialized successfully [total=4]
> >
> > I propose to keep the first one (IgniteConfiguration), remove the second
> > one (Data Regions Configured), and promote the last one to quiet mode
> while
> > also outputting the regions themselves like this:
> >  [11:30:36] Data Regions Initialized Successfully: 4
> >  [11:30:36]  ^-- plcWithMetrics [initSize=256,0 MiB, maxSize=6,3 GiB,
> > persistence=true, lazyMemoryAllocation=true]
> >  [11:30:36]  ^-- plcNoMetrics [initSize=256,0 MiB, maxSize=6,3 GiB,
> > persistence=true, lazyMemoryAllocation=true]
> >  [11:30:36]  ^-- sysMemPlc [initSize=40,0 MiB, maxSize=100,0 MiB,
> > persistence=true, lazyMemoryAllocation=false]
> >  [11:30:36]  ^-- volatileMemPlc [initSize=40,0 MiB, maxSize=100,0 MiB,
> > persistence=false, lazyMemoryAllocation=true]
> >
> > (maybe it will also include information about current usage of region in
> > line with IGNITE-9305's "Metrics for local node"
> >
> > WDYT?
> >
> > Regards,
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>

[Ilya Kasnacheev]

Hello!

I have filed a ticket about this improvement, plan to start on coding.

https://issues.apache.org/jira/browse/IGNITE-12505

Regards,
--
Ilya Kasnacheev


чт, 26 дек. 2019 г. в 12:50, Ilya Kasnacheev <[hidden email]>:

> Hello!
>
> Okay, I will mark default region.
>
> We already log information about internal memory regions, so removing it
> will require a lot of consensus. However, I can clearly map them as system
> regions when printing.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> чт, 26 дек. 2019 г. в 12:10, Ivan Pavlukhin <[hidden email]>:
>
>> Ilya,
>>
>> Indeed the matters can be improved.
>>
>> Is not it useful to mark what region is default? Also some doubts
>> about internal memory regions. It is not obvious that we should print
>> an information about them for every user. If we need to have some
>> determinism about offheap memory than I can think about logging
>> amounts for internal needs of total ones (a sum for all regions).
>>
>> вт, 24 дек. 2019 г. в 15:38, Ilya Kasnacheev <[hidden email]>:
>> >
>> > Hello!
>> >
>> > It came to my attention that we output data regions' configurations
>> twice
>> > when starting node, but we never output list of data regions (including
>> > system, etc) that were actually started.
>> >
>> > First we have IgniteConfiguration printed (quiet=false):
>> > 2019-07-24 02:33:33.918[INFO
>> ][Thread-139][o.a.i.i.IgniteKernal%GridNodeName
>> > ] IgniteConfiguration [... dfltDataRegConf=DataRegionConfiguration
>> [name=
>> > mem_plc, maxSize=635655159808, initSize=268435456, swapPath=null,
>> > pageEvictionMode=DISABLED, evictionThreshold=0.9,
>> emptyPagesPoolSize=100,
>> > metricsEnabled=true, metricsSubIntervalCount=5,
>> metricsRateTimeInterval=1000
>> > , persistenceEnabled=true, checkpointPageBufSize=17179869184],
>> storagePath=/
>> > ssd/data, checkpointFreq=30000, lockWaitTime=10000, checkpointThreads=4,
>> > checkpointWriteOrder=SEQUENTIAL, walHistSize=2147483647, walSegments=10,
>> > walSegmentSize=1073741824, walPath=/ssd/data/wal, walArchivePath=/sas/
>> > wal_archive, metricsEnabled=false, walMode=LOG_ONLY, walTlbSize=131072,
>> > walBuffSize=5242880, walFlushFreq=2000, walFsyncDelay=1000,
>> > walRecordIterBuffSize=67108864, alwaysWriteFullPages=false,
>> fileIOFactory=
>> > org.apache.ignite.internal.processors.cache.persistence.file.
>> > AsyncFileIOFactory@3612c49a, metricsSubIntervalCnt=5,
>> > metricsRateTimeInterval=60000, walAutoArchiveAfterInactivity=-1,
>> > writeThrottlingEnabled=false, walCompactionEnabled=true,
>> walCompactionLevel=
>> > 1], ...]
>> >
>> > Then we have all configured Data Regions printed per IGNITE-8803
>> > (quiet=true):
>> >  [11:30:36] Data Regions Configured:
>> >  [11:30:36]  ^-- plcWithMetrics [initSize=256,0 MiB, maxSize=6,3 GiB,
>> > persistence=false, lazyMemoryAllocation=true]
>> >  [11:30:36]  ^-- plcNoMetrics [initSize=256,0 MiB, maxSize=6,3 GiB,
>> > persistence=false, lazyMemoryAllocation=true]
>> >
>> > Then we print number of Data Regions that were initialized as per
>> > IGNITE-7196, but not regions themselves (quiet=false):
>> > Configured data regions initialized successfully [total=4]
>> >
>> > I propose to keep the first one (IgniteConfiguration), remove the second
>> > one (Data Regions Configured), and promote the last one to quiet mode
>> while
>> > also outputting the regions themselves like this:
>> >  [11:30:36] Data Regions Initialized Successfully: 4
>> >  [11:30:36]  ^-- plcWithMetrics [initSize=256,0 MiB, maxSize=6,3 GiB,
>> > persistence=true, lazyMemoryAllocation=true]
>> >  [11:30:36]  ^-- plcNoMetrics [initSize=256,0 MiB, maxSize=6,3 GiB,
>> > persistence=true, lazyMemoryAllocation=true]
>> >  [11:30:36]  ^-- sysMemPlc [initSize=40,0 MiB, maxSize=100,0 MiB,
>> > persistence=true, lazyMemoryAllocation=false]
>> >  [11:30:36]  ^-- volatileMemPlc [initSize=40,0 MiB, maxSize=100,0 MiB,
>> > persistence=false, lazyMemoryAllocation=true]
>> >
>> > (maybe it will also include information about current usage of region in
>> > line with IGNITE-9305's "Metrics for local node"
>> >
>> > WDYT?
>> >
>> > Regards,
>>
>>
>>
>> --
>> Best regards,
>> Ivan Pavlukhin
>>
>

[Ilya Kasnacheev]

Hello!

I have prepared a patch, please review. It will look like this:

Data Regions Started: 4
    ^--   sysMemPlc region [type=internal, persistence=true, lazyAlloc=false,
      ...  initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%,
allocRam=100MB, allocTotal=0MB]
    ^--   default region [type=default, persistence=true, lazyAlloc=true,
      ...  initCfg=30MB, maxCfg=30MB, usedRam=0MB, freeRam=100%,
allocRam=0MB, allocTotal=0MB]
    ^--   metastoreMemPlc region [type=internal, persistence=true,
lazyAlloc=false,
      ...  initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%,
allocRam=100MB, allocTotal=0MB]
    ^--   TxLog region [type=internal, persistence=true, lazyAlloc=false,
      ...  initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%,
allocRam=100MB, allocTotal=0MB]
    ^-- Ignite persistence [used=0MB]
...
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=9e6c8e45,
name=internal.GridNodeMetricsLogPdsSelfTest0, uptime=00:00:12.271]
    ^-- Cluster [hosts=1, CPUs=8, servers=2, clients=0, topVer=2, minorTopVer=3]
    ^-- Network [addrs=[127.0.0.1], localHost=127.0.0.1,
discoPort=47500, commPort=45010]
    ^-- CPU [CPUs=8, curLoad=2.7%, avgLoad=3.38%, GC=0%]
    ^-- Heap [used=48MB, free=99.32%, comm=564MB]
    ^-- Off-heap memory [used=0MB, free=99.96%, allocated=230MB]
    ^-- Page memory [pages=34]
    ^--   sysMemPlc region [type=internal, persistence=true, lazyAlloc=false,
      ...  initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.98%,
allocRam=100MB, allocTotal=0MB]
    ^--   default region [type=default, persistence=true, lazyAlloc=true,
      ...  initCfg=30MB, maxCfg=30MB, usedRam=0MB, freeRam=99.75%,
allocRam=30MB, allocTotal=0MB]
    ^--   metastoreMemPlc region [type=internal, persistence=true,
lazyAlloc=false,
      ...  initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.96%,
allocRam=0MB, allocTotal=0MB]
    ^--   TxLog region [type=internal, persistence=true, lazyAlloc=false,
      ...  initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%,
allocRam=100MB, allocTotal=0MB]
    ^-- Ignite persistence [used=0MB]
    ^-- Outbound messages queue [size=0]
    ^-- Public thread pool [active=0, idle=6, qSize=0]
    ^-- System thread pool [active=0, idle=8, qSize=0]
    ^-- Custom executor 0 [active=0, idle=0, qSize=0]
    ^-- Custom executor 1 [active=0, idle=0, qSize=0]


Regards,
--
Ilya Kasnacheev


пт, 27 дек. 2019 г. в 15:03, Ilya Kasnacheev <[hidden email]>:

> Hello!
>
> I have filed a ticket about this improvement, plan to start on coding.
>
> https://issues.apache.org/jira/browse/IGNITE-12505
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> чт, 26 дек. 2019 г. в 12:50, Ilya Kasnacheev <[hidden email]>:
>
>> Hello!
>>
>> Okay, I will mark default region.
>>
>> We already log information about internal memory regions, so removing it
>> will require a lot of consensus. However, I can clearly map them as system
>> regions when printing.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> чт, 26 дек. 2019 г. в 12:10, Ivan Pavlukhin <[hidden email]>:
>>
>>> Ilya,
>>>
>>> Indeed the matters can be improved.
>>>
>>> Is not it useful to mark what region is default? Also some doubts
>>> about internal memory regions. It is not obvious that we should print
>>> an information about them for every user. If we need to have some
>>> determinism about offheap memory than I can think about logging
>>> amounts for internal needs of total ones (a sum for all regions).
>>>
>>> вт, 24 дек. 2019 г. в 15:38, Ilya Kasnacheev <[hidden email]>:
>>> >
>>> > Hello!
>>> >
>>> > It came to my attention that we output data regions' configurations
>>> twice
>>> > when starting node, but we never output list of data regions (including
>>> > system, etc) that were actually started.
>>> >
>>> > First we have IgniteConfiguration printed (quiet=false):
>>> > 2019-07-24 02:33:33.918[INFO
>>> ][Thread-139][o.a.i.i.IgniteKernal%GridNodeName
>>> > ] IgniteConfiguration [... dfltDataRegConf=DataRegionConfiguration
>>> [name=
>>> > mem_plc, maxSize=635655159808, initSize=268435456, swapPath=null,
>>> > pageEvictionMode=DISABLED, evictionThreshold=0.9,
>>> emptyPagesPoolSize=100,
>>> > metricsEnabled=true, metricsSubIntervalCount=5,
>>> metricsRateTimeInterval=1000
>>> > , persistenceEnabled=true, checkpointPageBufSize=17179869184],
>>> storagePath=/
>>> > ssd/data, checkpointFreq=30000, lockWaitTime=10000,
>>> checkpointThreads=4,
>>> > checkpointWriteOrder=SEQUENTIAL, walHistSize=2147483647,
>>> walSegments=10,
>>> > walSegmentSize=1073741824, walPath=/ssd/data/wal, walArchivePath=/sas/
>>> > wal_archive, metricsEnabled=false, walMode=LOG_ONLY, walTlbSize=131072,
>>> > walBuffSize=5242880, walFlushFreq=2000, walFsyncDelay=1000,
>>> > walRecordIterBuffSize=67108864, alwaysWriteFullPages=false,
>>> fileIOFactory=
>>> > org.apache.ignite.internal.processors.cache.persistence.file.
>>> > AsyncFileIOFactory@3612c49a, metricsSubIntervalCnt=5,
>>> > metricsRateTimeInterval=60000, walAutoArchiveAfterInactivity=-1,
>>> > writeThrottlingEnabled=false, walCompactionEnabled=true,
>>> walCompactionLevel=
>>> > 1], ...]
>>> >
>>> > Then we have all configured Data Regions printed per IGNITE-8803
>>> > (quiet=true):
>>> >  [11:30:36] Data Regions Configured:
>>> >  [11:30:36]  ^-- plcWithMetrics [initSize=256,0 MiB, maxSize=6,3 GiB,
>>> > persistence=false, lazyMemoryAllocation=true]
>>> >  [11:30:36]  ^-- plcNoMetrics [initSize=256,0 MiB, maxSize=6,3 GiB,
>>> > persistence=false, lazyMemoryAllocation=true]
>>> >
>>> > Then we print number of Data Regions that were initialized as per
>>> > IGNITE-7196, but not regions themselves (quiet=false):
>>> > Configured data regions initialized successfully [total=4]
>>> >
>>> > I propose to keep the first one (IgniteConfiguration), remove the
>>> second
>>> > one (Data Regions Configured), and promote the last one to quiet mode
>>> while
>>> > also outputting the regions themselves like this:
>>> >  [11:30:36] Data Regions Initialized Successfully: 4
>>> >  [11:30:36]  ^-- plcWithMetrics [initSize=256,0 MiB, maxSize=6,3 GiB,
>>> > persistence=true, lazyMemoryAllocation=true]
>>> >  [11:30:36]  ^-- plcNoMetrics [initSize=256,0 MiB, maxSize=6,3 GiB,
>>> > persistence=true, lazyMemoryAllocation=true]
>>> >  [11:30:36]  ^-- sysMemPlc [initSize=40,0 MiB, maxSize=100,0 MiB,
>>> > persistence=true, lazyMemoryAllocation=false]
>>> >  [11:30:36]  ^-- volatileMemPlc [initSize=40,0 MiB, maxSize=100,0 MiB,
>>> > persistence=false, lazyMemoryAllocation=true]
>>> >
>>> > (maybe it will also include information about current usage of region
>>> in
>>> > line with IGNITE-9305's "Metrics for local node"
>>> >
>>> > WDYT?
>>> >
>>> > Regards,
>>>
>>>
>>>
>>> --
>>> Best regards,
>>> Ivan Pavlukhin
>>>
>>

[Ilya Kasnacheev]

Hello again!

TC passes, review is ready.

Your last chance to act before merge!

Regards,
--
Ilya Kasnacheev


вт, 31 дек. 2019 г. в 16:01, Ilya Kasnacheev <[hidden email]>:

> Hello!
>
> I have prepared a patch, please review. It will look like this:
>
> Data Regions Started: 4
>     ^--   sysMemPlc region [type=internal, persistence=true, lazyAlloc=false,
>       ...  initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, allocRam=100MB, allocTotal=0MB]
>     ^--   default region [type=default, persistence=true, lazyAlloc=true,
>       ...  initCfg=30MB, maxCfg=30MB, usedRam=0MB, freeRam=100%, allocRam=0MB, allocTotal=0MB]
>     ^--   metastoreMemPlc region [type=internal, persistence=true, lazyAlloc=false,
>       ...  initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, allocRam=100MB, allocTotal=0MB]
>     ^--   TxLog region [type=internal, persistence=true, lazyAlloc=false,
>       ...  initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, allocRam=100MB, allocTotal=0MB]
>     ^-- Ignite persistence [used=0MB]
> ...
> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
>     ^-- Node [id=9e6c8e45, name=internal.GridNodeMetricsLogPdsSelfTest0, uptime=00:00:12.271]
>     ^-- Cluster [hosts=1, CPUs=8, servers=2, clients=0, topVer=2, minorTopVer=3]
>     ^-- Network [addrs=[127.0.0.1], localHost=127.0.0.1, discoPort=47500, commPort=45010]
>     ^-- CPU [CPUs=8, curLoad=2.7%, avgLoad=3.38%, GC=0%]
>     ^-- Heap [used=48MB, free=99.32%, comm=564MB]
>     ^-- Off-heap memory [used=0MB, free=99.96%, allocated=230MB]
>     ^-- Page memory [pages=34]
>     ^--   sysMemPlc region [type=internal, persistence=true, lazyAlloc=false,
>       ...  initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.98%, allocRam=100MB, allocTotal=0MB]
>     ^--   default region [type=default, persistence=true, lazyAlloc=true,
>       ...  initCfg=30MB, maxCfg=30MB, usedRam=0MB, freeRam=99.75%, allocRam=30MB, allocTotal=0MB]
>     ^--   metastoreMemPlc region [type=internal, persistence=true, lazyAlloc=false,
>       ...  initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.96%, allocRam=0MB, allocTotal=0MB]
>     ^--   TxLog region [type=internal, persistence=true, lazyAlloc=false,
>       ...  initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, allocRam=100MB, allocTotal=0MB]
>     ^-- Ignite persistence [used=0MB]
>     ^-- Outbound messages queue [size=0]
>     ^-- Public thread pool [active=0, idle=6, qSize=0]
>     ^-- System thread pool [active=0, idle=8, qSize=0]
>     ^-- Custom executor 0 [active=0, idle=0, qSize=0]
>     ^-- Custom executor 1 [active=0, idle=0, qSize=0]
>
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 27 дек. 2019 г. в 15:03, Ilya Kasnacheev <[hidden email]>:
>
>> Hello!
>>
>> I have filed a ticket about this improvement, plan to start on coding.
>>
>> https://issues.apache.org/jira/browse/IGNITE-12505
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> чт, 26 дек. 2019 г. в 12:50, Ilya Kasnacheev <[hidden email]>:
>>
>>> Hello!
>>>
>>> Okay, I will mark default region.
>>>
>>> We already log information about internal memory regions, so removing it
>>> will require a lot of consensus. However, I can clearly map them as system
>>> regions when printing.
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> чт, 26 дек. 2019 г. в 12:10, Ivan Pavlukhin <[hidden email]>:
>>>
>>>> Ilya,
>>>>
>>>> Indeed the matters can be improved.
>>>>
>>>> Is not it useful to mark what region is default? Also some doubts
>>>> about internal memory regions. It is not obvious that we should print
>>>> an information about them for every user. If we need to have some
>>>> determinism about offheap memory than I can think about logging
>>>> amounts for internal needs of total ones (a sum for all regions).
>>>>
>>>> вт, 24 дек. 2019 г. в 15:38, Ilya Kasnacheev <[hidden email]>:
>>>> >
>>>> > Hello!
>>>> >
>>>> > It came to my attention that we output data regions' configurations
>>>> twice
>>>> > when starting node, but we never output list of data regions
>>>> (including
>>>> > system, etc) that were actually started.
>>>> >
>>>> > First we have IgniteConfiguration printed (quiet=false):
>>>> > 2019-07-24 02:33:33.918[INFO
>>>> ][Thread-139][o.a.i.i.IgniteKernal%GridNodeName
>>>> > ] IgniteConfiguration [... dfltDataRegConf=DataRegionConfiguration
>>>> [name=
>>>> > mem_plc, maxSize=635655159808, initSize=268435456, swapPath=null,
>>>> > pageEvictionMode=DISABLED, evictionThreshold=0.9,
>>>> emptyPagesPoolSize=100,
>>>> > metricsEnabled=true, metricsSubIntervalCount=5,
>>>> metricsRateTimeInterval=1000
>>>> > , persistenceEnabled=true, checkpointPageBufSize=17179869184],
>>>> storagePath=/
>>>> > ssd/data, checkpointFreq=30000, lockWaitTime=10000,
>>>> checkpointThreads=4,
>>>> > checkpointWriteOrder=SEQUENTIAL, walHistSize=2147483647,
>>>> walSegments=10,
>>>> > walSegmentSize=1073741824, walPath=/ssd/data/wal, walArchivePath=/sas/
>>>> > wal_archive, metricsEnabled=false, walMode=LOG_ONLY,
>>>> walTlbSize=131072,
>>>> > walBuffSize=5242880, walFlushFreq=2000, walFsyncDelay=1000,
>>>> > walRecordIterBuffSize=67108864, alwaysWriteFullPages=false,
>>>> fileIOFactory=
>>>> > org.apache.ignite.internal.processors.cache.persistence.file.
>>>> > AsyncFileIOFactory@3612c49a, metricsSubIntervalCnt=5,
>>>> > metricsRateTimeInterval=60000, walAutoArchiveAfterInactivity=-1,
>>>> > writeThrottlingEnabled=false, walCompactionEnabled=true,
>>>> walCompactionLevel=
>>>> > 1], ...]
>>>> >
>>>> > Then we have all configured Data Regions printed per IGNITE-8803
>>>> > (quiet=true):
>>>> >  [11:30:36] Data Regions Configured:
>>>> >  [11:30:36]  ^-- plcWithMetrics [initSize=256,0 MiB, maxSize=6,3 GiB,
>>>> > persistence=false, lazyMemoryAllocation=true]
>>>> >  [11:30:36]  ^-- plcNoMetrics [initSize=256,0 MiB, maxSize=6,3 GiB,
>>>> > persistence=false, lazyMemoryAllocation=true]
>>>> >
>>>> > Then we print number of Data Regions that were initialized as per
>>>> > IGNITE-7196, but not regions themselves (quiet=false):
>>>> > Configured data regions initialized successfully [total=4]
>>>> >
>>>> > I propose to keep the first one (IgniteConfiguration), remove the
>>>> second
>>>> > one (Data Regions Configured), and promote the last one to quiet mode
>>>> while
>>>> > also outputting the regions themselves like this:
>>>> >  [11:30:36] Data Regions Initialized Successfully: 4
>>>> >  [11:30:36]  ^-- plcWithMetrics [initSize=256,0 MiB, maxSize=6,3 GiB,
>>>> > persistence=true, lazyMemoryAllocation=true]
>>>> >  [11:30:36]  ^-- plcNoMetrics [initSize=256,0 MiB, maxSize=6,3 GiB,
>>>> > persistence=true, lazyMemoryAllocation=true]
>>>> >  [11:30:36]  ^-- sysMemPlc [initSize=40,0 MiB, maxSize=100,0 MiB,
>>>> > persistence=true, lazyMemoryAllocation=false]
>>>> >  [11:30:36]  ^-- volatileMemPlc [initSize=40,0 MiB, maxSize=100,0 MiB,
>>>> > persistence=false, lazyMemoryAllocation=true]
>>>> >
>>>> > (maybe it will also include information about current usage of region
>>>> in
>>>> > line with IGNITE-9305's "Metrics for local node"
>>>> >
>>>> > WDYT?
>>>> >
>>>> > Regards,
>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>> Ivan Pavlukhin
>>>>
>>>