Ignite PDS WAL analysis with human readable records

[Dmitriy Pavlov]

Hi Igniters,

Eduard Shangareev created WAL converter utility which shows WAL
archive/work directory segments content in human readable form. This
utility outputs all WAL records to output stream (may be redirected to
file). How to use tips can be found in issue
https://issues.apache.org/jira/browse/IGNITE-6277


This utility does not require Ignite node to be up and running and may be
used for offline analysis of copied files. Thanks to Alexey G. for idea and
review. Code is now available in modules/ignite-dev-tools.

1. Could you please write if utility can be useful for you?

2. Please respond if you need short wiki description.

3. What is your opinion what should be next steps to develop this utility?

4. Does utility need property file to refer Ignite Work directory using
conf file.

5. What do you think if utility remains console non interactive, or some
UI/Interactive console shell may be preferable?

Sincerely,

Dmitriy Pavlov

[dmagda]

Dmitriy, Eduard,

Sounds like a useful thing. I’ll be happy to give a feedback but need a documentation that explains how to use the tool, what’s the output format, how do decipher the output, etc (basic things). The Wiki description should be enough.

—
Denis

> On Sep 15, 2017, at 11:36 AM, Dmitry Pavlov <[hidden email]> wrote:
>
> Hi Igniters,
>
> Eduard Shangareev created WAL converter utility which shows WAL
> archive/work directory segments content in human readable form. This
> utility outputs all WAL records to output stream (may be redirected to
> file). How to use tips can be found in issue
> https://issues.apache.org/jira/browse/IGNITE-6277
>
>
> This utility does not require Ignite node to be up and running and may be
> used for offline analysis of copied files. Thanks to Alexey G. for idea and
> review. Code is now available in modules/ignite-dev-tools.
>
> 1. Could you please write if utility can be useful for you?
>
> 2. Please respond if you need short wiki description.
>
> 3. What is your opinion what should be next steps to develop this utility?
>
> 4. Does utility need property file to refer Ignite Work directory using
> conf file.
>
> 5. What do you think if utility remains console non interactive, or some
> UI/Interactive console shell may be preferable?
>
> Sincerely,
>
> Dmitriy Pavlov

[Dmitriy Pavlov]

Hi Denis,

Thank you for your feedback. It is very important to us to understand that
this tool may be useful not only for Ignite tests but for all developers.

I will create wiki page.

Sincerely,
Dmitriy Pavlov

сб, 16 сент. 2017 г. в 1:54, Denis Magda <[hidden email]>:

> Dmitriy, Eduard,
>
> Sounds like a useful thing. I’ll be happy to give a feedback but need a
> documentation that explains how to use the tool, what’s the output format,
> how do decipher the output, etc (basic things). The Wiki description should
> be enough.
>
> —
> Denis
>
> > On Sep 15, 2017, at 11:36 AM, Dmitry Pavlov <[hidden email]>
> wrote:
> >
> > Hi Igniters,
> >
> > Eduard Shangareev created WAL converter utility which shows WAL
> > archive/work directory segments content in human readable form. This
> > utility outputs all WAL records to output stream (may be redirected to
> > file). How to use tips can be found in issue
> > https://issues.apache.org/jira/browse/IGNITE-6277
> >
> >
> > This utility does not require Ignite node to be up and running and may be
> > used for offline analysis of copied files. Thanks to Alexey G. for idea
> and
> > review. Code is now available in modules/ignite-dev-tools.
> >
> > 1. Could you please write if utility can be useful for you?
> >
> > 2. Please respond if you need short wiki description.
> >
> > 3. What is your opinion what should be next steps to develop this
> utility?
> >
> > 4. Does utility need property file to refer Ignite Work directory using
> > conf file.
> >
> > 5. What do you think if utility remains console non interactive, or some
> > UI/Interactive console shell may be preferable?
> >
> > Sincerely,
> >
> > Dmitriy Pavlov
>
>

[Eduard Shangareev]

Dmitry,
thank you for starting this topic.

The IDEA of this utility is very simple - get the possibility to look into
your WAL.
But it also requires understanding what's going on in WAL mechanism (We
need to describe it in more details in the wiki).

It could be useful for a debug some weird things with recovery mechanism.
Initially, it was created for dealing with the issue in IGNITE-6342. It
helped to find out the error in how we deal with partition size.

Best regards,
Ed.




On Sat, Sep 16, 2017 at 2:09 AM, Dmitry Pavlov <[hidden email]>
wrote:

> Hi Denis,
>
> Thank you for your feedback. It is very important to us to understand that
> this tool may be useful not only for Ignite tests but for all developers.
>
> I will create wiki page.
>
> Sincerely,
> Dmitriy Pavlov
>
> сб, 16 сент. 2017 г. в 1:54, Denis Magda <[hidden email]>:
>
> > Dmitriy, Eduard,
> >
> > Sounds like a useful thing. I’ll be happy to give a feedback but need a
> > documentation that explains how to use the tool, what’s the output
> format,
> > how do decipher the output, etc (basic things). The Wiki description
> should
> > be enough.
> >
> > —
> > Denis
> >
> > > On Sep 15, 2017, at 11:36 AM, Dmitry Pavlov <[hidden email]>
> > wrote:
> > >
> > > Hi Igniters,
> > >
> > > Eduard Shangareev created WAL converter utility which shows WAL
> > > archive/work directory segments content in human readable form. This
> > > utility outputs all WAL records to output stream (may be redirected to
> > > file). How to use tips can be found in issue
> > > https://issues.apache.org/jira/browse/IGNITE-6277
> > >
> > >
> > > This utility does not require Ignite node to be up and running and may
> be
> > > used for offline analysis of copied files. Thanks to Alexey G. for idea
> > and
> > > review. Code is now available in modules/ignite-dev-tools.
> > >
> > > 1. Could you please write if utility can be useful for you?
> > >
> > > 2. Please respond if you need short wiki description.
> > >
> > > 3. What is your opinion what should be next steps to develop this
> > utility?
> > >
> > > 4. Does utility need property file to refer Ignite Work directory using
> > > conf file.
> > >
> > > 5. What do you think if utility remains console non interactive, or
> some
> > > UI/Interactive console shell may be preferable?
> > >
> > > Sincerely,
> > >
> > > Dmitriy Pavlov
> >
> >
>

[dsetrakyan]

Can we get a sample print out from this utility?

On Sun, Sep 17, 2017 at 3:42 PM, Eduard Shangareev <
[hidden email]> wrote:

> Dmitry,
> thank you for starting this topic.
>
> The IDEA of this utility is very simple - get the possibility to look into
> your WAL.
> But it also requires understanding what's going on in WAL mechanism (We
> need to describe it in more details in the wiki).
>
> It could be useful for a debug some weird things with recovery mechanism.
> Initially, it was created for dealing with the issue in IGNITE-6342. It
> helped to find out the error in how we deal with partition size.
>
> Best regards,
> Ed.
>
>
>
>
> On Sat, Sep 16, 2017 at 2:09 AM, Dmitry Pavlov <[hidden email]>
> wrote:
>
> > Hi Denis,
> >
> > Thank you for your feedback. It is very important to us to understand
> that
> > this tool may be useful not only for Ignite tests but for all developers.
> >
> > I will create wiki page.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > сб, 16 сент. 2017 г. в 1:54, Denis Magda <[hidden email]>:
> >
> > > Dmitriy, Eduard,
> > >
> > > Sounds like a useful thing. I’ll be happy to give a feedback but need a
> > > documentation that explains how to use the tool, what’s the output
> > format,
> > > how do decipher the output, etc (basic things). The Wiki description
> > should
> > > be enough.
> > >
> > > —
> > > Denis
> > >
> > > > On Sep 15, 2017, at 11:36 AM, Dmitry Pavlov <[hidden email]>
> > > wrote:
> > > >
> > > > Hi Igniters,
> > > >
> > > > Eduard Shangareev created WAL converter utility which shows WAL
> > > > archive/work directory segments content in human readable form. This
> > > > utility outputs all WAL records to output stream (may be redirected
> to
> > > > file). How to use tips can be found in issue
> > > > https://issues.apache.org/jira/browse/IGNITE-6277
> > > >
> > > >
> > > > This utility does not require Ignite node to be up and running and
> may
> > be
> > > > used for offline analysis of copied files. Thanks to Alexey G. for
> idea
> > > and
> > > > review. Code is now available in modules/ignite-dev-tools.
> > > >
> > > > 1. Could you please write if utility can be useful for you?
> > > >
> > > > 2. Please respond if you need short wiki description.
> > > >
> > > > 3. What is your opinion what should be next steps to develop this
> > > utility?
> > > >
> > > > 4. Does utility need property file to refer Ignite Work directory
> using
> > > > conf file.
> > > >
> > > > 5. What do you think if utility remains console non interactive, or
> > some
> > > > UI/Interactive console shell may be preferable?
> > > >
> > > > Sincerely,
> > > >
> > > > Dmitriy Pavlov
> > >
> > >
> >
>

[Dmitriy Pavlov]

Hi Igniters,

Page https://cwiki.apache.org/confluence/display/IGNITE/Ignite+WAL+reader to
collect related information about this prototype was created.

Sincerely,
Dmitriy Pavlov

пн, 18 сент. 2017 г. в 7:01, Dmitriy Setrakyan <[hidden email]>:

> Can we get a sample print out from this utility?
>
> On Sun, Sep 17, 2017 at 3:42 PM, Eduard Shangareev <
> [hidden email]> wrote:
>
> > Dmitry,
> > thank you for starting this topic.
> >
> > The IDEA of this utility is very simple - get the possibility to look
> into
> > your WAL.
> > But it also requires understanding what's going on in WAL mechanism (We
> > need to describe it in more details in the wiki).
> >
> > It could be useful for a debug some weird things with recovery mechanism.
> > Initially, it was created for dealing with the issue in IGNITE-6342. It
> > helped to find out the error in how we deal with partition size.
> >
> > Best regards,
> > Ed.
> >
> >
> >
> >
> > On Sat, Sep 16, 2017 at 2:09 AM, Dmitry Pavlov <[hidden email]>
> > wrote:
> >
> > > Hi Denis,
> > >
> > > Thank you for your feedback. It is very important to us to understand
> > that
> > > this tool may be useful not only for Ignite tests but for all
> developers.
> > >
> > > I will create wiki page.
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > > сб, 16 сент. 2017 г. в 1:54, Denis Magda <[hidden email]>:
> > >
> > > > Dmitriy, Eduard,
> > > >
> > > > Sounds like a useful thing. I’ll be happy to give a feedback but
> need a
> > > > documentation that explains how to use the tool, what’s the output
> > > format,
> > > > how do decipher the output, etc (basic things). The Wiki description
> > > should
> > > > be enough.
> > > >
> > > > —
> > > > Denis
> > > >
> > > > > On Sep 15, 2017, at 11:36 AM, Dmitry Pavlov <[hidden email]
> >
> > > > wrote:
> > > > >
> > > > > Hi Igniters,
> > > > >
> > > > > Eduard Shangareev created WAL converter utility which shows WAL
> > > > > archive/work directory segments content in human readable form.
> This
> > > > > utility outputs all WAL records to output stream (may be redirected
> > to
> > > > > file). How to use tips can be found in issue
> > > > > https://issues.apache.org/jira/browse/IGNITE-6277
> > > > >
> > > > >
> > > > > This utility does not require Ignite node to be up and running and
> > may
> > > be
> > > > > used for offline analysis of copied files. Thanks to Alexey G. for
> > idea
> > > > and
> > > > > review. Code is now available in modules/ignite-dev-tools.
> > > > >
> > > > > 1. Could you please write if utility can be useful for you?
> > > > >
> > > > > 2. Please respond if you need short wiki description.
> > > > >
> > > > > 3. What is your opinion what should be next steps to develop this
> > > > utility?
> > > > >
> > > > > 4. Does utility need property file to refer Ignite Work directory
> > using
> > > > > conf file.
> > > > >
> > > > > 5. What do you think if utility remains console non interactive, or
> > > some
> > > > > UI/Interactive console shell may be preferable?
> > > > >
> > > > > Sincerely,
> > > > >
> > > > > Dmitriy Pavlov
> > > >
> > > >
> > >
> >
>

[Eduard Shangareev]

Example of output (will add to wiki):

[W] InitNewPageRecord [newPageId=0001001c00000067, super=PageDeltaRecord
[grpId=2141373875, pageId=0001001c00000067, super=WALRecord [size=41,
chainSize=0, pos=FileWALPointer [idx=1, fileOffset=25406, len=41,
forceFlush=false], type=INIT_NEW_PAGE_RECORD]]]
[W] InitNewPageRecord [newPageId=0001001c00000068, super=PageDeltaRecord
[grpId=2141373875, pageId=0001001c00000068, super=WALRecord [size=41,
chainSize=0, pos=FileWALPointer [idx=1, fileOffset=25447, len=41,
forceFlush=false], type=INIT_NEW_PAGE_RECORD]]]
[W] PagesListAddPageRecord [dataPageId=0001001c0000002a,
super=PageDeltaRecord [grpId=2141373875, pageId=0001001c00000068,
super=WALRecord [size=37, chainSize=0, pos=FileWALPointer [idx=1,
fileOffset=25488, len=37, forceFlush=false], type=PAGES_LIST_ADD_PAGE]]]
[W] PageSnapshot [fullPageId = FullPageId [pageId=0001001c0000002a,
effectivePageId=0000001c0000002a, grpId=2141373875], page = [
Header [
type=1 (DataPageIO),
ver=1,
crc=0,
pageId=281595235794986(offset=0, flags=1, partId=28, index=42)
],
DataPageIO [
0001001c0000002a [4049, 4002, 3955, 3908, 3861, 3814, 3767, 3720, 3673,
3626][][free=3552, actualFree=-544
]],
super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
fileOffset=25525, len=4125, forceFlush=false], type=PAGE_RECORD]]]
[W] InsertRecord [idx=12, io=CacheIdAwareDataLeafIO[ver=1],
rightId=0000000000000000, super=PageDeltaRecord [grpId=2141373875,
pageId=0001001c00000004, super=WALRecord [size=59, chainSize=0,
pos=FileWALPointer [idx=1, fileOffset=29650, len=59, forceFlush=false],
type=BTREE_PAGE_INSERT]]]
[W] DataRecord [writeEntries=[DataEntry [cacheId=-1368047377, op=CREATE,
writeVer=GridCacheVersion [topVer=116892426, order=1505412445450,
nodeOrder=2], partId=28, partCnt=69]], super=WALRecord [size=100,
chainSize=0, pos=FileWALPointer [idx=1, fileOffset=29709, len=100,
forceFlush=false], type=DATA_RECORD]]
[W] PageSnapshot [fullPageId = FullPageId [pageId=000100000000000a,
effectivePageId=000000000000000a, grpId=2141373875], page = [
Header [
type=20 (PagePartitionCountersIO),
ver=1,
crc=0,
pageId=281474976710666(offset=0, flags=1, partId=0, index=10)
],
PagePartitionCounters [
count=1,
lastFlag=true,
nextCountersPageId=0000000000000000,
size={
-1368047377=313
}
]],
super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
fileOffset=29809, len=4125, forceFlush=false], type=PAGE_RECORD]]]
[W] PageSnapshot [fullPageId = FullPageId [pageId=0001000200000010,
effectivePageId=0000000200000010, grpId=2141373875], page = [
Header [
type=20 (PagePartitionCountersIO),
ver=1,
crc=0,
pageId=281483566645264(offset=0, flags=1, partId=2, index=16)
],
PagePartitionCounters [
count=1,
lastFlag=true,
nextCountersPageId=0000000000000000,
size={
-1368047377=313
}
]],
super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
fileOffset=33934, len=4125, forceFlush=false], type=PAGE_RECORD]]]
[W] PageSnapshot [fullPageId = FullPageId [pageId=000100050000000d,
effectivePageId=000000050000000d, grpId=2141373875], page = [
Header [
type=20 (PagePartitionCountersIO),
ver=1,
crc=0,
pageId=281496451547149(offset=0, flags=1, partId=5, index=13)
],
PagePartitionCounters [
count=1,
lastFlag=true,
nextCountersPageId=0000000000000000,
size={
-1368047377=313
}
]],
super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
fileOffset=38059, len=4125, forceFlush=false], type=PAGE_RECORD]]]
[W] PageSnapshot [fullPageId = FullPageId [pageId=0001000900000003,
effectivePageId=0000000900000003, grpId=2141373875], page = [
Header [
type=12 (PagesListMetaIO),
ver=1,
crc=0,
pageId=281513631416323(offset=0, flags=1, partId=9, index=3)
],
PagesListMeta [
nextMetaPageId=0000000000000000,
count=230,
bucketData={
bucket=163, list=GridLongList [idx=6,
arr=[281513631416558,281513631416559,281513631416560,281513631416561,281513631416562,281513631416563]]
bucket=166, list=GridLongList [idx=8,
arr=[281513631416550,281513631416551,281513631416552,281513631416553,281513631416554,281513631416555,281513631416556,281513631416557]]
bucket=169, list=GridLongList [idx=8,
arr=[281513631416542,281513631416543,281513631416544,281513631416545,281513631416546,281513631416547,281513631416548,281513631416549]]
bucket=172, list=GridLongList [idx=8,
arr=[281513631416534,281513631416535,281513631416536,281513631416537,281513631416538,281513631416539,281513631416540,281513631416541]]
bucket=175, list=GridLongList [idx=8,
arr=[281513631416526,281513631416527,281513631416528,281513631416529,281513631416530,281513631416531,281513631416532,281513631416533]]
bucket=178, list=GridLongList [idx=8,
arr=[281513631416512,281513631416513,281513631416520,281513631416521,281513631416522,281513631416523,281513631416524,281513631416525]]
bucket=181, list=GridLongList [idx=8,
arr=[281513631416510,281513631416511,281513631416514,281513631416515,281513631416516,281513631416517,281513631416518,281513631416519]]
bucket=184, list=GridLongList [idx=8,
arr=[281513631416500,281513631416503,281513631416504,281513631416505,281513631416506,281513631416507,281513631416508,281513631416509]]
bucket=187, list=GridLongList [idx=8,
arr=[281513631416494,281513631416495,281513631416496,281513631416497,281513631416498,281513631416499,281513631416501,281513631416502]]
bucket=190, list=GridLongList [idx=8,
arr=[281513631416484,281513631416485,281513631416486,281513631416487,281513631416488,281513631416489,281513631416492,281513631416493]]
bucket=193, list=GridLongList [idx=8,
arr=[281513631416448,281513631416449,281513631416480,281513631416481,281513631416482,281513631416483,281513631416490,281513631416491]]
bucket=196, list=GridLongList [idx=8,
arr=[281513631416446,281513631416447,281513631416474,281513631416475,281513631416476,281513631416477,281513631416478,281513631416479]]
bucket=199, list=GridLongList [idx=8,
arr=[281513631416444,281513631416445,281513631416468,281513631416469,281513631416470,281513631416471,281513631416472,281513631416473]]
bucket=202, list=GridLongList [idx=8,
arr=[281513631416436,281513631416437,281513631416462,281513631416463,281513631416464,281513631416465,281513631416466,281513631416467]]
bucket=205, list=GridLongList [idx=8,
arr=[281513631416434,281513631416435,281513631416456,281513631416457,281513631416458,281513631416459,281513631416460,281513631416461]]
bucket=209, list=GridLongList [idx=8,
arr=[281513631416426,281513631416427,281513631416450,281513631416451,281513631416452,281513631416453,281513631416454,281513631416455]]
bucket=212, list=GridLongList [idx=8,
arr=[281513631416418,281513631416419,281513631416438,281513631416439,281513631416440,281513631416441,281513631416442,281513631416443]]
bucket=215, list=GridLongList [idx=8,
arr=[281513631416416,281513631416417,281513631416428,281513631416429,281513631416430,281513631416431,281513631416432,281513631416433]]
bucket=218, list=GridLongList [idx=8,
arr=[281513631416414,281513631416415,281513631416420,281513631416421,281513631416422,281513631416423,281513631416424,281513631416425]]
bucket=221, list=GridLongList [idx=8,
arr=[281513631416406,281513631416407,281513631416408,281513631416409,281513631416410,281513631416411,281513631416412,281513631416413]]
bucket=224, list=GridLongList [idx=8,
arr=[281513631416398,281513631416399,281513631416400,281513631416401,281513631416402,281513631416403,281513631416404,281513631416405]]
bucket=227, list=GridLongList [idx=8,
arr=[281513631416390,281513631416391,281513631416392,281513631416393,281513631416394,281513631416395,281513631416396,281513631416397]]
bucket=230, list=GridLongList [idx=8,
arr=[281513631416381,281513631416382,281513631416383,281513631416384,281513631416385,281513631416386,281513631416387,281513631416388]]
bucket=233, list=GridLongList [idx=8,
arr=[281513631416373,281513631416374,281513631416375,281513631416376,281513631416377,281513631416378,281513631416379,281513631416380]]
bucket=236, list=GridLongList [idx=8,
arr=[281513631416365,281513631416366,281513631416367,281513631416368,281513631416369,281513631416370,281513631416371,281513631416372]]
bucket=239, list=GridLongList [idx=8,
arr=[281513631416351,281513631416352,281513631416359,281513631416360,281513631416361,281513631416362,281513631416363,281513631416364]]
bucket=242, list=GridLongList [idx=8,
arr=[281513631416348,281513631416349,281513631416353,281513631416354,281513631416355,281513631416356,281513631416357,281513631416358]]
bucket=245, list=GridLongList [idx=8,
arr=[281513631416335,281513631416336,281513631416337,281513631416338,281513631416339,281513631416340,281513631416346,281513631416347]]
bucket=248, list=GridLongList [idx=8,
arr=[281513631416326,281513631416327,281513631416328,281513631416329,281513631416330,281513631416331,281513631416343,281513631416344]]
}
]],
super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
fileOffset=42184, len=4125, forceFlush=false], type=PAGE_RECORD]]]
[W] PageSnapshot [fullPageId = FullPageId [pageId=000100090000000d,
effectivePageId=000000090000000d, grpId=2141373875], page = [
Header [
type=20 (PagePartitionCountersIO),
ver=1,
crc=0,
pageId=281513631416333(offset=0, flags=1, partId=9, index=13)
],
PagePartitionCounters [
count=1,
lastFlag=true,
nextCountersPageId=0000000000000000,
size={
-1368047377=198
}
]],
super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
fileOffset=46309, len=4125, forceFlush=false], type=PAGE_RECORD]]]
[W] PageSnapshot [fullPageId = FullPageId [pageId=0001000900000000,
effectivePageId=0000000900000000, grpId=2141373875], page = [
Header [
type=14 (PagePartitionMetaIO),
ver=1,
crc=0,
pageId=281513631416320(offset=0, flags=1, partId=9, index=0)
],
PageMeta[
treeRoot=281513631416323,
lastSuccessfulFullSnapshotId=0,
lastSuccessfulSnapshotId=0,
nextSnapshotTag=1,
lastSuccessfulSnapshotTag=0,
lastAllocatedPageCount=0,
candidatePageCount=0
],
PagePartitionMeta[
size=198,
updateCounter=198,
globalRemoveId=1505412422861000000,
partitionState=1(OWNING),
countersPageId=281513631416333
]],
super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
fileOffset=50434, len=4125, forceFlush=false], type=PAGE_RECORD]]]
[W] PageSnapshot [fullPageId = FullPageId [pageId=0001000b0000000c,
effectivePageId=0000000b0000000c, grpId=2141373875], page = [
Header [
type=20 (PagePartitionCountersIO),
ver=1,
crc=0,
pageId=281522221350924(offset=0, flags=1, partId=11, index=12)
],
PagePartitionCounters [
count=1,
lastFlag=true,
nextCountersPageId=0000000000000000,
size={
-1368047377=313
}
]],
super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
fileOffset=54559, len=4125, forceFlush=false], type=PAGE_RECORD]]]
[W] PageSnapshot [fullPageId = FullPageId [pageId=0001000c0000000f,
effectivePageId=0000000c0000000f, grpId=2141373875], page = [
Header [
type=20 (PagePartitionCountersIO),
ver=1,
crc=0,
pageId=281526516318223(offset=0, flags=1, partId=12, index=15)
],
PagePartitionCounters [
count=1,
lastFlag=true,
nextCountersPageId=0000000000000000,
size={
-1368047377=313
}
]],
super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
fileOffset=58684, len=4125, forceFlush=false], type=PAGE_RECORD]]]


On Mon, Sep 18, 2017 at 1:10 PM, Dmitry Pavlov <[hidden email]>
wrote:

> Hi Igniters,
>
> Page https://cwiki.apache.org/confluence/display/IGNITE/Ignite+WAL+reader
> to
> collect related information about this prototype was created.
>
> Sincerely,
> Dmitriy Pavlov
>
> пн, 18 сент. 2017 г. в 7:01, Dmitriy Setrakyan <[hidden email]>:
>
> > Can we get a sample print out from this utility?
> >
> > On Sun, Sep 17, 2017 at 3:42 PM, Eduard Shangareev <
> > [hidden email]> wrote:
> >
> > > Dmitry,
> > > thank you for starting this topic.
> > >
> > > The IDEA of this utility is very simple - get the possibility to look
> > into
> > > your WAL.
> > > But it also requires understanding what's going on in WAL mechanism (We
> > > need to describe it in more details in the wiki).
> > >
> > > It could be useful for a debug some weird things with recovery
> mechanism.
> > > Initially, it was created for dealing with the issue in IGNITE-6342. It
> > > helped to find out the error in how we deal with partition size.
> > >
> > > Best regards,
> > > Ed.
> > >
> > >
> > >
> > >
> > > On Sat, Sep 16, 2017 at 2:09 AM, Dmitry Pavlov <[hidden email]>
> > > wrote:
> > >
> > > > Hi Denis,
> > > >
> > > > Thank you for your feedback. It is very important to us to understand
> > > that
> > > > this tool may be useful not only for Ignite tests but for all
> > developers.
> > > >
> > > > I will create wiki page.
> > > >
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > > >
> > > > сб, 16 сент. 2017 г. в 1:54, Denis Magda <[hidden email]>:
> > > >
> > > > > Dmitriy, Eduard,
> > > > >
> > > > > Sounds like a useful thing. I’ll be happy to give a feedback but
> > need a
> > > > > documentation that explains how to use the tool, what’s the output
> > > > format,
> > > > > how do decipher the output, etc (basic things). The Wiki
> description
> > > > should
> > > > > be enough.
> > > > >
> > > > > —
> > > > > Denis
> > > > >
> > > > > > On Sep 15, 2017, at 11:36 AM, Dmitry Pavlov <
> [hidden email]
> > >
> > > > > wrote:
> > > > > >
> > > > > > Hi Igniters,
> > > > > >
> > > > > > Eduard Shangareev created WAL converter utility which shows WAL
> > > > > > archive/work directory segments content in human readable form.
> > This
> > > > > > utility outputs all WAL records to output stream (may be
> redirected
> > > to
> > > > > > file). How to use tips can be found in issue
> > > > > > https://issues.apache.org/jira/browse/IGNITE-6277
> > > > > >
> > > > > >
> > > > > > This utility does not require Ignite node to be up and running
> and
> > > may
> > > > be
> > > > > > used for offline analysis of copied files. Thanks to Alexey G.
> for
> > > idea
> > > > > and
> > > > > > review. Code is now available in modules/ignite-dev-tools.
> > > > > >
> > > > > > 1. Could you please write if utility can be useful for you?
> > > > > >
> > > > > > 2. Please respond if you need short wiki description.
> > > > > >
> > > > > > 3. What is your opinion what should be next steps to develop this
> > > > > utility?
> > > > > >
> > > > > > 4. Does utility need property file to refer Ignite Work directory
> > > using
> > > > > > conf file.
> > > > > >
> > > > > > 5. What do you think if utility remains console non interactive,
> or
> > > > some
> > > > > > UI/Interactive console shell may be preferable?
> > > > > >
> > > > > > Sincerely,
> > > > > >
> > > > > > Dmitriy Pavlov
> > > > >
> > > > >
> > > >
> > >
> >
>

[dsetrakyan]

Thanks!

After looking at the ticket, the way to launch this utility seems complex.
Does this utility have a shell script? If not, we should add one with
proper command line options, like we do for other scripts.

D.

On Mon, Sep 18, 2017 at 4:05 AM, Eduard Shangareev <
[hidden email]> wrote:

> Example of output (will add to wiki):
>
> [W] InitNewPageRecord [newPageId=0001001c00000067, super=PageDeltaRecord
> [grpId=2141373875, pageId=0001001c00000067, super=WALRecord [size=41,
> chainSize=0, pos=FileWALPointer [idx=1, fileOffset=25406, len=41,
> forceFlush=false], type=INIT_NEW_PAGE_RECORD]]]
> [W] InitNewPageRecord [newPageId=0001001c00000068, super=PageDeltaRecord
> [grpId=2141373875, pageId=0001001c00000068, super=WALRecord [size=41,
> chainSize=0, pos=FileWALPointer [idx=1, fileOffset=25447, len=41,
> forceFlush=false], type=INIT_NEW_PAGE_RECORD]]]
> [W] PagesListAddPageRecord [dataPageId=0001001c0000002a,
> super=PageDeltaRecord [grpId=2141373875, pageId=0001001c00000068,
> super=WALRecord [size=37, chainSize=0, pos=FileWALPointer [idx=1,
> fileOffset=25488, len=37, forceFlush=false], type=PAGES_LIST_ADD_PAGE]]]
> [W] PageSnapshot [fullPageId = FullPageId [pageId=0001001c0000002a,
> effectivePageId=0000001c0000002a, grpId=2141373875], page = [
> Header [
> type=1 (DataPageIO),
> ver=1,
> crc=0,
> pageId=281595235794986(offset=0, flags=1, partId=28, index=42)
> ],
> DataPageIO [
> 0001001c0000002a [4049, 4002, 3955, 3908, 3861, 3814, 3767, 3720, 3673,
> 3626][][free=3552, actualFree=-544
> ]],
> super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> fileOffset=25525, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> [W] InsertRecord [idx=12, io=CacheIdAwareDataLeafIO[ver=1],
> rightId=0000000000000000, super=PageDeltaRecord [grpId=2141373875,
> pageId=0001001c00000004, super=WALRecord [size=59, chainSize=0,
> pos=FileWALPointer [idx=1, fileOffset=29650, len=59, forceFlush=false],
> type=BTREE_PAGE_INSERT]]]
> [W] DataRecord [writeEntries=[DataEntry [cacheId=-1368047377, op=CREATE,
> writeVer=GridCacheVersion [topVer=116892426, order=1505412445450,
> nodeOrder=2], partId=28, partCnt=69]], super=WALRecord [size=100,
> chainSize=0, pos=FileWALPointer [idx=1, fileOffset=29709, len=100,
> forceFlush=false], type=DATA_RECORD]]
> [W] PageSnapshot [fullPageId = FullPageId [pageId=000100000000000a,
> effectivePageId=000000000000000a, grpId=2141373875], page = [
> Header [
> type=20 (PagePartitionCountersIO),
> ver=1,
> crc=0,
> pageId=281474976710666(offset=0, flags=1, partId=0, index=10)
> ],
> PagePartitionCounters [
> count=1,
> lastFlag=true,
> nextCountersPageId=0000000000000000,
> size={
> -1368047377=313
> }
> ]],
> super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> fileOffset=29809, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000200000010,
> effectivePageId=0000000200000010, grpId=2141373875], page = [
> Header [
> type=20 (PagePartitionCountersIO),
> ver=1,
> crc=0,
> pageId=281483566645264(offset=0, flags=1, partId=2, index=16)
> ],
> PagePartitionCounters [
> count=1,
> lastFlag=true,
> nextCountersPageId=0000000000000000,
> size={
> -1368047377=313
> }
> ]],
> super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> fileOffset=33934, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> [W] PageSnapshot [fullPageId = FullPageId [pageId=000100050000000d,
> effectivePageId=000000050000000d, grpId=2141373875], page = [
> Header [
> type=20 (PagePartitionCountersIO),
> ver=1,
> crc=0,
> pageId=281496451547149(offset=0, flags=1, partId=5, index=13)
> ],
> PagePartitionCounters [
> count=1,
> lastFlag=true,
> nextCountersPageId=0000000000000000,
> size={
> -1368047377=313
> }
> ]],
> super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> fileOffset=38059, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000900000003,
> effectivePageId=0000000900000003, grpId=2141373875], page = [
> Header [
> type=12 (PagesListMetaIO),
> ver=1,
> crc=0,
> pageId=281513631416323(offset=0, flags=1, partId=9, index=3)
> ],
> PagesListMeta [
> nextMetaPageId=0000000000000000,
> count=230,
> bucketData={
> bucket=163, list=GridLongList [idx=6,
> arr=[281513631416558,281513631416559,281513631416560,281513631416561,
> 281513631416562,281513631416563]]
> bucket=166, list=GridLongList [idx=8,
> arr=[281513631416550,281513631416551,281513631416552,281513631416553,
> 281513631416554,281513631416555,281513631416556,281513631416557]]
> bucket=169, list=GridLongList [idx=8,
> arr=[281513631416542,281513631416543,281513631416544,281513631416545,
> 281513631416546,281513631416547,281513631416548,281513631416549]]
> bucket=172, list=GridLongList [idx=8,
> arr=[281513631416534,281513631416535,281513631416536,281513631416537,
> 281513631416538,281513631416539,281513631416540,281513631416541]]
> bucket=175, list=GridLongList [idx=8,
> arr=[281513631416526,281513631416527,281513631416528,281513631416529,
> 281513631416530,281513631416531,281513631416532,281513631416533]]
> bucket=178, list=GridLongList [idx=8,
> arr=[281513631416512,281513631416513,281513631416520,281513631416521,
> 281513631416522,281513631416523,281513631416524,281513631416525]]
> bucket=181, list=GridLongList [idx=8,
> arr=[281513631416510,281513631416511,281513631416514,281513631416515,
> 281513631416516,281513631416517,281513631416518,281513631416519]]
> bucket=184, list=GridLongList [idx=8,
> arr=[281513631416500,281513631416503,281513631416504,281513631416505,
> 281513631416506,281513631416507,281513631416508,281513631416509]]
> bucket=187, list=GridLongList [idx=8,
> arr=[281513631416494,281513631416495,281513631416496,281513631416497,
> 281513631416498,281513631416499,281513631416501,281513631416502]]
> bucket=190, list=GridLongList [idx=8,
> arr=[281513631416484,281513631416485,281513631416486,281513631416487,
> 281513631416488,281513631416489,281513631416492,281513631416493]]
> bucket=193, list=GridLongList [idx=8,
> arr=[281513631416448,281513631416449,281513631416480,281513631416481,
> 281513631416482,281513631416483,281513631416490,281513631416491]]
> bucket=196, list=GridLongList [idx=8,
> arr=[281513631416446,281513631416447,281513631416474,281513631416475,
> 281513631416476,281513631416477,281513631416478,281513631416479]]
> bucket=199, list=GridLongList [idx=8,
> arr=[281513631416444,281513631416445,281513631416468,281513631416469,
> 281513631416470,281513631416471,281513631416472,281513631416473]]
> bucket=202, list=GridLongList [idx=8,
> arr=[281513631416436,281513631416437,281513631416462,281513631416463,
> 281513631416464,281513631416465,281513631416466,281513631416467]]
> bucket=205, list=GridLongList [idx=8,
> arr=[281513631416434,281513631416435,281513631416456,281513631416457,
> 281513631416458,281513631416459,281513631416460,281513631416461]]
> bucket=209, list=GridLongList [idx=8,
> arr=[281513631416426,281513631416427,281513631416450,281513631416451,
> 281513631416452,281513631416453,281513631416454,281513631416455]]
> bucket=212, list=GridLongList [idx=8,
> arr=[281513631416418,281513631416419,281513631416438,281513631416439,
> 281513631416440,281513631416441,281513631416442,281513631416443]]
> bucket=215, list=GridLongList [idx=8,
> arr=[281513631416416,281513631416417,281513631416428,281513631416429,
> 281513631416430,281513631416431,281513631416432,281513631416433]]
> bucket=218, list=GridLongList [idx=8,
> arr=[281513631416414,281513631416415,281513631416420,281513631416421,
> 281513631416422,281513631416423,281513631416424,281513631416425]]
> bucket=221, list=GridLongList [idx=8,
> arr=[281513631416406,281513631416407,281513631416408,281513631416409,
> 281513631416410,281513631416411,281513631416412,281513631416413]]
> bucket=224, list=GridLongList [idx=8,
> arr=[281513631416398,281513631416399,281513631416400,281513631416401,
> 281513631416402,281513631416403,281513631416404,281513631416405]]
> bucket=227, list=GridLongList [idx=8,
> arr=[281513631416390,281513631416391,281513631416392,281513631416393,
> 281513631416394,281513631416395,281513631416396,281513631416397]]
> bucket=230, list=GridLongList [idx=8,
> arr=[281513631416381,281513631416382,281513631416383,281513631416384,
> 281513631416385,281513631416386,281513631416387,281513631416388]]
> bucket=233, list=GridLongList [idx=8,
> arr=[281513631416373,281513631416374,281513631416375,281513631416376,
> 281513631416377,281513631416378,281513631416379,281513631416380]]
> bucket=236, list=GridLongList [idx=8,
> arr=[281513631416365,281513631416366,281513631416367,281513631416368,
> 281513631416369,281513631416370,281513631416371,281513631416372]]
> bucket=239, list=GridLongList [idx=8,
> arr=[281513631416351,281513631416352,281513631416359,281513631416360,
> 281513631416361,281513631416362,281513631416363,281513631416364]]
> bucket=242, list=GridLongList [idx=8,
> arr=[281513631416348,281513631416349,281513631416353,281513631416354,
> 281513631416355,281513631416356,281513631416357,281513631416358]]
> bucket=245, list=GridLongList [idx=8,
> arr=[281513631416335,281513631416336,281513631416337,281513631416338,
> 281513631416339,281513631416340,281513631416346,281513631416347]]
> bucket=248, list=GridLongList [idx=8,
> arr=[281513631416326,281513631416327,281513631416328,281513631416329,
> 281513631416330,281513631416331,281513631416343,281513631416344]]
> }
> ]],
> super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> fileOffset=42184, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> [W] PageSnapshot [fullPageId = FullPageId [pageId=000100090000000d,
> effectivePageId=000000090000000d, grpId=2141373875], page = [
> Header [
> type=20 (PagePartitionCountersIO),
> ver=1,
> crc=0,
> pageId=281513631416333(offset=0, flags=1, partId=9, index=13)
> ],
> PagePartitionCounters [
> count=1,
> lastFlag=true,
> nextCountersPageId=0000000000000000,
> size={
> -1368047377=198
> }
> ]],
> super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> fileOffset=46309, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000900000000,
> effectivePageId=0000000900000000, grpId=2141373875], page = [
> Header [
> type=14 (PagePartitionMetaIO),
> ver=1,
> crc=0,
> pageId=281513631416320(offset=0, flags=1, partId=9, index=0)
> ],
> PageMeta[
> treeRoot=281513631416323,
> lastSuccessfulFullSnapshotId=0,
> lastSuccessfulSnapshotId=0,
> nextSnapshotTag=1,
> lastSuccessfulSnapshotTag=0,
> lastAllocatedPageCount=0,
> candidatePageCount=0
> ],
> PagePartitionMeta[
> size=198,
> updateCounter=198,
> globalRemoveId=1505412422861000000,
> partitionState=1(OWNING),
> countersPageId=281513631416333
> ]],
> super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> fileOffset=50434, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000b0000000c,
> effectivePageId=0000000b0000000c, grpId=2141373875], page = [
> Header [
> type=20 (PagePartitionCountersIO),
> ver=1,
> crc=0,
> pageId=281522221350924(offset=0, flags=1, partId=11, index=12)
> ],
> PagePartitionCounters [
> count=1,
> lastFlag=true,
> nextCountersPageId=0000000000000000,
> size={
> -1368047377=313
> }
> ]],
> super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> fileOffset=54559, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000c0000000f,
> effectivePageId=0000000c0000000f, grpId=2141373875], page = [
> Header [
> type=20 (PagePartitionCountersIO),
> ver=1,
> crc=0,
> pageId=281526516318223(offset=0, flags=1, partId=12, index=15)
> ],
> PagePartitionCounters [
> count=1,
> lastFlag=true,
> nextCountersPageId=0000000000000000,
> size={
> -1368047377=313
> }
> ]],
> super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> fileOffset=58684, len=4125, forceFlush=false], type=PAGE_RECORD]]]
>
>
> On Mon, Sep 18, 2017 at 1:10 PM, Dmitry Pavlov <[hidden email]>
> wrote:
>
> > Hi Igniters,
> >
> > Page https://cwiki.apache.org/confluence/display/IGNITE/
> Ignite+WAL+reader
> > to
> > collect related information about this prototype was created.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > пн, 18 сент. 2017 г. в 7:01, Dmitriy Setrakyan <[hidden email]>:
> >
> > > Can we get a sample print out from this utility?
> > >
> > > On Sun, Sep 17, 2017 at 3:42 PM, Eduard Shangareev <
> > > [hidden email]> wrote:
> > >
> > > > Dmitry,
> > > > thank you for starting this topic.
> > > >
> > > > The IDEA of this utility is very simple - get the possibility to look
> > > into
> > > > your WAL.
> > > > But it also requires understanding what's going on in WAL mechanism
> (We
> > > > need to describe it in more details in the wiki).
> > > >
> > > > It could be useful for a debug some weird things with recovery
> > mechanism.
> > > > Initially, it was created for dealing with the issue in IGNITE-6342.
> It
> > > > helped to find out the error in how we deal with partition size.
> > > >
> > > > Best regards,
> > > > Ed.
> > > >
> > > >
> > > >
> > > >
> > > > On Sat, Sep 16, 2017 at 2:09 AM, Dmitry Pavlov <
> [hidden email]>
> > > > wrote:
> > > >
> > > > > Hi Denis,
> > > > >
> > > > > Thank you for your feedback. It is very important to us to
> understand
> > > > that
> > > > > this tool may be useful not only for Ignite tests but for all
> > > developers.
> > > > >
> > > > > I will create wiki page.
> > > > >
> > > > > Sincerely,
> > > > > Dmitriy Pavlov
> > > > >
> > > > > сб, 16 сент. 2017 г. в 1:54, Denis Magda <[hidden email]>:
> > > > >
> > > > > > Dmitriy, Eduard,
> > > > > >
> > > > > > Sounds like a useful thing. I’ll be happy to give a feedback but
> > > need a
> > > > > > documentation that explains how to use the tool, what’s the
> output
> > > > > format,
> > > > > > how do decipher the output, etc (basic things). The Wiki
> > description
> > > > > should
> > > > > > be enough.
> > > > > >
> > > > > > —
> > > > > > Denis
> > > > > >
> > > > > > > On Sep 15, 2017, at 11:36 AM, Dmitry Pavlov <
> > [hidden email]
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi Igniters,
> > > > > > >
> > > > > > > Eduard Shangareev created WAL converter utility which shows WAL
> > > > > > > archive/work directory segments content in human readable form.
> > > This
> > > > > > > utility outputs all WAL records to output stream (may be
> > redirected
> > > > to
> > > > > > > file). How to use tips can be found in issue
> > > > > > > https://issues.apache.org/jira/browse/IGNITE-6277
> > > > > > >
> > > > > > >
> > > > > > > This utility does not require Ignite node to be up and running
> > and
> > > > may
> > > > > be
> > > > > > > used for offline analysis of copied files. Thanks to Alexey G.
> > for
> > > > idea
> > > > > > and
> > > > > > > review. Code is now available in modules/ignite-dev-tools.
> > > > > > >
> > > > > > > 1. Could you please write if utility can be useful for you?
> > > > > > >
> > > > > > > 2. Please respond if you need short wiki description.
> > > > > > >
> > > > > > > 3. What is your opinion what should be next steps to develop
> this
> > > > > > utility?
> > > > > > >
> > > > > > > 4. Does utility need property file to refer Ignite Work
> directory
> > > > using
> > > > > > > conf file.
> > > > > > >
> > > > > > > 5. What do you think if utility remains console non
> interactive,
> > or
> > > > > some
> > > > > > > UI/Interactive console shell may be preferable?
> > > > > > >
> > > > > > > Sincerely,
> > > > > > >
> > > > > > > Dmitriy Pavlov
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Dmitriy Pavlov]

Dmitriy,

Thank you for your feedback I've created IGNITE-6421 for this change.

Is it common practice to use property file in Ignite utils? I'd like to
support parse and display data entries and its fields, but it is required
to configure several folders: 'binary_meta' and 'marshaller' folders.

My idea is to provide .conf file to configure reader once.

Sincerely,
Dmitriy Pavlov

пн, 18 сент. 2017 г. в 16:10, Dmitriy Setrakyan <[hidden email]>:

> Thanks!
>
> After looking at the ticket, the way to launch this utility seems complex.
> Does this utility have a shell script? If not, we should add one with
> proper command line options, like we do for other scripts.
>
> D.
>
> On Mon, Sep 18, 2017 at 4:05 AM, Eduard Shangareev <
> [hidden email]> wrote:
>
> > Example of output (will add to wiki):
> >
> > [W] InitNewPageRecord [newPageId=0001001c00000067, super=PageDeltaRecord
> > [grpId=2141373875, pageId=0001001c00000067, super=WALRecord [size=41,
> > chainSize=0, pos=FileWALPointer [idx=1, fileOffset=25406, len=41,
> > forceFlush=false], type=INIT_NEW_PAGE_RECORD]]]
> > [W] InitNewPageRecord [newPageId=0001001c00000068, super=PageDeltaRecord
> > [grpId=2141373875, pageId=0001001c00000068, super=WALRecord [size=41,
> > chainSize=0, pos=FileWALPointer [idx=1, fileOffset=25447, len=41,
> > forceFlush=false], type=INIT_NEW_PAGE_RECORD]]]
> > [W] PagesListAddPageRecord [dataPageId=0001001c0000002a,
> > super=PageDeltaRecord [grpId=2141373875, pageId=0001001c00000068,
> > super=WALRecord [size=37, chainSize=0, pos=FileWALPointer [idx=1,
> > fileOffset=25488, len=37, forceFlush=false], type=PAGES_LIST_ADD_PAGE]]]
> > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001001c0000002a,
> > effectivePageId=0000001c0000002a, grpId=2141373875], page = [
> > Header [
> > type=1 (DataPageIO),
> > ver=1,
> > crc=0,
> > pageId=281595235794986(offset=0, flags=1, partId=28, index=42)
> > ],
> > DataPageIO [
> > 0001001c0000002a [4049, 4002, 3955, 3908, 3861, 3814, 3767, 3720, 3673,
> > 3626][][free=3552, actualFree=-544
> > ]],
> > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > fileOffset=25525, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > [W] InsertRecord [idx=12, io=CacheIdAwareDataLeafIO[ver=1],
> > rightId=0000000000000000, super=PageDeltaRecord [grpId=2141373875,
> > pageId=0001001c00000004, super=WALRecord [size=59, chainSize=0,
> > pos=FileWALPointer [idx=1, fileOffset=29650, len=59, forceFlush=false],
> > type=BTREE_PAGE_INSERT]]]
> > [W] DataRecord [writeEntries=[DataEntry [cacheId=-1368047377, op=CREATE,
> > writeVer=GridCacheVersion [topVer=116892426, order=1505412445450,
> > nodeOrder=2], partId=28, partCnt=69]], super=WALRecord [size=100,
> > chainSize=0, pos=FileWALPointer [idx=1, fileOffset=29709, len=100,
> > forceFlush=false], type=DATA_RECORD]]
> > [W] PageSnapshot [fullPageId = FullPageId [pageId=000100000000000a,
> > effectivePageId=000000000000000a, grpId=2141373875], page = [
> > Header [
> > type=20 (PagePartitionCountersIO),
> > ver=1,
> > crc=0,
> > pageId=281474976710666(offset=0, flags=1, partId=0, index=10)
> > ],
> > PagePartitionCounters [
> > count=1,
> > lastFlag=true,
> > nextCountersPageId=0000000000000000,
> > size={
> > -1368047377=313
> > }
> > ]],
> > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > fileOffset=29809, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000200000010,
> > effectivePageId=0000000200000010, grpId=2141373875], page = [
> > Header [
> > type=20 (PagePartitionCountersIO),
> > ver=1,
> > crc=0,
> > pageId=281483566645264(offset=0, flags=1, partId=2, index=16)
> > ],
> > PagePartitionCounters [
> > count=1,
> > lastFlag=true,
> > nextCountersPageId=0000000000000000,
> > size={
> > -1368047377=313
> > }
> > ]],
> > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > fileOffset=33934, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > [W] PageSnapshot [fullPageId = FullPageId [pageId=000100050000000d,
> > effectivePageId=000000050000000d, grpId=2141373875], page = [
> > Header [
> > type=20 (PagePartitionCountersIO),
> > ver=1,
> > crc=0,
> > pageId=281496451547149(offset=0, flags=1, partId=5, index=13)
> > ],
> > PagePartitionCounters [
> > count=1,
> > lastFlag=true,
> > nextCountersPageId=0000000000000000,
> > size={
> > -1368047377=313
> > }
> > ]],
> > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > fileOffset=38059, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000900000003,
> > effectivePageId=0000000900000003, grpId=2141373875], page = [
> > Header [
> > type=12 (PagesListMetaIO),
> > ver=1,
> > crc=0,
> > pageId=281513631416323(offset=0, flags=1, partId=9, index=3)
> > ],
> > PagesListMeta [
> > nextMetaPageId=0000000000000000,
> > count=230,
> > bucketData={
> > bucket=163, list=GridLongList [idx=6,
> > arr=[281513631416558,281513631416559,281513631416560,281513631416561,
> > 281513631416562,281513631416563]]
> > bucket=166, list=GridLongList [idx=8,
> > arr=[281513631416550,281513631416551,281513631416552,281513631416553,
> > 281513631416554,281513631416555,281513631416556,281513631416557]]
> > bucket=169, list=GridLongList [idx=8,
> > arr=[281513631416542,281513631416543,281513631416544,281513631416545,
> > 281513631416546,281513631416547,281513631416548,281513631416549]]
> > bucket=172, list=GridLongList [idx=8,
> > arr=[281513631416534,281513631416535,281513631416536,281513631416537,
> > 281513631416538,281513631416539,281513631416540,281513631416541]]
> > bucket=175, list=GridLongList [idx=8,
> > arr=[281513631416526,281513631416527,281513631416528,281513631416529,
> > 281513631416530,281513631416531,281513631416532,281513631416533]]
> > bucket=178, list=GridLongList [idx=8,
> > arr=[281513631416512,281513631416513,281513631416520,281513631416521,
> > 281513631416522,281513631416523,281513631416524,281513631416525]]
> > bucket=181, list=GridLongList [idx=8,
> > arr=[281513631416510,281513631416511,281513631416514,281513631416515,
> > 281513631416516,281513631416517,281513631416518,281513631416519]]
> > bucket=184, list=GridLongList [idx=8,
> > arr=[281513631416500,281513631416503,281513631416504,281513631416505,
> > 281513631416506,281513631416507,281513631416508,281513631416509]]
> > bucket=187, list=GridLongList [idx=8,
> > arr=[281513631416494,281513631416495,281513631416496,281513631416497,
> > 281513631416498,281513631416499,281513631416501,281513631416502]]
> > bucket=190, list=GridLongList [idx=8,
> > arr=[281513631416484,281513631416485,281513631416486,281513631416487,
> > 281513631416488,281513631416489,281513631416492,281513631416493]]
> > bucket=193, list=GridLongList [idx=8,
> > arr=[281513631416448,281513631416449,281513631416480,281513631416481,
> > 281513631416482,281513631416483,281513631416490,281513631416491]]
> > bucket=196, list=GridLongList [idx=8,
> > arr=[281513631416446,281513631416447,281513631416474,281513631416475,
> > 281513631416476,281513631416477,281513631416478,281513631416479]]
> > bucket=199, list=GridLongList [idx=8,
> > arr=[281513631416444,281513631416445,281513631416468,281513631416469,
> > 281513631416470,281513631416471,281513631416472,281513631416473]]
> > bucket=202, list=GridLongList [idx=8,
> > arr=[281513631416436,281513631416437,281513631416462,281513631416463,
> > 281513631416464,281513631416465,281513631416466,281513631416467]]
> > bucket=205, list=GridLongList [idx=8,
> > arr=[281513631416434,281513631416435,281513631416456,281513631416457,
> > 281513631416458,281513631416459,281513631416460,281513631416461]]
> > bucket=209, list=GridLongList [idx=8,
> > arr=[281513631416426,281513631416427,281513631416450,281513631416451,
> > 281513631416452,281513631416453,281513631416454,281513631416455]]
> > bucket=212, list=GridLongList [idx=8,
> > arr=[281513631416418,281513631416419,281513631416438,281513631416439,
> > 281513631416440,281513631416441,281513631416442,281513631416443]]
> > bucket=215, list=GridLongList [idx=8,
> > arr=[281513631416416,281513631416417,281513631416428,281513631416429,
> > 281513631416430,281513631416431,281513631416432,281513631416433]]
> > bucket=218, list=GridLongList [idx=8,
> > arr=[281513631416414,281513631416415,281513631416420,281513631416421,
> > 281513631416422,281513631416423,281513631416424,281513631416425]]
> > bucket=221, list=GridLongList [idx=8,
> > arr=[281513631416406,281513631416407,281513631416408,281513631416409,
> > 281513631416410,281513631416411,281513631416412,281513631416413]]
> > bucket=224, list=GridLongList [idx=8,
> > arr=[281513631416398,281513631416399,281513631416400,281513631416401,
> > 281513631416402,281513631416403,281513631416404,281513631416405]]
> > bucket=227, list=GridLongList [idx=8,
> > arr=[281513631416390,281513631416391,281513631416392,281513631416393,
> > 281513631416394,281513631416395,281513631416396,281513631416397]]
> > bucket=230, list=GridLongList [idx=8,
> > arr=[281513631416381,281513631416382,281513631416383,281513631416384,
> > 281513631416385,281513631416386,281513631416387,281513631416388]]
> > bucket=233, list=GridLongList [idx=8,
> > arr=[281513631416373,281513631416374,281513631416375,281513631416376,
> > 281513631416377,281513631416378,281513631416379,281513631416380]]
> > bucket=236, list=GridLongList [idx=8,
> > arr=[281513631416365,281513631416366,281513631416367,281513631416368,
> > 281513631416369,281513631416370,281513631416371,281513631416372]]
> > bucket=239, list=GridLongList [idx=8,
> > arr=[281513631416351,281513631416352,281513631416359,281513631416360,
> > 281513631416361,281513631416362,281513631416363,281513631416364]]
> > bucket=242, list=GridLongList [idx=8,
> > arr=[281513631416348,281513631416349,281513631416353,281513631416354,
> > 281513631416355,281513631416356,281513631416357,281513631416358]]
> > bucket=245, list=GridLongList [idx=8,
> > arr=[281513631416335,281513631416336,281513631416337,281513631416338,
> > 281513631416339,281513631416340,281513631416346,281513631416347]]
> > bucket=248, list=GridLongList [idx=8,
> > arr=[281513631416326,281513631416327,281513631416328,281513631416329,
> > 281513631416330,281513631416331,281513631416343,281513631416344]]
> > }
> > ]],
> > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > fileOffset=42184, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > [W] PageSnapshot [fullPageId = FullPageId [pageId=000100090000000d,
> > effectivePageId=000000090000000d, grpId=2141373875], page = [
> > Header [
> > type=20 (PagePartitionCountersIO),
> > ver=1,
> > crc=0,
> > pageId=281513631416333(offset=0, flags=1, partId=9, index=13)
> > ],
> > PagePartitionCounters [
> > count=1,
> > lastFlag=true,
> > nextCountersPageId=0000000000000000,
> > size={
> > -1368047377=198
> > }
> > ]],
> > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > fileOffset=46309, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000900000000,
> > effectivePageId=0000000900000000, grpId=2141373875], page = [
> > Header [
> > type=14 (PagePartitionMetaIO),
> > ver=1,
> > crc=0,
> > pageId=281513631416320(offset=0, flags=1, partId=9, index=0)
> > ],
> > PageMeta[
> > treeRoot=281513631416323,
> > lastSuccessfulFullSnapshotId=0,
> > lastSuccessfulSnapshotId=0,
> > nextSnapshotTag=1,
> > lastSuccessfulSnapshotTag=0,
> > lastAllocatedPageCount=0,
> > candidatePageCount=0
> > ],
> > PagePartitionMeta[
> > size=198,
> > updateCounter=198,
> > globalRemoveId=1505412422861000000,
> > partitionState=1(OWNING),
> > countersPageId=281513631416333
> > ]],
> > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > fileOffset=50434, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000b0000000c,
> > effectivePageId=0000000b0000000c, grpId=2141373875], page = [
> > Header [
> > type=20 (PagePartitionCountersIO),
> > ver=1,
> > crc=0,
> > pageId=281522221350924(offset=0, flags=1, partId=11, index=12)
> > ],
> > PagePartitionCounters [
> > count=1,
> > lastFlag=true,
> > nextCountersPageId=0000000000000000,
> > size={
> > -1368047377=313
> > }
> > ]],
> > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > fileOffset=54559, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000c0000000f,
> > effectivePageId=0000000c0000000f, grpId=2141373875], page = [
> > Header [
> > type=20 (PagePartitionCountersIO),
> > ver=1,
> > crc=0,
> > pageId=281526516318223(offset=0, flags=1, partId=12, index=15)
> > ],
> > PagePartitionCounters [
> > count=1,
> > lastFlag=true,
> > nextCountersPageId=0000000000000000,
> > size={
> > -1368047377=313
> > }
> > ]],
> > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > fileOffset=58684, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> >
> >
> > On Mon, Sep 18, 2017 at 1:10 PM, Dmitry Pavlov <[hidden email]>
> > wrote:
> >
> > > Hi Igniters,
> > >
> > > Page https://cwiki.apache.org/confluence/display/IGNITE/
> > Ignite+WAL+reader
> > > to
> > > collect related information about this prototype was created.
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > > пн, 18 сент. 2017 г. в 7:01, Dmitriy Setrakyan <[hidden email]
> >:
> > >
> > > > Can we get a sample print out from this utility?
> > > >
> > > > On Sun, Sep 17, 2017 at 3:42 PM, Eduard Shangareev <
> > > > [hidden email]> wrote:
> > > >
> > > > > Dmitry,
> > > > > thank you for starting this topic.
> > > > >
> > > > > The IDEA of this utility is very simple - get the possibility to
> look
> > > > into
> > > > > your WAL.
> > > > > But it also requires understanding what's going on in WAL mechanism
> > (We
> > > > > need to describe it in more details in the wiki).
> > > > >
> > > > > It could be useful for a debug some weird things with recovery
> > > mechanism.
> > > > > Initially, it was created for dealing with the issue in
> IGNITE-6342.
> > It
> > > > > helped to find out the error in how we deal with partition size.
> > > > >
> > > > > Best regards,
> > > > > Ed.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Sat, Sep 16, 2017 at 2:09 AM, Dmitry Pavlov <
> > [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > Hi Denis,
> > > > > >
> > > > > > Thank you for your feedback. It is very important to us to
> > understand
> > > > > that
> > > > > > this tool may be useful not only for Ignite tests but for all
> > > > developers.
> > > > > >
> > > > > > I will create wiki page.
> > > > > >
> > > > > > Sincerely,
> > > > > > Dmitriy Pavlov
> > > > > >
> > > > > > сб, 16 сент. 2017 г. в 1:54, Denis Magda <[hidden email]>:
> > > > > >
> > > > > > > Dmitriy, Eduard,
> > > > > > >
> > > > > > > Sounds like a useful thing. I’ll be happy to give a feedback
> but
> > > > need a
> > > > > > > documentation that explains how to use the tool, what’s the
> > output
> > > > > > format,
> > > > > > > how do decipher the output, etc (basic things). The Wiki
> > > description
> > > > > > should
> > > > > > > be enough.
> > > > > > >
> > > > > > > —
> > > > > > > Denis
> > > > > > >
> > > > > > > > On Sep 15, 2017, at 11:36 AM, Dmitry Pavlov <
> > > [hidden email]
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi Igniters,
> > > > > > > >
> > > > > > > > Eduard Shangareev created WAL converter utility which shows
> WAL
> > > > > > > > archive/work directory segments content in human readable
> form.
> > > > This
> > > > > > > > utility outputs all WAL records to output stream (may be
> > > redirected
> > > > > to
> > > > > > > > file). How to use tips can be found in issue
> > > > > > > > https://issues.apache.org/jira/browse/IGNITE-6277
> > > > > > > >
> > > > > > > >
> > > > > > > > This utility does not require Ignite node to be up and
> running
> > > and
> > > > > may
> > > > > > be
> > > > > > > > used for offline analysis of copied files. Thanks to Alexey
> G.
> > > for
> > > > > idea
> > > > > > > and
> > > > > > > > review. Code is now available in modules/ignite-dev-tools.
> > > > > > > >
> > > > > > > > 1. Could you please write if utility can be useful for you?
> > > > > > > >
> > > > > > > > 2. Please respond if you need short wiki description.
> > > > > > > >
> > > > > > > > 3. What is your opinion what should be next steps to develop
> > this
> > > > > > > utility?
> > > > > > > >
> > > > > > > > 4. Does utility need property file to refer Ignite Work
> > directory
> > > > > using
> > > > > > > > conf file.
> > > > > > > >
> > > > > > > > 5. What do you think if utility remains console non
> > interactive,
> > > or
> > > > > > some
> > > > > > > > UI/Interactive console shell may be preferable?
> > > > > > > >
> > > > > > > > Sincerely,
> > > > > > > >
> > > > > > > > Dmitriy Pavlov
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[dsetrakyan]

Dmitriy, can you please describe your idea in the ticket?

Here is the link for easier access:
https://issues.apache.org/jira/browse/IGNITE-6421

D.

On Mon, Sep 18, 2017 at 7:02 AM, Dmitry Pavlov <[hidden email]>
wrote:

> Dmitriy,
>
> Thank you for your feedback I've created IGNITE-6421 for this change.
>
> Is it common practice to use property file in Ignite utils? I'd like to
> support parse and display data entries and its fields, but it is required
> to configure several folders: 'binary_meta' and 'marshaller' folders.
>
> My idea is to provide .conf file to configure reader once.
>
> Sincerely,
> Dmitriy Pavlov
>
> пн, 18 сент. 2017 г. в 16:10, Dmitriy Setrakyan <[hidden email]>:
>
> > Thanks!
> >
> > After looking at the ticket, the way to launch this utility seems
> complex.
> > Does this utility have a shell script? If not, we should add one with
> > proper command line options, like we do for other scripts.
> >
> > D.
> >
> > On Mon, Sep 18, 2017 at 4:05 AM, Eduard Shangareev <
> > [hidden email]> wrote:
> >
> > > Example of output (will add to wiki):
> > >
> > > [W] InitNewPageRecord [newPageId=0001001c00000067,
> super=PageDeltaRecord
> > > [grpId=2141373875, pageId=0001001c00000067, super=WALRecord [size=41,
> > > chainSize=0, pos=FileWALPointer [idx=1, fileOffset=25406, len=41,
> > > forceFlush=false], type=INIT_NEW_PAGE_RECORD]]]
> > > [W] InitNewPageRecord [newPageId=0001001c00000068,
> super=PageDeltaRecord
> > > [grpId=2141373875, pageId=0001001c00000068, super=WALRecord [size=41,
> > > chainSize=0, pos=FileWALPointer [idx=1, fileOffset=25447, len=41,
> > > forceFlush=false], type=INIT_NEW_PAGE_RECORD]]]
> > > [W] PagesListAddPageRecord [dataPageId=0001001c0000002a,
> > > super=PageDeltaRecord [grpId=2141373875, pageId=0001001c00000068,
> > > super=WALRecord [size=37, chainSize=0, pos=FileWALPointer [idx=1,
> > > fileOffset=25488, len=37, forceFlush=false],
> type=PAGES_LIST_ADD_PAGE]]]
> > > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001001c0000002a,
> > > effectivePageId=0000001c0000002a, grpId=2141373875], page = [
> > > Header [
> > > type=1 (DataPageIO),
> > > ver=1,
> > > crc=0,
> > > pageId=281595235794986(offset=0, flags=1, partId=28, index=42)
> > > ],
> > > DataPageIO [
> > > 0001001c0000002a [4049, 4002, 3955, 3908, 3861, 3814, 3767, 3720, 3673,
> > > 3626][][free=3552, actualFree=-544
> > > ]],
> > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > > fileOffset=25525, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > > [W] InsertRecord [idx=12, io=CacheIdAwareDataLeafIO[ver=1],
> > > rightId=0000000000000000, super=PageDeltaRecord [grpId=2141373875,
> > > pageId=0001001c00000004, super=WALRecord [size=59, chainSize=0,
> > > pos=FileWALPointer [idx=1, fileOffset=29650, len=59, forceFlush=false],
> > > type=BTREE_PAGE_INSERT]]]
> > > [W] DataRecord [writeEntries=[DataEntry [cacheId=-1368047377,
> op=CREATE,
> > > writeVer=GridCacheVersion [topVer=116892426, order=1505412445450,
> > > nodeOrder=2], partId=28, partCnt=69]], super=WALRecord [size=100,
> > > chainSize=0, pos=FileWALPointer [idx=1, fileOffset=29709, len=100,
> > > forceFlush=false], type=DATA_RECORD]]
> > > [W] PageSnapshot [fullPageId = FullPageId [pageId=000100000000000a,
> > > effectivePageId=000000000000000a, grpId=2141373875], page = [
> > > Header [
> > > type=20 (PagePartitionCountersIO),
> > > ver=1,
> > > crc=0,
> > > pageId=281474976710666(offset=0, flags=1, partId=0, index=10)
> > > ],
> > > PagePartitionCounters [
> > > count=1,
> > > lastFlag=true,
> > > nextCountersPageId=0000000000000000,
> > > size={
> > > -1368047377=313
> > > }
> > > ]],
> > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > > fileOffset=29809, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000200000010,
> > > effectivePageId=0000000200000010, grpId=2141373875], page = [
> > > Header [
> > > type=20 (PagePartitionCountersIO),
> > > ver=1,
> > > crc=0,
> > > pageId=281483566645264(offset=0, flags=1, partId=2, index=16)
> > > ],
> > > PagePartitionCounters [
> > > count=1,
> > > lastFlag=true,
> > > nextCountersPageId=0000000000000000,
> > > size={
> > > -1368047377=313
> > > }
> > > ]],
> > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > > fileOffset=33934, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > > [W] PageSnapshot [fullPageId = FullPageId [pageId=000100050000000d,
> > > effectivePageId=000000050000000d, grpId=2141373875], page = [
> > > Header [
> > > type=20 (PagePartitionCountersIO),
> > > ver=1,
> > > crc=0,
> > > pageId=281496451547149(offset=0, flags=1, partId=5, index=13)
> > > ],
> > > PagePartitionCounters [
> > > count=1,
> > > lastFlag=true,
> > > nextCountersPageId=0000000000000000,
> > > size={
> > > -1368047377=313
> > > }
> > > ]],
> > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > > fileOffset=38059, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000900000003,
> > > effectivePageId=0000000900000003, grpId=2141373875], page = [
> > > Header [
> > > type=12 (PagesListMetaIO),
> > > ver=1,
> > > crc=0,
> > > pageId=281513631416323(offset=0, flags=1, partId=9, index=3)
> > > ],
> > > PagesListMeta [
> > > nextMetaPageId=0000000000000000,
> > > count=230,
> > > bucketData={
> > > bucket=163, list=GridLongList [idx=6,
> > > arr=[281513631416558,281513631416559,281513631416560,281513631416561,
> > > 281513631416562,281513631416563]]
> > > bucket=166, list=GridLongList [idx=8,
> > > arr=[281513631416550,281513631416551,281513631416552,281513631416553,
> > > 281513631416554,281513631416555,281513631416556,281513631416557]]
> > > bucket=169, list=GridLongList [idx=8,
> > > arr=[281513631416542,281513631416543,281513631416544,281513631416545,
> > > 281513631416546,281513631416547,281513631416548,281513631416549]]
> > > bucket=172, list=GridLongList [idx=8,
> > > arr=[281513631416534,281513631416535,281513631416536,281513631416537,
> > > 281513631416538,281513631416539,281513631416540,281513631416541]]
> > > bucket=175, list=GridLongList [idx=8,
> > > arr=[281513631416526,281513631416527,281513631416528,281513631416529,
> > > 281513631416530,281513631416531,281513631416532,281513631416533]]
> > > bucket=178, list=GridLongList [idx=8,
> > > arr=[281513631416512,281513631416513,281513631416520,281513631416521,
> > > 281513631416522,281513631416523,281513631416524,281513631416525]]
> > > bucket=181, list=GridLongList [idx=8,
> > > arr=[281513631416510,281513631416511,281513631416514,281513631416515,
> > > 281513631416516,281513631416517,281513631416518,281513631416519]]
> > > bucket=184, list=GridLongList [idx=8,
> > > arr=[281513631416500,281513631416503,281513631416504,281513631416505,
> > > 281513631416506,281513631416507,281513631416508,281513631416509]]
> > > bucket=187, list=GridLongList [idx=8,
> > > arr=[281513631416494,281513631416495,281513631416496,281513631416497,
> > > 281513631416498,281513631416499,281513631416501,281513631416502]]
> > > bucket=190, list=GridLongList [idx=8,
> > > arr=[281513631416484,281513631416485,281513631416486,281513631416487,
> > > 281513631416488,281513631416489,281513631416492,281513631416493]]
> > > bucket=193, list=GridLongList [idx=8,
> > > arr=[281513631416448,281513631416449,281513631416480,281513631416481,
> > > 281513631416482,281513631416483,281513631416490,281513631416491]]
> > > bucket=196, list=GridLongList [idx=8,
> > > arr=[281513631416446,281513631416447,281513631416474,281513631416475,
> > > 281513631416476,281513631416477,281513631416478,281513631416479]]
> > > bucket=199, list=GridLongList [idx=8,
> > > arr=[281513631416444,281513631416445,281513631416468,281513631416469,
> > > 281513631416470,281513631416471,281513631416472,281513631416473]]
> > > bucket=202, list=GridLongList [idx=8,
> > > arr=[281513631416436,281513631416437,281513631416462,281513631416463,
> > > 281513631416464,281513631416465,281513631416466,281513631416467]]
> > > bucket=205, list=GridLongList [idx=8,
> > > arr=[281513631416434,281513631416435,281513631416456,281513631416457,
> > > 281513631416458,281513631416459,281513631416460,281513631416461]]
> > > bucket=209, list=GridLongList [idx=8,
> > > arr=[281513631416426,281513631416427,281513631416450,281513631416451,
> > > 281513631416452,281513631416453,281513631416454,281513631416455]]
> > > bucket=212, list=GridLongList [idx=8,
> > > arr=[281513631416418,281513631416419,281513631416438,281513631416439,
> > > 281513631416440,281513631416441,281513631416442,281513631416443]]
> > > bucket=215, list=GridLongList [idx=8,
> > > arr=[281513631416416,281513631416417,281513631416428,281513631416429,
> > > 281513631416430,281513631416431,281513631416432,281513631416433]]
> > > bucket=218, list=GridLongList [idx=8,
> > > arr=[281513631416414,281513631416415,281513631416420,281513631416421,
> > > 281513631416422,281513631416423,281513631416424,281513631416425]]
> > > bucket=221, list=GridLongList [idx=8,
> > > arr=[281513631416406,281513631416407,281513631416408,281513631416409,
> > > 281513631416410,281513631416411,281513631416412,281513631416413]]
> > > bucket=224, list=GridLongList [idx=8,
> > > arr=[281513631416398,281513631416399,281513631416400,281513631416401,
> > > 281513631416402,281513631416403,281513631416404,281513631416405]]
> > > bucket=227, list=GridLongList [idx=8,
> > > arr=[281513631416390,281513631416391,281513631416392,281513631416393,
> > > 281513631416394,281513631416395,281513631416396,281513631416397]]
> > > bucket=230, list=GridLongList [idx=8,
> > > arr=[281513631416381,281513631416382,281513631416383,281513631416384,
> > > 281513631416385,281513631416386,281513631416387,281513631416388]]
> > > bucket=233, list=GridLongList [idx=8,
> > > arr=[281513631416373,281513631416374,281513631416375,281513631416376,
> > > 281513631416377,281513631416378,281513631416379,281513631416380]]
> > > bucket=236, list=GridLongList [idx=8,
> > > arr=[281513631416365,281513631416366,281513631416367,281513631416368,
> > > 281513631416369,281513631416370,281513631416371,281513631416372]]
> > > bucket=239, list=GridLongList [idx=8,
> > > arr=[281513631416351,281513631416352,281513631416359,281513631416360,
> > > 281513631416361,281513631416362,281513631416363,281513631416364]]
> > > bucket=242, list=GridLongList [idx=8,
> > > arr=[281513631416348,281513631416349,281513631416353,281513631416354,
> > > 281513631416355,281513631416356,281513631416357,281513631416358]]
> > > bucket=245, list=GridLongList [idx=8,
> > > arr=[281513631416335,281513631416336,281513631416337,281513631416338,
> > > 281513631416339,281513631416340,281513631416346,281513631416347]]
> > > bucket=248, list=GridLongList [idx=8,
> > > arr=[281513631416326,281513631416327,281513631416328,281513631416329,
> > > 281513631416330,281513631416331,281513631416343,281513631416344]]
> > > }
> > > ]],
> > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > > fileOffset=42184, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > > [W] PageSnapshot [fullPageId = FullPageId [pageId=000100090000000d,
> > > effectivePageId=000000090000000d, grpId=2141373875], page = [
> > > Header [
> > > type=20 (PagePartitionCountersIO),
> > > ver=1,
> > > crc=0,
> > > pageId=281513631416333(offset=0, flags=1, partId=9, index=13)
> > > ],
> > > PagePartitionCounters [
> > > count=1,
> > > lastFlag=true,
> > > nextCountersPageId=0000000000000000,
> > > size={
> > > -1368047377=198
> > > }
> > > ]],
> > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > > fileOffset=46309, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000900000000,
> > > effectivePageId=0000000900000000, grpId=2141373875], page = [
> > > Header [
> > > type=14 (PagePartitionMetaIO),
> > > ver=1,
> > > crc=0,
> > > pageId=281513631416320(offset=0, flags=1, partId=9, index=0)
> > > ],
> > > PageMeta[
> > > treeRoot=281513631416323,
> > > lastSuccessfulFullSnapshotId=0,
> > > lastSuccessfulSnapshotId=0,
> > > nextSnapshotTag=1,
> > > lastSuccessfulSnapshotTag=0,
> > > lastAllocatedPageCount=0,
> > > candidatePageCount=0
> > > ],
> > > PagePartitionMeta[
> > > size=198,
> > > updateCounter=198,
> > > globalRemoveId=1505412422861000000,
> > > partitionState=1(OWNING),
> > > countersPageId=281513631416333
> > > ]],
> > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > > fileOffset=50434, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000b0000000c,
> > > effectivePageId=0000000b0000000c, grpId=2141373875], page = [
> > > Header [
> > > type=20 (PagePartitionCountersIO),
> > > ver=1,
> > > crc=0,
> > > pageId=281522221350924(offset=0, flags=1, partId=11, index=12)
> > > ],
> > > PagePartitionCounters [
> > > count=1,
> > > lastFlag=true,
> > > nextCountersPageId=0000000000000000,
> > > size={
> > > -1368047377=313
> > > }
> > > ]],
> > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > > fileOffset=54559, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000c0000000f,
> > > effectivePageId=0000000c0000000f, grpId=2141373875], page = [
> > > Header [
> > > type=20 (PagePartitionCountersIO),
> > > ver=1,
> > > crc=0,
> > > pageId=281526516318223(offset=0, flags=1, partId=12, index=15)
> > > ],
> > > PagePartitionCounters [
> > > count=1,
> > > lastFlag=true,
> > > nextCountersPageId=0000000000000000,
> > > size={
> > > -1368047377=313
> > > }
> > > ]],
> > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > > fileOffset=58684, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > >
> > >
> > > On Mon, Sep 18, 2017 at 1:10 PM, Dmitry Pavlov <[hidden email]>
> > > wrote:
> > >
> > > > Hi Igniters,
> > > >
> > > > Page https://cwiki.apache.org/confluence/display/IGNITE/
> > > Ignite+WAL+reader
> > > > to
> > > > collect related information about this prototype was created.
> > > >
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > > >
> > > > пн, 18 сент. 2017 г. в 7:01, Dmitriy Setrakyan <
> [hidden email]
> > >:
> > > >
> > > > > Can we get a sample print out from this utility?
> > > > >
> > > > > On Sun, Sep 17, 2017 at 3:42 PM, Eduard Shangareev <
> > > > > [hidden email]> wrote:
> > > > >
> > > > > > Dmitry,
> > > > > > thank you for starting this topic.
> > > > > >
> > > > > > The IDEA of this utility is very simple - get the possibility to
> > look
> > > > > into
> > > > > > your WAL.
> > > > > > But it also requires understanding what's going on in WAL
> mechanism
> > > (We
> > > > > > need to describe it in more details in the wiki).
> > > > > >
> > > > > > It could be useful for a debug some weird things with recovery
> > > > mechanism.
> > > > > > Initially, it was created for dealing with the issue in
> > IGNITE-6342.
> > > It
> > > > > > helped to find out the error in how we deal with partition size.
> > > > > >
> > > > > > Best regards,
> > > > > > Ed.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sat, Sep 16, 2017 at 2:09 AM, Dmitry Pavlov <
> > > [hidden email]>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Denis,
> > > > > > >
> > > > > > > Thank you for your feedback. It is very important to us to
> > > understand
> > > > > > that
> > > > > > > this tool may be useful not only for Ignite tests but for all
> > > > > developers.
> > > > > > >
> > > > > > > I will create wiki page.
> > > > > > >
> > > > > > > Sincerely,
> > > > > > > Dmitriy Pavlov
> > > > > > >
> > > > > > > сб, 16 сент. 2017 г. в 1:54, Denis Magda <[hidden email]>:
> > > > > > >
> > > > > > > > Dmitriy, Eduard,
> > > > > > > >
> > > > > > > > Sounds like a useful thing. I’ll be happy to give a feedback
> > but
> > > > > need a
> > > > > > > > documentation that explains how to use the tool, what’s the
> > > output
> > > > > > > format,
> > > > > > > > how do decipher the output, etc (basic things). The Wiki
> > > > description
> > > > > > > should
> > > > > > > > be enough.
> > > > > > > >
> > > > > > > > —
> > > > > > > > Denis
> > > > > > > >
> > > > > > > > > On Sep 15, 2017, at 11:36 AM, Dmitry Pavlov <
> > > > [hidden email]
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi Igniters,
> > > > > > > > >
> > > > > > > > > Eduard Shangareev created WAL converter utility which shows
> > WAL
> > > > > > > > > archive/work directory segments content in human readable
> > form.
> > > > > This
> > > > > > > > > utility outputs all WAL records to output stream (may be
> > > > redirected
> > > > > > to
> > > > > > > > > file). How to use tips can be found in issue
> > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-6277
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > This utility does not require Ignite node to be up and
> > running
> > > > and
> > > > > > may
> > > > > > > be
> > > > > > > > > used for offline analysis of copied files. Thanks to Alexey
> > G.
> > > > for
> > > > > > idea
> > > > > > > > and
> > > > > > > > > review. Code is now available in modules/ignite-dev-tools.
> > > > > > > > >
> > > > > > > > > 1. Could you please write if utility can be useful for you?
> > > > > > > > >
> > > > > > > > > 2. Please respond if you need short wiki description.
> > > > > > > > >
> > > > > > > > > 3. What is your opinion what should be next steps to
> develop
> > > this
> > > > > > > > utility?
> > > > > > > > >
> > > > > > > > > 4. Does utility need property file to refer Ignite Work
> > > directory
> > > > > > using
> > > > > > > > > conf file.
> > > > > > > > >
> > > > > > > > > 5. What do you think if utility remains console non
> > > interactive,
> > > > or
> > > > > > > some
> > > > > > > > > UI/Interactive console shell may be preferable?
> > > > > > > > >
> > > > > > > > > Sincerely,
> > > > > > > > >
> > > > > > > > > Dmitriy Pavlov
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Dmitriy Pavlov]

I've updated issue description.

Igniters,

who have already faced with necessity to look into WAL content and/or
latest records?

Sincerely,
Dmitriy Pavlov

пн, 18 сент. 2017 г. в 23:23, Dmitriy Setrakyan <[hidden email]>:

> Dmitriy, can you please describe your idea in the ticket?
>
> Here is the link for easier access:
> https://issues.apache.org/jira/browse/IGNITE-6421
>
> D.
>
> On Mon, Sep 18, 2017 at 7:02 AM, Dmitry Pavlov <[hidden email]>
> wrote:
>
> > Dmitriy,
> >
> > Thank you for your feedback I've created IGNITE-6421 for this change.
> >
> > Is it common practice to use property file in Ignite utils? I'd like to
> > support parse and display data entries and its fields, but it is required
> > to configure several folders: 'binary_meta' and 'marshaller' folders.
> >
> > My idea is to provide .conf file to configure reader once.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > пн, 18 сент. 2017 г. в 16:10, Dmitriy Setrakyan <[hidden email]>:
> >
> > > Thanks!
> > >
> > > After looking at the ticket, the way to launch this utility seems
> > complex.
> > > Does this utility have a shell script? If not, we should add one with
> > > proper command line options, like we do for other scripts.
> > >
> > > D.
> > >
> > > On Mon, Sep 18, 2017 at 4:05 AM, Eduard Shangareev <
> > > [hidden email]> wrote:
> > >
> > > > Example of output (will add to wiki):
> > > >
> > > > [W] InitNewPageRecord [newPageId=0001001c00000067,
> > super=PageDeltaRecord
> > > > [grpId=2141373875, pageId=0001001c00000067, super=WALRecord [size=41,
> > > > chainSize=0, pos=FileWALPointer [idx=1, fileOffset=25406, len=41,
> > > > forceFlush=false], type=INIT_NEW_PAGE_RECORD]]]
> > > > [W] InitNewPageRecord [newPageId=0001001c00000068,
> > super=PageDeltaRecord
> > > > [grpId=2141373875, pageId=0001001c00000068, super=WALRecord [size=41,
> > > > chainSize=0, pos=FileWALPointer [idx=1, fileOffset=25447, len=41,
> > > > forceFlush=false], type=INIT_NEW_PAGE_RECORD]]]
> > > > [W] PagesListAddPageRecord [dataPageId=0001001c0000002a,
> > > > super=PageDeltaRecord [grpId=2141373875, pageId=0001001c00000068,
> > > > super=WALRecord [size=37, chainSize=0, pos=FileWALPointer [idx=1,
> > > > fileOffset=25488, len=37, forceFlush=false],
> > type=PAGES_LIST_ADD_PAGE]]]
> > > > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001001c0000002a,
> > > > effectivePageId=0000001c0000002a, grpId=2141373875], page = [
> > > > Header [
> > > > type=1 (DataPageIO),
> > > > ver=1,
> > > > crc=0,
> > > > pageId=281595235794986(offset=0, flags=1, partId=28, index=42)
> > > > ],
> > > > DataPageIO [
> > > > 0001001c0000002a [4049, 4002, 3955, 3908, 3861, 3814, 3767, 3720,
> 3673,
> > > > 3626][][free=3552, actualFree=-544
> > > > ]],
> > > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer
> [idx=1,
> > > > fileOffset=25525, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > > > [W] InsertRecord [idx=12, io=CacheIdAwareDataLeafIO[ver=1],
> > > > rightId=0000000000000000, super=PageDeltaRecord [grpId=2141373875,
> > > > pageId=0001001c00000004, super=WALRecord [size=59, chainSize=0,
> > > > pos=FileWALPointer [idx=1, fileOffset=29650, len=59,
> forceFlush=false],
> > > > type=BTREE_PAGE_INSERT]]]
> > > > [W] DataRecord [writeEntries=[DataEntry [cacheId=-1368047377,
> > op=CREATE,
> > > > writeVer=GridCacheVersion [topVer=116892426, order=1505412445450,
> > > > nodeOrder=2], partId=28, partCnt=69]], super=WALRecord [size=100,
> > > > chainSize=0, pos=FileWALPointer [idx=1, fileOffset=29709, len=100,
> > > > forceFlush=false], type=DATA_RECORD]]
> > > > [W] PageSnapshot [fullPageId = FullPageId [pageId=000100000000000a,
> > > > effectivePageId=000000000000000a, grpId=2141373875], page = [
> > > > Header [
> > > > type=20 (PagePartitionCountersIO),
> > > > ver=1,
> > > > crc=0,
> > > > pageId=281474976710666(offset=0, flags=1, partId=0, index=10)
> > > > ],
> > > > PagePartitionCounters [
> > > > count=1,
> > > > lastFlag=true,
> > > > nextCountersPageId=0000000000000000,
> > > > size={
> > > > -1368047377=313
> > > > }
> > > > ]],
> > > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer
> [idx=1,
> > > > fileOffset=29809, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > > > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000200000010,
> > > > effectivePageId=0000000200000010, grpId=2141373875], page = [
> > > > Header [
> > > > type=20 (PagePartitionCountersIO),
> > > > ver=1,
> > > > crc=0,
> > > > pageId=281483566645264(offset=0, flags=1, partId=2, index=16)
> > > > ],
> > > > PagePartitionCounters [
> > > > count=1,
> > > > lastFlag=true,
> > > > nextCountersPageId=0000000000000000,
> > > > size={
> > > > -1368047377=313
> > > > }
> > > > ]],
> > > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer
> [idx=1,
> > > > fileOffset=33934, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > > > [W] PageSnapshot [fullPageId = FullPageId [pageId=000100050000000d,
> > > > effectivePageId=000000050000000d, grpId=2141373875], page = [
> > > > Header [
> > > > type=20 (PagePartitionCountersIO),
> > > > ver=1,
> > > > crc=0,
> > > > pageId=281496451547149(offset=0, flags=1, partId=5, index=13)
> > > > ],
> > > > PagePartitionCounters [
> > > > count=1,
> > > > lastFlag=true,
> > > > nextCountersPageId=0000000000000000,
> > > > size={
> > > > -1368047377=313
> > > > }
> > > > ]],
> > > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer
> [idx=1,
> > > > fileOffset=38059, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > > > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000900000003,
> > > > effectivePageId=0000000900000003, grpId=2141373875], page = [
> > > > Header [
> > > > type=12 (PagesListMetaIO),
> > > > ver=1,
> > > > crc=0,
> > > > pageId=281513631416323(offset=0, flags=1, partId=9, index=3)
> > > > ],
> > > > PagesListMeta [
> > > > nextMetaPageId=0000000000000000,
> > > > count=230,
> > > > bucketData={
> > > > bucket=163, list=GridLongList [idx=6,
> > > > arr=[281513631416558,281513631416559,281513631416560,281513631416561,
> > > > 281513631416562,281513631416563]]
> > > > bucket=166, list=GridLongList [idx=8,
> > > > arr=[281513631416550,281513631416551,281513631416552,281513631416553,
> > > > 281513631416554,281513631416555,281513631416556,281513631416557]]
> > > > bucket=169, list=GridLongList [idx=8,
> > > > arr=[281513631416542,281513631416543,281513631416544,281513631416545,
> > > > 281513631416546,281513631416547,281513631416548,281513631416549]]
> > > > bucket=172, list=GridLongList [idx=8,
> > > > arr=[281513631416534,281513631416535,281513631416536,281513631416537,
> > > > 281513631416538,281513631416539,281513631416540,281513631416541]]
> > > > bucket=175, list=GridLongList [idx=8,
> > > > arr=[281513631416526,281513631416527,281513631416528,281513631416529,
> > > > 281513631416530,281513631416531,281513631416532,281513631416533]]
> > > > bucket=178, list=GridLongList [idx=8,
> > > > arr=[281513631416512,281513631416513,281513631416520,281513631416521,
> > > > 281513631416522,281513631416523,281513631416524,281513631416525]]
> > > > bucket=181, list=GridLongList [idx=8,
> > > > arr=[281513631416510,281513631416511,281513631416514,281513631416515,
> > > > 281513631416516,281513631416517,281513631416518,281513631416519]]
> > > > bucket=184, list=GridLongList [idx=8,
> > > > arr=[281513631416500,281513631416503,281513631416504,281513631416505,
> > > > 281513631416506,281513631416507,281513631416508,281513631416509]]
> > > > bucket=187, list=GridLongList [idx=8,
> > > > arr=[281513631416494,281513631416495,281513631416496,281513631416497,
> > > > 281513631416498,281513631416499,281513631416501,281513631416502]]
> > > > bucket=190, list=GridLongList [idx=8,
> > > > arr=[281513631416484,281513631416485,281513631416486,281513631416487,
> > > > 281513631416488,281513631416489,281513631416492,281513631416493]]
> > > > bucket=193, list=GridLongList [idx=8,
> > > > arr=[281513631416448,281513631416449,281513631416480,281513631416481,
> > > > 281513631416482,281513631416483,281513631416490,281513631416491]]
> > > > bucket=196, list=GridLongList [idx=8,
> > > > arr=[281513631416446,281513631416447,281513631416474,281513631416475,
> > > > 281513631416476,281513631416477,281513631416478,281513631416479]]
> > > > bucket=199, list=GridLongList [idx=8,
> > > > arr=[281513631416444,281513631416445,281513631416468,281513631416469,
> > > > 281513631416470,281513631416471,281513631416472,281513631416473]]
> > > > bucket=202, list=GridLongList [idx=8,
> > > > arr=[281513631416436,281513631416437,281513631416462,281513631416463,
> > > > 281513631416464,281513631416465,281513631416466,281513631416467]]
> > > > bucket=205, list=GridLongList [idx=8,
> > > > arr=[281513631416434,281513631416435,281513631416456,281513631416457,
> > > > 281513631416458,281513631416459,281513631416460,281513631416461]]
> > > > bucket=209, list=GridLongList [idx=8,
> > > > arr=[281513631416426,281513631416427,281513631416450,281513631416451,
> > > > 281513631416452,281513631416453,281513631416454,281513631416455]]
> > > > bucket=212, list=GridLongList [idx=8,
> > > > arr=[281513631416418,281513631416419,281513631416438,281513631416439,
> > > > 281513631416440,281513631416441,281513631416442,281513631416443]]
> > > > bucket=215, list=GridLongList [idx=8,
> > > > arr=[281513631416416,281513631416417,281513631416428,281513631416429,
> > > > 281513631416430,281513631416431,281513631416432,281513631416433]]
> > > > bucket=218, list=GridLongList [idx=8,
> > > > arr=[281513631416414,281513631416415,281513631416420,281513631416421,
> > > > 281513631416422,281513631416423,281513631416424,281513631416425]]
> > > > bucket=221, list=GridLongList [idx=8,
> > > > arr=[281513631416406,281513631416407,281513631416408,281513631416409,
> > > > 281513631416410,281513631416411,281513631416412,281513631416413]]
> > > > bucket=224, list=GridLongList [idx=8,
> > > > arr=[281513631416398,281513631416399,281513631416400,281513631416401,
> > > > 281513631416402,281513631416403,281513631416404,281513631416405]]
> > > > bucket=227, list=GridLongList [idx=8,
> > > > arr=[281513631416390,281513631416391,281513631416392,281513631416393,
> > > > 281513631416394,281513631416395,281513631416396,281513631416397]]
> > > > bucket=230, list=GridLongList [idx=8,
> > > > arr=[281513631416381,281513631416382,281513631416383,281513631416384,
> > > > 281513631416385,281513631416386,281513631416387,281513631416388]]
> > > > bucket=233, list=GridLongList [idx=8,
> > > > arr=[281513631416373,281513631416374,281513631416375,281513631416376,
> > > > 281513631416377,281513631416378,281513631416379,281513631416380]]
> > > > bucket=236, list=GridLongList [idx=8,
> > > > arr=[281513631416365,281513631416366,281513631416367,281513631416368,
> > > > 281513631416369,281513631416370,281513631416371,281513631416372]]
> > > > bucket=239, list=GridLongList [idx=8,
> > > > arr=[281513631416351,281513631416352,281513631416359,281513631416360,
> > > > 281513631416361,281513631416362,281513631416363,281513631416364]]
> > > > bucket=242, list=GridLongList [idx=8,
> > > > arr=[281513631416348,281513631416349,281513631416353,281513631416354,
> > > > 281513631416355,281513631416356,281513631416357,281513631416358]]
> > > > bucket=245, list=GridLongList [idx=8,
> > > > arr=[281513631416335,281513631416336,281513631416337,281513631416338,
> > > > 281513631416339,281513631416340,281513631416346,281513631416347]]
> > > > bucket=248, list=GridLongList [idx=8,
> > > > arr=[281513631416326,281513631416327,281513631416328,281513631416329,
> > > > 281513631416330,281513631416331,281513631416343,281513631416344]]
> > > > }
> > > > ]],
> > > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer
> [idx=1,
> > > > fileOffset=42184, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > > > [W] PageSnapshot [fullPageId = FullPageId [pageId=000100090000000d,
> > > > effectivePageId=000000090000000d, grpId=2141373875], page = [
> > > > Header [
> > > > type=20 (PagePartitionCountersIO),
> > > > ver=1,
> > > > crc=0,
> > > > pageId=281513631416333(offset=0, flags=1, partId=9, index=13)
> > > > ],
> > > > PagePartitionCounters [
> > > > count=1,
> > > > lastFlag=true,
> > > > nextCountersPageId=0000000000000000,
> > > > size={
> > > > -1368047377=198
> > > > }
> > > > ]],
> > > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer
> [idx=1,
> > > > fileOffset=46309, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > > > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000900000000,
> > > > effectivePageId=0000000900000000, grpId=2141373875], page = [
> > > > Header [
> > > > type=14 (PagePartitionMetaIO),
> > > > ver=1,
> > > > crc=0,
> > > > pageId=281513631416320(offset=0, flags=1, partId=9, index=0)
> > > > ],
> > > > PageMeta[
> > > > treeRoot=281513631416323,
> > > > lastSuccessfulFullSnapshotId=0,
> > > > lastSuccessfulSnapshotId=0,
> > > > nextSnapshotTag=1,
> > > > lastSuccessfulSnapshotTag=0,
> > > > lastAllocatedPageCount=0,
> > > > candidatePageCount=0
> > > > ],
> > > > PagePartitionMeta[
> > > > size=198,
> > > > updateCounter=198,
> > > > globalRemoveId=1505412422861000000,
> > > > partitionState=1(OWNING),
> > > > countersPageId=281513631416333
> > > > ]],
> > > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer
> [idx=1,
> > > > fileOffset=50434, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > > > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000b0000000c,
> > > > effectivePageId=0000000b0000000c, grpId=2141373875], page = [
> > > > Header [
> > > > type=20 (PagePartitionCountersIO),
> > > > ver=1,
> > > > crc=0,
> > > > pageId=281522221350924(offset=0, flags=1, partId=11, index=12)
> > > > ],
> > > > PagePartitionCounters [
> > > > count=1,
> > > > lastFlag=true,
> > > > nextCountersPageId=0000000000000000,
> > > > size={
> > > > -1368047377=313
> > > > }
> > > > ]],
> > > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer
> [idx=1,
> > > > fileOffset=54559, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > > > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000c0000000f,
> > > > effectivePageId=0000000c0000000f, grpId=2141373875], page = [
> > > > Header [
> > > > type=20 (PagePartitionCountersIO),
> > > > ver=1,
> > > > crc=0,
> > > > pageId=281526516318223(offset=0, flags=1, partId=12, index=15)
> > > > ],
> > > > PagePartitionCounters [
> > > > count=1,
> > > > lastFlag=true,
> > > > nextCountersPageId=0000000000000000,
> > > > size={
> > > > -1368047377=313
> > > > }
> > > > ]],
> > > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer
> [idx=1,
> > > > fileOffset=58684, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > > >
> > > >
> > > > On Mon, Sep 18, 2017 at 1:10 PM, Dmitry Pavlov <
> [hidden email]>
> > > > wrote:
> > > >
> > > > > Hi Igniters,
> > > > >
> > > > > Page https://cwiki.apache.org/confluence/display/IGNITE/
> > > > Ignite+WAL+reader
> > > > > to
> > > > > collect related information about this prototype was created.
> > > > >
> > > > > Sincerely,
> > > > > Dmitriy Pavlov
> > > > >
> > > > > пн, 18 сент. 2017 г. в 7:01, Dmitriy Setrakyan <
> > [hidden email]
> > > >:
> > > > >
> > > > > > Can we get a sample print out from this utility?
> > > > > >
> > > > > > On Sun, Sep 17, 2017 at 3:42 PM, Eduard Shangareev <
> > > > > > [hidden email]> wrote:
> > > > > >
> > > > > > > Dmitry,
> > > > > > > thank you for starting this topic.
> > > > > > >
> > > > > > > The IDEA of this utility is very simple - get the possibility
> to
> > > look
> > > > > > into
> > > > > > > your WAL.
> > > > > > > But it also requires understanding what's going on in WAL
> > mechanism
> > > > (We
> > > > > > > need to describe it in more details in the wiki).
> > > > > > >
> > > > > > > It could be useful for a debug some weird things with recovery
> > > > > mechanism.
> > > > > > > Initially, it was created for dealing with the issue in
> > > IGNITE-6342.
> > > > It
> > > > > > > helped to find out the error in how we deal with partition
> size.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Ed.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Sat, Sep 16, 2017 at 2:09 AM, Dmitry Pavlov <
> > > > [hidden email]>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Denis,
> > > > > > > >
> > > > > > > > Thank you for your feedback. It is very important to us to
> > > > understand
> > > > > > > that
> > > > > > > > this tool may be useful not only for Ignite tests but for all
> > > > > > developers.
> > > > > > > >
> > > > > > > > I will create wiki page.
> > > > > > > >
> > > > > > > > Sincerely,
> > > > > > > > Dmitriy Pavlov
> > > > > > > >
> > > > > > > > сб, 16 сент. 2017 г. в 1:54, Denis Magda <[hidden email]
> >:
> > > > > > > >
> > > > > > > > > Dmitriy, Eduard,
> > > > > > > > >
> > > > > > > > > Sounds like a useful thing. I’ll be happy to give a
> feedback
> > > but
> > > > > > need a
> > > > > > > > > documentation that explains how to use the tool, what’s the
> > > > output
> > > > > > > > format,
> > > > > > > > > how do decipher the output, etc (basic things). The Wiki
> > > > > description
> > > > > > > > should
> > > > > > > > > be enough.
> > > > > > > > >
> > > > > > > > > —
> > > > > > > > > Denis
> > > > > > > > >
> > > > > > > > > > On Sep 15, 2017, at 11:36 AM, Dmitry Pavlov <
> > > > > [hidden email]
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Igniters,
> > > > > > > > > >
> > > > > > > > > > Eduard Shangareev created WAL converter utility which
> shows
> > > WAL
> > > > > > > > > > archive/work directory segments content in human readable
> > > form.
> > > > > > This
> > > > > > > > > > utility outputs all WAL records to output stream (may be
> > > > > redirected
> > > > > > > to
> > > > > > > > > > file). How to use tips can be found in issue
> > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-6277
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > This utility does not require Ignite node to be up and
> > > running
> > > > > and
> > > > > > > may
> > > > > > > > be
> > > > > > > > > > used for offline analysis of copied files. Thanks to
> Alexey
> > > G.
> > > > > for
> > > > > > > idea
> > > > > > > > > and
> > > > > > > > > > review. Code is now available in
> modules/ignite-dev-tools.
> > > > > > > > > >
> > > > > > > > > > 1. Could you please write if utility can be useful for
> you?
> > > > > > > > > >
> > > > > > > > > > 2. Please respond if you need short wiki description.
> > > > > > > > > >
> > > > > > > > > > 3. What is your opinion what should be next steps to
> > develop
> > > > this
> > > > > > > > > utility?
> > > > > > > > > >
> > > > > > > > > > 4. Does utility need property file to refer Ignite Work
> > > > directory
> > > > > > > using
> > > > > > > > > > conf file.
> > > > > > > > > >
> > > > > > > > > > 5. What do you think if utility remains console non
> > > > interactive,
> > > > > or
> > > > > > > > some
> > > > > > > > > > UI/Interactive console shell may be preferable?
> > > > > > > > > >
> > > > > > > > > > Sincerely,
> > > > > > > > > >
> > > > > > > > > > Dmitriy Pavlov
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>