Moving binary metadata to PDS storage folder

[Данилов Семён]

Hello!

I would like to propose moving /binary_meta and /marshaller folders to the PDS folder.

Motivation: data, directly related to the persistence, is stored outside the persistence dir, which can lead to various issues and also is not very convenient to use. In particular, with k8s, deployment disk that is attached to a container can not be accessed from other containers or outside of k8s. In case if support will need to drop persistence except data, there will be no way to recover due to binary metadata is required to process PDS files.

I created an issue (https://issues.apache.org/jira/browse/IGNITE-12994) and a pull request(https://github.com/apache/ignite/pull/7792) that fixes the issue.

In that PR I made the following:


*     store binary meta and marshaller data inside db/ folder
*     if binary meta of marshaller are found in "legacy" locations -- safely move them to new locations during the node startup


Kind regards,

Semyon Danilov.

[slava.koptilin]

Hello Semyon,

This is a good and long-awaited improvement! Thank you for your efforts!

Thanks,
S.

вт, 12 мая 2020 г. в 15:11, Данилов Семён <[hidden email]>:

> Hello!
>
> I would like to propose moving /binary_meta and /marshaller folders to the
> PDS folder.
>
> Motivation: data, directly related to the persistence, is stored outside
> the persistence dir, which can lead to various issues and also is not very
> convenient to use. In particular, with k8s, deployment disk that is
> attached to a container can not be accessed from other containers or
> outside of k8s. In case if support will need to drop persistence except
> data, there will be no way to recover due to binary metadata is required to
> process PDS files.
>
> I created an issue (https://issues.apache.org/jira/browse/IGNITE-12994) and a
> pull request(https://github.com/apache/ignite/pull/7792) that fixes the
> issue.
>
> In that PR I made the following:
>
>
> *     store binary meta and marshaller data inside db/ folder
> *     if binary meta of marshaller are found in "legacy" locations --
> safely move them to new locations during the node startup
>
>
> Kind regards,
>
> Semyon Danilov.
>

[antonovsergey93]

Hello Semyon,

This is a good idea!

вт, 12 мая 2020 г. в 15:53, Вячеслав Коптилин <[hidden email]>:

> Hello Semyon,
>
> This is a good and long-awaited improvement! Thank you for your efforts!
>
> Thanks,
> S.
>
> вт, 12 мая 2020 г. в 15:11, Данилов Семён <[hidden email]>:
>
> > Hello!
> >
> > I would like to propose moving /binary_meta and /marshaller folders to
> the
> > PDS folder.
> >
> > Motivation: data, directly related to the persistence, is stored outside
> > the persistence dir, which can lead to various issues and also is not
> very
> > convenient to use. In particular, with k8s, deployment disk that is
> > attached to a container can not be accessed from other containers or
> > outside of k8s. In case if support will need to drop persistence except
> > data, there will be no way to recover due to binary metadata is required
> to
> > process PDS files.
> >
> > I created an issue (https://issues.apache.org/jira/browse/IGNITE-12994)
> and a
> > pull request(https://github.com/apache/ignite/pull/7792) that fixes the
> > issue.
> >
> > In that PR I made the following:
> >
> >
> > *     store binary meta and marshaller data inside db/ folder
> > *     if binary meta of marshaller are found in "legacy" locations --
> > safely move them to new locations during the node startup
> >
> >
> > Kind regards,
> >
> > Semyon Danilov.
> >
>

--
BR, Sergey Antonov

[Denis Mekhanikov]

Sounds great!

It happens pretty frequently that users migrate to a new version of
Ignite and preserve persistence files only without caring too much about
the work folder. But it turns out, that the work folder actually has
some important stuff.
This improvement should help with this issue.

What's about in-memory mode? As far as I know, we write binary metadata
to disk even when no persistence is configured. Do you plan to address
it in any way?

Denis

On 12.05.2020 15:56, Sergey Antonov wrote:

> Hello Semyon,
>
> This is a good idea!
>
> вт, 12 мая 2020 г. в 15:53, Вячеслав Коптилин <[hidden email]>:
>
>> Hello Semyon,
>>
>> This is a good and long-awaited improvement! Thank you for your efforts!
>>
>> Thanks,
>> S.
>>
>> вт, 12 мая 2020 г. в 15:11, Данилов Семён <[hidden email]>:
>>
>>> Hello!
>>>
>>> I would like to propose moving /binary_meta and /marshaller folders to
>> the
>>> PDS folder.
>>>
>>> Motivation: data, directly related to the persistence, is stored outside
>>> the persistence dir, which can lead to various issues and also is not
>> very
>>> convenient to use. In particular, with k8s, deployment disk that is
>>> attached to a container can not be accessed from other containers or
>>> outside of k8s. In case if support will need to drop persistence except
>>> data, there will be no way to recover due to binary metadata is required
>> to
>>> process PDS files.
>>>
>>> I created an issue (https://issues.apache.org/jira/browse/IGNITE-12994)
>> and a
>>> pull request(https://github.com/apache/ignite/pull/7792) that fixes the
>>> issue.
>>>
>>> In that PR I made the following:
>>>
>>>
>>> *     store binary meta and marshaller data inside db/ folder
>>> *     if binary meta of marshaller are found in "legacy" locations --
>>> safely move them to new locations during the node startup
>>>
>>>
>>> Kind regards,
>>>
>>> Semyon Danilov.
>>>
>

[Maxim Muzafarov]

Folks,

I think it's important to discuss the following question regarding this thread:
Should we consider moving the migration procedure from the java
production code to migration scripts?

From my understanding, keeping all such things in java production
source code has some disadvantages:
1. It executes only once at the migration stage.
2. It affects the complexity of the source code and code maintenance.
3. Node crash cases must be covered during the migration procedure.
4. It affects the production usage e.g. the process doesn't have the
right access to the old directory (migration already completed) and
will fail the node start.


The right behavior from my point should be:
1. Change the default path of binary/marshaller directory to the new one.
2. Provide migration scripts for users.

WDYT?

On Wed, 13 May 2020 at 21:10, Denis Mekhanikov <[hidden email]> wrote:

>
> Sounds great!
>
> It happens pretty frequently that users migrate to a new version of
> Ignite and preserve persistence files only without caring too much about
> the work folder. But it turns out, that the work folder actually has
> some important stuff.
> This improvement should help with this issue.
>
> What's about in-memory mode? As far as I know, we write binary metadata
> to disk even when no persistence is configured. Do you plan to address
> it in any way?
>
> Denis
>
> On 12.05.2020 15:56, Sergey Antonov wrote:
> > Hello Semyon,
> >
> > This is a good idea!
> >
> > вт, 12 мая 2020 г. в 15:53, Вячеслав Коптилин <[hidden email]>:
> >
> >> Hello Semyon,
> >>
> >> This is a good and long-awaited improvement! Thank you for your efforts!
> >>
> >> Thanks,
> >> S.
> >>
> >> вт, 12 мая 2020 г. в 15:11, Данилов Семён <[hidden email]>:
> >>
> >>> Hello!
> >>>
> >>> I would like to propose moving /binary_meta and /marshaller folders to
> >> the
> >>> PDS folder.
> >>>
> >>> Motivation: data, directly related to the persistence, is stored outside
> >>> the persistence dir, which can lead to various issues and also is not
> >> very
> >>> convenient to use. In particular, with k8s, deployment disk that is
> >>> attached to a container can not be accessed from other containers or
> >>> outside of k8s. In case if support will need to drop persistence except
> >>> data, there will be no way to recover due to binary metadata is required
> >> to
> >>> process PDS files.
> >>>
> >>> I created an issue (https://issues.apache.org/jira/browse/IGNITE-12994)
> >> and a
> >>> pull request(https://github.com/apache/ignite/pull/7792) that fixes the
> >>> issue.
> >>>
> >>> In that PR I made the following:
> >>>
> >>>
> >>> *     store binary meta and marshaller data inside db/ folder
> >>> *     if binary meta of marshaller are found in "legacy" locations --
> >>> safely move them to new locations during the node startup
> >>>
> >>>
> >>> Kind regards,
> >>>
> >>> Semyon Danilov.
> >>>
> >

[Denis Mekhanikov]

Maxim,

This way we'll introduce a migration procedure between versions, which
we currently don't have. Different Ignite versions are compatible by
persistence storage. If we add a migration script, we need to decide,
whether we need to run it every time when the version is upgraded, or
only some specific versions are affected.

I suggest having a procedure that will look for metadata in the work
directory, and if it finds it there, then the node will use it.
Otherwise the persistence directory is used.

Denis

On 13.05.2020 21:40, Maxim Muzafarov wrote:

> Folks,
>
> I think it's important to discuss the following question regarding this thread:
> Should we consider moving the migration procedure from the java
> production code to migration scripts?
>
>  From my understanding, keeping all such things in java production
> source code has some disadvantages:
> 1. It executes only once at the migration stage.
> 2. It affects the complexity of the source code and code maintenance.
> 3. Node crash cases must be covered during the migration procedure.
> 4. It affects the production usage e.g. the process doesn't have the
> right access to the old directory (migration already completed) and
> will fail the node start.
>
>
> The right behavior from my point should be:
> 1. Change the default path of binary/marshaller directory to the new one.
> 2. Provide migration scripts for users.
>
> WDYT?
>
> On Wed, 13 May 2020 at 21:10, Denis Mekhanikov <[hidden email]> wrote:
>> Sounds great!
>>
>> It happens pretty frequently that users migrate to a new version of
>> Ignite and preserve persistence files only without caring too much about
>> the work folder. But it turns out, that the work folder actually has
>> some important stuff.
>> This improvement should help with this issue.
>>
>> What's about in-memory mode? As far as I know, we write binary metadata
>> to disk even when no persistence is configured. Do you plan to address
>> it in any way?
>>
>> Denis
>>
>> On 12.05.2020 15:56, Sergey Antonov wrote:
>>> Hello Semyon,
>>>
>>> This is a good idea!
>>>
>>> вт, 12 мая 2020 г. в 15:53, Вячеслав Коптилин <[hidden email]>:
>>>
>>>> Hello Semyon,
>>>>
>>>> This is a good and long-awaited improvement! Thank you for your efforts!
>>>>
>>>> Thanks,
>>>> S.
>>>>
>>>> вт, 12 мая 2020 г. в 15:11, Данилов Семён <[hidden email]>:
>>>>
>>>>> Hello!
>>>>>
>>>>> I would like to propose moving /binary_meta and /marshaller folders to
>>>> the
>>>>> PDS folder.
>>>>>
>>>>> Motivation: data, directly related to the persistence, is stored outside
>>>>> the persistence dir, which can lead to various issues and also is not
>>>> very
>>>>> convenient to use. In particular, with k8s, deployment disk that is
>>>>> attached to a container can not be accessed from other containers or
>>>>> outside of k8s. In case if support will need to drop persistence except
>>>>> data, there will be no way to recover due to binary metadata is required
>>>> to
>>>>> process PDS files.
>>>>>
>>>>> I created an issue (https://issues.apache.org/jira/browse/IGNITE-12994)
>>>> and a
>>>>> pull request(https://github.com/apache/ignite/pull/7792) that fixes the
>>>>> issue.
>>>>>
>>>>> In that PR I made the following:
>>>>>
>>>>>
>>>>> *     store binary meta and marshaller data inside db/ folder
>>>>> *     if binary meta of marshaller are found in "legacy" locations --
>>>>> safely move them to new locations during the node startup
>>>>>
>>>>>
>>>>> Kind regards,
>>>>>
>>>>> Semyon Danilov.
>>>>>

[Данилов Семён]

Hello Denis,

I found out that without persistence being enabled, no metadata is written on disk (where is an if clause at the start of every method of BinaryMetadataFileStore) and everything is stored in metadataLocalCache.

Semyon.

13.05.2020, 21:10, "Denis Mekhanikov" <[hidden email]>:

> Sounds great!
>
> It happens pretty frequently that users migrate to a new version of
> Ignite and preserve persistence files only without caring too much about
> the work folder. But it turns out, that the work folder actually has
> some important stuff.
> This improvement should help with this issue.
>
> What's about in-memory mode? As far as I know, we write binary metadata
> to disk even when no persistence is configured. Do you plan to address
> it in any way?
>
> Denis
>
> On 12.05.2020 15:56, Sergey Antonov wrote:
>>  Hello Semyon,
>>
>>  This is a good idea!
>>
>>  вт, 12 мая 2020 г. в 15:53, Вячеслав Коптилин <[hidden email]>:
>>
>>>  Hello Semyon,
>>>
>>>  This is a good and long-awaited improvement! Thank you for your efforts!
>>>
>>>  Thanks,
>>>  S.
>>>
>>>  вт, 12 мая 2020 г. в 15:11, Данилов Семён <[hidden email]>:
>>>
>>>>  Hello!
>>>>
>>>>  I would like to propose moving /binary_meta and /marshaller folders to
>>>  the
>>>>  PDS folder.
>>>>
>>>>  Motivation: data, directly related to the persistence, is stored outside
>>>>  the persistence dir, which can lead to various issues and also is not
>>>  very
>>>>  convenient to use. In particular, with k8s, deployment disk that is
>>>>  attached to a container can not be accessed from other containers or
>>>>  outside of k8s. In case if support will need to drop persistence except
>>>>  data, there will be no way to recover due to binary metadata is required
>>>  to
>>>>  process PDS files.
>>>>
>>>>  I created an issue (https://issues.apache.org/jira/browse/IGNITE-12994)
>>>  and a
>>>>  pull request(https://github.com/apache/ignite/pull/7792) that fixes the
>>>>  issue.
>>>>
>>>>  In that PR I made the following:
>>>>
>>>>  * store binary meta and marshaller data inside db/ folder
>>>>  * if binary meta of marshaller are found in "legacy" locations --
>>>>  safely move them to new locations during the node startup
>>>>
>>>>  Kind regards,
>>>>
>>>>  Semyon Danilov.

[Anton Kalashnikov]

Hello,

I agree with the described disadvantages(which Maxim mentioned) and I think it is a good idea to discuss a new way of pds migration in an extra ticket. But also, I agree with Denis that the script will add ambiguity(it's not clear which script should be used and when) if we add it. In my opinion, maybe the better way is to store some migration metainformation in ignite metastorage, and in case, when this information not found, execute some migration code - but of course, there is a lot to discuss.

According to this task, I don't see any problem if we left it as is because the overhead is minimum right now(only on start and only if the old folder exists). Also, we can easily drop(or move) this piece of code if we implement a new migration way in some time.

-- 
Best regards,
Anton Kalashnikov


13.05.2020, 21:59, "Denis Mekhanikov" <[hidden email]>:

> Maxim,
>
> This way we'll introduce a migration procedure between versions, which
> we currently don't have. Different Ignite versions are compatible by
> persistence storage. If we add a migration script, we need to decide,
> whether we need to run it every time when the version is upgraded, or
> only some specific versions are affected.
>
> I suggest having a procedure that will look for metadata in the work
> directory, and if it finds it there, then the node will use it.
> Otherwise the persistence directory is used.
>
> Denis
>
> On 13.05.2020 21:40, Maxim Muzafarov wrote:
>>  Folks,
>>
>>  I think it's important to discuss the following question regarding this thread:
>>  Should we consider moving the migration procedure from the java
>>  production code to migration scripts?
>>
>>   From my understanding, keeping all such things in java production
>>  source code has some disadvantages:
>>  1. It executes only once at the migration stage.
>>  2. It affects the complexity of the source code and code maintenance.
>>  3. Node crash cases must be covered during the migration procedure.
>>  4. It affects the production usage e.g. the process doesn't have the
>>  right access to the old directory (migration already completed) and
>>  will fail the node start.
>>
>>  The right behavior from my point should be:
>>  1. Change the default path of binary/marshaller directory to the new one.
>>  2. Provide migration scripts for users.
>>
>>  WDYT?
>>
>>  On Wed, 13 May 2020 at 21:10, Denis Mekhanikov <[hidden email]> wrote:
>>>  Sounds great!
>>>
>>>  It happens pretty frequently that users migrate to a new version of
>>>  Ignite and preserve persistence files only without caring too much about
>>>  the work folder. But it turns out, that the work folder actually has
>>>  some important stuff.
>>>  This improvement should help with this issue.
>>>
>>>  What's about in-memory mode? As far as I know, we write binary metadata
>>>  to disk even when no persistence is configured. Do you plan to address
>>>  it in any way?
>>>
>>>  Denis
>>>
>>>  On 12.05.2020 15:56, Sergey Antonov wrote:
>>>>  Hello Semyon,
>>>>
>>>>  This is a good idea!
>>>>
>>>>  вт, 12 мая 2020 г. в 15:53, Вячеслав Коптилин <[hidden email]>:
>>>>
>>>>>  Hello Semyon,
>>>>>
>>>>>  This is a good and long-awaited improvement! Thank you for your efforts!
>>>>>
>>>>>  Thanks,
>>>>>  S.
>>>>>
>>>>>  вт, 12 мая 2020 г. в 15:11, Данилов Семён <[hidden email]>:
>>>>>
>>>>>>  Hello!
>>>>>>
>>>>>>  I would like to propose moving /binary_meta and /marshaller folders to
>>>>>  the
>>>>>>  PDS folder.
>>>>>>
>>>>>>  Motivation: data, directly related to the persistence, is stored outside
>>>>>>  the persistence dir, which can lead to various issues and also is not
>>>>>  very
>>>>>>  convenient to use. In particular, with k8s, deployment disk that is
>>>>>>  attached to a container can not be accessed from other containers or
>>>>>>  outside of k8s. In case if support will need to drop persistence except
>>>>>>  data, there will be no way to recover due to binary metadata is required
>>>>>  to
>>>>>>  process PDS files.
>>>>>>
>>>>>>  I created an issue (https://issues.apache.org/jira/browse/IGNITE-12994)
>>>>>  and a
>>>>>>  pull request(https://github.com/apache/ignite/pull/7792) that fixes the
>>>>>>  issue.
>>>>>>
>>>>>>  In that PR I made the following:
>>>>>>
>>>>>>  * store binary meta and marshaller data inside db/ folder
>>>>>>  * if binary meta of marshaller are found in "legacy" locations --
>>>>>>  safely move them to new locations during the node startup
>>>>>>
>>>>>>  Kind regards,
>>>>>>
>>>>>>  Semyon Danilov.

[Maxim Muzafarov]

Folks,


I don't agree that it should be a single script for the whole migration process.
I don't think we should have some kind of automated `decision-maker`
processes for the migration procedure.

We should have well-defined documentation pages + the `migration`
directory with a single .sh per single operation.
For example:
1. binary_marshaller_move.sh -- move to binary, marshaller directories.
2. nodeId_to_consistentId.sh -- change a node directory to the new format.

The migration for each new version will require different sets of
these scripts (supposedly less than 10 in total).
I think users will be able to run each script manually to achieve PDS
compatibility according to their version.

On Fri, 15 May 2020 at 17:19, Anton Kalashnikov <[hidden email]> wrote:

>
> Hello,
>
> I agree with the described disadvantages(which Maxim mentioned) and I think it is a good idea to discuss a new way of pds migration in an extra ticket. But also, I agree with Denis that the script will add ambiguity(it's not clear which script should be used and when) if we add it. In my opinion, maybe the better way is to store some migration metainformation in ignite metastorage, and in case, when this information not found, execute some migration code - but of course, there is a lot to discuss.
>
> According to this task, I don't see any problem if we left it as is because the overhead is minimum right now(only on start and only if the old folder exists). Also, we can easily drop(or move) this piece of code if we implement a new migration way in some time.
>
> --
> Best regards,
> Anton Kalashnikov
>
>
> 13.05.2020, 21:59, "Denis Mekhanikov" <[hidden email]>:
> > Maxim,
> >
> > This way we'll introduce a migration procedure between versions, which
> > we currently don't have. Different Ignite versions are compatible by
> > persistence storage. If we add a migration script, we need to decide,
> > whether we need to run it every time when the version is upgraded, or
> > only some specific versions are affected.
> >
> > I suggest having a procedure that will look for metadata in the work
> > directory, and if it finds it there, then the node will use it.
> > Otherwise the persistence directory is used.
> >
> > Denis
> >
> > On 13.05.2020 21:40, Maxim Muzafarov wrote:
> >>  Folks,
> >>
> >>  I think it's important to discuss the following question regarding this thread:
> >>  Should we consider moving the migration procedure from the java
> >>  production code to migration scripts?
> >>
> >>   From my understanding, keeping all such things in java production
> >>  source code has some disadvantages:
> >>  1. It executes only once at the migration stage.
> >>  2. It affects the complexity of the source code and code maintenance.
> >>  3. Node crash cases must be covered during the migration procedure.
> >>  4. It affects the production usage e.g. the process doesn't have the
> >>  right access to the old directory (migration already completed) and
> >>  will fail the node start.
> >>
> >>  The right behavior from my point should be:
> >>  1. Change the default path of binary/marshaller directory to the new one.
> >>  2. Provide migration scripts for users.
> >>
> >>  WDYT?
> >>
> >>  On Wed, 13 May 2020 at 21:10, Denis Mekhanikov <[hidden email]> wrote:
> >>>  Sounds great!
> >>>
> >>>  It happens pretty frequently that users migrate to a new version of
> >>>  Ignite and preserve persistence files only without caring too much about
> >>>  the work folder. But it turns out, that the work folder actually has
> >>>  some important stuff.
> >>>  This improvement should help with this issue.
> >>>
> >>>  What's about in-memory mode? As far as I know, we write binary metadata
> >>>  to disk even when no persistence is configured. Do you plan to address
> >>>  it in any way?
> >>>
> >>>  Denis
> >>>
> >>>  On 12.05.2020 15:56, Sergey Antonov wrote:
> >>>>  Hello Semyon,
> >>>>
> >>>>  This is a good idea!
> >>>>
> >>>>  вт, 12 мая 2020 г. в 15:53, Вячеслав Коптилин <[hidden email]>:
> >>>>
> >>>>>  Hello Semyon,
> >>>>>
> >>>>>  This is a good and long-awaited improvement! Thank you for your efforts!
> >>>>>
> >>>>>  Thanks,
> >>>>>  S.
> >>>>>
> >>>>>  вт, 12 мая 2020 г. в 15:11, Данилов Семён <[hidden email]>:
> >>>>>
> >>>>>>  Hello!
> >>>>>>
> >>>>>>  I would like to propose moving /binary_meta and /marshaller folders to
> >>>>>  the
> >>>>>>  PDS folder.
> >>>>>>
> >>>>>>  Motivation: data, directly related to the persistence, is stored outside
> >>>>>>  the persistence dir, which can lead to various issues and also is not
> >>>>>  very
> >>>>>>  convenient to use. In particular, with k8s, deployment disk that is
> >>>>>>  attached to a container can not be accessed from other containers or
> >>>>>>  outside of k8s. In case if support will need to drop persistence except
> >>>>>>  data, there will be no way to recover due to binary metadata is required
> >>>>>  to
> >>>>>>  process PDS files.
> >>>>>>
> >>>>>>  I created an issue (https://issues.apache.org/jira/browse/IGNITE-12994)
> >>>>>  and a
> >>>>>>  pull request(https://github.com/apache/ignite/pull/7792) that fixes the
> >>>>>>  issue.
> >>>>>>
> >>>>>>  In that PR I made the following:
> >>>>>>
> >>>>>>  * store binary meta and marshaller data inside db/ folder
> >>>>>>  * if binary meta of marshaller are found in "legacy" locations --
> >>>>>>  safely move them to new locations during the node startup
> >>>>>>
> >>>>>>  Kind regards,
> >>>>>>
> >>>>>>  Semyon Danilov.

[Denis Mekhanikov]

Users shouldn't care about binary_meta and marshaller directories. Those
are internal details. Having to go through the documentation on those
things to figure out whether you need to run each of the provided scripts
or not will lead to terrible user experience.
If we add the binary_marshaller_move.sh script for this specific case, it
will stay with us forever just because migration from Ignite 2.8 to 2.9
required running this thing.

Denis

[Данилов Семён]

Hello everyone,

can we move forward with the merge of IGNITE-12994?

Semyon

18.05.2020, 19:08, "Denis Mekhanikov" <[hidden email]>:
> Users shouldn't care about binary_meta and marshaller directories. Those
> are internal details. Having to go through the documentation on those
> things to figure out whether you need to run each of the provided scripts
> or not will lead to terrible user experience.
> If we add the binary_marshaller_move.sh script for this specific case, it
> will stay with us forever just because migration from Ignite 2.8 to 2.9
> required running this thing.
>
> Denis