[DISCUSSION] System cache persistence.

[Andrew Mashenkov]

Hi Igniters,

I'd like to start a discussion to avoid system cache usage with persistence.

System cache is used in number of component internals.  No one cares
system cache can have stale data after grid restart as it wasn't impossible
before 2.1.
From Ignite 2.1 version it is possible to be persistent that may affect
components behavior:
Compute, IGFS, ServiceGrid, DataStructures.

What's wrong?
1. System cache persistent only if default region is configured as
persistent (and vice versa).
This is non-obvious and can causes unpredictable issues.

2. Any change in system cache requires distributed transaction that may
causes a deadlock.
We already avoid its usage in BinaryMarshaller and (almost) in recently
reworked ServiceGrid due to the "deadlock" reason.


What has been affected? and we can do?
3. IGFS
AFAIK, IGFS support is going to be discontinued. There is nothing to do if
IGFS will be removed in 3.0.

4. DataStrucutres
Its looks broken (may be partially) as I see  CacheDataStructuresManager
uses on-heap maps for id->structure mapping for some structures.
Look like it is safe to deprecate persistence for datastructures for now
and rework them separately.
Also, from user perspective, I'd expect datastructures persistence be
configured in some separate place or in datastructure configuration.

5. Compute
Let's rework this to use Metastore.

6. ServiceGrid.
We can use Metastore and drop old-services later.

5. Some 3-rd party plugins may be affected.
Of course, there is no compatibility guarantee if someone uses internal
components, but the issue #1 can make user frustrated.
We can prevent system cache being persistent.

Do we really ever need System cache with persistence enabled?
Thoughts?

I've create a ticket for this [1].

[1] https://issues.apache.org/jira/browse/IGNITE-11483

--
Best regards,
Andrey V. Mashenkov

[daradurvs]

Hi, Andrey!

>> 6. ServiceGrid.
>> We can use Metastore and drop old-services later.

As you mentioned new Service Grid does not use system cache.

Legacy implementation (GridServiceProcessor), uses system cache
WITHOUT persistence since 2.3 release and does not restore services
state at node restart [1].

[1] https://issues.apache.org/jira/browse/IGNITE-6629

On Tue, Mar 5, 2019 at 4:21 PM Andrey Mashenkov
<[hidden email]> wrote:

>
> Hi Igniters,
>
> I'd like to start a discussion to avoid system cache usage with persistence.
>
> System cache is used in number of component internals.  No one cares
> system cache can have stale data after grid restart as it wasn't impossible
> before 2.1.
> From Ignite 2.1 version it is possible to be persistent that may affect
> components behavior:
> Compute, IGFS, ServiceGrid, DataStructures.
>
> What's wrong?
> 1. System cache persistent only if default region is configured as
> persistent (and vice versa).
> This is non-obvious and can causes unpredictable issues.
>
> 2. Any change in system cache requires distributed transaction that may
> causes a deadlock.
> We already avoid its usage in BinaryMarshaller and (almost) in recently
> reworked ServiceGrid due to the "deadlock" reason.
>
>
> What has been affected? and we can do?
> 3. IGFS
> AFAIK, IGFS support is going to be discontinued. There is nothing to do if
> IGFS will be removed in 3.0.
>
> 4. DataStrucutres
> Its looks broken (may be partially) as I see  CacheDataStructuresManager
> uses on-heap maps for id->structure mapping for some structures.
> Look like it is safe to deprecate persistence for datastructures for now
> and rework them separately.
> Also, from user perspective, I'd expect datastructures persistence be
> configured in some separate place or in datastructure configuration.
>
> 5. Compute
> Let's rework this to use Metastore.
>
> 6. ServiceGrid.
> We can use Metastore and drop old-services later.
>
> 5. Some 3-rd party plugins may be affected.
> Of course, there is no compatibility guarantee if someone uses internal
> components, but the issue #1 can make user frustrated.
> We can prevent system cache being persistent.
>
> Do we really ever need System cache with persistence enabled?
> Thoughts?
>
> I've create a ticket for this [1].
>
> [1] https://issues.apache.org/jira/browse/IGNITE-11483
>
> --
> Best regards,
> Andrey V. Mashenkov

--
Best Regards, Vyacheslav D.