|
|
(I posted the same question on the users forum but got no answer, so I try here. I'm sorry if it's not the right place.)
I need a data structure that holds 1 billion of objects. Each object will consist of 4 integers. This will be used to model a large 2-dimensional grid.
The structure is initially empty (or can be full but have objects with all fields set to 0), but will be filled in with elements inserted at random places. The data structure is immutable in the sense that each index can be written only once. This invariant should be guaranteed even when there are multiple concurrent clients using it - writing to an already written index should return an error. All writes should be fast and have low latency.
Another important feature is that the memory layout of the structure matches the array indexes - element with index n is before the element n + 1. There will be a need to read 16384 consecutive elements quickly and that memory layout will be very cache friendly for such operation.
Elements written to the structure need to be durable. The plan is to replicate the writes on 2 nodes. Writes to about 1% of the indexes are more important than others and it should be guaranteed that acknowledged writes for that indexes survive a complete crash of one node. There should be no time window during which such writes are lost. But it's OK for the rest of 99% other writes to have a chance of data loss for performance reasons. So we need async replication with a possibility to enable synchronous replication for certain puts.
This all sounds like a replicated, durable array which isn't provided by Ignite, but I can imagine implementing this as a specialised map/key-value store, with ints as keys, disabled hashing of keys and the storage allocated off-heap to guarantee that keys that are close to each other have consecutive locations in RAM. Ideally, it should also be possible to specify whether synchronous replication is needed or not for each write operation. Is it possible to achieve using Apache Ignite?
|
Locked
