HDFS CacheStore

[Cristian Malinescu]

Hello folks - reloading on the topic to add a HDFS CacheStore module, is
anybody interested in brainstorming upon the idea? I want to be consistent
and make sure I capture most of the ideas, opinions and options before
proceeding brute force :). As long as HDFS paradigm is append-only, maybe a
plain integration with HBase would be a better option as going this path
Ignite will offload the need to manage HDFS compaction to HBase.
Cheers
Cris

[Konstantin Boudnik-2]

Cristian,

Great idea!
a couple of comments:
 - HDFS supports truncate since at least v. 2.7.0 (see HDFS-3107)
 - integration with HBase would increase the operational complexity, which is
   already high in HBase, to new levels. Besides, using HBase as the
   compaction mechanism shouldn't be called HDFS CacheStore
 - do you think this can be combined with IGFS on top of HDFS?

Thoughts?
  Cos

On Fri, May 13, 2016 at 10:35AM, Cristian Malinescu wrote:
> Hello folks - reloading on the topic to add a HDFS CacheStore module, is
> anybody interested in brainstorming upon the idea? I want to be consistent
> and make sure I capture most of the ideas, opinions and options before
> proceeding brute force :). As long as HDFS paradigm is append-only, maybe a
> plain integration with HBase would be a better option as going this path
> Ignite will offload the need to manage HDFS compaction to HBase.
> Cheers
> Cris