May be I am missing something but here's how HDFS works at this level (in a
very drafty description)...
- a client ask NN to create new file
- a client asks for a first block and starts writing the data into a
designated DN
- the data is written in chunks, that are reported to the NN to record
'available to read' length. It is more complex that just that because
involves a DN-pipelining, but isn't material for this conversation
- if a client crashes in a middle of the writing, the lease will remain and
can be recovered if a client were to re-connect and continued the writing
- if client never comes back the block replicas will remain incomplete and NN
will treat them accordingly
Basically, meta and data are always written separately by design. Looks like
you want to implement an improvement on top of what is already there, if I
read this correctly. Perhaps, the JIRA needs a bit more detailed description
or this is just me?
Thanks,
Cos
On Fri, Aug 28, 2015 at 05:21PM, Ivan Veselovskiy wrote:
> Hi, colleagues,
> working on IGFS failover functionality (IGNITE-1298, IGNITE-1277) we
> encountered the following issue: we would like to track IGFS meta cache
> updates in Ignite cluster to be able to track write progress of IGFS files
> -- this will allow us to clean up garbage data blocks belonging to
> partially written files when a writing node has crashed.
> To do that we would like to add a new field into IgfsFileInfo class, but
> that will break backwards compatibility in case old node works with new
> nodes in the same cluster, because of [de]serialization issues.
> So, the question is: is that okay to break the compatibility in such a way?
> If not, what is the nearest major release where this compatibility can be
> broken?
> Thanks.
> --ivan
Locked
