Why portable marshaller handle Date and Timestamp in the same way?

[Vladimir Ozerov]

Igniters,

For some reason we "merged" Date and Timestamp types in portable
marshaller. They are both written in the same format with the same type ID.
And how date is interpreted on read side - as Date or as Timestamp -
depends on configuration flag "use timestamp".

Is there are reason why we do this? Transparent conversion from Timestamp
to Date is invalid use case because it leads to data loss. Looks like we
can separate these types from each other and remove this strange
configuration parameter.

Thoughts?

Vladimir.

[Denis Magda]

I would definitely remove such a mapping if no one explains a reason we
have it.

--
Denis

On 10/6/2015 10:26 AM, Vladimir Ozerov wrote:

> Igniters,
>
> For some reason we "merged" Date and Timestamp types in portable
> marshaller. They are both written in the same format with the same type ID.
> And how date is interpreted on read side - as Date or as Timestamp -
> depends on configuration flag "use timestamp".
>
> Is there are reason why we do this? Transparent conversion from Timestamp
> to Date is invalid use case because it leads to data loss. Looks like we
> can separate these types from each other and remove this strange
> configuration parameter.
>
> Thoughts?
>
> Vladimir.
>

[Pavel Tupitsyn-3]

Keep in mind that separating them can introduce difficulties for other
platforms.
For example, DateTime in .Net has more precision (100ns vs 1ms in Java).
Serializing this in Java format will lead to data loss. Serializing .Net
DateTime as Timestamp will preserve precision, but may hurt
interoperability.

Thanks,

On Tue, Oct 6, 2015 at 10:29 AM, Denis Magda <[hidden email]> wrote:

> I would definitely remove such a mapping if no one explains a reason we
> have it.
>
> --
> Denis
>
>
> On 10/6/2015 10:26 AM, Vladimir Ozerov wrote:
>
>> Igniters,
>>
>> For some reason we "merged" Date and Timestamp types in portable
>> marshaller. They are both written in the same format with the same type
>> ID.
>> And how date is interpreted on read side - as Date or as Timestamp -
>> depends on configuration flag "use timestamp".
>>
>> Is there are reason why we do this? Transparent conversion from Timestamp
>> to Date is invalid use case because it leads to data loss. Looks like we
>> can separate these types from each other and remove this strange
>> configuration parameter.
>>
>> Thoughts?
>>
>> Vladimir.
>>
>>
>

--
--
Pavel Tupitsyn
GridGain Systems, Inc.
www.gridgain.com

[dsetrakyan]

On Tue, Oct 6, 2015 at 1:39 AM, Pavel Tupitsyn <[hidden email]>
wrote:

> Keep in mind that separating them can introduce difficulties for other
> platforms.
> For example, DateTime in .Net has more precision (100ns vs 1ms in Java).
> Serializing this in Java format will lead to data loss. Serializing .Net
> DateTime as Timestamp will preserve precision, but may hurt
> interoperability.
>

Thanks Pavel. This is exactly the reason why Date vs Timestamp selection it
is implemented right now via a configuration flag.

>
> Thanks,
>
> On Tue, Oct 6, 2015 at 10:29 AM, Denis Magda <[hidden email]> wrote:
>
> > I would definitely remove such a mapping if no one explains a reason we
> > have it.
> >
> > --
> > Denis
> >
> >
> > On 10/6/2015 10:26 AM, Vladimir Ozerov wrote:
> >
> >> Igniters,
> >>
> >> For some reason we "merged" Date and Timestamp types in portable
> >> marshaller. They are both written in the same format with the same type
> >> ID.
> >> And how date is interpreted on read side - as Date or as Timestamp -
> >> depends on configuration flag "use timestamp".
> >>
> >> Is there are reason why we do this? Transparent conversion from
> Timestamp
> >> to Date is invalid use case because it leads to data loss. Looks like we
> >> can separate these types from each other and remove this strange
> >> configuration parameter.
> >>
> >> Thoughts?
> >>
> >> Vladimir.
> >>
> >>
> >
>
>
> --
> --
> Pavel Tupitsyn
> GridGain Systems, Inc.
> www.gridgain.com
>

[Vladimir Ozerov]

This doesn't answers the question. First, Java Timestamp has greater
precision than .Net DateTime, so silent data loss could happen in this case
as well. Second, "use timestamp" is defined on class level. It means we
cannot handle a class which have both Date and Timestamp fields.

Looks like a bug and/or invalid design for me.

On Tue, Oct 6, 2015 at 12:21 PM, Dmitriy Setrakyan <[hidden email]>
wrote:

> On Tue, Oct 6, 2015 at 1:39 AM, Pavel Tupitsyn <[hidden email]>
> wrote:
>
> > Keep in mind that separating them can introduce difficulties for other
> > platforms.
> > For example, DateTime in .Net has more precision (100ns vs 1ms in Java).
> > Serializing this in Java format will lead to data loss. Serializing .Net
> > DateTime as Timestamp will preserve precision, but may hurt
> > interoperability.
> >
>
> Thanks Pavel. This is exactly the reason why Date vs Timestamp selection it
> is implemented right now via a configuration flag.
>
>
> >
> > Thanks,
> >
> > On Tue, Oct 6, 2015 at 10:29 AM, Denis Magda <[hidden email]>
> wrote:
> >
> > > I would definitely remove such a mapping if no one explains a reason we
> > > have it.
> > >
> > > --
> > > Denis
> > >
> > >
> > > On 10/6/2015 10:26 AM, Vladimir Ozerov wrote:
> > >
> > >> Igniters,
> > >>
> > >> For some reason we "merged" Date and Timestamp types in portable
> > >> marshaller. They are both written in the same format with the same
> type
> > >> ID.
> > >> And how date is interpreted on read side - as Date or as Timestamp -
> > >> depends on configuration flag "use timestamp".
> > >>
> > >> Is there are reason why we do this? Transparent conversion from
> > Timestamp
> > >> to Date is invalid use case because it leads to data loss. Looks like
> we
> > >> can separate these types from each other and remove this strange
> > >> configuration parameter.
> > >>
> > >> Thoughts?
> > >>
> > >> Vladimir.
> > >>
> > >>
> > >
> >
> >
> > --
> > --
> > Pavel Tupitsyn
> > GridGain Systems, Inc.
> > www.gridgain.com
> >
>

[dsetrakyan]

On Tue, Oct 6, 2015 at 2:42 AM, Vladimir Ozerov <[hidden email]>
wrote:

> This doesn't answers the question. First, Java Timestamp has greater
> precision than .Net DateTime, so silent data loss could happen in this case
> as well. Second, "use timestamp" is defined on class level. It means we
> cannot handle a class which have both Date and Timestamp fields.
>
> Looks like a bug and/or invalid design for me.
>

Agree, current design is not ideal. Vladimir, do you have other suggestions?

>
> On Tue, Oct 6, 2015 at 12:21 PM, Dmitriy Setrakyan <[hidden email]>
> wrote:
>
> > On Tue, Oct 6, 2015 at 1:39 AM, Pavel Tupitsyn <[hidden email]>
> > wrote:
> >
> > > Keep in mind that separating them can introduce difficulties for other
> > > platforms.
> > > For example, DateTime in .Net has more precision (100ns vs 1ms in
> Java).
> > > Serializing this in Java format will lead to data loss. Serializing
> .Net
> > > DateTime as Timestamp will preserve precision, but may hurt
> > > interoperability.
> > >
> >
> > Thanks Pavel. This is exactly the reason why Date vs Timestamp selection
> it
> > is implemented right now via a configuration flag.
> >
> >
> > >
> > > Thanks,
> > >
> > > On Tue, Oct 6, 2015 at 10:29 AM, Denis Magda <[hidden email]>
> > wrote:
> > >
> > > > I would definitely remove such a mapping if no one explains a reason
> we
> > > > have it.
> > > >
> > > > --
> > > > Denis
> > > >
> > > >
> > > > On 10/6/2015 10:26 AM, Vladimir Ozerov wrote:
> > > >
> > > >> Igniters,
> > > >>
> > > >> For some reason we "merged" Date and Timestamp types in portable
> > > >> marshaller. They are both written in the same format with the same
> > type
> > > >> ID.
> > > >> And how date is interpreted on read side - as Date or as Timestamp -
> > > >> depends on configuration flag "use timestamp".
> > > >>
> > > >> Is there are reason why we do this? Transparent conversion from
> > > Timestamp
> > > >> to Date is invalid use case because it leads to data loss. Looks
> like
> > we
> > > >> can separate these types from each other and remove this strange
> > > >> configuration parameter.
> > > >>
> > > >> Thoughts?
> > > >>
> > > >> Vladimir.
> > > >>
> > > >>
> > > >
> > >
> > >
> > > --
> > > --
> > > Pavel Tupitsyn
> > > GridGain Systems, Inc.
> > > www.gridgain.com
> > >
> >
>

[yzhdanov]

2015-10-06 12:45 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:

> On Tue, Oct 6, 2015 at 2:42 AM, Vladimir Ozerov <[hidden email]>
> wrote:
>
> > This doesn't answers the question. First, Java Timestamp has greater
> > precision than .Net DateTime, so silent data loss could happen in this
> case
> > as well. Second, "use timestamp" is defined on class level. It means we
> > cannot handle a class which have both Date and Timestamp fields.
> >
> > Looks like a bug and/or invalid design for me.
> >
>
> Agree, current design is not ideal. Vladimir, do you have other
> suggestions?

how about writing at max precision possible (+ proper type ID) and
interpreting binary data on read depending on (a) portable reader method
call or (b) on actual field type.

--Yakov

[Vladimir Ozerov]

Yakov, this could work in .Net where you have real generics. But it will
not work in Java in general case due to type erasure - you simply cannot
infer the type.

Let's look closely to this:
Date      Java: 10^-3
Timestamp Java: 10^-9
DateTime  .Net: 10^-7

What we see here, is that mapping Java Date to .Net DateTime is alomst
certainly a bad thing because we loos too much data. But interoping between
Timestamp and DateTime is more or less sensible, we loose only 0.1-s of
microseconds.

I would suggest the following solution:
1) Fully decouple Date and Timestamp in Java. These are completely
different types from Java perspective, H2 perspective (see GridH2Date,
GridH2Timestamp), any data database perspective, etc..
2) Map .Net DateTime to Java Timestamp with warning about possible
precision loss.

Vladimir.

On Tue, Oct 6, 2015 at 1:08 PM, Yakov Zhdanov <[hidden email]> wrote:

> 2015-10-06 12:45 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
>
> > On Tue, Oct 6, 2015 at 2:42 AM, Vladimir Ozerov <[hidden email]>
> > wrote:
> >
> > > This doesn't answers the question. First, Java Timestamp has greater
> > > precision than .Net DateTime, so silent data loss could happen in this
> > case
> > > as well. Second, "use timestamp" is defined on class level. It means we
> > > cannot handle a class which have both Date and Timestamp fields.
> > >
> > > Looks like a bug and/or invalid design for me.
> > >
> >
> > Agree, current design is not ideal. Vladimir, do you have other
> > suggestions?
>
>
> how about writing at max precision possible (+ proper type ID) and
> interpreting binary data on read depending on (a) portable reader method
> call or (b) on actual field type.
>
> --Yakov
>

[Branko Čibej]

On 06.10.2015 12:26, Vladimir Ozerov wrote:

> Yakov, this could work in .Net where you have real generics. But it will
> not work in Java in general case due to type erasure - you simply cannot
> infer the type.
>
> Let's look closely to this:
> Date      Java: 10^-3
> Timestamp Java: 10^-9
> DateTime  .Net: 10^-7
>
> What we see here, is that mapping Java Date to .Net DateTime is alomst
> certainly a bad thing because we loos too much data. But interoping between
> Timestamp and DateTime is more or less sensible, we loose only 0.1-s of
> microseconds.
>
> I would suggest the following solution:
> 1) Fully decouple Date and Timestamp in Java. These are completely
> different types from Java perspective, H2 perspective (see GridH2Date,
> GridH2Timestamp), any data database perspective, etc..
> 2) Map .Net DateTime to Java Timestamp with warning about possible
> precision loss.

From the peanut gallery ... it seems like a really bad idea to design a
marshalling format based on what some language standard library happens
to provide. IMO, the way to do this is to define your own max precision
timestamp type, marshal it at full precision, and provide conversions to
standard types. That way your users can choose to use your type which
provides full precision on all platforms, or decide to use the standard
types with the potential loss of precision that entails; but it becomes
*their* decision, not a limitation set by the library.

-- Brane

> On Tue, Oct 6, 2015 at 1:08 PM, Yakov Zhdanov <[hidden email]> wrote:
>
>> 2015-10-06 12:45 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
>>
>>> On Tue, Oct 6, 2015 at 2:42 AM, Vladimir Ozerov <[hidden email]>
>>> wrote:
>>>
>>>> This doesn't answers the question. First, Java Timestamp has greater
>>>> precision than .Net DateTime, so silent data loss could happen in this
>>> case
>>>> as well. Second, "use timestamp" is defined on class level. It means we
>>>> cannot handle a class which have both Date and Timestamp fields.
>>>>
>>>> Looks like a bug and/or invalid design for me.
>>>>
>>> Agree, current design is not ideal. Vladimir, do you have other
>>> suggestions?
>>
>> how about writing at max precision possible (+ proper type ID) and
>> interpreting binary data on read depending on (a) portable reader method
>> call or (b) on actual field type.
>>
>> --Yakov
>>

[Vladimir Ozerov]

Brane,

I agree that ideally no direct mapping should exist between types with
different size or precision. And of course we can introduce a new type
which will have the same characteristics on all platforms.
The downside is that users will have to learn new types and use them
instead of standard ones (like DateTime in .Net) what is not always
possible. Moreover, implicit mappings from .Net to Java types are important
for some Ignite internals, like indexing.
This is why I think it is better to map .Net DateTime to Java Timestamp.
The only situation when data loss is possible is when timestamp is created
in Java, but read in .Net. In this corner case we can advise users to use
their own types with no precision loss.

Vladimir.


On Tue, Oct 6, 2015 at 3:57 PM, Branko Čibej <[hidden email]> wrote:

> On 06.10.2015 12:26, Vladimir Ozerov wrote:
> > Yakov, this could work in .Net where you have real generics. But it will
> > not work in Java in general case due to type erasure - you simply cannot
> > infer the type.
> >
> > Let's look closely to this:
> > Date      Java: 10^-3
> > Timestamp Java: 10^-9
> > DateTime  .Net: 10^-7
> >
> > What we see here, is that mapping Java Date to .Net DateTime is alomst
> > certainly a bad thing because we loos too much data. But interoping
> between
> > Timestamp and DateTime is more or less sensible, we loose only 0.1-s of
> > microseconds.
> >
> > I would suggest the following solution:
> > 1) Fully decouple Date and Timestamp in Java. These are completely
> > different types from Java perspective, H2 perspective (see GridH2Date,
> > GridH2Timestamp), any data database perspective, etc..
> > 2) Map .Net DateTime to Java Timestamp with warning about possible
> > precision loss.
>
> From the peanut gallery ... it seems like a really bad idea to design a
> marshalling format based on what some language standard library happens
> to provide. IMO, the way to do this is to define your own max precision
> timestamp type, marshal it at full precision, and provide conversions to
> standard types. That way your users can choose to use your type which
> provides full precision on all platforms, or decide to use the standard
> types with the potential loss of precision that entails; but it becomes
> *their* decision, not a limitation set by the library.
>
> -- Brane
>
>
> > On Tue, Oct 6, 2015 at 1:08 PM, Yakov Zhdanov <[hidden email]>
> wrote:
> >
> >> 2015-10-06 12:45 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
> >>
> >>> On Tue, Oct 6, 2015 at 2:42 AM, Vladimir Ozerov <[hidden email]>
> >>> wrote:
> >>>
> >>>> This doesn't answers the question. First, Java Timestamp has greater
> >>>> precision than .Net DateTime, so silent data loss could happen in this
> >>> case
> >>>> as well. Second, "use timestamp" is defined on class level. It means
> we
> >>>> cannot handle a class which have both Date and Timestamp fields.
> >>>>
> >>>> Looks like a bug and/or invalid design for me.
> >>>>
> >>> Agree, current design is not ideal. Vladimir, do you have other
> >>> suggestions?
> >>
> >> how about writing at max precision possible (+ proper type ID) and
> >> interpreting binary data on read depending on (a) portable reader method
> >> call or (b) on actual field type.
> >>
> >> --Yakov
> >>
>
>