Sorting fields of Binarilyzable objects on write

classic Classic list List threaded Threaded
27 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Sorting fields of Binarilyzable objects on write

Vladimir Ozerov
Zapalniki,

Inspired by IGNITE-4669 (.NET: Sort binary object fields) [1].

Currently we sort binary object fields before when writing them to the
output stream in case of standard (Serializable) objects and
BinaryObjectBuilder. This makes sense as we have stable binary object
representation irrespective of fields order, which is very important e.g.
for DML. And it works fine from performance perspective as well:
- For standard classes we sort fields only once during initialization;
- For builder we have to maintain the whole object graph in memory before
writing anyway as builder is mutable, so sorting doesn't impose serious
performance hit.

But what to do with Binarilyzable classes? We can sort their fields as
well, but it means that:
1) We will not be able to write them directly to stream. Instead, we will
accumulate values in memory, and write only when the whole object graph is
known.
2) Currently reads are mostly sequential from memory perspective. With this
change reads will become random.

So we will loose both read and write serialization performance. How do you
think - do we need this change or not?

Vladimir.

[1] https://issues.apache.org/jira/browse/IGNITE-4669
Reply | Threaded
Open this post in threaded view
|

Re: Sorting fields of Binarilyzable objects on write

Sergi
Could you please elaborate why would we want to sort fields in Binarilyzable
classes?

If you are taking from stable binary representation perspective, then I
think it is a problem of user, but not ours.

Sergi

2017-04-10 17:53 GMT+03:00 Vladimir Ozerov <[hidden email]>:

> Zapalniki,
>
> Inspired by IGNITE-4669 (.NET: Sort binary object fields) [1].
>
> Currently we sort binary object fields before when writing them to the
> output stream in case of standard (Serializable) objects and
> BinaryObjectBuilder. This makes sense as we have stable binary object
> representation irrespective of fields order, which is very important e.g.
> for DML. And it works fine from performance perspective as well:
> - For standard classes we sort fields only once during initialization;
> - For builder we have to maintain the whole object graph in memory before
> writing anyway as builder is mutable, so sorting doesn't impose serious
> performance hit.
>
> But what to do with Binarilyzable classes? We can sort their fields as
> well, but it means that:
> 1) We will not be able to write them directly to stream. Instead, we will
> accumulate values in memory, and write only when the whole object graph is
> known.
> 2) Currently reads are mostly sequential from memory perspective. With this
> change reads will become random.
>
> So we will loose both read and write serialization performance. How do you
> think - do we need this change or not?
>
> Vladimir.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-4669
>
Reply | Threaded
Open this post in threaded view
|

Re: Sorting fields of Binarilyzable objects on write

Pavel Tupitsyn
I agree with performance concerns.
BinarYLizable (why don't we rename it to Binarizable already, btw?) used to
be
the fastest approach to serialization (and it makes sense that manual
approach is the best).
With sorting it won't be so anymore.

On the other hand, incorrect ordering will break DML in a very unobvious
way.
We don't want loads of "I've inserted with DML and cache.get does not work"
user list threads.

To solve this, we can add add field order check to Binarylizable, something
like
if (fieldName.compareTo(prevFieldName) < 0) throw ...
This should be fairly cheap and will make sure that binary representation
is stable.

Thoughts?

On Mon, Apr 10, 2017 at 6:07 PM, Sergi Vladykin <[hidden email]>
wrote:

> Could you please elaborate why would we want to sort fields in
> Binarilyzable
> classes?
>
> If you are taking from stable binary representation perspective, then I
> think it is a problem of user, but not ours.
>
> Sergi
>
> 2017-04-10 17:53 GMT+03:00 Vladimir Ozerov <[hidden email]>:
>
> > Zapalniki,
> >
> > Inspired by IGNITE-4669 (.NET: Sort binary object fields) [1].
> >
> > Currently we sort binary object fields before when writing them to the
> > output stream in case of standard (Serializable) objects and
> > BinaryObjectBuilder. This makes sense as we have stable binary object
> > representation irrespective of fields order, which is very important e.g.
> > for DML. And it works fine from performance perspective as well:
> > - For standard classes we sort fields only once during initialization;
> > - For builder we have to maintain the whole object graph in memory before
> > writing anyway as builder is mutable, so sorting doesn't impose serious
> > performance hit.
> >
> > But what to do with Binarilyzable classes? We can sort their fields as
> > well, but it means that:
> > 1) We will not be able to write them directly to stream. Instead, we will
> > accumulate values in memory, and write only when the whole object graph
> is
> > known.
> > 2) Currently reads are mostly sequential from memory perspective. With
> this
> > change reads will become random.
> >
> > So we will loose both read and write serialization performance. How do
> you
> > think - do we need this change or not?
> >
> > Vladimir.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-4669
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Sorting fields of Binarilyzable objects on write

dsetrakyan
In reply to this post by Sergi
I would agree that it should be on a user to always sort the fields, if we
make it a part of the contract. However, in this case, we should always
throw exception if user somehow provides fields in the wrong order.

D.

On Mon, Apr 10, 2017 at 8:07 AM, Sergi Vladykin <[hidden email]>
wrote:

> Could you please elaborate why would we want to sort fields in
> Binarilyzable
> classes?
>
> If you are taking from stable binary representation perspective, then I
> think it is a problem of user, but not ours.
>
> Sergi
>
> 2017-04-10 17:53 GMT+03:00 Vladimir Ozerov <[hidden email]>:
>
> > Zapalniki,
> >
> > Inspired by IGNITE-4669 (.NET: Sort binary object fields) [1].
> >
> > Currently we sort binary object fields before when writing them to the
> > output stream in case of standard (Serializable) objects and
> > BinaryObjectBuilder. This makes sense as we have stable binary object
> > representation irrespective of fields order, which is very important e.g.
> > for DML. And it works fine from performance perspective as well:
> > - For standard classes we sort fields only once during initialization;
> > - For builder we have to maintain the whole object graph in memory before
> > writing anyway as builder is mutable, so sorting doesn't impose serious
> > performance hit.
> >
> > But what to do with Binarilyzable classes? We can sort their fields as
> > well, but it means that:
> > 1) We will not be able to write them directly to stream. Instead, we will
> > accumulate values in memory, and write only when the whole object graph
> is
> > known.
> > 2) Currently reads are mostly sequential from memory perspective. With
> this
> > change reads will become random.
> >
> > So we will loose both read and write serialization performance. How do
> you
> > think - do we need this change or not?
> >
> > Vladimir.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-4669
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Sorting fields of Binarilyzable objects on write

Sergi
What is correct or incorrect ordering for DML?

Sergi

2017-04-10 18:14 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:

> I would agree that it should be on a user to always sort the fields, if we
> make it a part of the contract. However, in this case, we should always
> throw exception if user somehow provides fields in the wrong order.
>
> D.
>
> On Mon, Apr 10, 2017 at 8:07 AM, Sergi Vladykin <[hidden email]>
> wrote:
>
> > Could you please elaborate why would we want to sort fields in
> > Binarilyzable
> > classes?
> >
> > If you are taking from stable binary representation perspective, then I
> > think it is a problem of user, but not ours.
> >
> > Sergi
> >
> > 2017-04-10 17:53 GMT+03:00 Vladimir Ozerov <[hidden email]>:
> >
> > > Zapalniki,
> > >
> > > Inspired by IGNITE-4669 (.NET: Sort binary object fields) [1].
> > >
> > > Currently we sort binary object fields before when writing them to the
> > > output stream in case of standard (Serializable) objects and
> > > BinaryObjectBuilder. This makes sense as we have stable binary object
> > > representation irrespective of fields order, which is very important
> e.g.
> > > for DML. And it works fine from performance perspective as well:
> > > - For standard classes we sort fields only once during initialization;
> > > - For builder we have to maintain the whole object graph in memory
> before
> > > writing anyway as builder is mutable, so sorting doesn't impose serious
> > > performance hit.
> > >
> > > But what to do with Binarilyzable classes? We can sort their fields as
> > > well, but it means that:
> > > 1) We will not be able to write them directly to stream. Instead, we
> will
> > > accumulate values in memory, and write only when the whole object graph
> > is
> > > known.
> > > 2) Currently reads are mostly sequential from memory perspective. With
> > this
> > > change reads will become random.
> > >
> > > So we will loose both read and write serialization performance. How do
> > you
> > > think - do we need this change or not?
> > >
> > > Vladimir.
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-4669
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Sorting fields of Binarilyzable objects on write

Pavel Tupitsyn
Sergi, DML writes fields in alphabetic order and computes hash code
accordingly.
If user-defined Binarylizable impl uses different order, hash codes will be
inconsistent.

On Mon, Apr 10, 2017 at 6:18 PM, Sergi Vladykin <[hidden email]>
wrote:

> What is correct or incorrect ordering for DML?
>
> Sergi
>
> 2017-04-10 18:14 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
>
> > I would agree that it should be on a user to always sort the fields, if
> we
> > make it a part of the contract. However, in this case, we should always
> > throw exception if user somehow provides fields in the wrong order.
> >
> > D.
> >
> > On Mon, Apr 10, 2017 at 8:07 AM, Sergi Vladykin <
> [hidden email]>
> > wrote:
> >
> > > Could you please elaborate why would we want to sort fields in
> > > Binarilyzable
> > > classes?
> > >
> > > If you are taking from stable binary representation perspective, then I
> > > think it is a problem of user, but not ours.
> > >
> > > Sergi
> > >
> > > 2017-04-10 17:53 GMT+03:00 Vladimir Ozerov <[hidden email]>:
> > >
> > > > Zapalniki,
> > > >
> > > > Inspired by IGNITE-4669 (.NET: Sort binary object fields) [1].
> > > >
> > > > Currently we sort binary object fields before when writing them to
> the
> > > > output stream in case of standard (Serializable) objects and
> > > > BinaryObjectBuilder. This makes sense as we have stable binary object
> > > > representation irrespective of fields order, which is very important
> > e.g.
> > > > for DML. And it works fine from performance perspective as well:
> > > > - For standard classes we sort fields only once during
> initialization;
> > > > - For builder we have to maintain the whole object graph in memory
> > before
> > > > writing anyway as builder is mutable, so sorting doesn't impose
> serious
> > > > performance hit.
> > > >
> > > > But what to do with Binarilyzable classes? We can sort their fields
> as
> > > > well, but it means that:
> > > > 1) We will not be able to write them directly to stream. Instead, we
> > will
> > > > accumulate values in memory, and write only when the whole object
> graph
> > > is
> > > > known.
> > > > 2) Currently reads are mostly sequential from memory perspective.
> With
> > > this
> > > > change reads will become random.
> > > >
> > > > So we will loose both read and write serialization performance. How
> do
> > > you
> > > > think - do we need this change or not?
> > > >
> > > > Vladimir.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-4669
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Sorting fields of Binarilyzable objects on write

Sergi
The decision to use alphabetic order looks strange here. Using order
provided in QueryEntity and require from user to have the same order
in Binarylizable
looks more reasonable.

Sergi

2017-04-10 18:21 GMT+03:00 Pavel Tupitsyn <[hidden email]>:

> Sergi, DML writes fields in alphabetic order and computes hash code
> accordingly.
> If user-defined Binarylizable impl uses different order, hash codes will be
> inconsistent.
>
> On Mon, Apr 10, 2017 at 6:18 PM, Sergi Vladykin <[hidden email]>
> wrote:
>
> > What is correct or incorrect ordering for DML?
> >
> > Sergi
> >
> > 2017-04-10 18:14 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
> >
> > > I would agree that it should be on a user to always sort the fields, if
> > we
> > > make it a part of the contract. However, in this case, we should always
> > > throw exception if user somehow provides fields in the wrong order.
> > >
> > > D.
> > >
> > > On Mon, Apr 10, 2017 at 8:07 AM, Sergi Vladykin <
> > [hidden email]>
> > > wrote:
> > >
> > > > Could you please elaborate why would we want to sort fields in
> > > > Binarilyzable
> > > > classes?
> > > >
> > > > If you are taking from stable binary representation perspective,
> then I
> > > > think it is a problem of user, but not ours.
> > > >
> > > > Sergi
> > > >
> > > > 2017-04-10 17:53 GMT+03:00 Vladimir Ozerov <[hidden email]>:
> > > >
> > > > > Zapalniki,
> > > > >
> > > > > Inspired by IGNITE-4669 (.NET: Sort binary object fields) [1].
> > > > >
> > > > > Currently we sort binary object fields before when writing them to
> > the
> > > > > output stream in case of standard (Serializable) objects and
> > > > > BinaryObjectBuilder. This makes sense as we have stable binary
> object
> > > > > representation irrespective of fields order, which is very
> important
> > > e.g.
> > > > > for DML. And it works fine from performance perspective as well:
> > > > > - For standard classes we sort fields only once during
> > initialization;
> > > > > - For builder we have to maintain the whole object graph in memory
> > > before
> > > > > writing anyway as builder is mutable, so sorting doesn't impose
> > serious
> > > > > performance hit.
> > > > >
> > > > > But what to do with Binarilyzable classes? We can sort their fields
> > as
> > > > > well, but it means that:
> > > > > 1) We will not be able to write them directly to stream. Instead,
> we
> > > will
> > > > > accumulate values in memory, and write only when the whole object
> > graph
> > > > is
> > > > > known.
> > > > > 2) Currently reads are mostly sequential from memory perspective.
> > With
> > > > this
> > > > > change reads will become random.
> > > > >
> > > > > So we will loose both read and write serialization performance. How
> > do
> > > > you
> > > > > think - do we need this change or not?
> > > > >
> > > > > Vladimir.
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-4669
> > > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Sorting fields of Binarilyzable objects on write

dsetrakyan
In reply to this post by Pavel Tupitsyn
Does anyone disagree about having users sort the fields themselves, as
Sergi and I suggested above?

On Mon, Apr 10, 2017 at 8:21 AM, Pavel Tupitsyn <[hidden email]>
wrote:

> Sergi, DML writes fields in alphabetic order and computes hash code
> accordingly.
> If user-defined Binarylizable impl uses different order, hash codes will be
> inconsistent.
>
> On Mon, Apr 10, 2017 at 6:18 PM, Sergi Vladykin <[hidden email]>
> wrote:
>
> > What is correct or incorrect ordering for DML?
> >
> > Sergi
> >
> > 2017-04-10 18:14 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
> >
> > > I would agree that it should be on a user to always sort the fields, if
> > we
> > > make it a part of the contract. However, in this case, we should always
> > > throw exception if user somehow provides fields in the wrong order.
> > >
> > > D.
> > >
> > > On Mon, Apr 10, 2017 at 8:07 AM, Sergi Vladykin <
> > [hidden email]>
> > > wrote:
> > >
> > > > Could you please elaborate why would we want to sort fields in
> > > > Binarilyzable
> > > > classes?
> > > >
> > > > If you are taking from stable binary representation perspective,
> then I
> > > > think it is a problem of user, but not ours.
> > > >
> > > > Sergi
> > > >
> > > > 2017-04-10 17:53 GMT+03:00 Vladimir Ozerov <[hidden email]>:
> > > >
> > > > > Zapalniki,
> > > > >
> > > > > Inspired by IGNITE-4669 (.NET: Sort binary object fields) [1].
> > > > >
> > > > > Currently we sort binary object fields before when writing them to
> > the
> > > > > output stream in case of standard (Serializable) objects and
> > > > > BinaryObjectBuilder. This makes sense as we have stable binary
> object
> > > > > representation irrespective of fields order, which is very
> important
> > > e.g.
> > > > > for DML. And it works fine from performance perspective as well:
> > > > > - For standard classes we sort fields only once during
> > initialization;
> > > > > - For builder we have to maintain the whole object graph in memory
> > > before
> > > > > writing anyway as builder is mutable, so sorting doesn't impose
> > serious
> > > > > performance hit.
> > > > >
> > > > > But what to do with Binarilyzable classes? We can sort their fields
> > as
> > > > > well, but it means that:
> > > > > 1) We will not be able to write them directly to stream. Instead,
> we
> > > will
> > > > > accumulate values in memory, and write only when the whole object
> > graph
> > > > is
> > > > > known.
> > > > > 2) Currently reads are mostly sequential from memory perspective.
> > With
> > > > this
> > > > > change reads will become random.
> > > > >
> > > > > So we will loose both read and write serialization performance. How
> > do
> > > > you
> > > > > think - do we need this change or not?
> > > > >
> > > > > Vladimir.
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-4669
> > > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Sorting fields of Binarilyzable objects on write

Sergi
I even think that in DML we have to just calculate hashCode in order of
QueryEntity regardless of order in Binarylizable.

Sergi

2017-04-10 18:29 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:

> Does anyone disagree about having users sort the fields themselves, as
> Sergi and I suggested above?
>
> On Mon, Apr 10, 2017 at 8:21 AM, Pavel Tupitsyn <[hidden email]>
> wrote:
>
> > Sergi, DML writes fields in alphabetic order and computes hash code
> > accordingly.
> > If user-defined Binarylizable impl uses different order, hash codes will
> be
> > inconsistent.
> >
> > On Mon, Apr 10, 2017 at 6:18 PM, Sergi Vladykin <
> [hidden email]>
> > wrote:
> >
> > > What is correct or incorrect ordering for DML?
> > >
> > > Sergi
> > >
> > > 2017-04-10 18:14 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
> > >
> > > > I would agree that it should be on a user to always sort the fields,
> if
> > > we
> > > > make it a part of the contract. However, in this case, we should
> always
> > > > throw exception if user somehow provides fields in the wrong order.
> > > >
> > > > D.
> > > >
> > > > On Mon, Apr 10, 2017 at 8:07 AM, Sergi Vladykin <
> > > [hidden email]>
> > > > wrote:
> > > >
> > > > > Could you please elaborate why would we want to sort fields in
> > > > > Binarilyzable
> > > > > classes?
> > > > >
> > > > > If you are taking from stable binary representation perspective,
> > then I
> > > > > think it is a problem of user, but not ours.
> > > > >
> > > > > Sergi
> > > > >
> > > > > 2017-04-10 17:53 GMT+03:00 Vladimir Ozerov <[hidden email]>:
> > > > >
> > > > > > Zapalniki,
> > > > > >
> > > > > > Inspired by IGNITE-4669 (.NET: Sort binary object fields) [1].
> > > > > >
> > > > > > Currently we sort binary object fields before when writing them
> to
> > > the
> > > > > > output stream in case of standard (Serializable) objects and
> > > > > > BinaryObjectBuilder. This makes sense as we have stable binary
> > object
> > > > > > representation irrespective of fields order, which is very
> > important
> > > > e.g.
> > > > > > for DML. And it works fine from performance perspective as well:
> > > > > > - For standard classes we sort fields only once during
> > > initialization;
> > > > > > - For builder we have to maintain the whole object graph in
> memory
> > > > before
> > > > > > writing anyway as builder is mutable, so sorting doesn't impose
> > > serious
> > > > > > performance hit.
> > > > > >
> > > > > > But what to do with Binarilyzable classes? We can sort their
> fields
> > > as
> > > > > > well, but it means that:
> > > > > > 1) We will not be able to write them directly to stream. Instead,
> > we
> > > > will
> > > > > > accumulate values in memory, and write only when the whole object
> > > graph
> > > > > is
> > > > > > known.
> > > > > > 2) Currently reads are mostly sequential from memory perspective.
> > > With
> > > > > this
> > > > > > change reads will become random.
> > > > > >
> > > > > > So we will loose both read and write serialization performance.
> How
> > > do
> > > > > you
> > > > > > think - do we need this change or not?
> > > > > >
> > > > > > Vladimir.
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-4669
> > > > > >
> > > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Sorting fields of Binarilyzable objects on write

dsetrakyan
In reply to this post by Sergi
On Mon, Apr 10, 2017 at 8:28 AM, Sergi Vladykin <[hidden email]>
wrote:

> The decision to use alphabetic order looks strange here. Using order
> provided in QueryEntity and require from user to have the same order
> in Binarylizable
> looks more reasonable.
>

I think this would be much harder to verify. Alphabetical order is more
intuitive, no?
Reply | Threaded
Open this post in threaded view
|

Re: Sorting fields of Binarilyzable objects on write

Pavel Tupitsyn-3
QueryEntity sorting is not an option for "regular" classes with reflective
serialization.
We have to use alphabetical.
Also, same class can participate in multiple query entities.

10 апр. 2017 г. 18:52 пользователь "Dmitriy Setrakyan" <
[hidden email]> написал:

On Mon, Apr 10, 2017 at 8:28 AM, Sergi Vladykin <[hidden email]>
wrote:

> The decision to use alphabetic order looks strange here. Using order
> provided in QueryEntity and require from user to have the same order
> in Binarylizable
> looks more reasonable.
>

I think this would be much harder to verify. Alphabetical order is more
intuitive, no?
Reply | Threaded
Open this post in threaded view
|

Re: Sorting fields of Binarilyzable objects on write

Sergi
Why "regular" are different here?

Sergi

2017-04-10 18:59 GMT+03:00 Pavel Tupitsyn <[hidden email]>:

> QueryEntity sorting is not an option for "regular" classes with reflective
> serialization.
> We have to use alphabetical.
> Also, same class can participate in multiple query entities.
>
> 10 апр. 2017 г. 18:52 пользователь "Dmitriy Setrakyan" <
> [hidden email]> написал:
>
> On Mon, Apr 10, 2017 at 8:28 AM, Sergi Vladykin <[hidden email]>
> wrote:
>
> > The decision to use alphabetic order looks strange here. Using order
> > provided in QueryEntity and require from user to have the same order
> > in Binarylizable
> > looks more reasonable.
> >
>
> I think this would be much harder to verify. Alphabetical order is more
> intuitive, no?
>
Reply | Threaded
Open this post in threaded view
|

Re: Sorting fields of Binarilyzable objects on write

Vladimir Ozerov
Guys,

The problem is that order of fields serialization is unknown for regular
objects, where "regular" stands for non-Binarilyzable class. Reflection
returns fields in unpredictable order. For this reason you cannot match
fields order between class and QueryEntity. This is why we introduced
sorting, and this is why idea to rely on QueryEntity doesn't work.

On Mon, Apr 10, 2017 at 7:01 PM, Sergi Vladykin <[hidden email]>
wrote:

> Why "regular" are different here?
>
> Sergi
>
> 2017-04-10 18:59 GMT+03:00 Pavel Tupitsyn <[hidden email]>:
>
> > QueryEntity sorting is not an option for "regular" classes with
> reflective
> > serialization.
> > We have to use alphabetical.
> > Also, same class can participate in multiple query entities.
> >
> > 10 апр. 2017 г. 18:52 пользователь "Dmitriy Setrakyan" <
> > [hidden email]> написал:
> >
> > On Mon, Apr 10, 2017 at 8:28 AM, Sergi Vladykin <
> [hidden email]>
> > wrote:
> >
> > > The decision to use alphabetic order looks strange here. Using order
> > > provided in QueryEntity and require from user to have the same order
> > > in Binarylizable
> > > looks more reasonable.
> > >
> >
> > I think this would be much harder to verify. Alphabetical order is more
> > intuitive, no?
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Sorting fields of Binarilyzable objects on write

Sergi
I'm sorry, looks like I do not really well understand this stuff, but it is
still not clear to me why wouldn't we just take the order of key fields
given in QueryEntity and use it for both cases irrespectively to the order
of fields in regular Class or in Binarylizable?

I mean lets say we have a Class (with unpredictable order of fields
according to reflection):

   Person implements Serializable
      {int age, double salary}

Or we have a Binarylizable (with some unknown user defined order of fields):

  Person implements Binarylizable {
     writeBinary(w) {w.writeDouble("salary", salary),  w.writeInt(''age",
age)}

Also we have a QueryEntity (age, salary)

Why we can not take key fields names from QueryEntity in the given order
(age, salary) and get values from either "regular" Class or from
Binarylizable and calculate hash code?

Sergi

2017-04-10 19:56 GMT+03:00 Vladimir Ozerov <[hidden email]>:

> Guys,
>
> The problem is that order of fields serialization is unknown for regular
> objects, where "regular" stands for non-Binarilyzable class. Reflection
> returns fields in unpredictable order. For this reason you cannot match
> fields order between class and QueryEntity. This is why we introduced
> sorting, and this is why idea to rely on QueryEntity doesn't work.
>
> On Mon, Apr 10, 2017 at 7:01 PM, Sergi Vladykin <[hidden email]>
> wrote:
>
> > Why "regular" are different here?
> >
> > Sergi
> >
> > 2017-04-10 18:59 GMT+03:00 Pavel Tupitsyn <[hidden email]>:
> >
> > > QueryEntity sorting is not an option for "regular" classes with
> > reflective
> > > serialization.
> > > We have to use alphabetical.
> > > Also, same class can participate in multiple query entities.
> > >
> > > 10 апр. 2017 г. 18:52 пользователь "Dmitriy Setrakyan" <
> > > [hidden email]> написал:
> > >
> > > On Mon, Apr 10, 2017 at 8:28 AM, Sergi Vladykin <
> > [hidden email]>
> > > wrote:
> > >
> > > > The decision to use alphabetic order looks strange here. Using order
> > > > provided in QueryEntity and require from user to have the same order
> > > > in Binarylizable
> > > > looks more reasonable.
> > > >
> > >
> > > I think this would be much harder to verify. Alphabetical order is more
> > > intuitive, no?
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Sorting fields of Binarilyzable objects on write

dsetrakyan
Sergi, why do you not like the alphabetical order? Seems like it would be a
much easier to understand requirement from the user standpoint. Do you see
some issues here?

On Mon, Apr 10, 2017 at 10:43 AM, Sergi Vladykin <[hidden email]>
wrote:

> I'm sorry, looks like I do not really well understand this stuff, but it is
> still not clear to me why wouldn't we just take the order of key fields
> given in QueryEntity and use it for both cases irrespectively to the order
> of fields in regular Class or in Binarylizable?
>
> I mean lets say we have a Class (with unpredictable order of fields
> according to reflection):
>
>    Person implements Serializable
>       {int age, double salary}
>
> Or we have a Binarylizable (with some unknown user defined order of
> fields):
>
>   Person implements Binarylizable {
>      writeBinary(w) {w.writeDouble("salary", salary),  w.writeInt(''age",
> age)}
>
> Also we have a QueryEntity (age, salary)
>
> Why we can not take key fields names from QueryEntity in the given order
> (age, salary) and get values from either "regular" Class or from
> Binarylizable and calculate hash code?
>
> Sergi
>
> 2017-04-10 19:56 GMT+03:00 Vladimir Ozerov <[hidden email]>:
>
> > Guys,
> >
> > The problem is that order of fields serialization is unknown for regular
> > objects, where "regular" stands for non-Binarilyzable class. Reflection
> > returns fields in unpredictable order. For this reason you cannot match
> > fields order between class and QueryEntity. This is why we introduced
> > sorting, and this is why idea to rely on QueryEntity doesn't work.
> >
> > On Mon, Apr 10, 2017 at 7:01 PM, Sergi Vladykin <
> [hidden email]>
> > wrote:
> >
> > > Why "regular" are different here?
> > >
> > > Sergi
> > >
> > > 2017-04-10 18:59 GMT+03:00 Pavel Tupitsyn <[hidden email]>:
> > >
> > > > QueryEntity sorting is not an option for "regular" classes with
> > > reflective
> > > > serialization.
> > > > We have to use alphabetical.
> > > > Also, same class can participate in multiple query entities.
> > > >
> > > > 10 апр. 2017 г. 18:52 пользователь "Dmitriy Setrakyan" <
> > > > [hidden email]> написал:
> > > >
> > > > On Mon, Apr 10, 2017 at 8:28 AM, Sergi Vladykin <
> > > [hidden email]>
> > > > wrote:
> > > >
> > > > > The decision to use alphabetic order looks strange here. Using
> order
> > > > > provided in QueryEntity and require from user to have the same
> order
> > > > > in Binarylizable
> > > > > looks more reasonable.
> > > > >
> > > >
> > > > I think this would be much harder to verify. Alphabetical order is
> more
> > > > intuitive, no?
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Sorting fields of Binarilyzable objects on write

Sergi
I'm just trying to understand the current state of things and risks. May be
we need to do some adjustments here before 2.0 to be on the safe side.

Actually looks like this not really important, we just have to clearly
document that DML builds keys this way and require from user to do the same
to be able to use cache API.

Sergi

2017-04-11 21:47 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:

> Sergi, why do you not like the alphabetical order? Seems like it would be a
> much easier to understand requirement from the user standpoint. Do you see
> some issues here?
>
> On Mon, Apr 10, 2017 at 10:43 AM, Sergi Vladykin <[hidden email]
> >
> wrote:
>
> > I'm sorry, looks like I do not really well understand this stuff, but it
> is
> > still not clear to me why wouldn't we just take the order of key fields
> > given in QueryEntity and use it for both cases irrespectively to the
> order
> > of fields in regular Class or in Binarylizable?
> >
> > I mean lets say we have a Class (with unpredictable order of fields
> > according to reflection):
> >
> >    Person implements Serializable
> >       {int age, double salary}
> >
> > Or we have a Binarylizable (with some unknown user defined order of
> > fields):
> >
> >   Person implements Binarylizable {
> >      writeBinary(w) {w.writeDouble("salary", salary),  w.writeInt(''age",
> > age)}
> >
> > Also we have a QueryEntity (age, salary)
> >
> > Why we can not take key fields names from QueryEntity in the given order
> > (age, salary) and get values from either "regular" Class or from
> > Binarylizable and calculate hash code?
> >
> > Sergi
> >
> > 2017-04-10 19:56 GMT+03:00 Vladimir Ozerov <[hidden email]>:
> >
> > > Guys,
> > >
> > > The problem is that order of fields serialization is unknown for
> regular
> > > objects, where "regular" stands for non-Binarilyzable class. Reflection
> > > returns fields in unpredictable order. For this reason you cannot match
> > > fields order between class and QueryEntity. This is why we introduced
> > > sorting, and this is why idea to rely on QueryEntity doesn't work.
> > >
> > > On Mon, Apr 10, 2017 at 7:01 PM, Sergi Vladykin <
> > [hidden email]>
> > > wrote:
> > >
> > > > Why "regular" are different here?
> > > >
> > > > Sergi
> > > >
> > > > 2017-04-10 18:59 GMT+03:00 Pavel Tupitsyn <[hidden email]>:
> > > >
> > > > > QueryEntity sorting is not an option for "regular" classes with
> > > > reflective
> > > > > serialization.
> > > > > We have to use alphabetical.
> > > > > Also, same class can participate in multiple query entities.
> > > > >
> > > > > 10 апр. 2017 г. 18:52 пользователь "Dmitriy Setrakyan" <
> > > > > [hidden email]> написал:
> > > > >
> > > > > On Mon, Apr 10, 2017 at 8:28 AM, Sergi Vladykin <
> > > > [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > The decision to use alphabetic order looks strange here. Using
> > order
> > > > > > provided in QueryEntity and require from user to have the same
> > order
> > > > > > in Binarylizable
> > > > > > looks more reasonable.
> > > > > >
> > > > >
> > > > > I think this would be much harder to verify. Alphabetical order is
> > more
> > > > > intuitive, no?
> > > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Sorting fields of Binarilyzable objects on write

dsetrakyan
On Tue, Apr 11, 2017 at 1:28 PM, Sergi Vladykin <[hidden email]>
wrote:

> I'm just trying to understand the current state of things and risks. May be
> we need to do some adjustments here before 2.0 to be on the safe side.
>
> Actually looks like this not really important, we just have to clearly
> document that DML builds keys this way and require from user to do the same
> to be able to use cache API.
>


I think it is important from the usability stand point. A user can always
sort fields alphabetically in his or her mind. However, trying to remember
the field order from some QueryEntity is a lot harder.
Reply | Threaded
Open this post in threaded view
|

Re: Sorting fields of Binarilyzable objects on write

Pavel Tupitsyn
QueryEntity order is not only harder for the users, it will be nightmare to
implement.
What if there is no QueryEntity defined? What if the same class is used in
multiple QueryEntity?
I don't think serialization code has to be tied to QueryEntity in any way,
this violates separation of concerns.

So I guess we can agree on sorting fields alphabetically.

Let's get to the initial question:
* Should we do the sorting for the user (performance hit)?
* Should we at least validate user-defined order?

On Wed, Apr 12, 2017 at 2:12 AM, Dmitriy Setrakyan <[hidden email]>
wrote:

> On Tue, Apr 11, 2017 at 1:28 PM, Sergi Vladykin <[hidden email]>
> wrote:
>
> > I'm just trying to understand the current state of things and risks. May
> be
> > we need to do some adjustments here before 2.0 to be on the safe side.
> >
> > Actually looks like this not really important, we just have to clearly
> > document that DML builds keys this way and require from user to do the
> same
> > to be able to use cache API.
> >
>
>
> I think it is important from the usability stand point. A user can always
> sort fields alphabetically in his or her mind. However, trying to remember
> the field order from some QueryEntity is a lot harder.
>
Reply | Threaded
Open this post in threaded view
|

Re: Sorting fields of Binarilyzable objects on write

dsetrakyan
I think it is OK for users to do their own sorting, but we should
definitely validate the correct order and throw an exception if it is not.

D.

On Tue, Apr 11, 2017 at 11:02 PM, Pavel Tupitsyn <[hidden email]>
wrote:

> QueryEntity order is not only harder for the users, it will be nightmare to
> implement.
> What if there is no QueryEntity defined? What if the same class is used in
> multiple QueryEntity?
> I don't think serialization code has to be tied to QueryEntity in any way,
> this violates separation of concerns.
>
> So I guess we can agree on sorting fields alphabetically.
>
> Let's get to the initial question:
> * Should we do the sorting for the user (performance hit)?
> * Should we at least validate user-defined order?
>
> On Wed, Apr 12, 2017 at 2:12 AM, Dmitriy Setrakyan <[hidden email]>
> wrote:
>
> > On Tue, Apr 11, 2017 at 1:28 PM, Sergi Vladykin <
> [hidden email]>
> > wrote:
> >
> > > I'm just trying to understand the current state of things and risks.
> May
> > be
> > > we need to do some adjustments here before 2.0 to be on the safe side.
> > >
> > > Actually looks like this not really important, we just have to clearly
> > > document that DML builds keys this way and require from user to do the
> > same
> > > to be able to use cache API.
> > >
> >
> >
> > I think it is important from the usability stand point. A user can always
> > sort fields alphabetically in his or her mind. However, trying to
> remember
> > the field order from some QueryEntity is a lot harder.
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Sorting fields of Binarilyzable objects on write

Vladimir Ozerov
Consider the following code:

void writeBinary(BinaryWriter w) {
    w.writeBoolean("C", c);

    if (c)
        w.writeInt("A", a)
    else
        w.writeInt("B", b)
}

How are we going to force user to follow the contract in this case?

On Wed, Apr 12, 2017 at 9:16 AM, Dmitriy Setrakyan <[hidden email]>
wrote:

> I think it is OK for users to do their own sorting, but we should
> definitely validate the correct order and throw an exception if it is not.
>
> D.
>
> On Tue, Apr 11, 2017 at 11:02 PM, Pavel Tupitsyn <[hidden email]>
> wrote:
>
> > QueryEntity order is not only harder for the users, it will be nightmare
> to
> > implement.
> > What if there is no QueryEntity defined? What if the same class is used
> in
> > multiple QueryEntity?
> > I don't think serialization code has to be tied to QueryEntity in any
> way,
> > this violates separation of concerns.
> >
> > So I guess we can agree on sorting fields alphabetically.
> >
> > Let's get to the initial question:
> > * Should we do the sorting for the user (performance hit)?
> > * Should we at least validate user-defined order?
> >
> > On Wed, Apr 12, 2017 at 2:12 AM, Dmitriy Setrakyan <
> [hidden email]>
> > wrote:
> >
> > > On Tue, Apr 11, 2017 at 1:28 PM, Sergi Vladykin <
> > [hidden email]>
> > > wrote:
> > >
> > > > I'm just trying to understand the current state of things and risks.
> > May
> > > be
> > > > we need to do some adjustments here before 2.0 to be on the safe
> side.
> > > >
> > > > Actually looks like this not really important, we just have to
> clearly
> > > > document that DML builds keys this way and require from user to do
> the
> > > same
> > > > to be able to use cache API.
> > > >
> > >
> > >
> > > I think it is important from the usability stand point. A user can
> always
> > > sort fields alphabetically in his or her mind. However, trying to
> > remember
> > > the field order from some QueryEntity is a lot harder.
> > >
> >
>
12