Support for packed int and long primitives in raw binary API

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Support for packed int and long primitives in raw binary API

Alexei Scherbakov
Guys,

I notices we do not have support for packed ints and longs in raw binary
API [1] [2]

Such methods are essential for implementing efficient custom compression
schemes.

Their addition can simplify implementing custom serializers for the cases
then default binary marshaller is not enough, without additional library
dependencies.

Proposed API extension for rawReader/rawWriter:

org.apache.ignite.binary.BinaryRawWriter#writePackedInt

org.apache.ignite.binary.BinaryRawWriter#writePackedLong

org.apache.ignite.binary.BinaryRawReader#readPackedInt

org.apache.ignite.binary.BinaryRawReader#readPackedLong

JIRA ticket: [3]

Thoughs ?

[1] org.apache.ignite.binary.BinaryRawReader

[2] org.apache.ignite.binary.BinaryRawWriter

[3] https://issues.apache.org/jira/browse/IGNITE-6426

--

Best regards,
Alexei Scherbakov
Reply | Threaded
Open this post in threaded view
|

Re: Support for packed int and long primitives in raw binary API

dsetrakyan
Alexey, are you talking about arrays of ints and longs?

On Fri, Sep 29, 2017 at 3:29 AM, Alexei Scherbakov <
[hidden email]> wrote:

> Guys,
>
> I notices we do not have support for packed ints and longs in raw binary
> API [1] [2]
>
> Such methods are essential for implementing efficient custom compression
> schemes.
>
> Their addition can simplify implementing custom serializers for the cases
> then default binary marshaller is not enough, without additional library
> dependencies.
>
> Proposed API extension for rawReader/rawWriter:
>
> org.apache.ignite.binary.BinaryRawWriter#writePackedInt
>
> org.apache.ignite.binary.BinaryRawWriter#writePackedLong
>
> org.apache.ignite.binary.BinaryRawReader#readPackedInt
>
> org.apache.ignite.binary.BinaryRawReader#readPackedLong
>
> JIRA ticket: [3]
>
> Thoughs ?
>
> [1] org.apache.ignite.binary.BinaryRawReader
>
> [2] org.apache.ignite.binary.BinaryRawWriter
>
> [3] https://issues.apache.org/jira/browse/IGNITE-6426
>
> --
>
> Best regards,
> Alexei Scherbakov
>
Reply | Threaded
Open this post in threaded view
|

Re: Support for packed int and long primitives in raw binary API

Alexei Scherbakov
Dmitriy,

Not arrays, just primitives.

Using special binary representation, ints and longs can be represented by
1-9 bytes, depending on their cardinality.

I want to add such implementation in raw binary API to help implementing
custom serializers.



2017-09-29 15:12 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:

> Alexey, are you talking about arrays of ints and longs?
>
> On Fri, Sep 29, 2017 at 3:29 AM, Alexei Scherbakov <
> [hidden email]> wrote:
>
> > Guys,
> >
> > I notices we do not have support for packed ints and longs in raw binary
> > API [1] [2]
> >
> > Such methods are essential for implementing efficient custom compression
> > schemes.
> >
> > Their addition can simplify implementing custom serializers for the cases
> > then default binary marshaller is not enough, without additional library
> > dependencies.
> >
> > Proposed API extension for rawReader/rawWriter:
> >
> > org.apache.ignite.binary.BinaryRawWriter#writePackedInt
> >
> > org.apache.ignite.binary.BinaryRawWriter#writePackedLong
> >
> > org.apache.ignite.binary.BinaryRawReader#readPackedInt
> >
> > org.apache.ignite.binary.BinaryRawReader#readPackedLong
> >
> > JIRA ticket: [3]
> >
> > Thoughs ?
> >
> > [1] org.apache.ignite.binary.BinaryRawReader
> >
> > [2] org.apache.ignite.binary.BinaryRawWriter
> >
> > [3] https://issues.apache.org/jira/browse/IGNITE-6426
> >
> > --
> >
> > Best regards,
> > Alexei Scherbakov
> >
>



--

Best regards,
Alexei Scherbakov
Reply | Threaded
Open this post in threaded view
|

Re: Support for packed int and long primitives in raw binary API

Valentin Kulichenko
Alexey,

Any reason why you propose this only for raw data? I think this should be
an option for binary marshaller (probably per type). If we also add special
methods to Binarylizable, then they should be applied to both raw and
non-raw data.

BTW, there are couple other tickets regarding this, and looks like there is
already some progress:
- https://issues.apache.org/jira/browse/IGNITE-5097
- https://issues.apache.org/jira/browse/IGNITE-6418

-Val

On Fri, Sep 29, 2017 at 5:37 AM, Alexei Scherbakov <
[hidden email]> wrote:

> Dmitriy,
>
> Not arrays, just primitives.
>
> Using special binary representation, ints and longs can be represented by
> 1-9 bytes, depending on their cardinality.
>
> I want to add such implementation in raw binary API to help implementing
> custom serializers.
>
>
>
> 2017-09-29 15:12 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
>
> > Alexey, are you talking about arrays of ints and longs?
> >
> > On Fri, Sep 29, 2017 at 3:29 AM, Alexei Scherbakov <
> > [hidden email]> wrote:
> >
> > > Guys,
> > >
> > > I notices we do not have support for packed ints and longs in raw
> binary
> > > API [1] [2]
> > >
> > > Such methods are essential for implementing efficient custom
> compression
> > > schemes.
> > >
> > > Their addition can simplify implementing custom serializers for the
> cases
> > > then default binary marshaller is not enough, without additional
> library
> > > dependencies.
> > >
> > > Proposed API extension for rawReader/rawWriter:
> > >
> > > org.apache.ignite.binary.BinaryRawWriter#writePackedInt
> > >
> > > org.apache.ignite.binary.BinaryRawWriter#writePackedLong
> > >
> > > org.apache.ignite.binary.BinaryRawReader#readPackedInt
> > >
> > > org.apache.ignite.binary.BinaryRawReader#readPackedLong
> > >
> > > JIRA ticket: [3]
> > >
> > > Thoughs ?
> > >
> > > [1] org.apache.ignite.binary.BinaryRawReader
> > >
> > > [2] org.apache.ignite.binary.BinaryRawWriter
> > >
> > > [3] https://issues.apache.org/jira/browse/IGNITE-6426
> > >
> > > --
> > >
> > > Best regards,
> > > Alexei Scherbakov
> > >
> >
>
>
>
> --
>
> Best regards,
> Alexei Scherbakov
>