Apache Arrow and Apache Ignite

Hello all - are there any common points between the two? Where can they
be made to work together, if at all?

Thanks!

[dsetrakyan]

I am a bit unclear from the Apache Arrow website. Is Apache Arrow a
columnar in-memory database or is it a processing engine?

On Fri, Feb 19, 2016 at 6:31 PM, Dood@ODDO <[hidden email]> wrote:

> Hello all - are there any common points between the two? Where can they be
> made to work together, if at all?
>
> Thanks!
>

Dmitriy, that's a good question - according to [1] it seems to be a
"collection of algorithms and formats" - it is not a storage engine nor
is it a processing engine. Weird.

[1].
http://www.zdnet.com/article/apache-arrow-unifies-in-memory-big-data-systems/


On 2/19/2016 8:56 PM, Dmitriy Setrakyan wrote:

> I am a bit unclear from the Apache Arrow website. Is Apache Arrow a
> columnar in-memory database or is it a processing engine?
>
> On Fri, Feb 19, 2016 at 6:31 PM, Dood@ODDO <[hidden email]> wrote:
>
>> Hello all - are there any common points between the two? Where can they be
>> made to work together, if at all?
>>
>> Thanks!
>>

[Roman Shtykh]

It's a standard (including format and algorithms) to share in-memory data between several big data platforms.
In my understanding, it is like placing data in in-memory columns in Parquet and being able to use it in Kudu without serializing/deserializing.

-Roman

[dsetrakyan]

It sounds to me that it does not store it’s own data, but allows to process
data from somewhere else, right? If yes, will I be able to take Ignite data
with Arrow and convert it to a columnar format for some other processing?

On Sat, Feb 20, 2016 at 2:48 AM, Roman Shtykh <[hidden email]>
wrote:

> It's a standard (including format and algorithms) to share in-memory data
> between several big data platforms.
> In my understanding, it is like placing data in in-memory columns in
> Parquet and being able to use it in Kudu without serializing/deserializing.
>
> -Roman
>

That's the million dollar question - I think we should approach the
Arrow people and get a conversation going. We want to be ahead of the
curve, not behind it - Arrow seems to be making quite a stir, not to
mention that it was fast-tracked to mature project status apparently
solely based on the people/companies involved ;-)

On 2/20/2016 11:14 AM, Dmitriy Setrakyan wrote:

> It sounds to me that it does not store it’s own data, but allows to process
> data from somewhere else, right? If yes, will I be able to take Ignite data
> with Arrow and convert it to a columnar format for some other processing?
>
> On Sat, Feb 20, 2016 at 2:48 AM, Roman Shtykh <[hidden email]>
> wrote:
>
>> It's a standard (including format and algorithms) to share in-memory data
>> between several big data platforms.
>> In my understanding, it is like placing data in in-memory columns in
>> Parquet and being able to use it in Kudu without serializing/deserializing.
>>
>> -Roman
>>

[dsetrakyan]

Agree. Feel free to initiate a discussion on the Arrow list and cross-copy
this list. I will also think about it some more and will propose something
next week.

On Sat, Feb 20, 2016 at 12:13 PM, Dood@ODDO <[hidden email]> wrote:

> That's the million dollar question - I think we should approach the Arrow
> people and get a conversation going. We want to be ahead of the curve, not
> behind it - Arrow seems to be making quite a stir, not to mention that it
> was fast-tracked to mature project status apparently solely based on the
> people/companies involved ;-)
>
>
> On 2/20/2016 11:14 AM, Dmitriy Setrakyan wrote:
>
>> It sounds to me that it does not store it’s own data, but allows to
>> process
>> data from somewhere else, right? If yes, will I be able to take Ignite
>> data
>> with Arrow and convert it to a columnar format for some other processing?
>>
>> On Sat, Feb 20, 2016 at 2:48 AM, Roman Shtykh <[hidden email]>
>> wrote:
>>
>> It's a standard (including format and algorithms) to share in-memory data
>>> between several big data platforms.
>>> In my understanding, it is like placing data in in-memory columns in
>>> Parquet and being able to use it in Kudu without
>>> serializing/deserializing.
>>>
>>> -Roman
>>>
>>>
>

[Konstantin Boudnik-2]

On Sat, Feb 20, 2016 at 02:13PM, Dood@ODDO wrote:
> That's the million dollar question - I think we should approach the
> Arrow people and get a conversation going. We want to be ahead of
> the curve, not behind it - Arrow seems to be making quite a stir,
> not to mention that it was fast-tracked to mature project status
> apparently solely based on the people/companies involved ;-)

I am afraid you misunderstand the point of the 'fast tracking'. To start with,
ASF doesn't case what companies are involved nor there's no such thing as fast
tracking. However, there's 'direct to TLP' track, that allows for a community
that has enough ASF members and other people, with lengthy involvement into
Apache projects, to go directly into TLP. The reason is simple: these people
are well versed in Apache Way and the incubation isn't required for me.

So, please stop using this 'fast-tracked' confusion.
  Cos

P.S. Another point: TLP != 'mature project status'. Please educate yourself
about the Apache incubation, before making statement like this in the future.

> On 2/20/2016 11:14 AM, Dmitriy Setrakyan wrote:
> >It sounds to me that it does not store it’s own data, but allows to process
> >data from somewhere else, right? If yes, will I be able to take Ignite data
> >with Arrow and convert it to a columnar format for some other processing?
> >
> >On Sat, Feb 20, 2016 at 2:48 AM, Roman Shtykh <[hidden email]>
> >wrote:
> >
> >>It's a standard (including format and algorithms) to share in-memory data
> >>between several big data platforms.
> >>In my understanding, it is like placing data in in-memory columns in
> >>Parquet and being able to use it in Kudu without serializing/deserializing.
> >>
> >>-Roman
> >>
>

On 2/24/2016 1:31 PM, Konstantin Boudnik wrote:

> On Sat, Feb 20, 2016 at 02:13PM, Dood@ODDO wrote:
>> That's the million dollar question - I think we should approach the
>> Arrow people and get a conversation going. We want to be ahead of
>> the curve, not behind it - Arrow seems to be making quite a stir,
>> not to mention that it was fast-tracked to mature project status
>> apparently solely based on the people/companies involved ;-)
> I am afraid you misunderstand the point of the 'fast tracking'. To start with,
> ASF doesn't case what companies are involved nor there's no such thing as fast
> tracking. However, there's 'direct to TLP' track, that allows for a community
> that has enough ASF members and other people, with lengthy involvement into
> Apache projects, to go directly into TLP. The reason is simple: these people
> are well versed in Apache Way and the incubation isn't required for me.
>
> So, please stop using this 'fast-tracked' confusion.
>    Cos
>
> P.S. Another point: TLP != 'mature project status'. Please educate yourself
> about the Apache incubation, before making statement like this in the future.
>

I will educate myself. What you need to do is learn to be polite. I am
not sure you can teach that though ;(

[Konstantin Boudnik-2]

On Wed, Feb 24, 2016 at 02:02PM, Dood@ODDO wrote:

> On 2/24/2016 1:31 PM, Konstantin Boudnik wrote:
> >On Sat, Feb 20, 2016 at 02:13PM, Dood@ODDO wrote:
> >>That's the million dollar question - I think we should approach the
> >>Arrow people and get a conversation going. We want to be ahead of
> >>the curve, not behind it - Arrow seems to be making quite a stir,
> >>not to mention that it was fast-tracked to mature project status
> >>apparently solely based on the people/companies involved ;-)
> >I am afraid you misunderstand the point of the 'fast tracking'. To start with,
> >ASF doesn't case what companies are involved nor there's no such thing as fast
> >tracking. However, there's 'direct to TLP' track, that allows for a community
> >that has enough ASF members and other people, with lengthy involvement into
> >Apache projects, to go directly into TLP. The reason is simple: these people
> >are well versed in Apache Way and the incubation isn't required for me.
> >
> >So, please stop using this 'fast-tracked' confusion.
> >   Cos
> >
> >P.S. Another point: TLP != 'mature project status'. Please educate yourself
> >about the Apache incubation, before making statement like this in the future.
> >
>
> I will educate myself. What you need to do is learn to be polite. I
> am not sure you can teach that though ;(

If you don't mind elaborating on what particularly was impolite in my email, you'll
put me forever into your debts. And teach me a lesson of good manners too.

Cos

On 2/25/2016 11:06 AM, Konstantin Boudnik wrote:

> On Wed, Feb 24, 2016 at 02:02PM, Dood@ODDO wrote:
>> On 2/24/2016 1:31 PM, Konstantin Boudnik wrote:
>>> On Sat, Feb 20, 2016 at 02:13PM, Dood@ODDO wrote:
>>>> That's the million dollar question - I think we should approach the
>>>> Arrow people and get a conversation going. We want to be ahead of
>>>> the curve, not behind it - Arrow seems to be making quite a stir,
>>>> not to mention that it was fast-tracked to mature project status
>>>> apparently solely based on the people/companies involved ;-)
>>> I am afraid you misunderstand the point of the 'fast tracking'. To start with,
>>> ASF doesn't case what companies are involved nor there's no such thing as fast
>>> tracking. However, there's 'direct to TLP' track, that allows for a community
>>> that has enough ASF members and other people, with lengthy involvement into
>>> Apache projects, to go directly into TLP. The reason is simple: these people
>>> are well versed in Apache Way and the incubation isn't required for me.
>>>
>>> So, please stop using this 'fast-tracked' confusion.
>>>    Cos
>>>
>>> P.S. Another point: TLP != 'mature project status'. Please educate yourself
>>> about the Apache incubation, before making statement like this in the future.
>>>
>> I will educate myself. What you need to do is learn to be polite. I
>> am not sure you can teach that though ;(
> If you don't mind elaborating on what particularly was impolite in my email, you'll
> put me forever into your debts. And teach me a lesson of good manners too.
>
> Cos

I don't know, you just came off as angered by what I said. You took the
time to supposedly educate me how "fast tracking" is not the same as "
taking a project directly to TLP" (which I think is obviously the same
as fast tracking since it skipped the incubating phase). Then you took
more time to educate me how people/companies are not what Apache looked
at when fast tracking this project (pardon, taking the route called
"direct to TLP") but you said that people experienced in the ways of
Apache are what allowed for this project to be fast tracked (pardon
again, taken "direct to TLP").

Incidentally, who can join the dev list on the Arrow project at this
stage? I tried a week ago and was unsuccessful. I was under the
impression that anyone can join it.

[Konstantin Boudnik-2]

On Fri, Feb 26, 2016 at 09:22PM, Dood@ODDO wrote:

> On 2/25/2016 11:06 AM, Konstantin Boudnik wrote:
> >On Wed, Feb 24, 2016 at 02:02PM, Dood@ODDO wrote:
> >>On 2/24/2016 1:31 PM, Konstantin Boudnik wrote:
> >>>On Sat, Feb 20, 2016 at 02:13PM, Dood@ODDO wrote:
> >>>>That's the million dollar question - I think we should approach the
> >>>>Arrow people and get a conversation going. We want to be ahead of
> >>>>the curve, not behind it - Arrow seems to be making quite a stir,
> >>>>not to mention that it was fast-tracked to mature project status
> >>>>apparently solely based on the people/companies involved ;-)
> >>>I am afraid you misunderstand the point of the 'fast tracking'. To start with,
> >>>ASF doesn't case what companies are involved nor there's no such thing as fast
> >>>tracking. However, there's 'direct to TLP' track, that allows for a community
> >>>that has enough ASF members and other people, with lengthy involvement into
> >>>Apache projects, to go directly into TLP. The reason is simple: these people
> >>>are well versed in Apache Way and the incubation isn't required for me.
> >>>
> >>>So, please stop using this 'fast-tracked' confusion.
> >>>   Cos
> >>>
> >>>P.S. Another point: TLP != 'mature project status'. Please educate yourself
> >>>about the Apache incubation, before making statement like this in the future.
> >>>
> >>I will educate myself. What you need to do is learn to be polite. I
> >>am not sure you can teach that though ;(
> >If you don't mind elaborating on what particularly was impolite in my email, you'll
> >put me forever into your debts. And teach me a lesson of good manners too.
> >
> >Cos
> I don't know, you just came off as angered by what I said. You took
> the time to supposedly educate me how "fast tracking" is not the
> same as " taking a project directly to TLP" (which I think is
> obviously the same as fast tracking since it skipped the incubating
> phase). Then you took more time to educate me how people/companies
> are not what Apache looked at when fast tracking this project
> (pardon, taking the route called "direct to TLP") but you said that
> people experienced in the ways of Apache are what allowed for this
> project to be fast tracked (pardon again, taken "direct to TLP").

See, I too time to educate you on a variety of topics and provided you with
the pointers for future reading. How polite is that?

> Incidentally, who can join the dev list on the Arrow project at this
> stage? I tried a week ago and was unsuccessful. I was under the
> impression that anyone can join it.

dev@ lists are indeed open for everyone. If you email to
[hidden email] hasn't been answered, try once again - sometimes
emails get dropped. You also can write to [hidden email] and let them
know, that your request wasn't addressed.

Cos