Gianfranco,
I agree that integration with Drools would be useful. I think Yakov is questioning where this integration should be hosted. It looks like Infinispan is hosting it separately from their main code base. We can host it in Apache repo, or we can create a separate project for it, for example, here: https://github.com/apacheignite Looks like Drools is licensed under ASL2.0, so we could really do it either way. Would be interesting do find out what others think. D. On Tue, Sep 29, 2015 at 5:55 PM, Gianfranco Murador <[hidden email]> wrote: > Yakov, > So I guess that integration should only affect the way JBPM interacts with > Ignite ? > I think that a module dedicated to ignite could provide APIs more advanced > than simple get and put on a grid. Infinispan already provides a second > level cache Ignite does, but that does not mean that > Ignite does not have to provide similar feature. > Interacting JBPM with a dedicated module of ignite we could do processing > on the workflow instance in advanced mode, exploiting the near cache in a > distributed environment. > Maybe the module might not be ready for the release of version 1.5. > Ultimately, who knows better than you that Ignite is more advanced than > Infinspan ? > > > > 2015-09-29 16:52 GMT+02:00 Yakov Zhdanov <[hidden email]>: > > > Igniters, > > > > Given we somehow agreed on most of the scope, can everyone please cleanup > > JIRA queues accordingly? > > > > --Yakov > > > > > > -- > Gianfranco Murador > Igniter and Software Engineer. > |
In reply to this post by Konstantin Boudnik-2
Cos - why mixing the two? I think if we are embark on this route - why not
create two different assemblies, one for Hadoop bits, and another for Spark integration? Two would rarely intersect IMHO. -- Nikita Ivanov On Mon, Sep 28, 2015 at 10:40 PM, Konstantin Boudnik <[hidden email]> wrote: > While a minor modification which won't affect anything, I'd like to put > on the table the consolidation of hadoodp and spark bits into > ignite-accelerator assembly. The fix is ready and I was just waiting for > 1.4 to come out. > > Cos > > On September 28, 2015 11:11:44 AM PDT, Vladimir Ozerov < > [hidden email]> wrote: > >I'd like to add several things. > >1) It looks like fix for IBM JVM will brake backward compatibility > >because > >the problem is caused by missing serialVersionId, right? > >2) HP-UX users are also concerned, see last comments in > >https://issues.apache.org/jira/browse/IGNITE-1493 I think we must > >adress > >Unsafe refactoring in this release as well. > > > >On Mon, Sep 28, 2015 at 7:57 PM, Dmitriy Setrakyan > ><[hidden email]> > >wrote: > > > >> Gianfranco, > >> > >> I am not too familiar with JBPM. Can you make a case on why we may > >need > >> this integration and how it will benefit Ignite users? > >> > >> Thanks, > >> D. > >> > >> On Mon, Sep 28, 2015 at 6:56 PM, Gianfranco Murador > ><[hidden email]> > >> wrote: > >> > >> > I would like to propose the integration with JBPM , > >> > I think I can finish in a couple of weeks since I have a bit of > >time to > >> > devote to Ignite. > >> > Let me know if you agree. > >> > Regards, Gianfranco > >> > > >> > 2015-09-28 18:50 GMT+02:00 Dmitriy Setrakyan > ><[hidden email]>: > >> > > >> > > I would like to add one more: > >> > > 10. OSGI integration - Raul Kripalani > >> > > > >> > > Raul, hopefully you will have time to finish it. > >> > > > >> > > Also, if anyone in the community wants to propose a feature they > >are > >> > > working on which can be finished within next couple of weeks, it > >should > >> > get > >> > > into 1.5 as well. > >> > > > >> > > D. > >> > > > >> > > On Mon, Sep 28, 2015 at 6:43 PM, Yakov Zhdanov > ><[hidden email]> > >> > > wrote: > >> > > > >> > > > Guys! > >> > > > > >> > > > We have finalized 1.4 release! > >> > > > > >> > > > Thanks to everyone involved! That was very important release > >for our > >> > > > project. > >> > > > > >> > > > Let's discuss scope for 1.5. > >> > > > > >> > > > I think the main focus areas should be: > >> > > > 1. Optimized marshaller rework - Alex Goncharuk works on this. > >> > > > 2. .Net and Cpp API release (depends on 3) - Vladimir Ozerov > >> > > > 3. Portable API release - Denis Magda and Vladimir > >> > > > 4. Failover for continuous query notifications - Nick Tikhonov > >> > > > 5. Cache Rebalancing optimizations - Anton Vinogradov > >> > > > 6. Fix compatibility with IBM JDK > >> > > > 7. Cache stability and performance fixes (as many as possible) > >> > > > > >> > > > Nice to have: > >> > > > 8. Finish vert.x integration - Andrey Gura works on this. > >> > > > 9. Finish node.js integration - I think Semyon should pass it > >over to > >> > > > someone else, given he is busy with cache. Anyone? Andrey Gura? > >> > > > > >> > > > Comments are welcome! > >> > > > > >> > > > [1] https://issues.apache.org/jira/browse/IGNITE-950 > >> > > > [2] https://issues.apache.org/jira/browse/IGNITE-1282 > >> > > > [3] https://issues.apache.org/jira/browse/IGNITE-1258 > >> > > > [4] https://issues.apache.org/jira/browse/IGNITE-426 > >> > > > [5] https://issues.apache.org/jira/browse/IGNITE-1093 > >> > > > [5] https://issues.apache.org/jira/browse/IGNITE-324 > >> > > > [6] https://issues.apache.org/jira/browse/IGNITE-1526 > >> > > > [7] https://issues.apache.org/jira/browse/IGNITE-1345 > >> > > > [8] https://issues.apache.org/jira/browse/IGNITE-1079 > >> > > > [9] https://issues.apache.org/jira/browse/IGNITE-961 > >> > > > > >> > > > --Yakov > >> > > > > >> > > > >> > > >> > > >> > > >> > -- > >> > Gianfranco Murador > >> > Igniter and Software Engineer. > >> > > >> > |
One of the reasons is that I don't believe that running two builds to get two
different module assemblies is a right or useful idea. I can use reactor to build just what's needed, but looks like it won't produce specific assemblies, because spark is included into core package. hence, I would have to move the jars selection logic into the packaging, which seems orthogonal to the case at hands. As for the intersection of the two: will Spark more often intersect with the data fabric core, you think? Cos On Tue, Sep 29, 2015 at 10:58AM, Nikita Ivanov wrote: > Cos - why mixing the two? I think if we are embark on this route - why not > create two different assemblies, one for Hadoop bits, and another for Spark > integration? Two would rarely intersect IMHO. > > -- > Nikita Ivanov > > > On Mon, Sep 28, 2015 at 10:40 PM, Konstantin Boudnik <[hidden email]> wrote: > > > While a minor modification which won't affect anything, I'd like to put > > on the table the consolidation of hadoodp and spark bits into > > ignite-accelerator assembly. The fix is ready and I was just waiting for > > 1.4 to come out. > > > > Cos > > > > On September 28, 2015 11:11:44 AM PDT, Vladimir Ozerov < > > [hidden email]> wrote: > > >I'd like to add several things. > > >1) It looks like fix for IBM JVM will brake backward compatibility > > >because > > >the problem is caused by missing serialVersionId, right? > > >2) HP-UX users are also concerned, see last comments in > > >https://issues.apache.org/jira/browse/IGNITE-1493 I think we must > > >adress > > >Unsafe refactoring in this release as well. > > > > > >On Mon, Sep 28, 2015 at 7:57 PM, Dmitriy Setrakyan > > ><[hidden email]> > > >wrote: > > > > > >> Gianfranco, > > >> > > >> I am not too familiar with JBPM. Can you make a case on why we may > > >need > > >> this integration and how it will benefit Ignite users? > > >> > > >> Thanks, > > >> D. > > >> > > >> On Mon, Sep 28, 2015 at 6:56 PM, Gianfranco Murador > > ><[hidden email]> > > >> wrote: > > >> > > >> > I would like to propose the integration with JBPM , > > >> > I think I can finish in a couple of weeks since I have a bit of > > >time to > > >> > devote to Ignite. > > >> > Let me know if you agree. > > >> > Regards, Gianfranco > > >> > > > >> > 2015-09-28 18:50 GMT+02:00 Dmitriy Setrakyan > > ><[hidden email]>: > > >> > > > >> > > I would like to add one more: > > >> > > 10. OSGI integration - Raul Kripalani > > >> > > > > >> > > Raul, hopefully you will have time to finish it. > > >> > > > > >> > > Also, if anyone in the community wants to propose a feature they > > >are > > >> > > working on which can be finished within next couple of weeks, it > > >should > > >> > get > > >> > > into 1.5 as well. > > >> > > > > >> > > D. > > >> > > > > >> > > On Mon, Sep 28, 2015 at 6:43 PM, Yakov Zhdanov > > ><[hidden email]> > > >> > > wrote: > > >> > > > > >> > > > Guys! > > >> > > > > > >> > > > We have finalized 1.4 release! > > >> > > > > > >> > > > Thanks to everyone involved! That was very important release > > >for our > > >> > > > project. > > >> > > > > > >> > > > Let's discuss scope for 1.5. > > >> > > > > > >> > > > I think the main focus areas should be: > > >> > > > 1. Optimized marshaller rework - Alex Goncharuk works on this. > > >> > > > 2. .Net and Cpp API release (depends on 3) - Vladimir Ozerov > > >> > > > 3. Portable API release - Denis Magda and Vladimir > > >> > > > 4. Failover for continuous query notifications - Nick Tikhonov > > >> > > > 5. Cache Rebalancing optimizations - Anton Vinogradov > > >> > > > 6. Fix compatibility with IBM JDK > > >> > > > 7. Cache stability and performance fixes (as many as possible) > > >> > > > > > >> > > > Nice to have: > > >> > > > 8. Finish vert.x integration - Andrey Gura works on this. > > >> > > > 9. Finish node.js integration - I think Semyon should pass it > > >over to > > >> > > > someone else, given he is busy with cache. Anyone? Andrey Gura? > > >> > > > > > >> > > > Comments are welcome! > > >> > > > > > >> > > > [1] https://issues.apache.org/jira/browse/IGNITE-950 > > >> > > > [2] https://issues.apache.org/jira/browse/IGNITE-1282 > > >> > > > [3] https://issues.apache.org/jira/browse/IGNITE-1258 > > >> > > > [4] https://issues.apache.org/jira/browse/IGNITE-426 > > >> > > > [5] https://issues.apache.org/jira/browse/IGNITE-1093 > > >> > > > [5] https://issues.apache.org/jira/browse/IGNITE-324 > > >> > > > [6] https://issues.apache.org/jira/browse/IGNITE-1526 > > >> > > > [7] https://issues.apache.org/jira/browse/IGNITE-1345 > > >> > > > [8] https://issues.apache.org/jira/browse/IGNITE-1079 > > >> > > > [9] https://issues.apache.org/jira/browse/IGNITE-961 > > >> > > > > > >> > > > --Yakov > > >> > > > > > >> > > > > >> > > > >> > > > >> > > > >> > -- > > >> > Gianfranco Murador > > >> > Igniter and Software Engineer. > > >> > > > >> > > |
Well, my thinking at this point is that none of these three modules would
interest in real life. So, I was thinking of giving users a clear direction on getting a) fabric only, b) hadoop accelerator, and c) spark integration. -- Nikita Ivanov On Tue, Sep 29, 2015 at 5:29 PM, Konstantin Boudnik <[hidden email]> wrote: > One of the reasons is that I don't believe that running two builds to get > two > different module assemblies is a right or useful idea. I can use reactor to > build just what's needed, but looks like it won't produce specific > assemblies, > because spark is included into core package. hence, I would have to move > the > jars selection logic into the packaging, which seems orthogonal to the case > at hands. > > As for the intersection of the two: will Spark more often intersect with > the > data fabric core, you think? > > Cos > > On Tue, Sep 29, 2015 at 10:58AM, Nikita Ivanov wrote: > > Cos - why mixing the two? I think if we are embark on this route - why > not > > create two different assemblies, one for Hadoop bits, and another for > Spark > > integration? Two would rarely intersect IMHO. > > > > -- > > Nikita Ivanov > > > > > > On Mon, Sep 28, 2015 at 10:40 PM, Konstantin Boudnik <[hidden email]> > wrote: > > > > > While a minor modification which won't affect anything, I'd like to > put > > > on the table the consolidation of hadoodp and spark bits into > > > ignite-accelerator assembly. The fix is ready and I was just waiting > for > > > 1.4 to come out. > > > > > > Cos > > > > > > On September 28, 2015 11:11:44 AM PDT, Vladimir Ozerov < > > > [hidden email]> wrote: > > > >I'd like to add several things. > > > >1) It looks like fix for IBM JVM will brake backward compatibility > > > >because > > > >the problem is caused by missing serialVersionId, right? > > > >2) HP-UX users are also concerned, see last comments in > > > >https://issues.apache.org/jira/browse/IGNITE-1493 I think we must > > > >adress > > > >Unsafe refactoring in this release as well. > > > > > > > >On Mon, Sep 28, 2015 at 7:57 PM, Dmitriy Setrakyan > > > ><[hidden email]> > > > >wrote: > > > > > > > >> Gianfranco, > > > >> > > > >> I am not too familiar with JBPM. Can you make a case on why we may > > > >need > > > >> this integration and how it will benefit Ignite users? > > > >> > > > >> Thanks, > > > >> D. > > > >> > > > >> On Mon, Sep 28, 2015 at 6:56 PM, Gianfranco Murador > > > ><[hidden email]> > > > >> wrote: > > > >> > > > >> > I would like to propose the integration with JBPM , > > > >> > I think I can finish in a couple of weeks since I have a bit of > > > >time to > > > >> > devote to Ignite. > > > >> > Let me know if you agree. > > > >> > Regards, Gianfranco > > > >> > > > > >> > 2015-09-28 18:50 GMT+02:00 Dmitriy Setrakyan > > > ><[hidden email]>: > > > >> > > > > >> > > I would like to add one more: > > > >> > > 10. OSGI integration - Raul Kripalani > > > >> > > > > > >> > > Raul, hopefully you will have time to finish it. > > > >> > > > > > >> > > Also, if anyone in the community wants to propose a feature they > > > >are > > > >> > > working on which can be finished within next couple of weeks, it > > > >should > > > >> > get > > > >> > > into 1.5 as well. > > > >> > > > > > >> > > D. > > > >> > > > > > >> > > On Mon, Sep 28, 2015 at 6:43 PM, Yakov Zhdanov > > > ><[hidden email]> > > > >> > > wrote: > > > >> > > > > > >> > > > Guys! > > > >> > > > > > > >> > > > We have finalized 1.4 release! > > > >> > > > > > > >> > > > Thanks to everyone involved! That was very important release > > > >for our > > > >> > > > project. > > > >> > > > > > > >> > > > Let's discuss scope for 1.5. > > > >> > > > > > > >> > > > I think the main focus areas should be: > > > >> > > > 1. Optimized marshaller rework - Alex Goncharuk works on this. > > > >> > > > 2. .Net and Cpp API release (depends on 3) - Vladimir Ozerov > > > >> > > > 3. Portable API release - Denis Magda and Vladimir > > > >> > > > 4. Failover for continuous query notifications - Nick Tikhonov > > > >> > > > 5. Cache Rebalancing optimizations - Anton Vinogradov > > > >> > > > 6. Fix compatibility with IBM JDK > > > >> > > > 7. Cache stability and performance fixes (as many as possible) > > > >> > > > > > > >> > > > Nice to have: > > > >> > > > 8. Finish vert.x integration - Andrey Gura works on this. > > > >> > > > 9. Finish node.js integration - I think Semyon should pass it > > > >over to > > > >> > > > someone else, given he is busy with cache. Anyone? Andrey > Gura? > > > >> > > > > > > >> > > > Comments are welcome! > > > >> > > > > > > >> > > > [1] https://issues.apache.org/jira/browse/IGNITE-950 > > > >> > > > [2] https://issues.apache.org/jira/browse/IGNITE-1282 > > > >> > > > [3] https://issues.apache.org/jira/browse/IGNITE-1258 > > > >> > > > [4] https://issues.apache.org/jira/browse/IGNITE-426 > > > >> > > > [5] https://issues.apache.org/jira/browse/IGNITE-1093 > > > >> > > > [5] https://issues.apache.org/jira/browse/IGNITE-324 > > > >> > > > [6] https://issues.apache.org/jira/browse/IGNITE-1526 > > > >> > > > [7] https://issues.apache.org/jira/browse/IGNITE-1345 > > > >> > > > [8] https://issues.apache.org/jira/browse/IGNITE-1079 > > > >> > > > [9] https://issues.apache.org/jira/browse/IGNITE-961 > > > >> > > > > > > >> > > > --Yakov > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > -- > > > >> > Gianfranco Murador > > > >> > Igniter and Software Engineer. > > > >> > > > > >> > > > > |
How about we rename the hadoop-accelerator to "in-memory file system" and
include spark and map-reduce there as well. "In-memory file system" is a known entity and will bring more interest. We can position the map-reduce and spark RDD as additional features there. Thoughts? D. On Wed, Sep 30, 2015 at 4:00 AM, Nikita Ivanov <[hidden email]> wrote: > Well, my thinking at this point is that none of these three modules would > interest in real life. So, I was thinking of giving users a clear direction > on getting a) fabric only, b) hadoop accelerator, and c) spark integration. > > -- > Nikita Ivanov > > > On Tue, Sep 29, 2015 at 5:29 PM, Konstantin Boudnik <[hidden email]> > wrote: > > > One of the reasons is that I don't believe that running two builds to get > > two > > different module assemblies is a right or useful idea. I can use reactor > to > > build just what's needed, but looks like it won't produce specific > > assemblies, > > because spark is included into core package. hence, I would have to move > > the > > jars selection logic into the packaging, which seems orthogonal to the > case > > at hands. > > > > As for the intersection of the two: will Spark more often intersect with > > the > > data fabric core, you think? > > > > Cos > > > > On Tue, Sep 29, 2015 at 10:58AM, Nikita Ivanov wrote: > > > Cos - why mixing the two? I think if we are embark on this route - why > > not > > > create two different assemblies, one for Hadoop bits, and another for > > Spark > > > integration? Two would rarely intersect IMHO. > > > > > > -- > > > Nikita Ivanov > > > > > > > > > On Mon, Sep 28, 2015 at 10:40 PM, Konstantin Boudnik <[hidden email]> > > wrote: > > > > > > > While a minor modification which won't affect anything, I'd like to > > put > > > > on the table the consolidation of hadoodp and spark bits into > > > > ignite-accelerator assembly. The fix is ready and I was just waiting > > for > > > > 1.4 to come out. > > > > > > > > Cos > > > > > > > > On September 28, 2015 11:11:44 AM PDT, Vladimir Ozerov < > > > > [hidden email]> wrote: > > > > >I'd like to add several things. > > > > >1) It looks like fix for IBM JVM will brake backward compatibility > > > > >because > > > > >the problem is caused by missing serialVersionId, right? > > > > >2) HP-UX users are also concerned, see last comments in > > > > >https://issues.apache.org/jira/browse/IGNITE-1493 I think we must > > > > >adress > > > > >Unsafe refactoring in this release as well. > > > > > > > > > >On Mon, Sep 28, 2015 at 7:57 PM, Dmitriy Setrakyan > > > > ><[hidden email]> > > > > >wrote: > > > > > > > > > >> Gianfranco, > > > > >> > > > > >> I am not too familiar with JBPM. Can you make a case on why we may > > > > >need > > > > >> this integration and how it will benefit Ignite users? > > > > >> > > > > >> Thanks, > > > > >> D. > > > > >> > > > > >> On Mon, Sep 28, 2015 at 6:56 PM, Gianfranco Murador > > > > ><[hidden email]> > > > > >> wrote: > > > > >> > > > > >> > I would like to propose the integration with JBPM , > > > > >> > I think I can finish in a couple of weeks since I have a bit of > > > > >time to > > > > >> > devote to Ignite. > > > > >> > Let me know if you agree. > > > > >> > Regards, Gianfranco > > > > >> > > > > > >> > 2015-09-28 18:50 GMT+02:00 Dmitriy Setrakyan > > > > ><[hidden email]>: > > > > >> > > > > > >> > > I would like to add one more: > > > > >> > > 10. OSGI integration - Raul Kripalani > > > > >> > > > > > > >> > > Raul, hopefully you will have time to finish it. > > > > >> > > > > > > >> > > Also, if anyone in the community wants to propose a feature > they > > > > >are > > > > >> > > working on which can be finished within next couple of weeks, > it > > > > >should > > > > >> > get > > > > >> > > into 1.5 as well. > > > > >> > > > > > > >> > > D. > > > > >> > > > > > > >> > > On Mon, Sep 28, 2015 at 6:43 PM, Yakov Zhdanov > > > > ><[hidden email]> > > > > >> > > wrote: > > > > >> > > > > > > >> > > > Guys! > > > > >> > > > > > > > >> > > > We have finalized 1.4 release! > > > > >> > > > > > > > >> > > > Thanks to everyone involved! That was very important release > > > > >for our > > > > >> > > > project. > > > > >> > > > > > > > >> > > > Let's discuss scope for 1.5. > > > > >> > > > > > > > >> > > > I think the main focus areas should be: > > > > >> > > > 1. Optimized marshaller rework - Alex Goncharuk works on > this. > > > > >> > > > 2. .Net and Cpp API release (depends on 3) - Vladimir Ozerov > > > > >> > > > 3. Portable API release - Denis Magda and Vladimir > > > > >> > > > 4. Failover for continuous query notifications - Nick > Tikhonov > > > > >> > > > 5. Cache Rebalancing optimizations - Anton Vinogradov > > > > >> > > > 6. Fix compatibility with IBM JDK > > > > >> > > > 7. Cache stability and performance fixes (as many as > possible) > > > > >> > > > > > > > >> > > > Nice to have: > > > > >> > > > 8. Finish vert.x integration - Andrey Gura works on this. > > > > >> > > > 9. Finish node.js integration - I think Semyon should pass > it > > > > >over to > > > > >> > > > someone else, given he is busy with cache. Anyone? Andrey > > Gura? > > > > >> > > > > > > > >> > > > Comments are welcome! > > > > >> > > > > > > > >> > > > [1] https://issues.apache.org/jira/browse/IGNITE-950 > > > > >> > > > [2] https://issues.apache.org/jira/browse/IGNITE-1282 > > > > >> > > > [3] https://issues.apache.org/jira/browse/IGNITE-1258 > > > > >> > > > [4] https://issues.apache.org/jira/browse/IGNITE-426 > > > > >> > > > [5] https://issues.apache.org/jira/browse/IGNITE-1093 > > > > >> > > > [5] https://issues.apache.org/jira/browse/IGNITE-324 > > > > >> > > > [6] https://issues.apache.org/jira/browse/IGNITE-1526 > > > > >> > > > [7] https://issues.apache.org/jira/browse/IGNITE-1345 > > > > >> > > > [8] https://issues.apache.org/jira/browse/IGNITE-1079 > > > > >> > > > [9] https://issues.apache.org/jira/browse/IGNITE-961 > > > > >> > > > > > > > >> > > > --Yakov > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > -- > > > > >> > Gianfranco Murador > > > > >> > Igniter and Software Engineer. > > > > >> > > > > > >> > > > > > > > |
I'm not sure how a file system concept corresponds to shared RDD or custom
MapReduce implementation... May we should have another "assembly" just for in-memory file system? -- Nikita Ivanov On Tue, Sep 29, 2015 at 7:04 PM, Dmitriy Setrakyan <[hidden email]> wrote: > How about we rename the hadoop-accelerator to "in-memory file system" and > include spark and map-reduce there as well. > > "In-memory file system" is a known entity and will bring more interest. We > can position the map-reduce and spark RDD as additional features there. > > Thoughts? > > D. > > On Wed, Sep 30, 2015 at 4:00 AM, Nikita Ivanov <[hidden email]> > wrote: > > > Well, my thinking at this point is that none of these three modules would > > interest in real life. So, I was thinking of giving users a clear > direction > > on getting a) fabric only, b) hadoop accelerator, and c) spark > integration. > > > > -- > > Nikita Ivanov > > > > > > On Tue, Sep 29, 2015 at 5:29 PM, Konstantin Boudnik <[hidden email]> > > wrote: > > > > > One of the reasons is that I don't believe that running two builds to > get > > > two > > > different module assemblies is a right or useful idea. I can use > reactor > > to > > > build just what's needed, but looks like it won't produce specific > > > assemblies, > > > because spark is included into core package. hence, I would have to > move > > > the > > > jars selection logic into the packaging, which seems orthogonal to the > > case > > > at hands. > > > > > > As for the intersection of the two: will Spark more often intersect > with > > > the > > > data fabric core, you think? > > > > > > Cos > > > > > > On Tue, Sep 29, 2015 at 10:58AM, Nikita Ivanov wrote: > > > > Cos - why mixing the two? I think if we are embark on this route - > why > > > not > > > > create two different assemblies, one for Hadoop bits, and another for > > > Spark > > > > integration? Two would rarely intersect IMHO. > > > > > > > > -- > > > > Nikita Ivanov > > > > > > > > > > > > On Mon, Sep 28, 2015 at 10:40 PM, Konstantin Boudnik <[hidden email] > > > > > wrote: > > > > > > > > > While a minor modification which won't affect anything, I'd like > to > > > put > > > > > on the table the consolidation of hadoodp and spark bits into > > > > > ignite-accelerator assembly. The fix is ready and I was just > waiting > > > for > > > > > 1.4 to come out. > > > > > > > > > > Cos > > > > > > > > > > On September 28, 2015 11:11:44 AM PDT, Vladimir Ozerov < > > > > > [hidden email]> wrote: > > > > > >I'd like to add several things. > > > > > >1) It looks like fix for IBM JVM will brake backward compatibility > > > > > >because > > > > > >the problem is caused by missing serialVersionId, right? > > > > > >2) HP-UX users are also concerned, see last comments in > > > > > >https://issues.apache.org/jira/browse/IGNITE-1493 I think we must > > > > > >adress > > > > > >Unsafe refactoring in this release as well. > > > > > > > > > > > >On Mon, Sep 28, 2015 at 7:57 PM, Dmitriy Setrakyan > > > > > ><[hidden email]> > > > > > >wrote: > > > > > > > > > > > >> Gianfranco, > > > > > >> > > > > > >> I am not too familiar with JBPM. Can you make a case on why we > may > > > > > >need > > > > > >> this integration and how it will benefit Ignite users? > > > > > >> > > > > > >> Thanks, > > > > > >> D. > > > > > >> > > > > > >> On Mon, Sep 28, 2015 at 6:56 PM, Gianfranco Murador > > > > > ><[hidden email]> > > > > > >> wrote: > > > > > >> > > > > > >> > I would like to propose the integration with JBPM , > > > > > >> > I think I can finish in a couple of weeks since I have a bit > of > > > > > >time to > > > > > >> > devote to Ignite. > > > > > >> > Let me know if you agree. > > > > > >> > Regards, Gianfranco > > > > > >> > > > > > > >> > 2015-09-28 18:50 GMT+02:00 Dmitriy Setrakyan > > > > > ><[hidden email]>: > > > > > >> > > > > > > >> > > I would like to add one more: > > > > > >> > > 10. OSGI integration - Raul Kripalani > > > > > >> > > > > > > > >> > > Raul, hopefully you will have time to finish it. > > > > > >> > > > > > > > >> > > Also, if anyone in the community wants to propose a feature > > they > > > > > >are > > > > > >> > > working on which can be finished within next couple of > weeks, > > it > > > > > >should > > > > > >> > get > > > > > >> > > into 1.5 as well. > > > > > >> > > > > > > > >> > > D. > > > > > >> > > > > > > > >> > > On Mon, Sep 28, 2015 at 6:43 PM, Yakov Zhdanov > > > > > ><[hidden email]> > > > > > >> > > wrote: > > > > > >> > > > > > > > >> > > > Guys! > > > > > >> > > > > > > > > >> > > > We have finalized 1.4 release! > > > > > >> > > > > > > > > >> > > > Thanks to everyone involved! That was very important > release > > > > > >for our > > > > > >> > > > project. > > > > > >> > > > > > > > > >> > > > Let's discuss scope for 1.5. > > > > > >> > > > > > > > > >> > > > I think the main focus areas should be: > > > > > >> > > > 1. Optimized marshaller rework - Alex Goncharuk works on > > this. > > > > > >> > > > 2. .Net and Cpp API release (depends on 3) - Vladimir > Ozerov > > > > > >> > > > 3. Portable API release - Denis Magda and Vladimir > > > > > >> > > > 4. Failover for continuous query notifications - Nick > > Tikhonov > > > > > >> > > > 5. Cache Rebalancing optimizations - Anton Vinogradov > > > > > >> > > > 6. Fix compatibility with IBM JDK > > > > > >> > > > 7. Cache stability and performance fixes (as many as > > possible) > > > > > >> > > > > > > > > >> > > > Nice to have: > > > > > >> > > > 8. Finish vert.x integration - Andrey Gura works on this. > > > > > >> > > > 9. Finish node.js integration - I think Semyon should pass > > it > > > > > >over to > > > > > >> > > > someone else, given he is busy with cache. Anyone? Andrey > > > Gura? > > > > > >> > > > > > > > > >> > > > Comments are welcome! > > > > > >> > > > > > > > > >> > > > [1] https://issues.apache.org/jira/browse/IGNITE-950 > > > > > >> > > > [2] https://issues.apache.org/jira/browse/IGNITE-1282 > > > > > >> > > > [3] https://issues.apache.org/jira/browse/IGNITE-1258 > > > > > >> > > > [4] https://issues.apache.org/jira/browse/IGNITE-426 > > > > > >> > > > [5] https://issues.apache.org/jira/browse/IGNITE-1093 > > > > > >> > > > [5] https://issues.apache.org/jira/browse/IGNITE-324 > > > > > >> > > > [6] https://issues.apache.org/jira/browse/IGNITE-1526 > > > > > >> > > > [7] https://issues.apache.org/jira/browse/IGNITE-1345 > > > > > >> > > > [8] https://issues.apache.org/jira/browse/IGNITE-1079 > > > > > >> > > > [9] https://issues.apache.org/jira/browse/IGNITE-961 > > > > > >> > > > > > > > > >> > > > --Yakov > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > -- > > > > > >> > Gianfranco Murador > > > > > >> > Igniter and Software Engineer. > > > > > >> > > > > > > >> > > > > > > > > > > > |
On Wed, Sep 30, 2015 at 4:34 AM, Nikita Ivanov <[hidden email]> wrote:
> I'm not sure how a file system concept corresponds to shared RDD or custom > MapReduce implementation... May we should have another "assembly" just for > in-memory file system? > And how would users use IgniteRDD and IGFS in the same project? There could be a use case when RDDs are used for objects and IGFS for files in the same application. > > -- > Nikita Ivanov > > > On Tue, Sep 29, 2015 at 7:04 PM, Dmitriy Setrakyan <[hidden email]> > wrote: > > > How about we rename the hadoop-accelerator to "in-memory file system" and > > include spark and map-reduce there as well. > > > > "In-memory file system" is a known entity and will bring more interest. > We > > can position the map-reduce and spark RDD as additional features there. > > > > Thoughts? > > > > D. > > > > On Wed, Sep 30, 2015 at 4:00 AM, Nikita Ivanov <[hidden email]> > > wrote: > > > > > Well, my thinking at this point is that none of these three modules > would > > > interest in real life. So, I was thinking of giving users a clear > > direction > > > on getting a) fabric only, b) hadoop accelerator, and c) spark > > integration. > > > > > > -- > > > Nikita Ivanov > > > > > > > > > On Tue, Sep 29, 2015 at 5:29 PM, Konstantin Boudnik <[hidden email]> > > > wrote: > > > > > > > One of the reasons is that I don't believe that running two builds to > > get > > > > two > > > > different module assemblies is a right or useful idea. I can use > > reactor > > > to > > > > build just what's needed, but looks like it won't produce specific > > > > assemblies, > > > > because spark is included into core package. hence, I would have to > > move > > > > the > > > > jars selection logic into the packaging, which seems orthogonal to > the > > > case > > > > at hands. > > > > > > > > As for the intersection of the two: will Spark more often intersect > > with > > > > the > > > > data fabric core, you think? > > > > > > > > Cos > > > > > > > > On Tue, Sep 29, 2015 at 10:58AM, Nikita Ivanov wrote: > > > > > Cos - why mixing the two? I think if we are embark on this route - > > why > > > > not > > > > > create two different assemblies, one for Hadoop bits, and another > for > > > > Spark > > > > > integration? Two would rarely intersect IMHO. > > > > > > > > > > -- > > > > > Nikita Ivanov > > > > > > > > > > > > > > > On Mon, Sep 28, 2015 at 10:40 PM, Konstantin Boudnik < > [hidden email] > > > > > > > wrote: > > > > > > > > > > > While a minor modification which won't affect anything, I'd like > > to > > > > put > > > > > > on the table the consolidation of hadoodp and spark bits into > > > > > > ignite-accelerator assembly. The fix is ready and I was just > > waiting > > > > for > > > > > > 1.4 to come out. > > > > > > > > > > > > Cos > > > > > > > > > > > > On September 28, 2015 11:11:44 AM PDT, Vladimir Ozerov < > > > > > > [hidden email]> wrote: > > > > > > >I'd like to add several things. > > > > > > >1) It looks like fix for IBM JVM will brake backward > compatibility > > > > > > >because > > > > > > >the problem is caused by missing serialVersionId, right? > > > > > > >2) HP-UX users are also concerned, see last comments in > > > > > > >https://issues.apache.org/jira/browse/IGNITE-1493 I think we > must > > > > > > >adress > > > > > > >Unsafe refactoring in this release as well. > > > > > > > > > > > > > >On Mon, Sep 28, 2015 at 7:57 PM, Dmitriy Setrakyan > > > > > > ><[hidden email]> > > > > > > >wrote: > > > > > > > > > > > > > >> Gianfranco, > > > > > > >> > > > > > > >> I am not too familiar with JBPM. Can you make a case on why we > > may > > > > > > >need > > > > > > >> this integration and how it will benefit Ignite users? > > > > > > >> > > > > > > >> Thanks, > > > > > > >> D. > > > > > > >> > > > > > > >> On Mon, Sep 28, 2015 at 6:56 PM, Gianfranco Murador > > > > > > ><[hidden email]> > > > > > > >> wrote: > > > > > > >> > > > > > > >> > I would like to propose the integration with JBPM , > > > > > > >> > I think I can finish in a couple of weeks since I have a > bit > > of > > > > > > >time to > > > > > > >> > devote to Ignite. > > > > > > >> > Let me know if you agree. > > > > > > >> > Regards, Gianfranco > > > > > > >> > > > > > > > >> > 2015-09-28 18:50 GMT+02:00 Dmitriy Setrakyan > > > > > > ><[hidden email]>: > > > > > > >> > > > > > > > >> > > I would like to add one more: > > > > > > >> > > 10. OSGI integration - Raul Kripalani > > > > > > >> > > > > > > > > >> > > Raul, hopefully you will have time to finish it. > > > > > > >> > > > > > > > > >> > > Also, if anyone in the community wants to propose a > feature > > > they > > > > > > >are > > > > > > >> > > working on which can be finished within next couple of > > weeks, > > > it > > > > > > >should > > > > > > >> > get > > > > > > >> > > into 1.5 as well. > > > > > > >> > > > > > > > > >> > > D. > > > > > > >> > > > > > > > > >> > > On Mon, Sep 28, 2015 at 6:43 PM, Yakov Zhdanov > > > > > > ><[hidden email]> > > > > > > >> > > wrote: > > > > > > >> > > > > > > > > >> > > > Guys! > > > > > > >> > > > > > > > > > >> > > > We have finalized 1.4 release! > > > > > > >> > > > > > > > > > >> > > > Thanks to everyone involved! That was very important > > release > > > > > > >for our > > > > > > >> > > > project. > > > > > > >> > > > > > > > > > >> > > > Let's discuss scope for 1.5. > > > > > > >> > > > > > > > > > >> > > > I think the main focus areas should be: > > > > > > >> > > > 1. Optimized marshaller rework - Alex Goncharuk works on > > > this. > > > > > > >> > > > 2. .Net and Cpp API release (depends on 3) - Vladimir > > Ozerov > > > > > > >> > > > 3. Portable API release - Denis Magda and Vladimir > > > > > > >> > > > 4. Failover for continuous query notifications - Nick > > > Tikhonov > > > > > > >> > > > 5. Cache Rebalancing optimizations - Anton Vinogradov > > > > > > >> > > > 6. Fix compatibility with IBM JDK > > > > > > >> > > > 7. Cache stability and performance fixes (as many as > > > possible) > > > > > > >> > > > > > > > > > >> > > > Nice to have: > > > > > > >> > > > 8. Finish vert.x integration - Andrey Gura works on > this. > > > > > > >> > > > 9. Finish node.js integration - I think Semyon should > pass > > > it > > > > > > >over to > > > > > > >> > > > someone else, given he is busy with cache. Anyone? > Andrey > > > > Gura? > > > > > > >> > > > > > > > > > >> > > > Comments are welcome! > > > > > > >> > > > > > > > > > >> > > > [1] https://issues.apache.org/jira/browse/IGNITE-950 > > > > > > >> > > > [2] https://issues.apache.org/jira/browse/IGNITE-1282 > > > > > > >> > > > [3] https://issues.apache.org/jira/browse/IGNITE-1258 > > > > > > >> > > > [4] https://issues.apache.org/jira/browse/IGNITE-426 > > > > > > >> > > > [5] https://issues.apache.org/jira/browse/IGNITE-1093 > > > > > > >> > > > [5] https://issues.apache.org/jira/browse/IGNITE-324 > > > > > > >> > > > [6] https://issues.apache.org/jira/browse/IGNITE-1526 > > > > > > >> > > > [7] https://issues.apache.org/jira/browse/IGNITE-1345 > > > > > > >> > > > [8] https://issues.apache.org/jira/browse/IGNITE-1079 > > > > > > >> > > > [9] https://issues.apache.org/jira/browse/IGNITE-961 > > > > > > >> > > > > > > > > > >> > > > --Yakov > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > -- > > > > > > >> > Gianfranco Murador > > > > > > >> > Igniter and Software Engineer. > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > |
With my Ignite hat on, I think having them together in the same assembly makes
sense. After all, this is why I have offered the patch in the form it is right now. With my Bigtop hat on: I don't care one way or another. As soon as I can produce one or more assemblies that will have all the required bits in it - I will hack them anyway to produce seamless native installation experience for the users. I am happy to to change the patch to produce two assemblies or six as far as we can unblock the Bigtop change. Thanks, Cos On Wed, Sep 30, 2015 at 09:14AM, Dmitriy Setrakyan wrote: > On Wed, Sep 30, 2015 at 4:34 AM, Nikita Ivanov <[hidden email]> wrote: > > > I'm not sure how a file system concept corresponds to shared RDD or custom > > MapReduce implementation... May we should have another "assembly" just for > > in-memory file system? > > > > And how would users use IgniteRDD and IGFS in the same project? There could > be a use case when RDDs are used for objects and IGFS for files in the same > application. > > > > > > -- > > Nikita Ivanov > > > > > > On Tue, Sep 29, 2015 at 7:04 PM, Dmitriy Setrakyan <[hidden email]> > > wrote: > > > > > How about we rename the hadoop-accelerator to "in-memory file system" and > > > include spark and map-reduce there as well. > > > > > > "In-memory file system" is a known entity and will bring more interest. > > We > > > can position the map-reduce and spark RDD as additional features there. > > > > > > Thoughts? > > > > > > D. > > > > > > On Wed, Sep 30, 2015 at 4:00 AM, Nikita Ivanov <[hidden email]> > > > wrote: > > > > > > > Well, my thinking at this point is that none of these three modules > > would > > > > interest in real life. So, I was thinking of giving users a clear > > > direction > > > > on getting a) fabric only, b) hadoop accelerator, and c) spark > > > integration. > > > > > > > > -- > > > > Nikita Ivanov > > > > > > > > > > > > On Tue, Sep 29, 2015 at 5:29 PM, Konstantin Boudnik <[hidden email]> > > > > wrote: > > > > > > > > > One of the reasons is that I don't believe that running two builds to > > > get > > > > > two > > > > > different module assemblies is a right or useful idea. I can use > > > reactor > > > > to > > > > > build just what's needed, but looks like it won't produce specific > > > > > assemblies, > > > > > because spark is included into core package. hence, I would have to > > > move > > > > > the > > > > > jars selection logic into the packaging, which seems orthogonal to > > the > > > > case > > > > > at hands. > > > > > > > > > > As for the intersection of the two: will Spark more often intersect > > > with > > > > > the > > > > > data fabric core, you think? > > > > > > > > > > Cos > > > > > > > > > > On Tue, Sep 29, 2015 at 10:58AM, Nikita Ivanov wrote: > > > > > > Cos - why mixing the two? I think if we are embark on this route - > > > why > > > > > not > > > > > > create two different assemblies, one for Hadoop bits, and another > > for > > > > > Spark > > > > > > integration? Two would rarely intersect IMHO. > > > > > > > > > > > > -- > > > > > > Nikita Ivanov > > > > > > > > > > > > > > > > > > On Mon, Sep 28, 2015 at 10:40 PM, Konstantin Boudnik < > > [hidden email] > > > > > > > > > wrote: > > > > > > > > > > > > > While a minor modification which won't affect anything, I'd like > > > to > > > > > put > > > > > > > on the table the consolidation of hadoodp and spark bits into > > > > > > > ignite-accelerator assembly. The fix is ready and I was just > > > waiting > > > > > for > > > > > > > 1.4 to come out. > > > > > > > > > > > > > > Cos > > > > > > > > > > > > > > On September 28, 2015 11:11:44 AM PDT, Vladimir Ozerov < > > > > > > > [hidden email]> wrote: > > > > > > > >I'd like to add several things. > > > > > > > >1) It looks like fix for IBM JVM will brake backward > > compatibility > > > > > > > >because > > > > > > > >the problem is caused by missing serialVersionId, right? > > > > > > > >2) HP-UX users are also concerned, see last comments in > > > > > > > >https://issues.apache.org/jira/browse/IGNITE-1493 I think we > > must > > > > > > > >adress > > > > > > > >Unsafe refactoring in this release as well. > > > > > > > > > > > > > > > >On Mon, Sep 28, 2015 at 7:57 PM, Dmitriy Setrakyan > > > > > > > ><[hidden email]> > > > > > > > >wrote: > > > > > > > > > > > > > > > >> Gianfranco, > > > > > > > >> > > > > > > > >> I am not too familiar with JBPM. Can you make a case on why we > > > may > > > > > > > >need > > > > > > > >> this integration and how it will benefit Ignite users? > > > > > > > >> > > > > > > > >> Thanks, > > > > > > > >> D. > > > > > > > >> > > > > > > > >> On Mon, Sep 28, 2015 at 6:56 PM, Gianfranco Murador > > > > > > > ><[hidden email]> > > > > > > > >> wrote: > > > > > > > >> > > > > > > > >> > I would like to propose the integration with JBPM , > > > > > > > >> > I think I can finish in a couple of weeks since I have a > > bit > > > of > > > > > > > >time to > > > > > > > >> > devote to Ignite. > > > > > > > >> > Let me know if you agree. > > > > > > > >> > Regards, Gianfranco > > > > > > > >> > > > > > > > > >> > 2015-09-28 18:50 GMT+02:00 Dmitriy Setrakyan > > > > > > > ><[hidden email]>: > > > > > > > >> > > > > > > > > >> > > I would like to add one more: > > > > > > > >> > > 10. OSGI integration - Raul Kripalani > > > > > > > >> > > > > > > > > > >> > > Raul, hopefully you will have time to finish it. > > > > > > > >> > > > > > > > > > >> > > Also, if anyone in the community wants to propose a > > feature > > > > they > > > > > > > >are > > > > > > > >> > > working on which can be finished within next couple of > > > weeks, > > > > it > > > > > > > >should > > > > > > > >> > get > > > > > > > >> > > into 1.5 as well. > > > > > > > >> > > > > > > > > > >> > > D. > > > > > > > >> > > > > > > > > > >> > > On Mon, Sep 28, 2015 at 6:43 PM, Yakov Zhdanov > > > > > > > ><[hidden email]> > > > > > > > >> > > wrote: > > > > > > > >> > > > > > > > > > >> > > > Guys! > > > > > > > >> > > > > > > > > > > >> > > > We have finalized 1.4 release! > > > > > > > >> > > > > > > > > > > >> > > > Thanks to everyone involved! That was very important > > > release > > > > > > > >for our > > > > > > > >> > > > project. > > > > > > > >> > > > > > > > > > > >> > > > Let's discuss scope for 1.5. > > > > > > > >> > > > > > > > > > > >> > > > I think the main focus areas should be: > > > > > > > >> > > > 1. Optimized marshaller rework - Alex Goncharuk works on > > > > this. > > > > > > > >> > > > 2. .Net and Cpp API release (depends on 3) - Vladimir > > > Ozerov > > > > > > > >> > > > 3. Portable API release - Denis Magda and Vladimir > > > > > > > >> > > > 4. Failover for continuous query notifications - Nick > > > > Tikhonov > > > > > > > >> > > > 5. Cache Rebalancing optimizations - Anton Vinogradov > > > > > > > >> > > > 6. Fix compatibility with IBM JDK > > > > > > > >> > > > 7. Cache stability and performance fixes (as many as > > > > possible) > > > > > > > >> > > > > > > > > > > >> > > > Nice to have: > > > > > > > >> > > > 8. Finish vert.x integration - Andrey Gura works on > > this. > > > > > > > >> > > > 9. Finish node.js integration - I think Semyon should > > pass > > > > it > > > > > > > >over to > > > > > > > >> > > > someone else, given he is busy with cache. Anyone? > > Andrey > > > > > Gura? > > > > > > > >> > > > > > > > > > > >> > > > Comments are welcome! > > > > > > > >> > > > > > > > > > > >> > > > [1] https://issues.apache.org/jira/browse/IGNITE-950 > > > > > > > >> > > > [2] https://issues.apache.org/jira/browse/IGNITE-1282 > > > > > > > >> > > > [3] https://issues.apache.org/jira/browse/IGNITE-1258 > > > > > > > >> > > > [4] https://issues.apache.org/jira/browse/IGNITE-426 > > > > > > > >> > > > [5] https://issues.apache.org/jira/browse/IGNITE-1093 > > > > > > > >> > > > [5] https://issues.apache.org/jira/browse/IGNITE-324 > > > > > > > >> > > > [6] https://issues.apache.org/jira/browse/IGNITE-1526 > > > > > > > >> > > > [7] https://issues.apache.org/jira/browse/IGNITE-1345 > > > > > > > >> > > > [8] https://issues.apache.org/jira/browse/IGNITE-1079 > > > > > > > >> > > > [9] https://issues.apache.org/jira/browse/IGNITE-961 > > > > > > > >> > > > > > > > > > > >> > > > --Yakov > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > -- > > > > > > > >> > Gianfranco Murador > > > > > > > >> > Igniter and Software Engineer. > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > |
Free forum by Nabble | Edit this page |