Adding ML to Ignite, IGNITE-4572

[nivanov]

Igniters,
I'd like to kick off the conversation on adding ML capabilities to Ignite
as it relates to IGNITE-4572.

I started off some preliminary steps on this ticket and would like to
invite anyone interesting in implementing distributed core algebra on top
of Ignite to join me in this work. There's plenty to contribute from ideas
and math, to implementation, tests, and documentation.

Background on IGNITE-4572.
----------------------------------------
Throughout many conferences last year I have heard the sentiment that one
of the functional areas that is badly missing in Ignite is a library of
distributed Machine Learning algorithms. In fact, it was frequently noted
as the only reason not to finally migrate from Spark to Ignite...

ML, in general, is a pretty large undertaking and there are more than a few
ways to approach this in the context of Ignite.  One idea I'd like to chew
on and prototype is to look close at Apache Mahout (
http://mahout.apache.org/). After a deep look at Mahout codebase I don't
believe the outright code-level integration is feasible at this points -
but we can certainly borrow as many ideas from it as possible.

Thanks!
--
Nikita Ivanov

[nivanov]

Just a quick follow up...

We've made some good progress on the initial code implementation in the last couple of months and I think we'll make it on time to release a early "beta" version of the IgniteML (a.k.a distributed math) for 2.0 release.

[dmagda]

Unbelievable, that’s astonishing we can already release the distributed algebra support in the beta state. Please share the tasks you’re planning to work on right after the release.

—
Denis

> On Mar 28, 2017, at 11:12 AM, nivanov <[hidden email]> wrote:
>
> Just a quick follow up...
>
> We've made some good progress on the initial code implementation in the last
> couple of months and I think we'll make it on time to release a early "beta"
> version of the IgniteML (a.k.a distributed math) for 2.0 release.
>
>
>
> --
> View this message in context: http://apache-ignite-developers.2346864.n4.nabble.com/Adding-ML-to-Ignite-IGNITE-4572-tp13936p15858.html
> Sent from the Apache Ignite Developers mailing list archive at Nabble.com.

[dsetrakyan]

+100

Great feature! I too would like to see a list of tickets required for the
release.

D.

On Tue, Mar 28, 2017 at 11:24 AM, Denis Magda <[hidden email]> wrote:

> Unbelievable, that’s astonishing we can already release the distributed
> algebra support in the beta state. Please share the tasks you’re planning
> to work on right after the release.
>
> —
> Denis
>
> > On Mar 28, 2017, at 11:12 AM, nivanov <[hidden email]> wrote:
> >
> > Just a quick follow up...
> >
> > We've made some good progress on the initial code implementation in the
> last
> > couple of months and I think we'll make it on time to release a early
> "beta"
> > version of the IgniteML (a.k.a distributed math) for 2.0 release.
> >
> >
> >
> > --
> > View this message in context: http://apache-ignite-
> developers.2346864.n4.nabble.com/Adding-ML-to-Ignite-
> IGNITE-4572-tp13936p15858.html
> > Sent from the Apache Ignite Developers mailing list archive at
> Nabble.com.
>
>

[alpercakan]

Hi,
I am Alper. I am a computer engineering and mathematics (as double major) student with great interest in machine learning. I wish to participate in GSOC - 2017 and I've been interested in the issue https://issues.apache.org/jira/plugins/servlet/mobile#issue/IGNITE-4572 for a while. I think that my skills fits this project and that I can really be useful and contribute. Can you please help me about where I need to start from?
Thank you.

[nivanov]

Alper - we are very close to "pull request" our preliminary work into ignite 2.0 (finger crossed it will make it into it). Once we have it available in the main ignite 2.0 branch there will be plenty of interesting tasks to tackle. Stay tuned on this thread!

[alpercakan]

Thank you for your reply. I will be watching this thread.

However, GSOC applications are due Monday. So, some suggestions for what to write on the proposal would be great, if pull request will be later than Monday.

[anantbietec]

yeah thats good but GSOC application last till today only so any suggestion
how to write the proposal or what need to be included in the proposal.



--
View this message in context: http://apache-ignite-developers.2346864.n4.nabble.com/Adding-ML-to-Ignite-IGNITE-4572-tp13936p16070.html
Sent from the Apache Ignite Developers mailing list archive at Nabble.com.

[anantbietec]

Hi i am Anant khandelwal currently pursuing mastes in the area of machine learning and numerical optimization I am interested in this project as i have recently modify the map algorithm through the use of HYpersphere I am a student at IIT delhi also today is the last day to finalize application for GSOC 2017 so any suggestion please what to include in the proposal.

[dmagda]

These are some suggestions from Nikita that were sent to Alper privately that might be useful for other candidates:

Alper,
Here are some of the immediate tasks we'll be working on past 2.0 release (and you are 100% welcome to tackle them if you find them interesting):

(1) Linear regression algorithm on super-large distributed sparse dataset
(2) Logistic regression algorithm on super-large distributed sparse dataset

Thanks!

> On Apr 3, 2017, at 12:21 AM, anantbietec <[hidden email]> wrote:
>
> yeah thats good but GSOC application last till today only so any suggestion
> how to write the proposal or what need to be included in the proposal.
>
>
>
> --
> View this message in context: http://apache-ignite-developers.2346864.n4.nabble.com/Adding-ML-to-Ignite-IGNITE-4572-tp13936p16070.html
> Sent from the Apache Ignite Developers mailing list archive at Nabble.com.

[anantbietec]

I am very enthusiatic about this project but sir i have not submitted PROPOSAL FOR THIS IN gsoc 2017  can i still be allowed to do this project

As the reply comes very late i didnt know what are the tasks to do in this project . So i didnt prepare the proposal .

Can we still have the discussion and assign tasks to me so that i can contribute to the project and also tell me upto now what has been done so that i am also on the same page.

Thanks
Anant

[Denis Magda]

Sorry for the late reply. Sure, you're always welcomed to join our
community and participate in the development of modules of interest.

As for ML, please keep an eye on this discussion where will break down ML
activities a bit later.

Denis

On Wednesday, April 5, 2017, anantbietec <[hidden email]> wrote:

> I am very enthusiatic about this project but sir i have not submitted
> PROPOSAL FOR THIS IN gsoc 2017  can i still be allowed to do this project
>
> As the reply comes very late i didnt know what are the tasks to do in this
> project . So i didnt prepare the proposal .
>
> Can we still have the discussion and assign tasks to me so that i can
> contribute to the project and also tell me upto now what has been done so
> that i am also on the same page.
>
> Thanks
> Anant
>
>
>
> --
> View this message in context: http://apache-ignite-
> developers.2346864.n4.nabble.com/Adding-ML-to-Ignite-
> IGNITE-4572-tp13936p16190.html
> Sent from the Apache Ignite Developers mailing list archive at Nabble.com.
>

[Yuriy Babak]

Hi all

Till the end of this week we want to prepare early "beta" version of the IgniteML for merge with 2.0 release.

[Yuriy Babak]

Guys, a bit more details about the current state of ML lib and how we plan to release it.

Currently IgniteML is separate module and use java8. We depends only on ignite-core module. So we think that we could release lib as sources. It should be much easier than provide our lib binary.

[dmagda]

Yuri, thanks for an update.

Agree, that it makes sense to release ML in sources for now in order not to overcomplicate our build system right before the release.

*Anton*, would you help Yuri to add ML into our project build system? Yuri, going forward I would suggest you to learn more about Ignite build system and make up a way how to release it in binaries. The community will support you on this for sure.  

*Nikita*, *Yuri*, who of you will prepare the documentation on Ignite readme.io <http://readme.io/>? We should explain what the module is and how to use it in the current state.

—
Denis

> On Apr 12, 2017, at 11:33 AM, Yury Babak <[hidden email]> wrote:
>
> Guys, a bit more details about the current state of ML lib and how we plan to
> release it.
>
> Currently IgniteML is separate module and use java8. We depends only on
> ignite-core module. So we think that we could release lib as sources. It
> should be much easier than provide our lib binary.
>
>
>
> --
> View this message in context: http://apache-ignite-developers.2346864.n4.nabble.com/Adding-ML-to-Ignite-IGNITE-4572-tp13936p16532.html
> Sent from the Apache Ignite Developers mailing list archive at Nabble.com.

[Yuriy Babak]

As far as I know Nikita wants to provide this documentation.

[oignatenko]

Yury, Denis - me and Nikita are going to take care of readme.io docs.

I plan to start on this tomorrow, will keep you posted on the progress.

Current plan is to briefly introduce new package and examples and explain how to run examples and possibly unit tests.

With regards to build details, this will depend on how accurate is the description in project readme files: I am currently running build check to verify everything in details. If readme files are okay we will likely just refer readers to these for more details. In case if there are any discrepancies we will provide concrete build details at readme.io.

If you have anything to add or correct in above plan please let me know.

regards, Oleg

Yury Babak wrote

As far as I know Nikita wants to provide this documentation.

[dmagda]

Hi Oleg,

That’s great, thanks. I’ve created a JIRA ticket for this task and assigned on Nikita for now. Please create an account for yourself and reassign the ticket if needed:
https://issues.apache.org/jira/browse/IGNITE-4964

Plus, there is an invisible page on readme.io <http://readme.io/> created for you where you can document everything right away:
https://dash.readme.io/project/apacheignite/v1.9/docs/distributed-algebra

I’ll grant you editor rights to the readme today, you’ll receive a notification.

—
Denis
 

> On Apr 13, 2017, at 9:33 AM, oignatenko <[hidden email]> wrote:
>
> Yury, Denis - me and Nikita are going to take care of readme.io docs.
>
> I plan to start on this tomorrow, will keep you posted on the progress.
>
> Current plan is to briefly introduce new package and examples and explain
> how to run examples and possibly unit tests.
>
> With regards to build details, this will depend on how accurate is the
> description in project readme files: I am currently running build check to
> verify everything in details. If readme files are okay we will likely just
> refer readers to these for more details. In case if there are any
> discrepancies we will provide concrete build details at readme.io.
>
> If you have anything to add or correct in above plan please let me know.
>
> regards, Oleg
>
> Yury Babak wrote
>> As far as I know Nikita wants to provide this documentation.
>
>
>
>
>
> --
> View this message in context: http://apache-ignite-developers.2346864.n4.nabble.com/Adding-ML-to-Ignite-IGNITE-4572-tp13936p16603.html
> Sent from the Apache Ignite Developers mailing list archive at Nabble.com.

[oignatenko]

Thank you Denis, that sounds like a good plan. I am working on it now. I didn't reassign IGNITE-4964, just set a watch on it. Will comment on it when I have concrete details to share.

Side note my build check completed successfully, although it took quite a bit of time: I cleaned my local maven repo to make sure that things really work from the clean state.

[dmagda]

Oleg, Nikita, Yuri,

Frankly, I don’t like how the module is named presently which is 'ignite-math’.

I propose to rename it to ‘ignite-ml’. Presently, it holds a small fraction of all ML functionality we might have in the future and this is why, thinking of future, it will be a right decision to name it accordingly.

Any concerns?

BTW, have the module been fully merged? I want to try to build it and check the examples.

—
Denis

> On Apr 14, 2017, at 7:05 AM, oignatenko <[hidden email]> wrote:
>
> Thank you Denis, that sounds like a good plan. I am working on it now. I
> didn't reassign IGNITE-4964, just set a watch on it. Will comment on it when
> I have concrete details to share.
>
> Side note my build check completed successfully, although it took quite a
> bit of time: I cleaned my local maven repo to make sure that things really
> work from the clean state.
>
>
>
> --
> View this message in context: http://apache-ignite-developers.2346864.n4.nabble.com/Adding-ML-to-Ignite-IGNITE-4572-tp13936p16677.html
> Sent from the Apache Ignite Developers mailing list archive at Nabble.com.