Is ML module @IgniteExperimental?

[Nikolay Izhikov-2]

Hello, Igniters.

Can someone bring some light on the state of the ML module in Ignite?
As far as I know, for now, we have only 1 active contributor to this area -
Alexey Zinoviev.
I see how whole modules come and go from the module - [1]

Please, also note this quote:

> Also as a result of good testing from both side (from me and Stepan) we
> found a lot of bugs and CVEs in hadoop related components that should be
> removed in release branch too.

0. As far as I know, the ML module has no documentation. Is it correct? Do
we have plans to fix it?

1. Can someone related to the ML, please, give some examples of the CVE or
issues that can be fixed only with removing a bunch of modules?

3. Is ML module production ready?

4. Should we mark it with the @IgniteExperimental?

5. Should we move it to the ignite-extensions?

[1]
https://github.com/apache/ignite/commit/53e886b8ed38a6842cef8b44ace6851855dfad29

[Alexey Zinoviev]

Hello, Igniters, and you, Nikolay.

First of all, if you have real interest to the ML module and its state, I
could make call with you and explain this.


*As far as I know, for now, we have only 1 active contributor to this area
-Alexey Zinoviev.*
Currently, we have 2 active contributors, me and Ravil Galeeyev, a few
newbies, another guys who started tensorflow and another modules and
submodules don't visit the community for many months.

*Is ML module production ready?*
This release will be the first release, than ML is production ready and it
totally my work.


*Can someone related to the ML, please, give some examples of the CVE
orissues that can be fixed only with removing a bunch of modules?*
CVE is not the main reason to remove the "bunch of modules", but part of
the story.
The main reason, the modules are not work proper way, were experimental,
never released as a production-ready, support old, outdated version, the
external frameworks, like Tensorflow, move integration with ignite to the
special repos, they are not finished, the code there is broken and couldn't
be fixed, because and I have no power/C++ skills/permission to commit
something to them and time to support this broken modules.

Also broken TF module blocks the removal of IGFS.

Found CVE were related to the dependencies related to hadoop/tf/parquet and
so on.

*Should we mark it with the @IgniteExperimental? *
I don't know, we have no this RAW annotation a few weeks ago and I don't
know how we should use it.
It could be, if you finish the discussion about this annotation and the
write docs about it and share it with me.


* As far as I know, the ML module has no documentation. Is it correct? Dowe
have plans to fix it?*
The ML docs are here, on our Ignite documentation
https://apacheignite.readme.io/docs/machine-learning
of course, something could be wrong, 1.5 year we are not released Ignite
Yes, I have plan to fix. Of course, after fixing all bugs in release branch

*Should we move it to the ignite-extensions?*
No we shouldn't, I don't want this, and have a lot of arguments and
currently is not the time for discussion about this (they are too young and
have now real infrastructure and release-cycle)

P.S. Community, I understand that the removal of module looks strange, but
we could understand that ML was a strange experiment without roadmap and
this situation is finished.
Now, I have roadmap (will be published later), newbie tickets, the ability
to prepare correct docs, the understanding what could be used by clients
and first of all, production-ready ML (it could be run on Ignite-cluster,
really, it works.)

If you, Igniters, believe, that I could be a good maintainer for ML module,
please support me here in this thread
If you think, that I do something wrong, OK, please write it too, I'll read
carefully.

I spent a few months to fix bugs in components, which were abandoned by
their creators.

My goal: Ignite should have light-weight, easy-integrated ML without
strange and unfinished experiments which could be not maintained. It's a
part of common movement in Ignite (removal of modules or moving them to the
separate repos).


пн, 17 февр. 2020 г. в 12:10, Nikolay Izhikov <[hidden email]>:

> Hello, Igniters.
>
> Can someone bring some light on the state of the ML module in Ignite?
> As far as I know, for now, we have only 1 active contributor to this area -
> Alexey Zinoviev.
> I see how whole modules come and go from the module - [1]
>
> Please, also note this quote:
>
> > Also as a result of good testing from both side (from me and Stepan) we
> > found a lot of bugs and CVEs in hadoop related components that should be
> > removed in release branch too.
>
> 0. As far as I know, the ML module has no documentation. Is it correct? Do
> we have plans to fix it?
>
> 1. Can someone related to the ML, please, give some examples of the CVE or
> issues that can be fixed only with removing a bunch of modules?
>
> 3. Is ML module production ready?
>
> 4. Should we mark it with the @IgniteExperimental?
>
> 5. Should we move it to the ignite-extensions?
>
> [1]
>
> https://github.com/apache/ignite/commit/53e886b8ed38a6842cef8b44ace6851855dfad29
>

[Dmitry Pavlov]

Folks,

>  Do we have plans to fix it?
It is not how community works. It depends on contributors and their
interests  in maintaining and fixing particular bug/feature/module. If
someone would like to help, he/she is always welcomed.

Don't ask others to do stuff for you, in other words.

>CVE
CVEs  are not to be discussed at dev@ use security@ or private@. see
https://www.apache.org/security/committers.html

 > Should we mark it with the @IgniteExperimental
it may be agreed by the community during separate discussion/vote. I
personally not sure I agree.

Since Alexey has interest in maintaining, I support his idea on how module
should be developed. Apache motto is: "Let those who do the job make a
decisions".

Sincerely,
Dmitriy Pavlov


пн, 17 февр. 2020 г. в 13:49, Alexey Zinoviev <[hidden email]>:

> Hello, Igniters, and you, Nikolay.
>
> First of all, if you have real interest to the ML module and its state, I
> could make call with you and explain this.
>
>
> *As far as I know, for now, we have only 1 active contributor to this area
> -Alexey Zinoviev.*
> Currently, we have 2 active contributors, me and Ravil Galeeyev, a few
> newbies, another guys who started tensorflow and another modules and
> submodules don't visit the community for many months.
>
> *Is ML module production ready?*
> This release will be the first release, than ML is production ready and it
> totally my work.
>
>
> *Can someone related to the ML, please, give some examples of the CVE
> orissues that can be fixed only with removing a bunch of modules?*
> CVE is not the main reason to remove the "bunch of modules", but part of
> the story.
> The main reason, the modules are not work proper way, were experimental,
> never released as a production-ready, support old, outdated version, the
> external frameworks, like Tensorflow, move integration with ignite to the
> special repos, they are not finished, the code there is broken and couldn't
> be fixed, because and I have no power/C++ skills/permission to commit
> something to them and time to support this broken modules.
>
> Also broken TF module blocks the removal of IGFS.
>
> Found CVE were related to the dependencies related to hadoop/tf/parquet and
> so on.
>
> *Should we mark it with the @IgniteExperimental? *
> I don't know, we have no this RAW annotation a few weeks ago and I don't
> know how we should use it.
> It could be, if you finish the discussion about this annotation and the
> write docs about it and share it with me.
>
>
> * As far as I know, the ML module has no documentation. Is it correct? Dowe
> have plans to fix it?*
> The ML docs are here, on our Ignite documentation
> https://apacheignite.readme.io/docs/machine-learning
> of course, something could be wrong, 1.5 year we are not released Ignite
> Yes, I have plan to fix. Of course, after fixing all bugs in release branch
>
> *Should we move it to the ignite-extensions?*
> No we shouldn't, I don't want this, and have a lot of arguments and
> currently is not the time for discussion about this (they are too young and
> have now real infrastructure and release-cycle)
>
> P.S. Community, I understand that the removal of module looks strange, but
> we could understand that ML was a strange experiment without roadmap and
> this situation is finished.
> Now, I have roadmap (will be published later), newbie tickets, the ability
> to prepare correct docs, the understanding what could be used by clients
> and first of all, production-ready ML (it could be run on Ignite-cluster,
> really, it works.)
>
> If you, Igniters, believe, that I could be a good maintainer for ML module,
> please support me here in this thread
> If you think, that I do something wrong, OK, please write it too, I'll read
> carefully.
>
> I spent a few months to fix bugs in components, which were abandoned by
> their creators.
>
> My goal: Ignite should have light-weight, easy-integrated ML without
> strange and unfinished experiments which could be not maintained. It's a
> part of common movement in Ignite (removal of modules or moving them to the
> separate repos).
>
>
> пн, 17 февр. 2020 г. в 12:10, Nikolay Izhikov <[hidden email]>:
>
> > Hello, Igniters.
> >
> > Can someone bring some light on the state of the ML module in Ignite?
> > As far as I know, for now, we have only 1 active contributor to this
> area -
> > Alexey Zinoviev.
> > I see how whole modules come and go from the module - [1]
> >
> > Please, also note this quote:
> >
> > > Also as a result of good testing from both side (from me and Stepan) we
> > > found a lot of bugs and CVEs in hadoop related components that should
> be
> > > removed in release branch too.
> >
> > 0. As far as I know, the ML module has no documentation. Is it correct?
> Do
> > we have plans to fix it?
> >
> > 1. Can someone related to the ML, please, give some examples of the CVE
> or
> > issues that can be fixed only with removing a bunch of modules?
> >
> > 3. Is ML module production ready?
> >
> > 4. Should we mark it with the @IgniteExperimental?
> >
> > 5. Should we move it to the ignite-extensions?
> >
> > [1]
> >
> >
> https://github.com/apache/ignite/commit/53e886b8ed38a6842cef8b44ace6851855dfad29
> >
>

[Ravil Galeyev]

Hi Team,

First of all, let me introduce myself. I’m Ravil, I contribute to the ML
module since 2018 and from time to time I make talks about it. (I..e data
science summit in Warsaw [1]).

So, Alexey made a huge effort to develop the ML module but he is not alone.
If you check the repo you will find other contributors.

Therefore the ML module is alive and is able to run and has the roadmap.
For me, it means that it’s not a raw project.

Regarding documentation, it’d like to mention the code is the best
documentation :)

We have examples for most algorithms [2]. But if it needed I’m ready to
help the community with documentation in English German Polish or Russain.


[1] https://dssconf.pl/

[2]
https://github.com/apache/ignite/tree/master/examples/src/main/java/org/apache/ignite/examples/ml

Best regards,

Ravil


On Mon, 17 Feb 2020 at 11:49, Alexey Zinoviev <[hidden email]>
wrote:

> Hello, Igniters, and you, Nikolay.
>
> First of all, if you have real interest to the ML module and its state, I
> could make call with you and explain this.
>
>
> *As far as I know, for now, we have only 1 active contributor to this area
> -Alexey Zinoviev.*
> Currently, we have 2 active contributors, me and Ravil Galeeyev, a few
> newbies, another guys who started tensorflow and another modules and
> submodules don't visit the community for many months.
>
> *Is ML module production ready?*
> This release will be the first release, than ML is production ready and it
> totally my work.
>
>
> *Can someone related to the ML, please, give some examples of the CVE
> orissues that can be fixed only with removing a bunch of modules?*
> CVE is not the main reason to remove the "bunch of modules", but part of
> the story.
> The main reason, the modules are not work proper way, were experimental,
> never released as a production-ready, support old, outdated version, the
> external frameworks, like Tensorflow, move integration with ignite to the
> special repos, they are not finished, the code there is broken and couldn't
> be fixed, because and I have no power/C++ skills/permission to commit
> something to them and time to support this broken modules.
>
> Also broken TF module blocks the removal of IGFS.
>
> Found CVE were related to the dependencies related to hadoop/tf/parquet and
> so on.
>
> *Should we mark it with the @IgniteExperimental? *
> I don't know, we have no this RAW annotation a few weeks ago and I don't
> know how we should use it.
> It could be, if you finish the discussion about this annotation and the
> write docs about it and share it with me.
>
>
> * As far as I know, the ML module has no documentation. Is it correct? Dowe
> have plans to fix it?*
> The ML docs are here, on our Ignite documentation
> https://apacheignite.readme.io/docs/machine-learning
> of course, something could be wrong, 1.5 year we are not released Ignite
> Yes, I have plan to fix. Of course, after fixing all bugs in release branch
>
> *Should we move it to the ignite-extensions?*
> No we shouldn't, I don't want this, and have a lot of arguments and
> currently is not the time for discussion about this (they are too young and
> have now real infrastructure and release-cycle)
>
> P.S. Community, I understand that the removal of module looks strange, but
> we could understand that ML was a strange experiment without roadmap and
> this situation is finished.
> Now, I have roadmap (will be published later), newbie tickets, the ability
> to prepare correct docs, the understanding what could be used by clients
> and first of all, production-ready ML (it could be run on Ignite-cluster,
> really, it works.)
>
> If you, Igniters, believe, that I could be a good maintainer for ML module,
> please support me here in this thread
> If you think, that I do something wrong, OK, please write it too, I'll read
> carefully.
>
> I spent a few months to fix bugs in components, which were abandoned by
> their creators.
>
> My goal: Ignite should have light-weight, easy-integrated ML without
> strange and unfinished experiments which could be not maintained. It's a
> part of common movement in Ignite (removal of modules or moving them to the
> separate repos).
>
>
> пн, 17 февр. 2020 г. в 12:10, Nikolay Izhikov <[hidden email]>:
>
> > Hello, Igniters.
> >
> > Can someone bring some light on the state of the ML module in Ignite?
> > As far as I know, for now, we have only 1 active contributor to this
> area -
> > Alexey Zinoviev.
> > I see how whole modules come and go from the module - [1]
> >
> > Please, also note this quote:
> >
> > > Also as a result of good testing from both side (from me and Stepan) we
> > > found a lot of bugs and CVEs in hadoop related components that should
> be
> > > removed in release branch too.
> >
> > 0. As far as I know, the ML module has no documentation. Is it correct?
> Do
> > we have plans to fix it?
> >
> > 1. Can someone related to the ML, please, give some examples of the CVE
> or
> > issues that can be fixed only with removing a bunch of modules?
> >
> > 3. Is ML module production ready?
> >
> > 4. Should we mark it with the @IgniteExperimental?
> >
> > 5. Should we move it to the ignite-extensions?
> >
> > [1]
> >
> >
> https://github.com/apache/ignite/commit/53e886b8ed38a6842cef8b44ace6851855dfad29
> >
>

[Nikolay Izhikov-2]

Hello, Alexey.

> The main reason, the modules are not work proper way, were experimental,
> never released as a production-ready, support old, outdated version, the
> external frameworks, like Tensorflow, move integration with ignite to the
> special repos, they are not finished, the code there is broken and couldn't
> be fixed, because and I have no power/C++ skills/permission to commit
> something to them and time to support this broken modules.


Do we have some tickets or wider explanation for it?
It very uncommon for me that the decision to remove modules from the master and release is not discussed widely in the community.

> 17 февр. 2020 г., в 14:39, Ravil Galeyev <[hidden email]> написал(а):
>
> Hi Team,
>
> First of all, let me introduce myself. I’m Ravil, I contribute to the ML
> module since 2018 and from time to time I make talks about it. (I..e data
> science summit in Warsaw [1]).
>
> So, Alexey made a huge effort to develop the ML module but he is not alone.
> If you check the repo you will find other contributors.
>
> Therefore the ML module is alive and is able to run and has the roadmap.
> For me, it means that it’s not a raw project.
>
> Regarding documentation, it’d like to mention the code is the best
> documentation :)
>
> We have examples for most algorithms [2]. But if it needed I’m ready to
> help the community with documentation in English German Polish or Russain.
>
>
> [1] https://dssconf.pl/
>
> [2]
> https://github.com/apache/ignite/tree/master/examples/src/main/java/org/apache/ignite/examples/ml
>
> Best regards,
>
> Ravil
>
>
> On Mon, 17 Feb 2020 at 11:49, Alexey Zinoviev <[hidden email]>
> wrote:
>
>> Hello, Igniters, and you, Nikolay.
>>
>> First of all, if you have real interest to the ML module and its state, I
>> could make call with you and explain this.
>>
>>
>> *As far as I know, for now, we have only 1 active contributor to this area
>> -Alexey Zinoviev.*
>> Currently, we have 2 active contributors, me and Ravil Galeeyev, a few
>> newbies, another guys who started tensorflow and another modules and
>> submodules don't visit the community for many months.
>>
>> *Is ML module production ready?*
>> This release will be the first release, than ML is production ready and it
>> totally my work.
>>
>>
>> *Can someone related to the ML, please, give some examples of the CVE
>> orissues that can be fixed only with removing a bunch of modules?*
>> CVE is not the main reason to remove the "bunch of modules", but part of
>> the story.
>> The main reason, the modules are not work proper way, were experimental,
>> never released as a production-ready, support old, outdated version, the
>> external frameworks, like Tensorflow, move integration with ignite to the
>> special repos, they are not finished, the code there is broken and couldn't
>> be fixed, because and I have no power/C++ skills/permission to commit
>> something to them and time to support this broken modules.
>>
>> Also broken TF module blocks the removal of IGFS.
>>
>> Found CVE were related to the dependencies related to hadoop/tf/parquet and
>> so on.
>>
>> *Should we mark it with the @IgniteExperimental? *
>> I don't know, we have no this RAW annotation a few weeks ago and I don't
>> know how we should use it.
>> It could be, if you finish the discussion about this annotation and the
>> write docs about it and share it with me.
>>
>>
>> * As far as I know, the ML module has no documentation. Is it correct? Dowe
>> have plans to fix it?*
>> The ML docs are here, on our Ignite documentation
>> https://apacheignite.readme.io/docs/machine-learning
>> of course, something could be wrong, 1.5 year we are not released Ignite
>> Yes, I have plan to fix. Of course, after fixing all bugs in release branch
>>
>> *Should we move it to the ignite-extensions?*
>> No we shouldn't, I don't want this, and have a lot of arguments and
>> currently is not the time for discussion about this (they are too young and
>> have now real infrastructure and release-cycle)
>>
>> P.S. Community, I understand that the removal of module looks strange, but
>> we could understand that ML was a strange experiment without roadmap and
>> this situation is finished.
>> Now, I have roadmap (will be published later), newbie tickets, the ability
>> to prepare correct docs, the understanding what could be used by clients
>> and first of all, production-ready ML (it could be run on Ignite-cluster,
>> really, it works.)
>>
>> If you, Igniters, believe, that I could be a good maintainer for ML module,
>> please support me here in this thread
>> If you think, that I do something wrong, OK, please write it too, I'll read
>> carefully.
>>
>> I spent a few months to fix bugs in components, which were abandoned by
>> their creators.
>>
>> My goal: Ignite should have light-weight, easy-integrated ML without
>> strange and unfinished experiments which could be not maintained. It's a
>> part of common movement in Ignite (removal of modules or moving them to the
>> separate repos).
>>
>>
>> пн, 17 февр. 2020 г. в 12:10, Nikolay Izhikov <[hidden email]>:
>>
>>> Hello, Igniters.
>>>
>>> Can someone bring some light on the state of the ML module in Ignite?
>>> As far as I know, for now, we have only 1 active contributor to this
>> area -
>>> Alexey Zinoviev.
>>> I see how whole modules come and go from the module - [1]
>>>
>>> Please, also note this quote:
>>>
>>>> Also as a result of good testing from both side (from me and Stepan) we
>>>> found a lot of bugs and CVEs in hadoop related components that should
>> be
>>>> removed in release branch too.
>>>
>>> 0. As far as I know, the ML module has no documentation. Is it correct?
>> Do
>>> we have plans to fix it?
>>>
>>> 1. Can someone related to the ML, please, give some examples of the CVE
>> or
>>> issues that can be fixed only with removing a bunch of modules?
>>>
>>> 3. Is ML module production ready?
>>>
>>> 4. Should we mark it with the @IgniteExperimental?
>>>
>>> 5. Should we move it to the ignite-extensions?
>>>
>>> [1]
>>>
>>>
>> https://github.com/apache/ignite/commit/53e886b8ed38a6842cef8b44ace6851855dfad29
>>>
>>

[Alexey Zinoviev]

Ok, agree, that I should start discussion before making changes, but I was
limited by release 2.8 and trying don;'t be a delayed person for that.
During release I was focused on fixing bugs and don't tests TF and Ignite
together

I thought that as a maintainer of ML module I could do perform these
actions.

Below I will share my statement why it should be removed and why it should
be removed immediately

About TensorFlow module (reason for removal)

   1. This module is only one module that uses IGFS and needs in FileSystem
   on Ignte side due to TensorFlow API
   2. This module a part of bridge between Ignite ML and Tensorflow and its
   broken after changes in TensorFlow on TensorFlow side
   3. TensorFlow released new version without Ignite bridge, no chance to
   run them together for new releases
   4. This module wasn't complete and developer who did this, left the
   community
   5. The development skills for this story require python/C++/java
   programming together
   6. The module is a source of bugs which could be fixed for release 2.8
   and possibly for future releases (nobody in community could this)
   7. The release size reduced from 6 Gb to 4.5 due to removed dependencies
   8. TensorFlow now is not popular among Data Scientists, the PyTorch is
   the most popular tool for Deep Learning (like NetBeans and IDEA)
   9. Nobody uses that in production because it was developed between 2.7
   and 2.8 (2.7 has only proof-of-concept)

Nikolay, sorry for that, hope to share more information about the ML and
discuss here the main changes before actions.

пн, 17 февр. 2020 г. в 16:18, Nikolay Izhikov <[hidden email]>:

> Hello, Alexey.
>
> > The main reason, the modules are not work proper way, were experimental,
> > never released as a production-ready, support old, outdated version, the
> > external frameworks, like Tensorflow, move integration with ignite to the
> > special repos, they are not finished, the code there is broken and
> couldn't
> > be fixed, because and I have no power/C++ skills/permission to commit
> > something to them and time to support this broken modules.
>
>
> Do we have some tickets or wider explanation for it?
> It very uncommon for me that the decision to remove modules from the
> master and release is not discussed widely in the community.
>
> > 17 февр. 2020 г., в 14:39, Ravil Galeyev <[hidden email]>
> написал(а):
> >
> > Hi Team,
> >
> > First of all, let me introduce myself. I’m Ravil, I contribute to the ML
> > module since 2018 and from time to time I make talks about it. (I..e data
> > science summit in Warsaw [1]).
> >
> > So, Alexey made a huge effort to develop the ML module but he is not
> alone.
> > If you check the repo you will find other contributors.
> >
> > Therefore the ML module is alive and is able to run and has the roadmap.
> > For me, it means that it’s not a raw project.
> >
> > Regarding documentation, it’d like to mention the code is the best
> > documentation :)
> >
> > We have examples for most algorithms [2]. But if it needed I’m ready to
> > help the community with documentation in English German Polish or
> Russain.
> >
> >
> > [1] https://dssconf.pl/
> >
> > [2]
> >
> https://github.com/apache/ignite/tree/master/examples/src/main/java/org/apache/ignite/examples/ml
> >
> > Best regards,
> >
> > Ravil
> >
> >
> > On Mon, 17 Feb 2020 at 11:49, Alexey Zinoviev <[hidden email]>
> > wrote:
> >
> >> Hello, Igniters, and you, Nikolay.
> >>
> >> First of all, if you have real interest to the ML module and its state,
> I
> >> could make call with you and explain this.
> >>
> >>
> >> *As far as I know, for now, we have only 1 active contributor to this
> area
> >> -Alexey Zinoviev.*
> >> Currently, we have 2 active contributors, me and Ravil Galeeyev, a few
> >> newbies, another guys who started tensorflow and another modules and
> >> submodules don't visit the community for many months.
> >>
> >> *Is ML module production ready?*
> >> This release will be the first release, than ML is production ready and
> it
> >> totally my work.
> >>
> >>
> >> *Can someone related to the ML, please, give some examples of the CVE
> >> orissues that can be fixed only with removing a bunch of modules?*
> >> CVE is not the main reason to remove the "bunch of modules", but part of
> >> the story.
> >> The main reason, the modules are not work proper way, were experimental,
> >> never released as a production-ready, support old, outdated version, the
> >> external frameworks, like Tensorflow, move integration with ignite to
> the
> >> special repos, they are not finished, the code there is broken and
> couldn't
> >> be fixed, because and I have no power/C++ skills/permission to commit
> >> something to them and time to support this broken modules.
> >>
> >> Also broken TF module blocks the removal of IGFS.
> >>
> >> Found CVE were related to the dependencies related to hadoop/tf/parquet
> and
> >> so on.
> >>
> >> *Should we mark it with the @IgniteExperimental? *
> >> I don't know, we have no this RAW annotation a few weeks ago and I don't
> >> know how we should use it.
> >> It could be, if you finish the discussion about this annotation and the
> >> write docs about it and share it with me.
> >>
> >>
> >> * As far as I know, the ML module has no documentation. Is it correct?
> Dowe
> >> have plans to fix it?*
> >> The ML docs are here, on our Ignite documentation
> >> https://apacheignite.readme.io/docs/machine-learning
> >> of course, something could be wrong, 1.5 year we are not released Ignite
> >> Yes, I have plan to fix. Of course, after fixing all bugs in release
> branch
> >>
> >> *Should we move it to the ignite-extensions?*
> >> No we shouldn't, I don't want this, and have a lot of arguments and
> >> currently is not the time for discussion about this (they are too young
> and
> >> have now real infrastructure and release-cycle)
> >>
> >> P.S. Community, I understand that the removal of module looks strange,
> but
> >> we could understand that ML was a strange experiment without roadmap and
> >> this situation is finished.
> >> Now, I have roadmap (will be published later), newbie tickets, the
> ability
> >> to prepare correct docs, the understanding what could be used by clients
> >> and first of all, production-ready ML (it could be run on
> Ignite-cluster,
> >> really, it works.)
> >>
> >> If you, Igniters, believe, that I could be a good maintainer for ML
> module,
> >> please support me here in this thread
> >> If you think, that I do something wrong, OK, please write it too, I'll
> read
> >> carefully.
> >>
> >> I spent a few months to fix bugs in components, which were abandoned by
> >> their creators.
> >>
> >> My goal: Ignite should have light-weight, easy-integrated ML without
> >> strange and unfinished experiments which could be not maintained. It's a
> >> part of common movement in Ignite (removal of modules or moving them to
> the
> >> separate repos).
> >>
> >>
> >> пн, 17 февр. 2020 г. в 12:10, Nikolay Izhikov <[hidden email]>:
> >>
> >>> Hello, Igniters.
> >>>
> >>> Can someone bring some light on the state of the ML module in Ignite?
> >>> As far as I know, for now, we have only 1 active contributor to this
> >> area -
> >>> Alexey Zinoviev.
> >>> I see how whole modules come and go from the module - [1]
> >>>
> >>> Please, also note this quote:
> >>>
> >>>> Also as a result of good testing from both side (from me and Stepan)
> we
> >>>> found a lot of bugs and CVEs in hadoop related components that should
> >> be
> >>>> removed in release branch too.
> >>>
> >>> 0. As far as I know, the ML module has no documentation. Is it correct?
> >> Do
> >>> we have plans to fix it?
> >>>
> >>> 1. Can someone related to the ML, please, give some examples of the CVE
> >> or
> >>> issues that can be fixed only with removing a bunch of modules?
> >>>
> >>> 3. Is ML module production ready?
> >>>
> >>> 4. Should we mark it with the @IgniteExperimental?
> >>>
> >>> 5. Should we move it to the ignite-extensions?
> >>>
> >>> [1]
> >>>
> >>>
> >>
> https://github.com/apache/ignite/commit/53e886b8ed38a6842cef8b44ace6851855dfad29
> >>>
> >>
>
>

[dmagda]

Alexey,

I missed this thread and only now realized that TensorFlow, genetic
algorithms and some other APIs were expelled from 2.8. I would encourage us
to start a dedicated discussion for any APIs removal or significant changes
to let other community members share their opinions or take appropriate
actions (like proper documentation redirects setup for pages that are gone
and updates on the website like [1] and [2]). For instance, I have no glue
that the topic of TensorFlow removal was briefly mentioned in this
discussion thread.

I see the reasoning about TensorFlow but why have we removed generic
algorithms that had a dependency on the compute APIs only?

[1] https://ignite.apache.org/features/tensorflow.html
[2] https://ignite.apache.org/features/machinelearning.html#ga-grid

-
Denis


On Mon, Feb 17, 2020 at 5:39 AM Alexey Zinoviev <[hidden email]>
wrote:

> Ok, agree, that I should start discussion before making changes, but I was
> limited by release 2.8 and trying don;'t be a delayed person for that.
> During release I was focused on fixing bugs and don't tests TF and Ignite
> together
>
> I thought that as a maintainer of ML module I could do perform these
> actions.
>
> Below I will share my statement why it should be removed and why it should
> be removed immediately
>
> About TensorFlow module (reason for removal)
>
>    1. This module is only one module that uses IGFS and needs in FileSystem
>    on Ignte side due to TensorFlow API
>    2. This module a part of bridge between Ignite ML and Tensorflow and its
>    broken after changes in TensorFlow on TensorFlow side
>    3. TensorFlow released new version without Ignite bridge, no chance to
>    run them together for new releases
>    4. This module wasn't complete and developer who did this, left the
>    community
>    5. The development skills for this story require python/C++/java
>    programming together
>    6. The module is a source of bugs which could be fixed for release 2.8
>    and possibly for future releases (nobody in community could this)
>    7. The release size reduced from 6 Gb to 4.5 due to removed dependencies
>    8. TensorFlow now is not popular among Data Scientists, the PyTorch is
>    the most popular tool for Deep Learning (like NetBeans and IDEA)
>    9. Nobody uses that in production because it was developed between 2.7
>    and 2.8 (2.7 has only proof-of-concept)
>
> Nikolay, sorry for that, hope to share more information about the ML and
> discuss here the main changes before actions.
>
> пн, 17 февр. 2020 г. в 16:18, Nikolay Izhikov <[hidden email]>:
>
> > Hello, Alexey.
> >
> > > The main reason, the modules are not work proper way, were
> experimental,
> > > never released as a production-ready, support old, outdated version,
> the
> > > external frameworks, like Tensorflow, move integration with ignite to
> the
> > > special repos, they are not finished, the code there is broken and
> > couldn't
> > > be fixed, because and I have no power/C++ skills/permission to commit
> > > something to them and time to support this broken modules.
> >
> >
> > Do we have some tickets or wider explanation for it?
> > It very uncommon for me that the decision to remove modules from the
> > master and release is not discussed widely in the community.
> >
> > > 17 февр. 2020 г., в 14:39, Ravil Galeyev <[hidden email]>
> > написал(а):
> > >
> > > Hi Team,
> > >
> > > First of all, let me introduce myself. I’m Ravil, I contribute to the
> ML
> > > module since 2018 and from time to time I make talks about it. (I..e
> data
> > > science summit in Warsaw [1]).
> > >
> > > So, Alexey made a huge effort to develop the ML module but he is not
> > alone.
> > > If you check the repo you will find other contributors.
> > >
> > > Therefore the ML module is alive and is able to run and has the
> roadmap.
> > > For me, it means that it’s not a raw project.
> > >
> > > Regarding documentation, it’d like to mention the code is the best
> > > documentation :)
> > >
> > > We have examples for most algorithms [2]. But if it needed I’m ready to
> > > help the community with documentation in English German Polish or
> > Russain.
> > >
> > >
> > > [1] https://dssconf.pl/
> > >
> > > [2]
> > >
> >
> https://github.com/apache/ignite/tree/master/examples/src/main/java/org/apache/ignite/examples/ml
> > >
> > > Best regards,
> > >
> > > Ravil
> > >
> > >
> > > On Mon, 17 Feb 2020 at 11:49, Alexey Zinoviev <[hidden email]>
> > > wrote:
> > >
> > >> Hello, Igniters, and you, Nikolay.
> > >>
> > >> First of all, if you have real interest to the ML module and its
> state,
> > I
> > >> could make call with you and explain this.
> > >>
> > >>
> > >> *As far as I know, for now, we have only 1 active contributor to this
> > area
> > >> -Alexey Zinoviev.*
> > >> Currently, we have 2 active contributors, me and Ravil Galeeyev, a few
> > >> newbies, another guys who started tensorflow and another modules and
> > >> submodules don't visit the community for many months.
> > >>
> > >> *Is ML module production ready?*
> > >> This release will be the first release, than ML is production ready
> and
> > it
> > >> totally my work.
> > >>
> > >>
> > >> *Can someone related to the ML, please, give some examples of the CVE
> > >> orissues that can be fixed only with removing a bunch of modules?*
> > >> CVE is not the main reason to remove the "bunch of modules", but part
> of
> > >> the story.
> > >> The main reason, the modules are not work proper way, were
> experimental,
> > >> never released as a production-ready, support old, outdated version,
> the
> > >> external frameworks, like Tensorflow, move integration with ignite to
> > the
> > >> special repos, they are not finished, the code there is broken and
> > couldn't
> > >> be fixed, because and I have no power/C++ skills/permission to commit
> > >> something to them and time to support this broken modules.
> > >>
> > >> Also broken TF module blocks the removal of IGFS.
> > >>
> > >> Found CVE were related to the dependencies related to
> hadoop/tf/parquet
> > and
> > >> so on.
> > >>
> > >> *Should we mark it with the @IgniteExperimental? *
> > >> I don't know, we have no this RAW annotation a few weeks ago and I
> don't
> > >> know how we should use it.
> > >> It could be, if you finish the discussion about this annotation and
> the
> > >> write docs about it and share it with me.
> > >>
> > >>
> > >> * As far as I know, the ML module has no documentation. Is it correct?
> > Dowe
> > >> have plans to fix it?*
> > >> The ML docs are here, on our Ignite documentation
> > >> https://apacheignite.readme.io/docs/machine-learning
> > >> of course, something could be wrong, 1.5 year we are not released
> Ignite
> > >> Yes, I have plan to fix. Of course, after fixing all bugs in release
> > branch
> > >>
> > >> *Should we move it to the ignite-extensions?*
> > >> No we shouldn't, I don't want this, and have a lot of arguments and
> > >> currently is not the time for discussion about this (they are too
> young
> > and
> > >> have now real infrastructure and release-cycle)
> > >>
> > >> P.S. Community, I understand that the removal of module looks strange,
> > but
> > >> we could understand that ML was a strange experiment without roadmap
> and
> > >> this situation is finished.
> > >> Now, I have roadmap (will be published later), newbie tickets, the
> > ability
> > >> to prepare correct docs, the understanding what could be used by
> clients
> > >> and first of all, production-ready ML (it could be run on
> > Ignite-cluster,
> > >> really, it works.)
> > >>
> > >> If you, Igniters, believe, that I could be a good maintainer for ML
> > module,
> > >> please support me here in this thread
> > >> If you think, that I do something wrong, OK, please write it too, I'll
> > read
> > >> carefully.
> > >>
> > >> I spent a few months to fix bugs in components, which were abandoned
> by
> > >> their creators.
> > >>
> > >> My goal: Ignite should have light-weight, easy-integrated ML without
> > >> strange and unfinished experiments which could be not maintained.
> It's a
> > >> part of common movement in Ignite (removal of modules or moving them
> to
> > the
> > >> separate repos).
> > >>
> > >>
> > >> пн, 17 февр. 2020 г. в 12:10, Nikolay Izhikov <[hidden email]>:
> > >>
> > >>> Hello, Igniters.
> > >>>
> > >>> Can someone bring some light on the state of the ML module in Ignite?
> > >>> As far as I know, for now, we have only 1 active contributor to this
> > >> area -
> > >>> Alexey Zinoviev.
> > >>> I see how whole modules come and go from the module - [1]
> > >>>
> > >>> Please, also note this quote:
> > >>>
> > >>>> Also as a result of good testing from both side (from me and Stepan)
> > we
> > >>>> found a lot of bugs and CVEs in hadoop related components that
> should
> > >> be
> > >>>> removed in release branch too.
> > >>>
> > >>> 0. As far as I know, the ML module has no documentation. Is it
> correct?
> > >> Do
> > >>> we have plans to fix it?
> > >>>
> > >>> 1. Can someone related to the ML, please, give some examples of the
> CVE
> > >> or
> > >>> issues that can be fixed only with removing a bunch of modules?
> > >>>
> > >>> 3. Is ML module production ready?
> > >>>
> > >>> 4. Should we mark it with the @IgniteExperimental?
> > >>>
> > >>> 5. Should we move it to the ignite-extensions?
> > >>>
> > >>> [1]
> > >>>
> > >>>
> > >>
> >
> https://github.com/apache/ignite/commit/53e886b8ed38a6842cef8b44ace6851855dfad29
> > >>>
> > >>
> >
> >
>

[Alexey Zinoviev]

Hi, Denis!

Be honest, the significant amount of the ML contirbutors left the community
previous year in frustration with unfinished parts.
In this situation, I reduced the unsed and broken parts according our
previous discussions peer-to-peer (not on devlist, our mistake) to release
the stable core of ML which could be supported with reduced power.

The reasons for GA removal
1. It doesn't related to the ML topic
2. It has no intersection with the ML package (as you mentioned)
3. It doesn't support Ignite code and in many places Java codestyle
4. It was experimental package placed in ML in time of earliest experiments
in 2017
5. Nobody doesn't want to support this for the years

Genetic Algorithms could be moved to Ignite-extension (if somebody
interested in it)

A lot of things are changed since release 2.7

Lessons are learnt, I will start discussion topics next time for the
significant changes or removal in API, moreover, the next releases I hope
to use new @IgniteExperimental (it was added too late) and another
annotations for the release cycle.

вт, 24 мар. 2020 г. в 20:00, Denis Magda <[hidden email]>:

> Alexey,
>
> I missed this thread and only now realized that TensorFlow, genetic
> algorithms and some other APIs were expelled from 2.8. I would encourage us
> to start a dedicated discussion for any APIs removal or significant changes
> to let other community members share their opinions or take appropriate
> actions (like proper documentation redirects setup for pages that are gone
> and updates on the website like [1] and [2]). For instance, I have no glue
> that the topic of TensorFlow removal was briefly mentioned in this
> discussion thread.
>
> I see the reasoning about TensorFlow but why have we removed generic
> algorithms that had a dependency on the compute APIs only?
>
> [1] https://ignite.apache.org/features/tensorflow.html
> [2] https://ignite.apache.org/features/machinelearning.html#ga-grid
>
> -
> Denis
>
>
> On Mon, Feb 17, 2020 at 5:39 AM Alexey Zinoviev <[hidden email]>
> wrote:
>
> > Ok, agree, that I should start discussion before making changes, but I
> was
> > limited by release 2.8 and trying don;'t be a delayed person for that.
> > During release I was focused on fixing bugs and don't tests TF and Ignite
> > together
> >
> > I thought that as a maintainer of ML module I could do perform these
> > actions.
> >
> > Below I will share my statement why it should be removed and why it
> should
> > be removed immediately
> >
> > About TensorFlow module (reason for removal)
> >
> >    1. This module is only one module that uses IGFS and needs in
> FileSystem
> >    on Ignte side due to TensorFlow API
> >    2. This module a part of bridge between Ignite ML and Tensorflow and
> its
> >    broken after changes in TensorFlow on TensorFlow side
> >    3. TensorFlow released new version without Ignite bridge, no chance to
> >    run them together for new releases
> >    4. This module wasn't complete and developer who did this, left the
> >    community
> >    5. The development skills for this story require python/C++/java
> >    programming together
> >    6. The module is a source of bugs which could be fixed for release 2.8
> >    and possibly for future releases (nobody in community could this)
> >    7. The release size reduced from 6 Gb to 4.5 due to removed
> dependencies
> >    8. TensorFlow now is not popular among Data Scientists, the PyTorch is
> >    the most popular tool for Deep Learning (like NetBeans and IDEA)
> >    9. Nobody uses that in production because it was developed between 2.7
> >    and 2.8 (2.7 has only proof-of-concept)
> >
> > Nikolay, sorry for that, hope to share more information about the ML and
> > discuss here the main changes before actions.
> >
> > пн, 17 февр. 2020 г. в 16:18, Nikolay Izhikov <[hidden email]>:
> >
> > > Hello, Alexey.
> > >
> > > > The main reason, the modules are not work proper way, were
> > experimental,
> > > > never released as a production-ready, support old, outdated version,
> > the
> > > > external frameworks, like Tensorflow, move integration with ignite to
> > the
> > > > special repos, they are not finished, the code there is broken and
> > > couldn't
> > > > be fixed, because and I have no power/C++ skills/permission to commit
> > > > something to them and time to support this broken modules.
> > >
> > >
> > > Do we have some tickets or wider explanation for it?
> > > It very uncommon for me that the decision to remove modules from the
> > > master and release is not discussed widely in the community.
> > >
> > > > 17 февр. 2020 г., в 14:39, Ravil Galeyev <[hidden email]>
> > > написал(а):
> > > >
> > > > Hi Team,
> > > >
> > > > First of all, let me introduce myself. I’m Ravil, I contribute to the
> > ML
> > > > module since 2018 and from time to time I make talks about it. (I..e
> > data
> > > > science summit in Warsaw [1]).
> > > >
> > > > So, Alexey made a huge effort to develop the ML module but he is not
> > > alone.
> > > > If you check the repo you will find other contributors.
> > > >
> > > > Therefore the ML module is alive and is able to run and has the
> > roadmap.
> > > > For me, it means that it’s not a raw project.
> > > >
> > > > Regarding documentation, it’d like to mention the code is the best
> > > > documentation :)
> > > >
> > > > We have examples for most algorithms [2]. But if it needed I’m ready
> to
> > > > help the community with documentation in English German Polish or
> > > Russain.
> > > >
> > > >
> > > > [1] https://dssconf.pl/
> > > >
> > > > [2]
> > > >
> > >
> >
> https://github.com/apache/ignite/tree/master/examples/src/main/java/org/apache/ignite/examples/ml
> > > >
> > > > Best regards,
> > > >
> > > > Ravil
> > > >
> > > >
> > > > On Mon, 17 Feb 2020 at 11:49, Alexey Zinoviev <
> [hidden email]>
> > > > wrote:
> > > >
> > > >> Hello, Igniters, and you, Nikolay.
> > > >>
> > > >> First of all, if you have real interest to the ML module and its
> > state,
> > > I
> > > >> could make call with you and explain this.
> > > >>
> > > >>
> > > >> *As far as I know, for now, we have only 1 active contributor to
> this
> > > area
> > > >> -Alexey Zinoviev.*
> > > >> Currently, we have 2 active contributors, me and Ravil Galeeyev, a
> few
> > > >> newbies, another guys who started tensorflow and another modules and
> > > >> submodules don't visit the community for many months.
> > > >>
> > > >> *Is ML module production ready?*
> > > >> This release will be the first release, than ML is production ready
> > and
> > > it
> > > >> totally my work.
> > > >>
> > > >>
> > > >> *Can someone related to the ML, please, give some examples of the
> CVE
> > > >> orissues that can be fixed only with removing a bunch of modules?*
> > > >> CVE is not the main reason to remove the "bunch of modules", but
> part
> > of
> > > >> the story.
> > > >> The main reason, the modules are not work proper way, were
> > experimental,
> > > >> never released as a production-ready, support old, outdated version,
> > the
> > > >> external frameworks, like Tensorflow, move integration with ignite
> to
> > > the
> > > >> special repos, they are not finished, the code there is broken and
> > > couldn't
> > > >> be fixed, because and I have no power/C++ skills/permission to
> commit
> > > >> something to them and time to support this broken modules.
> > > >>
> > > >> Also broken TF module blocks the removal of IGFS.
> > > >>
> > > >> Found CVE were related to the dependencies related to
> > hadoop/tf/parquet
> > > and
> > > >> so on.
> > > >>
> > > >> *Should we mark it with the @IgniteExperimental? *
> > > >> I don't know, we have no this RAW annotation a few weeks ago and I
> > don't
> > > >> know how we should use it.
> > > >> It could be, if you finish the discussion about this annotation and
> > the
> > > >> write docs about it and share it with me.
> > > >>
> > > >>
> > > >> * As far as I know, the ML module has no documentation. Is it
> correct?
> > > Dowe
> > > >> have plans to fix it?*
> > > >> The ML docs are here, on our Ignite documentation
> > > >> https://apacheignite.readme.io/docs/machine-learning
> > > >> of course, something could be wrong, 1.5 year we are not released
> > Ignite
> > > >> Yes, I have plan to fix. Of course, after fixing all bugs in release
> > > branch
> > > >>
> > > >> *Should we move it to the ignite-extensions?*
> > > >> No we shouldn't, I don't want this, and have a lot of arguments and
> > > >> currently is not the time for discussion about this (they are too
> > young
> > > and
> > > >> have now real infrastructure and release-cycle)
> > > >>
> > > >> P.S. Community, I understand that the removal of module looks
> strange,
> > > but
> > > >> we could understand that ML was a strange experiment without roadmap
> > and
> > > >> this situation is finished.
> > > >> Now, I have roadmap (will be published later), newbie tickets, the
> > > ability
> > > >> to prepare correct docs, the understanding what could be used by
> > clients
> > > >> and first of all, production-ready ML (it could be run on
> > > Ignite-cluster,
> > > >> really, it works.)
> > > >>
> > > >> If you, Igniters, believe, that I could be a good maintainer for ML
> > > module,
> > > >> please support me here in this thread
> > > >> If you think, that I do something wrong, OK, please write it too,
> I'll
> > > read
> > > >> carefully.
> > > >>
> > > >> I spent a few months to fix bugs in components, which were abandoned
> > by
> > > >> their creators.
> > > >>
> > > >> My goal: Ignite should have light-weight, easy-integrated ML without
> > > >> strange and unfinished experiments which could be not maintained.
> > It's a
> > > >> part of common movement in Ignite (removal of modules or moving them
> > to
> > > the
> > > >> separate repos).
> > > >>
> > > >>
> > > >> пн, 17 февр. 2020 г. в 12:10, Nikolay Izhikov <[hidden email]
> >:
> > > >>
> > > >>> Hello, Igniters.
> > > >>>
> > > >>> Can someone bring some light on the state of the ML module in
> Ignite?
> > > >>> As far as I know, for now, we have only 1 active contributor to
> this
> > > >> area -
> > > >>> Alexey Zinoviev.
> > > >>> I see how whole modules come and go from the module - [1]
> > > >>>
> > > >>> Please, also note this quote:
> > > >>>
> > > >>>> Also as a result of good testing from both side (from me and
> Stepan)
> > > we
> > > >>>> found a lot of bugs and CVEs in hadoop related components that
> > should
> > > >> be
> > > >>>> removed in release branch too.
> > > >>>
> > > >>> 0. As far as I know, the ML module has no documentation. Is it
> > correct?
> > > >> Do
> > > >>> we have plans to fix it?
> > > >>>
> > > >>> 1. Can someone related to the ML, please, give some examples of the
> > CVE
> > > >> or
> > > >>> issues that can be fixed only with removing a bunch of modules?
> > > >>>
> > > >>> 3. Is ML module production ready?
> > > >>>
> > > >>> 4. Should we mark it with the @IgniteExperimental?
> > > >>>
> > > >>> 5. Should we move it to the ignite-extensions?
> > > >>>
> > > >>> [1]
> > > >>>
> > > >>>
> > > >>
> > >
> >
> https://github.com/apache/ignite/commit/53e886b8ed38a6842cef8b44ace6851855dfad29
> > > >>>
> > > >>
> > >
> > >
> >
>

[dmagda]

Alexey, thanks for sharing details and your reasoning behind the taken
actions. It makes sense. I've updated the machine learning pages on the new
website that will be released in several days.

-
Denis


On Tue, Mar 24, 2020 at 11:07 AM Alexey Zinoviev <[hidden email]>
wrote:

> Hi, Denis!
>
> Be honest, the significant amount of the ML contirbutors left the community
> previous year in frustration with unfinished parts.
> In this situation, I reduced the unsed and broken parts according our
> previous discussions peer-to-peer (not on devlist, our mistake) to release
> the stable core of ML which could be supported with reduced power.
>
> The reasons for GA removal
> 1. It doesn't related to the ML topic
> 2. It has no intersection with the ML package (as you mentioned)
> 3. It doesn't support Ignite code and in many places Java codestyle
> 4. It was experimental package placed in ML in time of earliest experiments
> in 2017
> 5. Nobody doesn't want to support this for the years
>
> Genetic Algorithms could be moved to Ignite-extension (if somebody
> interested in it)
>
> A lot of things are changed since release 2.7
>
> Lessons are learnt, I will start discussion topics next time for the
> significant changes or removal in API, moreover, the next releases I hope
> to use new @IgniteExperimental (it was added too late) and another
> annotations for the release cycle.
>
> вт, 24 мар. 2020 г. в 20:00, Denis Magda <[hidden email]>:
>
> > Alexey,
> >
> > I missed this thread and only now realized that TensorFlow, genetic
> > algorithms and some other APIs were expelled from 2.8. I would encourage
> us
> > to start a dedicated discussion for any APIs removal or significant
> changes
> > to let other community members share their opinions or take appropriate
> > actions (like proper documentation redirects setup for pages that are
> gone
> > and updates on the website like [1] and [2]). For instance, I have no
> glue
> > that the topic of TensorFlow removal was briefly mentioned in this
> > discussion thread.
> >
> > I see the reasoning about TensorFlow but why have we removed generic
> > algorithms that had a dependency on the compute APIs only?
> >
> > [1] https://ignite.apache.org/features/tensorflow.html
> > [2] https://ignite.apache.org/features/machinelearning.html#ga-grid
> >
> > -
> > Denis
> >
> >
> > On Mon, Feb 17, 2020 at 5:39 AM Alexey Zinoviev <[hidden email]>
> > wrote:
> >
> > > Ok, agree, that I should start discussion before making changes, but I
> > was
> > > limited by release 2.8 and trying don;'t be a delayed person for that.
> > > During release I was focused on fixing bugs and don't tests TF and
> Ignite
> > > together
> > >
> > > I thought that as a maintainer of ML module I could do perform these
> > > actions.
> > >
> > > Below I will share my statement why it should be removed and why it
> > should
> > > be removed immediately
> > >
> > > About TensorFlow module (reason for removal)
> > >
> > >    1. This module is only one module that uses IGFS and needs in
> > FileSystem
> > >    on Ignte side due to TensorFlow API
> > >    2. This module a part of bridge between Ignite ML and Tensorflow and
> > its
> > >    broken after changes in TensorFlow on TensorFlow side
> > >    3. TensorFlow released new version without Ignite bridge, no chance
> to
> > >    run them together for new releases
> > >    4. This module wasn't complete and developer who did this, left the
> > >    community
> > >    5. The development skills for this story require python/C++/java
> > >    programming together
> > >    6. The module is a source of bugs which could be fixed for release
> 2.8
> > >    and possibly for future releases (nobody in community could this)
> > >    7. The release size reduced from 6 Gb to 4.5 due to removed
> > dependencies
> > >    8. TensorFlow now is not popular among Data Scientists, the PyTorch
> is
> > >    the most popular tool for Deep Learning (like NetBeans and IDEA)
> > >    9. Nobody uses that in production because it was developed between
> 2.7
> > >    and 2.8 (2.7 has only proof-of-concept)
> > >
> > > Nikolay, sorry for that, hope to share more information about the ML
> and
> > > discuss here the main changes before actions.
> > >
> > > пн, 17 февр. 2020 г. в 16:18, Nikolay Izhikov <[hidden email]>:
> > >
> > > > Hello, Alexey.
> > > >
> > > > > The main reason, the modules are not work proper way, were
> > > experimental,
> > > > > never released as a production-ready, support old, outdated
> version,
> > > the
> > > > > external frameworks, like Tensorflow, move integration with ignite
> to
> > > the
> > > > > special repos, they are not finished, the code there is broken and
> > > > couldn't
> > > > > be fixed, because and I have no power/C++ skills/permission to
> commit
> > > > > something to them and time to support this broken modules.
> > > >
> > > >
> > > > Do we have some tickets or wider explanation for it?
> > > > It very uncommon for me that the decision to remove modules from the
> > > > master and release is not discussed widely in the community.
> > > >
> > > > > 17 февр. 2020 г., в 14:39, Ravil Galeyev <[hidden email]>
> > > > написал(а):
> > > > >
> > > > > Hi Team,
> > > > >
> > > > > First of all, let me introduce myself. I’m Ravil, I contribute to
> the
> > > ML
> > > > > module since 2018 and from time to time I make talks about it.
> (I..e
> > > data
> > > > > science summit in Warsaw [1]).
> > > > >
> > > > > So, Alexey made a huge effort to develop the ML module but he is
> not
> > > > alone.
> > > > > If you check the repo you will find other contributors.
> > > > >
> > > > > Therefore the ML module is alive and is able to run and has the
> > > roadmap.
> > > > > For me, it means that it’s not a raw project.
> > > > >
> > > > > Regarding documentation, it’d like to mention the code is the best
> > > > > documentation :)
> > > > >
> > > > > We have examples for most algorithms [2]. But if it needed I’m
> ready
> > to
> > > > > help the community with documentation in English German Polish or
> > > > Russain.
> > > > >
> > > > >
> > > > > [1] https://dssconf.pl/
> > > > >
> > > > > [2]
> > > > >
> > > >
> > >
> >
> https://github.com/apache/ignite/tree/master/examples/src/main/java/org/apache/ignite/examples/ml
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Ravil
> > > > >
> > > > >
> > > > > On Mon, 17 Feb 2020 at 11:49, Alexey Zinoviev <
> > [hidden email]>
> > > > > wrote:
> > > > >
> > > > >> Hello, Igniters, and you, Nikolay.
> > > > >>
> > > > >> First of all, if you have real interest to the ML module and its
> > > state,
> > > > I
> > > > >> could make call with you and explain this.
> > > > >>
> > > > >>
> > > > >> *As far as I know, for now, we have only 1 active contributor to
> > this
> > > > area
> > > > >> -Alexey Zinoviev.*
> > > > >> Currently, we have 2 active contributors, me and Ravil Galeeyev, a
> > few
> > > > >> newbies, another guys who started tensorflow and another modules
> and
> > > > >> submodules don't visit the community for many months.
> > > > >>
> > > > >> *Is ML module production ready?*
> > > > >> This release will be the first release, than ML is production
> ready
> > > and
> > > > it
> > > > >> totally my work.
> > > > >>
> > > > >>
> > > > >> *Can someone related to the ML, please, give some examples of the
> > CVE
> > > > >> orissues that can be fixed only with removing a bunch of modules?*
> > > > >> CVE is not the main reason to remove the "bunch of modules", but
> > part
> > > of
> > > > >> the story.
> > > > >> The main reason, the modules are not work proper way, were
> > > experimental,
> > > > >> never released as a production-ready, support old, outdated
> version,
> > > the
> > > > >> external frameworks, like Tensorflow, move integration with ignite
> > to
> > > > the
> > > > >> special repos, they are not finished, the code there is broken and
> > > > couldn't
> > > > >> be fixed, because and I have no power/C++ skills/permission to
> > commit
> > > > >> something to them and time to support this broken modules.
> > > > >>
> > > > >> Also broken TF module blocks the removal of IGFS.
> > > > >>
> > > > >> Found CVE were related to the dependencies related to
> > > hadoop/tf/parquet
> > > > and
> > > > >> so on.
> > > > >>
> > > > >> *Should we mark it with the @IgniteExperimental? *
> > > > >> I don't know, we have no this RAW annotation a few weeks ago and I
> > > don't
> > > > >> know how we should use it.
> > > > >> It could be, if you finish the discussion about this annotation
> and
> > > the
> > > > >> write docs about it and share it with me.
> > > > >>
> > > > >>
> > > > >> * As far as I know, the ML module has no documentation. Is it
> > correct?
> > > > Dowe
> > > > >> have plans to fix it?*
> > > > >> The ML docs are here, on our Ignite documentation
> > > > >> https://apacheignite.readme.io/docs/machine-learning
> > > > >> of course, something could be wrong, 1.5 year we are not released
> > > Ignite
> > > > >> Yes, I have plan to fix. Of course, after fixing all bugs in
> release
> > > > branch
> > > > >>
> > > > >> *Should we move it to the ignite-extensions?*
> > > > >> No we shouldn't, I don't want this, and have a lot of arguments
> and
> > > > >> currently is not the time for discussion about this (they are too
> > > young
> > > > and
> > > > >> have now real infrastructure and release-cycle)
> > > > >>
> > > > >> P.S. Community, I understand that the removal of module looks
> > strange,
> > > > but
> > > > >> we could understand that ML was a strange experiment without
> roadmap
> > > and
> > > > >> this situation is finished.
> > > > >> Now, I have roadmap (will be published later), newbie tickets, the
> > > > ability
> > > > >> to prepare correct docs, the understanding what could be used by
> > > clients
> > > > >> and first of all, production-ready ML (it could be run on
> > > > Ignite-cluster,
> > > > >> really, it works.)
> > > > >>
> > > > >> If you, Igniters, believe, that I could be a good maintainer for
> ML
> > > > module,
> > > > >> please support me here in this thread
> > > > >> If you think, that I do something wrong, OK, please write it too,
> > I'll
> > > > read
> > > > >> carefully.
> > > > >>
> > > > >> I spent a few months to fix bugs in components, which were
> abandoned
> > > by
> > > > >> their creators.
> > > > >>
> > > > >> My goal: Ignite should have light-weight, easy-integrated ML
> without
> > > > >> strange and unfinished experiments which could be not maintained.
> > > It's a
> > > > >> part of common movement in Ignite (removal of modules or moving
> them
> > > to
> > > > the
> > > > >> separate repos).
> > > > >>
> > > > >>
> > > > >> пн, 17 февр. 2020 г. в 12:10, Nikolay Izhikov <
> [hidden email]
> > >:
> > > > >>
> > > > >>> Hello, Igniters.
> > > > >>>
> > > > >>> Can someone bring some light on the state of the ML module in
> > Ignite?
> > > > >>> As far as I know, for now, we have only 1 active contributor to
> > this
> > > > >> area -
> > > > >>> Alexey Zinoviev.
> > > > >>> I see how whole modules come and go from the module - [1]
> > > > >>>
> > > > >>> Please, also note this quote:
> > > > >>>
> > > > >>>> Also as a result of good testing from both side (from me and
> > Stepan)
> > > > we
> > > > >>>> found a lot of bugs and CVEs in hadoop related components that
> > > should
> > > > >> be
> > > > >>>> removed in release branch too.
> > > > >>>
> > > > >>> 0. As far as I know, the ML module has no documentation. Is it
> > > correct?
> > > > >> Do
> > > > >>> we have plans to fix it?
> > > > >>>
> > > > >>> 1. Can someone related to the ML, please, give some examples of
> the
> > > CVE
> > > > >> or
> > > > >>> issues that can be fixed only with removing a bunch of modules?
> > > > >>>
> > > > >>> 3. Is ML module production ready?
> > > > >>>
> > > > >>> 4. Should we mark it with the @IgniteExperimental?
> > > > >>>
> > > > >>> 5. Should we move it to the ignite-extensions?
> > > > >>>
> > > > >>> [1]
> > > > >>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/ignite/commit/53e886b8ed38a6842cef8b44ace6851855dfad29
> > > > >>>
> > > > >>
> > > >
> > > >
> > >
> >
>

[Maxim Muzafarov]

Folks,

It seems to me we've forgotten to remove -Ptensorflow profile from the
~Build Apache Ignite~ suite [1] on the project build.
I've removed it. Please, revert it back if I'm missing something
during this discussion.

[1] https://ci.ignite.apache.org/admin/editBuildRunners.html?id=buildType:IgniteTests24Java8_BuildApacheIgnite

On Wed, 25 Mar 2020 at 02:21, Denis Magda <[hidden email]> wrote:

>
> Alexey, thanks for sharing details and your reasoning behind the taken
> actions. It makes sense. I've updated the machine learning pages on the new
> website that will be released in several days.
>
> -
> Denis
>
>
> On Tue, Mar 24, 2020 at 11:07 AM Alexey Zinoviev <[hidden email]>
> wrote:
>
> > Hi, Denis!
> >
> > Be honest, the significant amount of the ML contirbutors left the community
> > previous year in frustration with unfinished parts.
> > In this situation, I reduced the unsed and broken parts according our
> > previous discussions peer-to-peer (not on devlist, our mistake) to release
> > the stable core of ML which could be supported with reduced power.
> >
> > The reasons for GA removal
> > 1. It doesn't related to the ML topic
> > 2. It has no intersection with the ML package (as you mentioned)
> > 3. It doesn't support Ignite code and in many places Java codestyle
> > 4. It was experimental package placed in ML in time of earliest experiments
> > in 2017
> > 5. Nobody doesn't want to support this for the years
> >
> > Genetic Algorithms could be moved to Ignite-extension (if somebody
> > interested in it)
> >
> > A lot of things are changed since release 2.7
> >
> > Lessons are learnt, I will start discussion topics next time for the
> > significant changes or removal in API, moreover, the next releases I hope
> > to use new @IgniteExperimental (it was added too late) and another
> > annotations for the release cycle.
> >
> > вт, 24 мар. 2020 г. в 20:00, Denis Magda <[hidden email]>:
> >
> > > Alexey,
> > >
> > > I missed this thread and only now realized that TensorFlow, genetic
> > > algorithms and some other APIs were expelled from 2.8. I would encourage
> > us
> > > to start a dedicated discussion for any APIs removal or significant
> > changes
> > > to let other community members share their opinions or take appropriate
> > > actions (like proper documentation redirects setup for pages that are
> > gone
> > > and updates on the website like [1] and [2]). For instance, I have no
> > glue
> > > that the topic of TensorFlow removal was briefly mentioned in this
> > > discussion thread.
> > >
> > > I see the reasoning about TensorFlow but why have we removed generic
> > > algorithms that had a dependency on the compute APIs only?
> > >
> > > [1] https://ignite.apache.org/features/tensorflow.html
> > > [2] https://ignite.apache.org/features/machinelearning.html#ga-grid
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Mon, Feb 17, 2020 at 5:39 AM Alexey Zinoviev <[hidden email]>
> > > wrote:
> > >
> > > > Ok, agree, that I should start discussion before making changes, but I
> > > was
> > > > limited by release 2.8 and trying don;'t be a delayed person for that.
> > > > During release I was focused on fixing bugs and don't tests TF and
> > Ignite
> > > > together
> > > >
> > > > I thought that as a maintainer of ML module I could do perform these
> > > > actions.
> > > >
> > > > Below I will share my statement why it should be removed and why it
> > > should
> > > > be removed immediately
> > > >
> > > > About TensorFlow module (reason for removal)
> > > >
> > > >    1. This module is only one module that uses IGFS and needs in
> > > FileSystem
> > > >    on Ignte side due to TensorFlow API
> > > >    2. This module a part of bridge between Ignite ML and Tensorflow and
> > > its
> > > >    broken after changes in TensorFlow on TensorFlow side
> > > >    3. TensorFlow released new version without Ignite bridge, no chance
> > to
> > > >    run them together for new releases
> > > >    4. This module wasn't complete and developer who did this, left the
> > > >    community
> > > >    5. The development skills for this story require python/C++/java
> > > >    programming together
> > > >    6. The module is a source of bugs which could be fixed for release
> > 2.8
> > > >    and possibly for future releases (nobody in community could this)
> > > >    7. The release size reduced from 6 Gb to 4.5 due to removed
> > > dependencies
> > > >    8. TensorFlow now is not popular among Data Scientists, the PyTorch
> > is
> > > >    the most popular tool for Deep Learning (like NetBeans and IDEA)
> > > >    9. Nobody uses that in production because it was developed between
> > 2.7
> > > >    and 2.8 (2.7 has only proof-of-concept)
> > > >
> > > > Nikolay, sorry for that, hope to share more information about the ML
> > and
> > > > discuss here the main changes before actions.
> > > >
> > > > пн, 17 февр. 2020 г. в 16:18, Nikolay Izhikov <[hidden email]>:
> > > >
> > > > > Hello, Alexey.
> > > > >
> > > > > > The main reason, the modules are not work proper way, were
> > > > experimental,
> > > > > > never released as a production-ready, support old, outdated
> > version,
> > > > the
> > > > > > external frameworks, like Tensorflow, move integration with ignite
> > to
> > > > the
> > > > > > special repos, they are not finished, the code there is broken and
> > > > > couldn't
> > > > > > be fixed, because and I have no power/C++ skills/permission to
> > commit
> > > > > > something to them and time to support this broken modules.
> > > > >
> > > > >
> > > > > Do we have some tickets or wider explanation for it?
> > > > > It very uncommon for me that the decision to remove modules from the
> > > > > master and release is not discussed widely in the community.
> > > > >
> > > > > > 17 февр. 2020 г., в 14:39, Ravil Galeyev <[hidden email]>
> > > > > написал(а):
> > > > > >
> > > > > > Hi Team,
> > > > > >
> > > > > > First of all, let me introduce myself. I’m Ravil, I contribute to
> > the
> > > > ML
> > > > > > module since 2018 and from time to time I make talks about it.
> > (I..e
> > > > data
> > > > > > science summit in Warsaw [1]).
> > > > > >
> > > > > > So, Alexey made a huge effort to develop the ML module but he is
> > not
> > > > > alone.
> > > > > > If you check the repo you will find other contributors.
> > > > > >
> > > > > > Therefore the ML module is alive and is able to run and has the
> > > > roadmap.
> > > > > > For me, it means that it’s not a raw project.
> > > > > >
> > > > > > Regarding documentation, it’d like to mention the code is the best
> > > > > > documentation :)
> > > > > >
> > > > > > We have examples for most algorithms [2]. But if it needed I’m
> > ready
> > > to
> > > > > > help the community with documentation in English German Polish or
> > > > > Russain.
> > > > > >
> > > > > >
> > > > > > [1] https://dssconf.pl/
> > > > > >
> > > > > > [2]
> > > > > >
> > > > >
> > > >
> > >
> > https://github.com/apache/ignite/tree/master/examples/src/main/java/org/apache/ignite/examples/ml
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Ravil
> > > > > >
> > > > > >
> > > > > > On Mon, 17 Feb 2020 at 11:49, Alexey Zinoviev <
> > > [hidden email]>
> > > > > > wrote:
> > > > > >
> > > > > >> Hello, Igniters, and you, Nikolay.
> > > > > >>
> > > > > >> First of all, if you have real interest to the ML module and its
> > > > state,
> > > > > I
> > > > > >> could make call with you and explain this.
> > > > > >>
> > > > > >>
> > > > > >> *As far as I know, for now, we have only 1 active contributor to
> > > this
> > > > > area
> > > > > >> -Alexey Zinoviev.*
> > > > > >> Currently, we have 2 active contributors, me and Ravil Galeeyev, a
> > > few
> > > > > >> newbies, another guys who started tensorflow and another modules
> > and
> > > > > >> submodules don't visit the community for many months.
> > > > > >>
> > > > > >> *Is ML module production ready?*
> > > > > >> This release will be the first release, than ML is production
> > ready
> > > > and
> > > > > it
> > > > > >> totally my work.
> > > > > >>
> > > > > >>
> > > > > >> *Can someone related to the ML, please, give some examples of the
> > > CVE
> > > > > >> orissues that can be fixed only with removing a bunch of modules?*
> > > > > >> CVE is not the main reason to remove the "bunch of modules", but
> > > part
> > > > of
> > > > > >> the story.
> > > > > >> The main reason, the modules are not work proper way, were
> > > > experimental,
> > > > > >> never released as a production-ready, support old, outdated
> > version,
> > > > the
> > > > > >> external frameworks, like Tensorflow, move integration with ignite
> > > to
> > > > > the
> > > > > >> special repos, they are not finished, the code there is broken and
> > > > > couldn't
> > > > > >> be fixed, because and I have no power/C++ skills/permission to
> > > commit
> > > > > >> something to them and time to support this broken modules.
> > > > > >>
> > > > > >> Also broken TF module blocks the removal of IGFS.
> > > > > >>
> > > > > >> Found CVE were related to the dependencies related to
> > > > hadoop/tf/parquet
> > > > > and
> > > > > >> so on.
> > > > > >>
> > > > > >> *Should we mark it with the @IgniteExperimental? *
> > > > > >> I don't know, we have no this RAW annotation a few weeks ago and I
> > > > don't
> > > > > >> know how we should use it.
> > > > > >> It could be, if you finish the discussion about this annotation
> > and
> > > > the
> > > > > >> write docs about it and share it with me.
> > > > > >>
> > > > > >>
> > > > > >> * As far as I know, the ML module has no documentation. Is it
> > > correct?
> > > > > Dowe
> > > > > >> have plans to fix it?*
> > > > > >> The ML docs are here, on our Ignite documentation
> > > > > >> https://apacheignite.readme.io/docs/machine-learning
> > > > > >> of course, something could be wrong, 1.5 year we are not released
> > > > Ignite
> > > > > >> Yes, I have plan to fix. Of course, after fixing all bugs in
> > release
> > > > > branch
> > > > > >>
> > > > > >> *Should we move it to the ignite-extensions?*
> > > > > >> No we shouldn't, I don't want this, and have a lot of arguments
> > and
> > > > > >> currently is not the time for discussion about this (they are too
> > > > young
> > > > > and
> > > > > >> have now real infrastructure and release-cycle)
> > > > > >>
> > > > > >> P.S. Community, I understand that the removal of module looks
> > > strange,
> > > > > but
> > > > > >> we could understand that ML was a strange experiment without
> > roadmap
> > > > and
> > > > > >> this situation is finished.
> > > > > >> Now, I have roadmap (will be published later), newbie tickets, the
> > > > > ability
> > > > > >> to prepare correct docs, the understanding what could be used by
> > > > clients
> > > > > >> and first of all, production-ready ML (it could be run on
> > > > > Ignite-cluster,
> > > > > >> really, it works.)
> > > > > >>
> > > > > >> If you, Igniters, believe, that I could be a good maintainer for
> > ML
> > > > > module,
> > > > > >> please support me here in this thread
> > > > > >> If you think, that I do something wrong, OK, please write it too,
> > > I'll
> > > > > read
> > > > > >> carefully.
> > > > > >>
> > > > > >> I spent a few months to fix bugs in components, which were
> > abandoned
> > > > by
> > > > > >> their creators.
> > > > > >>
> > > > > >> My goal: Ignite should have light-weight, easy-integrated ML
> > without
> > > > > >> strange and unfinished experiments which could be not maintained.
> > > > It's a
> > > > > >> part of common movement in Ignite (removal of modules or moving
> > them
> > > > to
> > > > > the
> > > > > >> separate repos).
> > > > > >>
> > > > > >>
> > > > > >> пн, 17 февр. 2020 г. в 12:10, Nikolay Izhikov <
> > [hidden email]
> > > >:
> > > > > >>
> > > > > >>> Hello, Igniters.
> > > > > >>>
> > > > > >>> Can someone bring some light on the state of the ML module in
> > > Ignite?
> > > > > >>> As far as I know, for now, we have only 1 active contributor to
> > > this
> > > > > >> area -
> > > > > >>> Alexey Zinoviev.
> > > > > >>> I see how whole modules come and go from the module - [1]
> > > > > >>>
> > > > > >>> Please, also note this quote:
> > > > > >>>
> > > > > >>>> Also as a result of good testing from both side (from me and
> > > Stepan)
> > > > > we
> > > > > >>>> found a lot of bugs and CVEs in hadoop related components that
> > > > should
> > > > > >> be
> > > > > >>>> removed in release branch too.
> > > > > >>>
> > > > > >>> 0. As far as I know, the ML module has no documentation. Is it
> > > > correct?
> > > > > >> Do
> > > > > >>> we have plans to fix it?
> > > > > >>>
> > > > > >>> 1. Can someone related to the ML, please, give some examples of
> > the
> > > > CVE
> > > > > >> or
> > > > > >>> issues that can be fixed only with removing a bunch of modules?
> > > > > >>>
> > > > > >>> 3. Is ML module production ready?
> > > > > >>>
> > > > > >>> 4. Should we mark it with the @IgniteExperimental?
> > > > > >>>
> > > > > >>> 5. Should we move it to the ignite-extensions?
> > > > > >>>
> > > > > >>> [1]
> > > > > >>>
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> > https://github.com/apache/ignite/commit/53e886b8ed38a6842cef8b44ace6851855dfad29
> > > > > >>>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >

[Alexey Zinoviev]

No, that is right!

вс, 12 апр. 2020 г., 5:14 Maxim Muzafarov <[hidden email]>:

> Folks,
>
> It seems to me we've forgotten to remove -Ptensorflow profile from the
> ~Build Apache Ignite~ suite [1] on the project build.
> I've removed it. Please, revert it back if I'm missing something
> during this discussion.
>
> [1]
> https://ci.ignite.apache.org/admin/editBuildRunners.html?id=buildType:IgniteTests24Java8_BuildApacheIgnite
>
> On Wed, 25 Mar 2020 at 02:21, Denis Magda <[hidden email]> wrote:
> >
> > Alexey, thanks for sharing details and your reasoning behind the taken
> > actions. It makes sense. I've updated the machine learning pages on the
> new
> > website that will be released in several days.
> >
> > -
> > Denis
> >
> >
> > On Tue, Mar 24, 2020 at 11:07 AM Alexey Zinoviev <[hidden email]
> >
> > wrote:
> >
> > > Hi, Denis!
> > >
> > > Be honest, the significant amount of the ML contirbutors left the
> community
> > > previous year in frustration with unfinished parts.
> > > In this situation, I reduced the unsed and broken parts according our
> > > previous discussions peer-to-peer (not on devlist, our mistake) to
> release
> > > the stable core of ML which could be supported with reduced power.
> > >
> > > The reasons for GA removal
> > > 1. It doesn't related to the ML topic
> > > 2. It has no intersection with the ML package (as you mentioned)
> > > 3. It doesn't support Ignite code and in many places Java codestyle
> > > 4. It was experimental package placed in ML in time of earliest
> experiments
> > > in 2017
> > > 5. Nobody doesn't want to support this for the years
> > >
> > > Genetic Algorithms could be moved to Ignite-extension (if somebody
> > > interested in it)
> > >
> > > A lot of things are changed since release 2.7
> > >
> > > Lessons are learnt, I will start discussion topics next time for the
> > > significant changes or removal in API, moreover, the next releases I
> hope
> > > to use new @IgniteExperimental (it was added too late) and another
> > > annotations for the release cycle.
> > >
> > > вт, 24 мар. 2020 г. в 20:00, Denis Magda <[hidden email]>:
> > >
> > > > Alexey,
> > > >
> > > > I missed this thread and only now realized that TensorFlow, genetic
> > > > algorithms and some other APIs were expelled from 2.8. I would
> encourage
> > > us
> > > > to start a dedicated discussion for any APIs removal or significant
> > > changes
> > > > to let other community members share their opinions or take
> appropriate
> > > > actions (like proper documentation redirects setup for pages that are
> > > gone
> > > > and updates on the website like [1] and [2]). For instance, I have no
> > > glue
> > > > that the topic of TensorFlow removal was briefly mentioned in this
> > > > discussion thread.
> > > >
> > > > I see the reasoning about TensorFlow but why have we removed generic
> > > > algorithms that had a dependency on the compute APIs only?
> > > >
> > > > [1] https://ignite.apache.org/features/tensorflow.html
> > > > [2] https://ignite.apache.org/features/machinelearning.html#ga-grid
> > > >
> > > > -
> > > > Denis
> > > >
> > > >
> > > > On Mon, Feb 17, 2020 at 5:39 AM Alexey Zinoviev <
> [hidden email]>
> > > > wrote:
> > > >
> > > > > Ok, agree, that I should start discussion before making changes,
> but I
> > > > was
> > > > > limited by release 2.8 and trying don;'t be a delayed person for
> that.
> > > > > During release I was focused on fixing bugs and don't tests TF and
> > > Ignite
> > > > > together
> > > > >
> > > > > I thought that as a maintainer of ML module I could do perform
> these
> > > > > actions.
> > > > >
> > > > > Below I will share my statement why it should be removed and why it
> > > > should
> > > > > be removed immediately
> > > > >
> > > > > About TensorFlow module (reason for removal)
> > > > >
> > > > >    1. This module is only one module that uses IGFS and needs in
> > > > FileSystem
> > > > >    on Ignte side due to TensorFlow API
> > > > >    2. This module a part of bridge between Ignite ML and
> Tensorflow and
> > > > its
> > > > >    broken after changes in TensorFlow on TensorFlow side
> > > > >    3. TensorFlow released new version without Ignite bridge, no
> chance
> > > to
> > > > >    run them together for new releases
> > > > >    4. This module wasn't complete and developer who did this, left
> the
> > > > >    community
> > > > >    5. The development skills for this story require python/C++/java
> > > > >    programming together
> > > > >    6. The module is a source of bugs which could be fixed for
> release
> > > 2.8
> > > > >    and possibly for future releases (nobody in community could
> this)
> > > > >    7. The release size reduced from 6 Gb to 4.5 due to removed
> > > > dependencies
> > > > >    8. TensorFlow now is not popular among Data Scientists, the
> PyTorch
> > > is
> > > > >    the most popular tool for Deep Learning (like NetBeans and IDEA)
> > > > >    9. Nobody uses that in production because it was developed
> between
> > > 2.7
> > > > >    and 2.8 (2.7 has only proof-of-concept)
> > > > >
> > > > > Nikolay, sorry for that, hope to share more information about the
> ML
> > > and
> > > > > discuss here the main changes before actions.
> > > > >
> > > > > пн, 17 февр. 2020 г. в 16:18, Nikolay Izhikov <[hidden email]
> >:
> > > > >
> > > > > > Hello, Alexey.
> > > > > >
> > > > > > > The main reason, the modules are not work proper way, were
> > > > > experimental,
> > > > > > > never released as a production-ready, support old, outdated
> > > version,
> > > > > the
> > > > > > > external frameworks, like Tensorflow, move integration with
> ignite
> > > to
> > > > > the
> > > > > > > special repos, they are not finished, the code there is broken
> and
> > > > > > couldn't
> > > > > > > be fixed, because and I have no power/C++ skills/permission to
> > > commit
> > > > > > > something to them and time to support this broken modules.
> > > > > >
> > > > > >
> > > > > > Do we have some tickets or wider explanation for it?
> > > > > > It very uncommon for me that the decision to remove modules from
> the
> > > > > > master and release is not discussed widely in the community.
> > > > > >
> > > > > > > 17 февр. 2020 г., в 14:39, Ravil Galeyev <[hidden email]>
> > > > > > написал(а):
> > > > > > >
> > > > > > > Hi Team,
> > > > > > >
> > > > > > > First of all, let me introduce myself. I’m Ravil, I contribute
> to
> > > the
> > > > > ML
> > > > > > > module since 2018 and from time to time I make talks about it.
> > > (I..e
> > > > > data
> > > > > > > science summit in Warsaw [1]).
> > > > > > >
> > > > > > > So, Alexey made a huge effort to develop the ML module but he
> is
> > > not
> > > > > > alone.
> > > > > > > If you check the repo you will find other contributors.
> > > > > > >
> > > > > > > Therefore the ML module is alive and is able to run and has the
> > > > > roadmap.
> > > > > > > For me, it means that it’s not a raw project.
> > > > > > >
> > > > > > > Regarding documentation, it’d like to mention the code is the
> best
> > > > > > > documentation :)
> > > > > > >
> > > > > > > We have examples for most algorithms [2]. But if it needed I’m
> > > ready
> > > > to
> > > > > > > help the community with documentation in English German Polish
> or
> > > > > > Russain.
> > > > > > >
> > > > > > >
> > > > > > > [1] https://dssconf.pl/
> > > > > > >
> > > > > > > [2]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://github.com/apache/ignite/tree/master/examples/src/main/java/org/apache/ignite/examples/ml
> > > > > > >
> > > > > > > Best regards,
> > > > > > >
> > > > > > > Ravil
> > > > > > >
> > > > > > >
> > > > > > > On Mon, 17 Feb 2020 at 11:49, Alexey Zinoviev <
> > > > [hidden email]>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hello, Igniters, and you, Nikolay.
> > > > > > >>
> > > > > > >> First of all, if you have real interest to the ML module and
> its
> > > > > state,
> > > > > > I
> > > > > > >> could make call with you and explain this.
> > > > > > >>
> > > > > > >>
> > > > > > >> *As far as I know, for now, we have only 1 active contributor
> to
> > > > this
> > > > > > area
> > > > > > >> -Alexey Zinoviev.*
> > > > > > >> Currently, we have 2 active contributors, me and Ravil
> Galeeyev, a
> > > > few
> > > > > > >> newbies, another guys who started tensorflow and another
> modules
> > > and
> > > > > > >> submodules don't visit the community for many months.
> > > > > > >>
> > > > > > >> *Is ML module production ready?*
> > > > > > >> This release will be the first release, than ML is production
> > > ready
> > > > > and
> > > > > > it
> > > > > > >> totally my work.
> > > > > > >>
> > > > > > >>
> > > > > > >> *Can someone related to the ML, please, give some examples of
> the
> > > > CVE
> > > > > > >> orissues that can be fixed only with removing a bunch of
> modules?*
> > > > > > >> CVE is not the main reason to remove the "bunch of modules",
> but
> > > > part
> > > > > of
> > > > > > >> the story.
> > > > > > >> The main reason, the modules are not work proper way, were
> > > > > experimental,
> > > > > > >> never released as a production-ready, support old, outdated
> > > version,
> > > > > the
> > > > > > >> external frameworks, like Tensorflow, move integration with
> ignite
> > > > to
> > > > > > the
> > > > > > >> special repos, they are not finished, the code there is
> broken and
> > > > > > couldn't
> > > > > > >> be fixed, because and I have no power/C++ skills/permission to
> > > > commit
> > > > > > >> something to them and time to support this broken modules.
> > > > > > >>
> > > > > > >> Also broken TF module blocks the removal of IGFS.
> > > > > > >>
> > > > > > >> Found CVE were related to the dependencies related to
> > > > > hadoop/tf/parquet
> > > > > > and
> > > > > > >> so on.
> > > > > > >>
> > > > > > >> *Should we mark it with the @IgniteExperimental? *
> > > > > > >> I don't know, we have no this RAW annotation a few weeks ago
> and I
> > > > > don't
> > > > > > >> know how we should use it.
> > > > > > >> It could be, if you finish the discussion about this
> annotation
> > > and
> > > > > the
> > > > > > >> write docs about it and share it with me.
> > > > > > >>
> > > > > > >>
> > > > > > >> * As far as I know, the ML module has no documentation. Is it
> > > > correct?
> > > > > > Dowe
> > > > > > >> have plans to fix it?*
> > > > > > >> The ML docs are here, on our Ignite documentation
> > > > > > >> https://apacheignite.readme.io/docs/machine-learning
> > > > > > >> of course, something could be wrong, 1.5 year we are not
> released
> > > > > Ignite
> > > > > > >> Yes, I have plan to fix. Of course, after fixing all bugs in
> > > release
> > > > > > branch
> > > > > > >>
> > > > > > >> *Should we move it to the ignite-extensions?*
> > > > > > >> No we shouldn't, I don't want this, and have a lot of
> arguments
> > > and
> > > > > > >> currently is not the time for discussion about this (they are
> too
> > > > > young
> > > > > > and
> > > > > > >> have now real infrastructure and release-cycle)
> > > > > > >>
> > > > > > >> P.S. Community, I understand that the removal of module looks
> > > > strange,
> > > > > > but
> > > > > > >> we could understand that ML was a strange experiment without
> > > roadmap
> > > > > and
> > > > > > >> this situation is finished.
> > > > > > >> Now, I have roadmap (will be published later), newbie
> tickets, the
> > > > > > ability
> > > > > > >> to prepare correct docs, the understanding what could be used
> by
> > > > > clients
> > > > > > >> and first of all, production-ready ML (it could be run on
> > > > > > Ignite-cluster,
> > > > > > >> really, it works.)
> > > > > > >>
> > > > > > >> If you, Igniters, believe, that I could be a good maintainer
> for
> > > ML
> > > > > > module,
> > > > > > >> please support me here in this thread
> > > > > > >> If you think, that I do something wrong, OK, please write it
> too,
> > > > I'll
> > > > > > read
> > > > > > >> carefully.
> > > > > > >>
> > > > > > >> I spent a few months to fix bugs in components, which were
> > > abandoned
> > > > > by
> > > > > > >> their creators.
> > > > > > >>
> > > > > > >> My goal: Ignite should have light-weight, easy-integrated ML
> > > without
> > > > > > >> strange and unfinished experiments which could be not
> maintained.
> > > > > It's a
> > > > > > >> part of common movement in Ignite (removal of modules or
> moving
> > > them
> > > > > to
> > > > > > the
> > > > > > >> separate repos).
> > > > > > >>
> > > > > > >>
> > > > > > >> пн, 17 февр. 2020 г. в 12:10, Nikolay Izhikov <
> > > [hidden email]
> > > > >:
> > > > > > >>
> > > > > > >>> Hello, Igniters.
> > > > > > >>>
> > > > > > >>> Can someone bring some light on the state of the ML module in
> > > > Ignite?
> > > > > > >>> As far as I know, for now, we have only 1 active contributor
> to
> > > > this
> > > > > > >> area -
> > > > > > >>> Alexey Zinoviev.
> > > > > > >>> I see how whole modules come and go from the module - [1]
> > > > > > >>>
> > > > > > >>> Please, also note this quote:
> > > > > > >>>
> > > > > > >>>> Also as a result of good testing from both side (from me and
> > > > Stepan)
> > > > > > we
> > > > > > >>>> found a lot of bugs and CVEs in hadoop related components
> that
> > > > > should
> > > > > > >> be
> > > > > > >>>> removed in release branch too.
> > > > > > >>>
> > > > > > >>> 0. As far as I know, the ML module has no documentation. Is
> it
> > > > > correct?
> > > > > > >> Do
> > > > > > >>> we have plans to fix it?
> > > > > > >>>
> > > > > > >>> 1. Can someone related to the ML, please, give some examples
> of
> > > the
> > > > > CVE
> > > > > > >> or
> > > > > > >>> issues that can be fixed only with removing a bunch of
> modules?
> > > > > > >>>
> > > > > > >>> 3. Is ML module production ready?
> > > > > > >>>
> > > > > > >>> 4. Should we mark it with the @IgniteExperimental?
> > > > > > >>>
> > > > > > >>> 5. Should we move it to the ignite-extensions?
> > > > > > >>>
> > > > > > >>> [1]
> > > > > > >>>
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> https://github.com/apache/ignite/commit/53e886b8ed38a6842cef8b44ace6851855dfad29
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
>