[DISCUSS] Pub/Sub Streamer Implementation

[Emmanouil Gkatziouras]

Hello everyone,

I started using Ignite lately. Part of my work involves Pub/Sub, thus I
created a pub/sub streamer.
Pub/Sub is pretty close to Kafka.
Here's how the streamer works. Every node creates an instance of a Pub/Sub
client.
The clients subscribe to the same topic using the same subscription name.
A message is being sent and one of the nodes will receive the message,
process it and then forward it to Ignite using the extractor provided.

Any thoughts?

*Emmanouil Gkatziouras*
https://egkatzioura.com/ | https://www.linkedin.com/in/gkatziourasemmanouil/
https://github.com/gkatzioura

[dmagda]

Hello Emmanouil,

Are you referring to any specific implementation tailored for a cloud? If
I'm not mistaken you were mentioning AWS/GCE. Is there any Github source
code?

Here is a list of existing integrations with streaming and pub/sub
frameworks:
https://apacheignite-mix.readme.io/docs/overview

with the source code in Ignite master:
https://github.com/apache/ignite/tree/master/modules

Check out those resources to see what are the most suitable Ignite APIs for
such integrations.

@Saikat Maitra <[hidden email]> , @Roman <[hidden email]> , as
streaming technologies contributors/maintainers, could you please join this
thread?

-
Denis


On Thu, Oct 10, 2019 at 7:58 AM Emmanouil Gkatziouras <[hidden email]>
wrote:

> Hello everyone,
>
> I started using Ignite lately. Part of my work involves Pub/Sub, thus I
> created a pub/sub streamer.
> Pub/Sub is pretty close to Kafka.
> Here's how the streamer works. Every node creates an instance of a Pub/Sub
> client.
> The clients subscribe to the same topic using the same subscription name.
> A message is being sent and one of the nodes will receive the message,
> process it and then forward it to Ignite using the extractor provided.
>
> Any thoughts?
>
> *Emmanouil Gkatziouras*
> https://egkatzioura.com/ |
> https://www.linkedin.com/in/gkatziourasemmanouil/
> https://github.com/gkatzioura
>

[Emmanouil Gkatziouras]

Hi all,

By mentioning Pub/Sub I mean Google Cloud's Pub/Sub [1]
Yes it is indeed cloud specific. It is the messaging service provided by
GCP.
I have made the integration with it, but I haven't published it yet (I am
waiting for the discussion/ticket and proceed with the contribution).

Kind regards,
Emmanouil

[1] https://cloud.google.com/pubsub/docs/overview

*Emmanouil Gkatziouras*
https://egkatzioura.com/ | https://www.linkedin.com/in/gkatziourasemmanouil/
https://github.com/gkatzioura


On Fri, 11 Oct 2019 at 21:45, Denis Magda <[hidden email]> wrote:

> Hello Emmanouil,
>
> Are you referring to any specific implementation tailored for a cloud? If
> I'm not mistaken you were mentioning AWS/GCE. Is there any Github source
> code?
>
> Here is a list of existing integrations with streaming and pub/sub
> frameworks:
> https://apacheignite-mix.readme.io/docs/overview
>
> with the source code in Ignite master:
> https://github.com/apache/ignite/tree/master/modules
>
> Check out those resources to see what are the most suitable Ignite APIs for
> such integrations.
>
> @Saikat Maitra <[hidden email]> , @Roman <[hidden email]> , as
> streaming technologies contributors/maintainers, could you please join this
> thread?
>
> -
> Denis
>
>
> On Thu, Oct 10, 2019 at 7:58 AM Emmanouil Gkatziouras <
> [hidden email]>
> wrote:
>
> > Hello everyone,
> >
> > I started using Ignite lately. Part of my work involves Pub/Sub, thus I
> > created a pub/sub streamer.
> > Pub/Sub is pretty close to Kafka.
> > Here's how the streamer works. Every node creates an instance of a
> Pub/Sub
> > client.
> > The clients subscribe to the same topic using the same subscription name.
> > A message is being sent and one of the nodes will receive the message,
> > process it and then forward it to Ignite using the extractor provided.
> >
> > Any thoughts?
> >
> > *Emmanouil Gkatziouras*
> > https://egkatzioura.com/ |
> > https://www.linkedin.com/in/gkatziourasemmanouil/
> > https://github.com/gkatzioura
> >
>

[Saikat Maitra]

Hello Emmanouil,

Can you please share more info about the implementation?

We have similar implementation for data streamers with Kafka and RocketMQ

https://apacheignite-mix.readme.io/docs/kafka-streamer

https://apacheignite-mix.readme.io/docs/rocketmq-streamer

Please review and share if you have any questions.

Regards,
Saikat



On Fri, Oct 11, 2019 at 4:56 PM Emmanouil Gkatziouras <[hidden email]>
wrote:

> Hi all,
>
> By mentioning Pub/Sub I mean Google Cloud's Pub/Sub [1]
> Yes it is indeed cloud specific. It is the messaging service provided by
> GCP.
> I have made the integration with it, but I haven't published it yet (I am
> waiting for the discussion/ticket and proceed with the contribution).
>
> Kind regards,
> Emmanouil
>
> [1] https://cloud.google.com/pubsub/docs/overview
>
> *Emmanouil Gkatziouras*
> https://egkatzioura.com/ |
> https://www.linkedin.com/in/gkatziourasemmanouil/
> https://github.com/gkatzioura
>
>
> On Fri, 11 Oct 2019 at 21:45, Denis Magda <[hidden email]> wrote:
>
> > Hello Emmanouil,
> >
> > Are you referring to any specific implementation tailored for a cloud? If
> > I'm not mistaken you were mentioning AWS/GCE. Is there any Github source
> > code?
> >
> > Here is a list of existing integrations with streaming and pub/sub
> > frameworks:
> > https://apacheignite-mix.readme.io/docs/overview
> >
> > with the source code in Ignite master:
> > https://github.com/apache/ignite/tree/master/modules
> >
> > Check out those resources to see what are the most suitable Ignite APIs
> for
> > such integrations.
> >
> > @Saikat Maitra <[hidden email]> , @Roman <[hidden email]> ,
> as
> > streaming technologies contributors/maintainers, could you please join
> this
> > thread?
> >
> > -
> > Denis
> >
> >
> > On Thu, Oct 10, 2019 at 7:58 AM Emmanouil Gkatziouras <
> > [hidden email]>
> > wrote:
> >
> > > Hello everyone,
> > >
> > > I started using Ignite lately. Part of my work involves Pub/Sub, thus I
> > > created a pub/sub streamer.
> > > Pub/Sub is pretty close to Kafka.
> > > Here's how the streamer works. Every node creates an instance of a
> > Pub/Sub
> > > client.
> > > The clients subscribe to the same topic using the same subscription
> name.
> > > A message is being sent and one of the nodes will receive the message,
> > > process it and then forward it to Ignite using the extractor provided.
> > >
> > > Any thoughts?
> > >
> > > *Emmanouil Gkatziouras*
> > > https://egkatzioura.com/ |
> > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > https://github.com/gkatzioura
> > >
> >
>

[Emmanouil Gkatziouras]

Hi @Saikat Maitra

I have my implementation on an Ignite fork on my personal GitHub account [1]
It is based on the Kafka implementation which is of a polling nature.
GCP does not provide a service like Kafka. Instead it provides us with
Pub/Sub which is just like Kafka.
Therefore using the Kafka streamer as a guideline made more sense to me.

Apart from the implementation I have also added a unit test.
It definitely needs some polishing and a double check on my side to make
the review smooth.
For the tests a local Pub/Sub [2] server has to be setup however I am not
sure which is the best way to do so with regards to your CI/CD.

[1] https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub
[2] https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/

*Emmanouil Gkatziouras*
https://egkatzioura.com/ | https://www.linkedin.com/in/gkatziourasemmanouil/
https://github.com/gkatzioura


On Sat, 12 Oct 2019 at 02:11, Saikat Maitra <[hidden email]> wrote:

> Hello Emmanouil,
>
> Can you please share more info about the implementation?
>
> We have similar implementation for data streamers with Kafka and RocketMQ
>
> https://apacheignite-mix.readme.io/docs/kafka-streamer
>
> https://apacheignite-mix.readme.io/docs/rocketmq-streamer
>
> Please review and share if you have any questions.
>
> Regards,
> Saikat
>
>
>
> On Fri, Oct 11, 2019 at 4:56 PM Emmanouil Gkatziouras <
> [hidden email]>
> wrote:
>
> > Hi all,
> >
> > By mentioning Pub/Sub I mean Google Cloud's Pub/Sub [1]
> > Yes it is indeed cloud specific. It is the messaging service provided by
> > GCP.
> > I have made the integration with it, but I haven't published it yet (I am
> > waiting for the discussion/ticket and proceed with the contribution).
> >
> > Kind regards,
> > Emmanouil
> >
> > [1] https://cloud.google.com/pubsub/docs/overview
> >
> > *Emmanouil Gkatziouras*
> > https://egkatzioura.com/ |
> > https://www.linkedin.com/in/gkatziourasemmanouil/
> > https://github.com/gkatzioura
> >
> >
> > On Fri, 11 Oct 2019 at 21:45, Denis Magda <[hidden email]> wrote:
> >
> > > Hello Emmanouil,
> > >
> > > Are you referring to any specific implementation tailored for a cloud?
> If
> > > I'm not mistaken you were mentioning AWS/GCE. Is there any Github
> source
> > > code?
> > >
> > > Here is a list of existing integrations with streaming and pub/sub
> > > frameworks:
> > > https://apacheignite-mix.readme.io/docs/overview
> > >
> > > with the source code in Ignite master:
> > > https://github.com/apache/ignite/tree/master/modules
> > >
> > > Check out those resources to see what are the most suitable Ignite APIs
> > for
> > > such integrations.
> > >
> > > @Saikat Maitra <[hidden email]> , @Roman <[hidden email]>
> ,
> > as
> > > streaming technologies contributors/maintainers, could you please join
> > this
> > > thread?
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Thu, Oct 10, 2019 at 7:58 AM Emmanouil Gkatziouras <
> > > [hidden email]>
> > > wrote:
> > >
> > > > Hello everyone,
> > > >
> > > > I started using Ignite lately. Part of my work involves Pub/Sub,
> thus I
> > > > created a pub/sub streamer.
> > > > Pub/Sub is pretty close to Kafka.
> > > > Here's how the streamer works. Every node creates an instance of a
> > > Pub/Sub
> > > > client.
> > > > The clients subscribe to the same topic using the same subscription
> > name.
> > > > A message is being sent and one of the nodes will receive the
> message,
> > > > process it and then forward it to Ignite using the extractor
> provided.
> > > >
> > > > Any thoughts?
> > > >
> > > > *Emmanouil Gkatziouras*
> > > > https://egkatzioura.com/ |
> > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > > https://github.com/gkatzioura
> > > >
> > >
> >
>

[Saikat Maitra]

Hi Emmanouil,

The changes looks good to me. I wanted to check if you are also able to
validate that the once data is added to Ignite cluster you are also able to
access it using another Ignite client or using rest endpoints.

We use Teamcity for CI process, the details are as follows

Trigger validation of those test suites that have been affected by your
changes on TeamCity <http://ci.ignite.apache.org/>: <http://204.14.53.153/>

   - Locate a test suite you have to check, press button named "..." that
   is located on the left of "Run" button. "Run custom build" dialog will
   appear;
   - Go to "Changes" tab and choose "pull/<pull-request-number>/head" in
   "Build branch" dropdown list;
   - Press "Run build" button and monitor tests results.



@Roman [hidden email] - can you also please take a look?

Regards,
Saikat

On Sun, Oct 13, 2019 at 6:20 PM Emmanouil Gkatziouras <[hidden email]>
wrote:

> Hi @Saikat Maitra
>
> I have my implementation on an Ignite fork on my personal GitHub account
> [1]
> It is based on the Kafka implementation which is of a polling nature.
> GCP does not provide a service like Kafka. Instead it provides us with
> Pub/Sub which is just like Kafka.
> Therefore using the Kafka streamer as a guideline made more sense to me.
>
> Apart from the implementation I have also added a unit test.
> It definitely needs some polishing and a double check on my side to make
> the review smooth.
> For the tests a local Pub/Sub [2] server has to be setup however I am not
> sure which is the best way to do so with regards to your CI/CD.
>
> [1] https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub
> [2] https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/
>
> *Emmanouil Gkatziouras*
> https://egkatzioura.com/ |
> https://www.linkedin.com/in/gkatziourasemmanouil/
> https://github.com/gkatzioura
>
>
> On Sat, 12 Oct 2019 at 02:11, Saikat Maitra <[hidden email]>
> wrote:
>
> > Hello Emmanouil,
> >
> > Can you please share more info about the implementation?
> >
> > We have similar implementation for data streamers with Kafka and RocketMQ
> >
> > https://apacheignite-mix.readme.io/docs/kafka-streamer
> >
> > https://apacheignite-mix.readme.io/docs/rocketmq-streamer
> >
> > Please review and share if you have any questions.
> >
> > Regards,
> > Saikat
> >
> >
> >
> > On Fri, Oct 11, 2019 at 4:56 PM Emmanouil Gkatziouras <
> > [hidden email]>
> > wrote:
> >
> > > Hi all,
> > >
> > > By mentioning Pub/Sub I mean Google Cloud's Pub/Sub [1]
> > > Yes it is indeed cloud specific. It is the messaging service provided
> by
> > > GCP.
> > > I have made the integration with it, but I haven't published it yet (I
> am
> > > waiting for the discussion/ticket and proceed with the contribution).
> > >
> > > Kind regards,
> > > Emmanouil
> > >
> > > [1] https://cloud.google.com/pubsub/docs/overview
> > >
> > > *Emmanouil Gkatziouras*
> > > https://egkatzioura.com/ |
> > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > https://github.com/gkatzioura
> > >
> > >
> > > On Fri, 11 Oct 2019 at 21:45, Denis Magda <[hidden email]> wrote:
> > >
> > > > Hello Emmanouil,
> > > >
> > > > Are you referring to any specific implementation tailored for a
> cloud?
> > If
> > > > I'm not mistaken you were mentioning AWS/GCE. Is there any Github
> > source
> > > > code?
> > > >
> > > > Here is a list of existing integrations with streaming and pub/sub
> > > > frameworks:
> > > > https://apacheignite-mix.readme.io/docs/overview
> > > >
> > > > with the source code in Ignite master:
> > > > https://github.com/apache/ignite/tree/master/modules
> > > >
> > > > Check out those resources to see what are the most suitable Ignite
> APIs
> > > for
> > > > such integrations.
> > > >
> > > > @Saikat Maitra <[hidden email]> , @Roman <[hidden email]
> >
> > ,
> > > as
> > > > streaming technologies contributors/maintainers, could you please
> join
> > > this
> > > > thread?
> > > >
> > > > -
> > > > Denis
> > > >
> > > >
> > > > On Thu, Oct 10, 2019 at 7:58 AM Emmanouil Gkatziouras <
> > > > [hidden email]>
> > > > wrote:
> > > >
> > > > > Hello everyone,
> > > > >
> > > > > I started using Ignite lately. Part of my work involves Pub/Sub,
> > thus I
> > > > > created a pub/sub streamer.
> > > > > Pub/Sub is pretty close to Kafka.
> > > > > Here's how the streamer works. Every node creates an instance of a
> > > > Pub/Sub
> > > > > client.
> > > > > The clients subscribe to the same topic using the same subscription
> > > name.
> > > > > A message is being sent and one of the nodes will receive the
> > message,
> > > > > process it and then forward it to Ignite using the extractor
> > provided.
> > > > >
> > > > > Any thoughts?
> > > > >
> > > > > *Emmanouil Gkatziouras*
> > > > > https://egkatzioura.com/ |
> > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > > > https://github.com/gkatzioura
> > > > >
> > > >
> > >
> >
>

[dmagda]

Emmanouil, Saikat,

After contemplating on this, let me propose another way with this
particular integration that is aligned with our modularization endeavor.
The modularization [1] defines the Ignite core, Ignite modules, and
independent integrations. The first two are to be located in Ignite
repositories and to be tested/updated by the community for every release.
The modules are a subset of existing integrations that are adopted widely
and require community attention.

As for the independent integrations [1], those are supposed to be located
in a separate GitHub repo. I would advise us to start implementing this
concept with the Pub/Sub integration. Even though it will be located in a
separate Github repo, it will be listed among officially available
integrations on the Ignite website featuring Emmanouil as a contributor.

What do you think?


[1]
https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations



-
Denis


On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra <[hidden email]>
wrote:

> Hi Emmanouil,
>
> The changes looks good to me. I wanted to check if you are also able to
> validate that the once data is added to Ignite cluster you are also able to
> access it using another Ignite client or using rest endpoints.
>
> We use Teamcity for CI process, the details are as follows
>
> Trigger validation of those test suites that have been affected by your
> changes on TeamCity <http://ci.ignite.apache.org/>: <http://204.14.53.153/
> >
>
>    - Locate a test suite you have to check, press button named "..." that
>    is located on the left of "Run" button. "Run custom build" dialog will
>    appear;
>    - Go to "Changes" tab and choose "pull/<pull-request-number>/head" in
>    "Build branch" dropdown list;
>    - Press "Run build" button and monitor tests results.
>
>
>
> @Roman [hidden email] - can you also please take a look?
>
> Regards,
> Saikat
>
> On Sun, Oct 13, 2019 at 6:20 PM Emmanouil Gkatziouras <
> [hidden email]>
> wrote:
>
> > Hi @Saikat Maitra
> >
> > I have my implementation on an Ignite fork on my personal GitHub account
> > [1]
> > It is based on the Kafka implementation which is of a polling nature.
> > GCP does not provide a service like Kafka. Instead it provides us with
> > Pub/Sub which is just like Kafka.
> > Therefore using the Kafka streamer as a guideline made more sense to me.
> >
> > Apart from the implementation I have also added a unit test.
> > It definitely needs some polishing and a double check on my side to make
> > the review smooth.
> > For the tests a local Pub/Sub [2] server has to be setup however I am not
> > sure which is the best way to do so with regards to your CI/CD.
> >
> > [1] https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub
> > [2] https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/
> >
> > *Emmanouil Gkatziouras*
> > https://egkatzioura.com/ |
> > https://www.linkedin.com/in/gkatziourasemmanouil/
> > https://github.com/gkatzioura
> >
> >
> > On Sat, 12 Oct 2019 at 02:11, Saikat Maitra <[hidden email]>
> > wrote:
> >
> > > Hello Emmanouil,
> > >
> > > Can you please share more info about the implementation?
> > >
> > > We have similar implementation for data streamers with Kafka and
> RocketMQ
> > >
> > > https://apacheignite-mix.readme.io/docs/kafka-streamer
> > >
> > > https://apacheignite-mix.readme.io/docs/rocketmq-streamer
> > >
> > > Please review and share if you have any questions.
> > >
> > > Regards,
> > > Saikat
> > >
> > >
> > >
> > > On Fri, Oct 11, 2019 at 4:56 PM Emmanouil Gkatziouras <
> > > [hidden email]>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > By mentioning Pub/Sub I mean Google Cloud's Pub/Sub [1]
> > > > Yes it is indeed cloud specific. It is the messaging service provided
> > by
> > > > GCP.
> > > > I have made the integration with it, but I haven't published it yet
> (I
> > am
> > > > waiting for the discussion/ticket and proceed with the contribution).
> > > >
> > > > Kind regards,
> > > > Emmanouil
> > > >
> > > > [1] https://cloud.google.com/pubsub/docs/overview
> > > >
> > > > *Emmanouil Gkatziouras*
> > > > https://egkatzioura.com/ |
> > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > > https://github.com/gkatzioura
> > > >
> > > >
> > > > On Fri, 11 Oct 2019 at 21:45, Denis Magda <[hidden email]> wrote:
> > > >
> > > > > Hello Emmanouil,
> > > > >
> > > > > Are you referring to any specific implementation tailored for a
> > cloud?
> > > If
> > > > > I'm not mistaken you were mentioning AWS/GCE. Is there any Github
> > > source
> > > > > code?
> > > > >
> > > > > Here is a list of existing integrations with streaming and pub/sub
> > > > > frameworks:
> > > > > https://apacheignite-mix.readme.io/docs/overview
> > > > >
> > > > > with the source code in Ignite master:
> > > > > https://github.com/apache/ignite/tree/master/modules
> > > > >
> > > > > Check out those resources to see what are the most suitable Ignite
> > APIs
> > > > for
> > > > > such integrations.
> > > > >
> > > > > @Saikat Maitra <[hidden email]> , @Roman <
> [hidden email]
> > >
> > > ,
> > > > as
> > > > > streaming technologies contributors/maintainers, could you please
> > join
> > > > this
> > > > > thread?
> > > > >
> > > > > -
> > > > > Denis
> > > > >
> > > > >
> > > > > On Thu, Oct 10, 2019 at 7:58 AM Emmanouil Gkatziouras <
> > > > > [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > Hello everyone,
> > > > > >
> > > > > > I started using Ignite lately. Part of my work involves Pub/Sub,
> > > thus I
> > > > > > created a pub/sub streamer.
> > > > > > Pub/Sub is pretty close to Kafka.
> > > > > > Here's how the streamer works. Every node creates an instance of
> a
> > > > > Pub/Sub
> > > > > > client.
> > > > > > The clients subscribe to the same topic using the same
> subscription
> > > > name.
> > > > > > A message is being sent and one of the nodes will receive the
> > > message,
> > > > > > process it and then forward it to Ignite using the extractor
> > > provided.
> > > > > >
> > > > > > Any thoughts?
> > > > > >
> > > > > > *Emmanouil Gkatziouras*
> > > > > > https://egkatzioura.com/ |
> > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > > > > https://github.com/gkatzioura
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Emmanouil Gkatziouras]

Hi all!

Based on the discussions most probably this is going to be on another
GitHub repo. Therefore I will prepare a standalone project with the Pub/Sub
module and once the repository is created a pull request shall be issued.
If there is anything else I can do in order to facilitate this please let
me know.

If a ticket is created for this issue, I shall be able to create a pull
request and thus a build will take place on ignite's CI.
If everyone is aligned with adding a Pub/Sub implementation may I create a
ticket on JIRA?

My next question has to do on how Ignite configures the build jobs in Team
City. The unit tests for my implementation need a local Pub/Sub server up
and running (they provide one for testing purposes). Will this take effect
in a form of a build script inside my implementation or is it something
that should be configured on Team City?

Kind regards,
Emmanouil


*Emmanouil Gkatziouras*
https://egkatzioura.com/ | https://www.linkedin.com/in/gkatziourasemmanouil/
https://github.com/gkatzioura


On Tue, 15 Oct 2019 at 23:27, Denis Magda <[hidden email]> wrote:

> Emmanouil, Saikat,
>
> After contemplating on this, let me propose another way with this
> particular integration that is aligned with our modularization endeavor.
> The modularization [1] defines the Ignite core, Ignite modules, and
> independent integrations. The first two are to be located in Ignite
> repositories and to be tested/updated by the community for every release.
> The modules are a subset of existing integrations that are adopted widely
> and require community attention.
>
> As for the independent integrations [1], those are supposed to be located
> in a separate GitHub repo. I would advise us to start implementing this
> concept with the Pub/Sub integration. Even though it will be located in a
> separate Github repo, it will be listed among officially available
> integrations on the Ignite website featuring Emmanouil as a contributor.
>
> What do you think?
>
>
> [1]
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
>
>
>
> -
> Denis
>
>
> On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra <[hidden email]>
> wrote:
>
> > Hi Emmanouil,
> >
> > The changes looks good to me. I wanted to check if you are also able to
> > validate that the once data is added to Ignite cluster you are also able
> to
> > access it using another Ignite client or using rest endpoints.
> >
> > We use Teamcity for CI process, the details are as follows
> >
> > Trigger validation of those test suites that have been affected by your
> > changes on TeamCity <http://ci.ignite.apache.org/>: <
> http://204.14.53.153/
> > >
> >
> >    - Locate a test suite you have to check, press button named "..." that
> >    is located on the left of "Run" button. "Run custom build" dialog will
> >    appear;
> >    - Go to "Changes" tab and choose "pull/<pull-request-number>/head" in
> >    "Build branch" dropdown list;
> >    - Press "Run build" button and monitor tests results.
> >
> >
> >
> > @Roman [hidden email] - can you also please take a look?
> >
> > Regards,
> > Saikat
> >
> > On Sun, Oct 13, 2019 at 6:20 PM Emmanouil Gkatziouras <
> > [hidden email]>
> > wrote:
> >
> > > Hi @Saikat Maitra
> > >
> > > I have my implementation on an Ignite fork on my personal GitHub
> account
> > > [1]
> > > It is based on the Kafka implementation which is of a polling nature.
> > > GCP does not provide a service like Kafka. Instead it provides us with
> > > Pub/Sub which is just like Kafka.
> > > Therefore using the Kafka streamer as a guideline made more sense to
> me.
> > >
> > > Apart from the implementation I have also added a unit test.
> > > It definitely needs some polishing and a double check on my side to
> make
> > > the review smooth.
> > > For the tests a local Pub/Sub [2] server has to be setup however I am
> not
> > > sure which is the best way to do so with regards to your CI/CD.
> > >
> > > [1] https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub
> > > [2] https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/
> > >
> > > *Emmanouil Gkatziouras*
> > > https://egkatzioura.com/ |
> > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > https://github.com/gkatzioura
> > >
> > >
> > > On Sat, 12 Oct 2019 at 02:11, Saikat Maitra <[hidden email]>
> > > wrote:
> > >
> > > > Hello Emmanouil,
> > > >
> > > > Can you please share more info about the implementation?
> > > >
> > > > We have similar implementation for data streamers with Kafka and
> > RocketMQ
> > > >
> > > > https://apacheignite-mix.readme.io/docs/kafka-streamer
> > > >
> > > > https://apacheignite-mix.readme.io/docs/rocketmq-streamer
> > > >
> > > > Please review and share if you have any questions.
> > > >
> > > > Regards,
> > > > Saikat
> > > >
> > > >
> > > >
> > > > On Fri, Oct 11, 2019 at 4:56 PM Emmanouil Gkatziouras <
> > > > [hidden email]>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > By mentioning Pub/Sub I mean Google Cloud's Pub/Sub [1]
> > > > > Yes it is indeed cloud specific. It is the messaging service
> provided
> > > by
> > > > > GCP.
> > > > > I have made the integration with it, but I haven't published it yet
> > (I
> > > am
> > > > > waiting for the discussion/ticket and proceed with the
> contribution).
> > > > >
> > > > > Kind regards,
> > > > > Emmanouil
> > > > >
> > > > > [1] https://cloud.google.com/pubsub/docs/overview
> > > > >
> > > > > *Emmanouil Gkatziouras*
> > > > > https://egkatzioura.com/ |
> > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > > > https://github.com/gkatzioura
> > > > >
> > > > >
> > > > > On Fri, 11 Oct 2019 at 21:45, Denis Magda <[hidden email]>
> wrote:
> > > > >
> > > > > > Hello Emmanouil,
> > > > > >
> > > > > > Are you referring to any specific implementation tailored for a
> > > cloud?
> > > > If
> > > > > > I'm not mistaken you were mentioning AWS/GCE. Is there any Github
> > > > source
> > > > > > code?
> > > > > >
> > > > > > Here is a list of existing integrations with streaming and
> pub/sub
> > > > > > frameworks:
> > > > > > https://apacheignite-mix.readme.io/docs/overview
> > > > > >
> > > > > > with the source code in Ignite master:
> > > > > > https://github.com/apache/ignite/tree/master/modules
> > > > > >
> > > > > > Check out those resources to see what are the most suitable
> Ignite
> > > APIs
> > > > > for
> > > > > > such integrations.
> > > > > >
> > > > > > @Saikat Maitra <[hidden email]> , @Roman <
> > [hidden email]
> > > >
> > > > ,
> > > > > as
> > > > > > streaming technologies contributors/maintainers, could you please
> > > join
> > > > > this
> > > > > > thread?
> > > > > >
> > > > > > -
> > > > > > Denis
> > > > > >
> > > > > >
> > > > > > On Thu, Oct 10, 2019 at 7:58 AM Emmanouil Gkatziouras <
> > > > > > [hidden email]>
> > > > > > wrote:
> > > > > >
> > > > > > > Hello everyone,
> > > > > > >
> > > > > > > I started using Ignite lately. Part of my work involves
> Pub/Sub,
> > > > thus I
> > > > > > > created a pub/sub streamer.
> > > > > > > Pub/Sub is pretty close to Kafka.
> > > > > > > Here's how the streamer works. Every node creates an instance
> of
> > a
> > > > > > Pub/Sub
> > > > > > > client.
> > > > > > > The clients subscribe to the same topic using the same
> > subscription
> > > > > name.
> > > > > > > A message is being sent and one of the nodes will receive the
> > > > message,
> > > > > > > process it and then forward it to Ignite using the extractor
> > > > provided.
> > > > > > >
> > > > > > > Any thoughts?
> > > > > > >
> > > > > > > *Emmanouil Gkatziouras*
> > > > > > > https://egkatzioura.com/ |
> > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > > > > > https://github.com/gkatzioura
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Saikat Maitra]

Hi Denis, Emmanouil

Thank you for your email. I think creating a separate integration project
in github and also proposing it as an apache incubator project will be a
good move. The other separate integration modules
https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
can
be moved to this new apache incubator project.

There are similar integration available for Flink and Spark in Apache Bahir
https://bahir.apache.org/

Do you think we should reach out to Apache Bahir community with a proposal
or we should plan to create a separate Apache Incubator project?

Regards,
Saikat



On Wed, Oct 16, 2019 at 6:21 AM Emmanouil Gkatziouras <[hidden email]>
wrote:

> Hi all!
>
> Based on the discussions most probably this is going to be on another
> GitHub repo. Therefore I will prepare a standalone project with the Pub/Sub
> module and once the repository is created a pull request shall be issued.
> If there is anything else I can do in order to facilitate this please let
> me know.
>
> If a ticket is created for this issue, I shall be able to create a pull
> request and thus a build will take place on ignite's CI.
> If everyone is aligned with adding a Pub/Sub implementation may I create a
> ticket on JIRA?
>
> My next question has to do on how Ignite configures the build jobs in Team
> City. The unit tests for my implementation need a local Pub/Sub server up
> and running (they provide one for testing purposes). Will this take effect
> in a form of a build script inside my implementation or is it something
> that should be configured on Team City?
>
> Kind regards,
> Emmanouil
>
>
> *Emmanouil Gkatziouras*
> https://egkatzioura.com/ |
> https://www.linkedin.com/in/gkatziourasemmanouil/
> https://github.com/gkatzioura
>
>
> On Tue, 15 Oct 2019 at 23:27, Denis Magda <[hidden email]> wrote:
>
> > Emmanouil, Saikat,
> >
> > After contemplating on this, let me propose another way with this
> > particular integration that is aligned with our modularization endeavor.
> > The modularization [1] defines the Ignite core, Ignite modules, and
> > independent integrations. The first two are to be located in Ignite
> > repositories and to be tested/updated by the community for every release.
> > The modules are a subset of existing integrations that are adopted widely
> > and require community attention.
> >
> > As for the independent integrations [1], those are supposed to be located
> > in a separate GitHub repo. I would advise us to start implementing this
> > concept with the Pub/Sub integration. Even though it will be located in a
> > separate Github repo, it will be listed among officially available
> > integrations on the Ignite website featuring Emmanouil as a contributor.
> >
> > What do you think?
> >
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> >
> >
> >
> > -
> > Denis
> >
> >
> > On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra <[hidden email]>
> > wrote:
> >
> > > Hi Emmanouil,
> > >
> > > The changes looks good to me. I wanted to check if you are also able to
> > > validate that the once data is added to Ignite cluster you are also
> able
> > to
> > > access it using another Ignite client or using rest endpoints.
> > >
> > > We use Teamcity for CI process, the details are as follows
> > >
> > > Trigger validation of those test suites that have been affected by your
> > > changes on TeamCity <http://ci.ignite.apache.org/>: <
> > http://204.14.53.153/
> > > >
> > >
> > >    - Locate a test suite you have to check, press button named "..."
> that
> > >    is located on the left of "Run" button. "Run custom build" dialog
> will
> > >    appear;
> > >    - Go to "Changes" tab and choose "pull/<pull-request-number>/head"
> in
> > >    "Build branch" dropdown list;
> > >    - Press "Run build" button and monitor tests results.
> > >
> > >
> > >
> > > @Roman [hidden email] - can you also please take a look?
> > >
> > > Regards,
> > > Saikat
> > >
> > > On Sun, Oct 13, 2019 at 6:20 PM Emmanouil Gkatziouras <
> > > [hidden email]>
> > > wrote:
> > >
> > > > Hi @Saikat Maitra
> > > >
> > > > I have my implementation on an Ignite fork on my personal GitHub
> > account
> > > > [1]
> > > > It is based on the Kafka implementation which is of a polling nature.
> > > > GCP does not provide a service like Kafka. Instead it provides us
> with
> > > > Pub/Sub which is just like Kafka.
> > > > Therefore using the Kafka streamer as a guideline made more sense to
> > me.
> > > >
> > > > Apart from the implementation I have also added a unit test.
> > > > It definitely needs some polishing and a double check on my side to
> > make
> > > > the review smooth.
> > > > For the tests a local Pub/Sub [2] server has to be setup however I am
> > not
> > > > sure which is the best way to do so with regards to your CI/CD.
> > > >
> > > > [1] https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub
> > > > [2] https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/
> > > >
> > > > *Emmanouil Gkatziouras*
> > > > https://egkatzioura.com/ |
> > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > > https://github.com/gkatzioura
> > > >
> > > >
> > > > On Sat, 12 Oct 2019 at 02:11, Saikat Maitra <[hidden email]
> >
> > > > wrote:
> > > >
> > > > > Hello Emmanouil,
> > > > >
> > > > > Can you please share more info about the implementation?
> > > > >
> > > > > We have similar implementation for data streamers with Kafka and
> > > RocketMQ
> > > > >
> > > > > https://apacheignite-mix.readme.io/docs/kafka-streamer
> > > > >
> > > > > https://apacheignite-mix.readme.io/docs/rocketmq-streamer
> > > > >
> > > > > Please review and share if you have any questions.
> > > > >
> > > > > Regards,
> > > > > Saikat
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Oct 11, 2019 at 4:56 PM Emmanouil Gkatziouras <
> > > > > [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > By mentioning Pub/Sub I mean Google Cloud's Pub/Sub [1]
> > > > > > Yes it is indeed cloud specific. It is the messaging service
> > provided
> > > > by
> > > > > > GCP.
> > > > > > I have made the integration with it, but I haven't published it
> yet
> > > (I
> > > > am
> > > > > > waiting for the discussion/ticket and proceed with the
> > contribution).
> > > > > >
> > > > > > Kind regards,
> > > > > > Emmanouil
> > > > > >
> > > > > > [1] https://cloud.google.com/pubsub/docs/overview
> > > > > >
> > > > > > *Emmanouil Gkatziouras*
> > > > > > https://egkatzioura.com/ |
> > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > > > > https://github.com/gkatzioura
> > > > > >
> > > > > >
> > > > > > On Fri, 11 Oct 2019 at 21:45, Denis Magda <[hidden email]>
> > wrote:
> > > > > >
> > > > > > > Hello Emmanouil,
> > > > > > >
> > > > > > > Are you referring to any specific implementation tailored for a
> > > > cloud?
> > > > > If
> > > > > > > I'm not mistaken you were mentioning AWS/GCE. Is there any
> Github
> > > > > source
> > > > > > > code?
> > > > > > >
> > > > > > > Here is a list of existing integrations with streaming and
> > pub/sub
> > > > > > > frameworks:
> > > > > > > https://apacheignite-mix.readme.io/docs/overview
> > > > > > >
> > > > > > > with the source code in Ignite master:
> > > > > > > https://github.com/apache/ignite/tree/master/modules
> > > > > > >
> > > > > > > Check out those resources to see what are the most suitable
> > Ignite
> > > > APIs
> > > > > > for
> > > > > > > such integrations.
> > > > > > >
> > > > > > > @Saikat Maitra <[hidden email]> , @Roman <
> > > [hidden email]
> > > > >
> > > > > ,
> > > > > > as
> > > > > > > streaming technologies contributors/maintainers, could you
> please
> > > > join
> > > > > > this
> > > > > > > thread?
> > > > > > >
> > > > > > > -
> > > > > > > Denis
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Oct 10, 2019 at 7:58 AM Emmanouil Gkatziouras <
> > > > > > > [hidden email]>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello everyone,
> > > > > > > >
> > > > > > > > I started using Ignite lately. Part of my work involves
> > Pub/Sub,
> > > > > thus I
> > > > > > > > created a pub/sub streamer.
> > > > > > > > Pub/Sub is pretty close to Kafka.
> > > > > > > > Here's how the streamer works. Every node creates an instance
> > of
> > > a
> > > > > > > Pub/Sub
> > > > > > > > client.
> > > > > > > > The clients subscribe to the same topic using the same
> > > subscription
> > > > > > name.
> > > > > > > > A message is being sent and one of the nodes will receive the
> > > > > message,
> > > > > > > > process it and then forward it to Ignite using the extractor
> > > > > provided.
> > > > > > > >
> > > > > > > > Any thoughts?
> > > > > > > >
> > > > > > > > *Emmanouil Gkatziouras*
> > > > > > > > https://egkatzioura.com/ |
> > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > > > > > > https://github.com/gkatzioura
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Alexey Zinoviev]

Maybe we could move all our Streaming Integrations there, but what is about
maintaining and committer permissions to the new repositories?
I see the list of the committers and PMC members there
https://bahir.apache.org/community-members/

Could somebody from Ignite community be added to this list as a
PMC/committer to maintain Ignite integrations?
If yes, I'd happy to join this project and collaborate with these guys
together
If no, and we should start from the zero level with external PRs - hmmm,
it's better to have our own external repository with ApacheBahirr ideology.

Also, I agree, that all connectors could be moved there (in ApacheBahirr
like repository), but not all modules from the page
https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
because
they use different parts of core modules and could be developed with
different velocity.

In reality, before creation of new repositories we need wide discussion
based on the document mentioned above.

P.S. Of course, we could experiment in the next release 2.8 with one or two
integrations like twitter or ZeroMQ
Also, I'd like Denis Magda idea for the separate repositories in Apache
Github to maintain them by anybody.


чт, 17 окт. 2019 г. в 05:02, Saikat Maitra <[hidden email]>:

> Hi Denis, Emmanouil
>
> Thank you for your email. I think creating a separate integration project
> in github and also proposing it as an apache incubator project will be a
> good move. The other separate integration modules
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> can
> be moved to this new apache incubator project.
>
> There are similar integration available for Flink and Spark in Apache Bahir
> https://bahir.apache.org/
>
> Do you think we should reach out to Apache Bahir community with a proposal
> or we should plan to create a separate Apache Incubator project?
>
> Regards,
> Saikat
>
>
>
> On Wed, Oct 16, 2019 at 6:21 AM Emmanouil Gkatziouras <
> [hidden email]>
> wrote:
>
> > Hi all!
> >
> > Based on the discussions most probably this is going to be on another
> > GitHub repo. Therefore I will prepare a standalone project with the
> Pub/Sub
> > module and once the repository is created a pull request shall be issued.
> > If there is anything else I can do in order to facilitate this please let
> > me know.
> >
> > If a ticket is created for this issue, I shall be able to create a pull
> > request and thus a build will take place on ignite's CI.
> > If everyone is aligned with adding a Pub/Sub implementation may I create
> a
> > ticket on JIRA?
> >
> > My next question has to do on how Ignite configures the build jobs in
> Team
> > City. The unit tests for my implementation need a local Pub/Sub server up
> > and running (they provide one for testing purposes). Will this take
> effect
> > in a form of a build script inside my implementation or is it something
> > that should be configured on Team City?
> >
> > Kind regards,
> > Emmanouil
> >
> >
> > *Emmanouil Gkatziouras*
> > https://egkatzioura.com/ |
> > https://www.linkedin.com/in/gkatziourasemmanouil/
> > https://github.com/gkatzioura
> >
> >
> > On Tue, 15 Oct 2019 at 23:27, Denis Magda <[hidden email]> wrote:
> >
> > > Emmanouil, Saikat,
> > >
> > > After contemplating on this, let me propose another way with this
> > > particular integration that is aligned with our modularization
> endeavor.
> > > The modularization [1] defines the Ignite core, Ignite modules, and
> > > independent integrations. The first two are to be located in Ignite
> > > repositories and to be tested/updated by the community for every
> release.
> > > The modules are a subset of existing integrations that are adopted
> widely
> > > and require community attention.
> > >
> > > As for the independent integrations [1], those are supposed to be
> located
> > > in a separate GitHub repo. I would advise us to start implementing this
> > > concept with the Pub/Sub integration. Even though it will be located
> in a
> > > separate Github repo, it will be listed among officially available
> > > integrations on the Ignite website featuring Emmanouil as a
> contributor.
> > >
> > > What do you think?
> > >
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> > >
> > >
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra <[hidden email]
> >
> > > wrote:
> > >
> > > > Hi Emmanouil,
> > > >
> > > > The changes looks good to me. I wanted to check if you are also able
> to
> > > > validate that the once data is added to Ignite cluster you are also
> > able
> > > to
> > > > access it using another Ignite client or using rest endpoints.
> > > >
> > > > We use Teamcity for CI process, the details are as follows
> > > >
> > > > Trigger validation of those test suites that have been affected by
> your
> > > > changes on TeamCity <http://ci.ignite.apache.org/>: <
> > > http://204.14.53.153/
> > > > >
> > > >
> > > >    - Locate a test suite you have to check, press button named "..."
> > that
> > > >    is located on the left of "Run" button. "Run custom build" dialog
> > will
> > > >    appear;
> > > >    - Go to "Changes" tab and choose "pull/<pull-request-number>/head"
> > in
> > > >    "Build branch" dropdown list;
> > > >    - Press "Run build" button and monitor tests results.
> > > >
> > > >
> > > >
> > > > @Roman [hidden email] - can you also please take a look?
> > > >
> > > > Regards,
> > > > Saikat
> > > >
> > > > On Sun, Oct 13, 2019 at 6:20 PM Emmanouil Gkatziouras <
> > > > [hidden email]>
> > > > wrote:
> > > >
> > > > > Hi @Saikat Maitra
> > > > >
> > > > > I have my implementation on an Ignite fork on my personal GitHub
> > > account
> > > > > [1]
> > > > > It is based on the Kafka implementation which is of a polling
> nature.
> > > > > GCP does not provide a service like Kafka. Instead it provides us
> > with
> > > > > Pub/Sub which is just like Kafka.
> > > > > Therefore using the Kafka streamer as a guideline made more sense
> to
> > > me.
> > > > >
> > > > > Apart from the implementation I have also added a unit test.
> > > > > It definitely needs some polishing and a double check on my side to
> > > make
> > > > > the review smooth.
> > > > > For the tests a local Pub/Sub [2] server has to be setup however I
> am
> > > not
> > > > > sure which is the best way to do so with regards to your CI/CD.
> > > > >
> > > > > [1]
> https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub
> > > > > [2] https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/
> > > > >
> > > > > *Emmanouil Gkatziouras*
> > > > > https://egkatzioura.com/ |
> > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > > > https://github.com/gkatzioura
> > > > >
> > > > >
> > > > > On Sat, 12 Oct 2019 at 02:11, Saikat Maitra <
> [hidden email]
> > >
> > > > > wrote:
> > > > >
> > > > > > Hello Emmanouil,
> > > > > >
> > > > > > Can you please share more info about the implementation?
> > > > > >
> > > > > > We have similar implementation for data streamers with Kafka and
> > > > RocketMQ
> > > > > >
> > > > > > https://apacheignite-mix.readme.io/docs/kafka-streamer
> > > > > >
> > > > > > https://apacheignite-mix.readme.io/docs/rocketmq-streamer
> > > > > >
> > > > > > Please review and share if you have any questions.
> > > > > >
> > > > > > Regards,
> > > > > > Saikat
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Oct 11, 2019 at 4:56 PM Emmanouil Gkatziouras <
> > > > > > [hidden email]>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > By mentioning Pub/Sub I mean Google Cloud's Pub/Sub [1]
> > > > > > > Yes it is indeed cloud specific. It is the messaging service
> > > provided
> > > > > by
> > > > > > > GCP.
> > > > > > > I have made the integration with it, but I haven't published it
> > yet
> > > > (I
> > > > > am
> > > > > > > waiting for the discussion/ticket and proceed with the
> > > contribution).
> > > > > > >
> > > > > > > Kind regards,
> > > > > > > Emmanouil
> > > > > > >
> > > > > > > [1] https://cloud.google.com/pubsub/docs/overview
> > > > > > >
> > > > > > > *Emmanouil Gkatziouras*
> > > > > > > https://egkatzioura.com/ |
> > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > > > > > https://github.com/gkatzioura
> > > > > > >
> > > > > > >
> > > > > > > On Fri, 11 Oct 2019 at 21:45, Denis Magda <[hidden email]>
> > > wrote:
> > > > > > >
> > > > > > > > Hello Emmanouil,
> > > > > > > >
> > > > > > > > Are you referring to any specific implementation tailored
> for a
> > > > > cloud?
> > > > > > If
> > > > > > > > I'm not mistaken you were mentioning AWS/GCE. Is there any
> > Github
> > > > > > source
> > > > > > > > code?
> > > > > > > >
> > > > > > > > Here is a list of existing integrations with streaming and
> > > pub/sub
> > > > > > > > frameworks:
> > > > > > > > https://apacheignite-mix.readme.io/docs/overview
> > > > > > > >
> > > > > > > > with the source code in Ignite master:
> > > > > > > > https://github.com/apache/ignite/tree/master/modules
> > > > > > > >
> > > > > > > > Check out those resources to see what are the most suitable
> > > Ignite
> > > > > APIs
> > > > > > > for
> > > > > > > > such integrations.
> > > > > > > >
> > > > > > > > @Saikat Maitra <[hidden email]> , @Roman <
> > > > [hidden email]
> > > > > >
> > > > > > ,
> > > > > > > as
> > > > > > > > streaming technologies contributors/maintainers, could you
> > please
> > > > > join
> > > > > > > this
> > > > > > > > thread?
> > > > > > > >
> > > > > > > > -
> > > > > > > > Denis
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Oct 10, 2019 at 7:58 AM Emmanouil Gkatziouras <
> > > > > > > > [hidden email]>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello everyone,
> > > > > > > > >
> > > > > > > > > I started using Ignite lately. Part of my work involves
> > > Pub/Sub,
> > > > > > thus I
> > > > > > > > > created a pub/sub streamer.
> > > > > > > > > Pub/Sub is pretty close to Kafka.
> > > > > > > > > Here's how the streamer works. Every node creates an
> instance
> > > of
> > > > a
> > > > > > > > Pub/Sub
> > > > > > > > > client.
> > > > > > > > > The clients subscribe to the same topic using the same
> > > > subscription
> > > > > > > name.
> > > > > > > > > A message is being sent and one of the nodes will receive
> the
> > > > > > message,
> > > > > > > > > process it and then forward it to Ignite using the
> extractor
> > > > > > provided.
> > > > > > > > >
> > > > > > > > > Any thoughts?
> > > > > > > > >
> > > > > > > > > *Emmanouil Gkatziouras*
> > > > > > > > > https://egkatzioura.com/ |
> > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > > > > > > > https://github.com/gkatzioura
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Ilya Kasnacheev]

Hello!

Historically I support some integrations, so I can final review/merge
integrations commit until someone else steps in.

Regards,
--
Ilya Kasnacheev


чт, 17 окт. 2019 г. в 12:54, Alexey Zinoviev <[hidden email]>:

> Maybe we could move all our Streaming Integrations there, but what is about
> maintaining and committer permissions to the new repositories?
> I see the list of the committers and PMC members there
> https://bahir.apache.org/community-members/
>
> Could somebody from Ignite community be added to this list as a
> PMC/committer to maintain Ignite integrations?
> If yes, I'd happy to join this project and collaborate with these guys
> together
> If no, and we should start from the zero level with external PRs - hmmm,
> it's better to have our own external repository with ApacheBahirr ideology.
>
> Also, I agree, that all connectors could be moved there (in ApacheBahirr
> like repository), but not all modules from the page
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> because
> they use different parts of core modules and could be developed with
> different velocity.
>
> In reality, before creation of new repositories we need wide discussion
> based on the document mentioned above.
>
> P.S. Of course, we could experiment in the next release 2.8 with one or two
> integrations like twitter or ZeroMQ
> Also, I'd like Denis Magda idea for the separate repositories in Apache
> Github to maintain them by anybody.
>
>
> чт, 17 окт. 2019 г. в 05:02, Saikat Maitra <[hidden email]>:
>
> > Hi Denis, Emmanouil
> >
> > Thank you for your email. I think creating a separate integration project
> > in github and also proposing it as an apache incubator project will be a
> > good move. The other separate integration modules
> >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> > can
> > be moved to this new apache incubator project.
> >
> > There are similar integration available for Flink and Spark in Apache
> Bahir
> > https://bahir.apache.org/
> >
> > Do you think we should reach out to Apache Bahir community with a
> proposal
> > or we should plan to create a separate Apache Incubator project?
> >
> > Regards,
> > Saikat
> >
> >
> >
> > On Wed, Oct 16, 2019 at 6:21 AM Emmanouil Gkatziouras <
> > [hidden email]>
> > wrote:
> >
> > > Hi all!
> > >
> > > Based on the discussions most probably this is going to be on another
> > > GitHub repo. Therefore I will prepare a standalone project with the
> > Pub/Sub
> > > module and once the repository is created a pull request shall be
> issued.
> > > If there is anything else I can do in order to facilitate this please
> let
> > > me know.
> > >
> > > If a ticket is created for this issue, I shall be able to create a pull
> > > request and thus a build will take place on ignite's CI.
> > > If everyone is aligned with adding a Pub/Sub implementation may I
> create
> > a
> > > ticket on JIRA?
> > >
> > > My next question has to do on how Ignite configures the build jobs in
> > Team
> > > City. The unit tests for my implementation need a local Pub/Sub server
> up
> > > and running (they provide one for testing purposes). Will this take
> > effect
> > > in a form of a build script inside my implementation or is it something
> > > that should be configured on Team City?
> > >
> > > Kind regards,
> > > Emmanouil
> > >
> > >
> > > *Emmanouil Gkatziouras*
> > > https://egkatzioura.com/ |
> > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > https://github.com/gkatzioura
> > >
> > >
> > > On Tue, 15 Oct 2019 at 23:27, Denis Magda <[hidden email]> wrote:
> > >
> > > > Emmanouil, Saikat,
> > > >
> > > > After contemplating on this, let me propose another way with this
> > > > particular integration that is aligned with our modularization
> > endeavor.
> > > > The modularization [1] defines the Ignite core, Ignite modules, and
> > > > independent integrations. The first two are to be located in Ignite
> > > > repositories and to be tested/updated by the community for every
> > release.
> > > > The modules are a subset of existing integrations that are adopted
> > widely
> > > > and require community attention.
> > > >
> > > > As for the independent integrations [1], those are supposed to be
> > located
> > > > in a separate GitHub repo. I would advise us to start implementing
> this
> > > > concept with the Pub/Sub integration. Even though it will be located
> > in a
> > > > separate Github repo, it will be listed among officially available
> > > > integrations on the Ignite website featuring Emmanouil as a
> > contributor.
> > > >
> > > > What do you think?
> > > >
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> > > >
> > > >
> > > >
> > > > -
> > > > Denis
> > > >
> > > >
> > > > On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra <
> [hidden email]
> > >
> > > > wrote:
> > > >
> > > > > Hi Emmanouil,
> > > > >
> > > > > The changes looks good to me. I wanted to check if you are also
> able
> > to
> > > > > validate that the once data is added to Ignite cluster you are also
> > > able
> > > > to
> > > > > access it using another Ignite client or using rest endpoints.
> > > > >
> > > > > We use Teamcity for CI process, the details are as follows
> > > > >
> > > > > Trigger validation of those test suites that have been affected by
> > your
> > > > > changes on TeamCity <http://ci.ignite.apache.org/>: <
> > > > http://204.14.53.153/
> > > > > >
> > > > >
> > > > >    - Locate a test suite you have to check, press button named
> "..."
> > > that
> > > > >    is located on the left of "Run" button. "Run custom build"
> dialog
> > > will
> > > > >    appear;
> > > > >    - Go to "Changes" tab and choose
> "pull/<pull-request-number>/head"
> > > in
> > > > >    "Build branch" dropdown list;
> > > > >    - Press "Run build" button and monitor tests results.
> > > > >
> > > > >
> > > > >
> > > > > @Roman [hidden email] - can you also please take a look?
> > > > >
> > > > > Regards,
> > > > > Saikat
> > > > >
> > > > > On Sun, Oct 13, 2019 at 6:20 PM Emmanouil Gkatziouras <
> > > > > [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > Hi @Saikat Maitra
> > > > > >
> > > > > > I have my implementation on an Ignite fork on my personal GitHub
> > > > account
> > > > > > [1]
> > > > > > It is based on the Kafka implementation which is of a polling
> > nature.
> > > > > > GCP does not provide a service like Kafka. Instead it provides us
> > > with
> > > > > > Pub/Sub which is just like Kafka.
> > > > > > Therefore using the Kafka streamer as a guideline made more sense
> > to
> > > > me.
> > > > > >
> > > > > > Apart from the implementation I have also added a unit test.
> > > > > > It definitely needs some polishing and a double check on my side
> to
> > > > make
> > > > > > the review smooth.
> > > > > > For the tests a local Pub/Sub [2] server has to be setup however
> I
> > am
> > > > not
> > > > > > sure which is the best way to do so with regards to your CI/CD.
> > > > > >
> > > > > > [1]
> > https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub
> > > > > > [2] https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/
> > > > > >
> > > > > > *Emmanouil Gkatziouras*
> > > > > > https://egkatzioura.com/ |
> > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > > > > https://github.com/gkatzioura
> > > > > >
> > > > > >
> > > > > > On Sat, 12 Oct 2019 at 02:11, Saikat Maitra <
> > [hidden email]
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hello Emmanouil,
> > > > > > >
> > > > > > > Can you please share more info about the implementation?
> > > > > > >
> > > > > > > We have similar implementation for data streamers with Kafka
> and
> > > > > RocketMQ
> > > > > > >
> > > > > > > https://apacheignite-mix.readme.io/docs/kafka-streamer
> > > > > > >
> > > > > > > https://apacheignite-mix.readme.io/docs/rocketmq-streamer
> > > > > > >
> > > > > > > Please review and share if you have any questions.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Saikat
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Oct 11, 2019 at 4:56 PM Emmanouil Gkatziouras <
> > > > > > > [hidden email]>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > By mentioning Pub/Sub I mean Google Cloud's Pub/Sub [1]
> > > > > > > > Yes it is indeed cloud specific. It is the messaging service
> > > > provided
> > > > > > by
> > > > > > > > GCP.
> > > > > > > > I have made the integration with it, but I haven't published
> it
> > > yet
> > > > > (I
> > > > > > am
> > > > > > > > waiting for the discussion/ticket and proceed with the
> > > > contribution).
> > > > > > > >
> > > > > > > > Kind regards,
> > > > > > > > Emmanouil
> > > > > > > >
> > > > > > > > [1] https://cloud.google.com/pubsub/docs/overview
> > > > > > > >
> > > > > > > > *Emmanouil Gkatziouras*
> > > > > > > > https://egkatzioura.com/ |
> > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > > > > > > https://github.com/gkatzioura
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, 11 Oct 2019 at 21:45, Denis Magda <[hidden email]
> >
> > > > wrote:
> > > > > > > >
> > > > > > > > > Hello Emmanouil,
> > > > > > > > >
> > > > > > > > > Are you referring to any specific implementation tailored
> > for a
> > > > > > cloud?
> > > > > > > If
> > > > > > > > > I'm not mistaken you were mentioning AWS/GCE. Is there any
> > > Github
> > > > > > > source
> > > > > > > > > code?
> > > > > > > > >
> > > > > > > > > Here is a list of existing integrations with streaming and
> > > > pub/sub
> > > > > > > > > frameworks:
> > > > > > > > > https://apacheignite-mix.readme.io/docs/overview
> > > > > > > > >
> > > > > > > > > with the source code in Ignite master:
> > > > > > > > > https://github.com/apache/ignite/tree/master/modules
> > > > > > > > >
> > > > > > > > > Check out those resources to see what are the most suitable
> > > > Ignite
> > > > > > APIs
> > > > > > > > for
> > > > > > > > > such integrations.
> > > > > > > > >
> > > > > > > > > @Saikat Maitra <[hidden email]> , @Roman <
> > > > > [hidden email]
> > > > > > >
> > > > > > > ,
> > > > > > > > as
> > > > > > > > > streaming technologies contributors/maintainers, could you
> > > please
> > > > > > join
> > > > > > > > this
> > > > > > > > > thread?
> > > > > > > > >
> > > > > > > > > -
> > > > > > > > > Denis
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Oct 10, 2019 at 7:58 AM Emmanouil Gkatziouras <
> > > > > > > > > [hidden email]>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hello everyone,
> > > > > > > > > >
> > > > > > > > > > I started using Ignite lately. Part of my work involves
> > > > Pub/Sub,
> > > > > > > thus I
> > > > > > > > > > created a pub/sub streamer.
> > > > > > > > > > Pub/Sub is pretty close to Kafka.
> > > > > > > > > > Here's how the streamer works. Every node creates an
> > instance
> > > > of
> > > > > a
> > > > > > > > > Pub/Sub
> > > > > > > > > > client.
> > > > > > > > > > The clients subscribe to the same topic using the same
> > > > > subscription
> > > > > > > > name.
> > > > > > > > > > A message is being sent and one of the nodes will receive
> > the
> > > > > > > message,
> > > > > > > > > > process it and then forward it to Ignite using the
> > extractor
> > > > > > > provided.
> > > > > > > > > >
> > > > > > > > > > Any thoughts?
> > > > > > > > > >
> > > > > > > > > > *Emmanouil Gkatziouras*
> > > > > > > > > > https://egkatzioura.com/ |
> > > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > > > > > > > > https://github.com/gkatzioura
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[dmagda]

Folks,

The concept of Apache Bahir is precisely what we need! Saikat, thanks for
doing the research. Why I believe the * idea* fits us well:

   - All integrations can be stored in separate Github repositories and
   have their dev lifecycles. We've not obliged to couple all the integrations
   in a single repo. For instance, Spark can be located in a dedicated repo
   while streaming integrations might be in a single one. It's up to us.
   - All the repositories and integrations belong to ASF. We're not dumping
   them on Github but rather keep supporting and developing in accordance with
   ASF vision and practices.
   - There is a way to reward contributors via committership and PMC
   membership. You won't get this if a project is just one of the millions of
   Github projects.

Saikat, as Ignite PMC member, could you please kick-off a separate dev-list
discussion to involve more community members and referring to this thread?
I think the primary question the community needs to answer whether we
should join the Bahir or kick-off "Ignite Extensions" project?

-
Denis


On Thu, Oct 17, 2019 at 2:54 AM Alexey Zinoviev <[hidden email]>
wrote:

> Maybe we could move all our Streaming Integrations there, but what is about
> maintaining and committer permissions to the new repositories?
> I see the list of the committers and PMC members there
> https://bahir.apache.org/community-members/
>
> Could somebody from Ignite community be added to this list as a
> PMC/committer to maintain Ignite integrations?
> If yes, I'd happy to join this project and collaborate with these guys
> together
> If no, and we should start from the zero level with external PRs - hmmm,
> it's better to have our own external repository with ApacheBahirr ideology.
>
> Also, I agree, that all connectors could be moved there (in ApacheBahirr
> like repository), but not all modules from the page
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> because
> they use different parts of core modules and could be developed with
> different velocity.
>
> In reality, before creation of new repositories we need wide discussion
> based on the document mentioned above.
>
> P.S. Of course, we could experiment in the next release 2.8 with one or two
> integrations like twitter or ZeroMQ
> Also, I'd like Denis Magda idea for the separate repositories in Apache
> Github to maintain them by anybody.
>
>
> чт, 17 окт. 2019 г. в 05:02, Saikat Maitra <[hidden email]>:
>
> > Hi Denis, Emmanouil
> >
> > Thank you for your email. I think creating a separate integration project
> > in github and also proposing it as an apache incubator project will be a
> > good move. The other separate integration modules
> >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> > can
> > be moved to this new apache incubator project.
> >
> > There are similar integration available for Flink and Spark in Apache
> Bahir
> > https://bahir.apache.org/
> >
> > Do you think we should reach out to Apache Bahir community with a
> proposal
> > or we should plan to create a separate Apache Incubator project?
> >
> > Regards,
> > Saikat
> >
> >
> >
> > On Wed, Oct 16, 2019 at 6:21 AM Emmanouil Gkatziouras <
> > [hidden email]>
> > wrote:
> >
> > > Hi all!
> > >
> > > Based on the discussions most probably this is going to be on another
> > > GitHub repo. Therefore I will prepare a standalone project with the
> > Pub/Sub
> > > module and once the repository is created a pull request shall be
> issued.
> > > If there is anything else I can do in order to facilitate this please
> let
> > > me know.
> > >
> > > If a ticket is created for this issue, I shall be able to create a pull
> > > request and thus a build will take place on ignite's CI.
> > > If everyone is aligned with adding a Pub/Sub implementation may I
> create
> > a
> > > ticket on JIRA?
> > >
> > > My next question has to do on how Ignite configures the build jobs in
> > Team
> > > City. The unit tests for my implementation need a local Pub/Sub server
> up
> > > and running (they provide one for testing purposes). Will this take
> > effect
> > > in a form of a build script inside my implementation or is it something
> > > that should be configured on Team City?
> > >
> > > Kind regards,
> > > Emmanouil
> > >
> > >
> > > *Emmanouil Gkatziouras*
> > > https://egkatzioura.com/ |
> > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > https://github.com/gkatzioura
> > >
> > >
> > > On Tue, 15 Oct 2019 at 23:27, Denis Magda <[hidden email]> wrote:
> > >
> > > > Emmanouil, Saikat,
> > > >
> > > > After contemplating on this, let me propose another way with this
> > > > particular integration that is aligned with our modularization
> > endeavor.
> > > > The modularization [1] defines the Ignite core, Ignite modules, and
> > > > independent integrations. The first two are to be located in Ignite
> > > > repositories and to be tested/updated by the community for every
> > release.
> > > > The modules are a subset of existing integrations that are adopted
> > widely
> > > > and require community attention.
> > > >
> > > > As for the independent integrations [1], those are supposed to be
> > located
> > > > in a separate GitHub repo. I would advise us to start implementing
> this
> > > > concept with the Pub/Sub integration. Even though it will be located
> > in a
> > > > separate Github repo, it will be listed among officially available
> > > > integrations on the Ignite website featuring Emmanouil as a
> > contributor.
> > > >
> > > > What do you think?
> > > >
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> > > >
> > > >
> > > >
> > > > -
> > > > Denis
> > > >
> > > >
> > > > On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra <
> [hidden email]
> > >
> > > > wrote:
> > > >
> > > > > Hi Emmanouil,
> > > > >
> > > > > The changes looks good to me. I wanted to check if you are also
> able
> > to
> > > > > validate that the once data is added to Ignite cluster you are also
> > > able
> > > > to
> > > > > access it using another Ignite client or using rest endpoints.
> > > > >
> > > > > We use Teamcity for CI process, the details are as follows
> > > > >
> > > > > Trigger validation of those test suites that have been affected by
> > your
> > > > > changes on TeamCity <http://ci.ignite.apache.org/>: <
> > > > http://204.14.53.153/
> > > > > >
> > > > >
> > > > >    - Locate a test suite you have to check, press button named
> "..."
> > > that
> > > > >    is located on the left of "Run" button. "Run custom build"
> dialog
> > > will
> > > > >    appear;
> > > > >    - Go to "Changes" tab and choose
> "pull/<pull-request-number>/head"
> > > in
> > > > >    "Build branch" dropdown list;
> > > > >    - Press "Run build" button and monitor tests results.
> > > > >
> > > > >
> > > > >
> > > > > @Roman [hidden email] - can you also please take a look?
> > > > >
> > > > > Regards,
> > > > > Saikat
> > > > >
> > > > > On Sun, Oct 13, 2019 at 6:20 PM Emmanouil Gkatziouras <
> > > > > [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > Hi @Saikat Maitra
> > > > > >
> > > > > > I have my implementation on an Ignite fork on my personal GitHub
> > > > account
> > > > > > [1]
> > > > > > It is based on the Kafka implementation which is of a polling
> > nature.
> > > > > > GCP does not provide a service like Kafka. Instead it provides us
> > > with
> > > > > > Pub/Sub which is just like Kafka.
> > > > > > Therefore using the Kafka streamer as a guideline made more sense
> > to
> > > > me.
> > > > > >
> > > > > > Apart from the implementation I have also added a unit test.
> > > > > > It definitely needs some polishing and a double check on my side
> to
> > > > make
> > > > > > the review smooth.
> > > > > > For the tests a local Pub/Sub [2] server has to be setup however
> I
> > am
> > > > not
> > > > > > sure which is the best way to do so with regards to your CI/CD.
> > > > > >
> > > > > > [1]
> > https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub
> > > > > > [2] https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/
> > > > > >
> > > > > > *Emmanouil Gkatziouras*
> > > > > > https://egkatzioura.com/ |
> > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > > > > https://github.com/gkatzioura
> > > > > >
> > > > > >
> > > > > > On Sat, 12 Oct 2019 at 02:11, Saikat Maitra <
> > [hidden email]
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hello Emmanouil,
> > > > > > >
> > > > > > > Can you please share more info about the implementation?
> > > > > > >
> > > > > > > We have similar implementation for data streamers with Kafka
> and
> > > > > RocketMQ
> > > > > > >
> > > > > > > https://apacheignite-mix.readme.io/docs/kafka-streamer
> > > > > > >
> > > > > > > https://apacheignite-mix.readme.io/docs/rocketmq-streamer
> > > > > > >
> > > > > > > Please review and share if you have any questions.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Saikat
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Oct 11, 2019 at 4:56 PM Emmanouil Gkatziouras <
> > > > > > > [hidden email]>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > By mentioning Pub/Sub I mean Google Cloud's Pub/Sub [1]
> > > > > > > > Yes it is indeed cloud specific. It is the messaging service
> > > > provided
> > > > > > by
> > > > > > > > GCP.
> > > > > > > > I have made the integration with it, but I haven't published
> it
> > > yet
> > > > > (I
> > > > > > am
> > > > > > > > waiting for the discussion/ticket and proceed with the
> > > > contribution).
> > > > > > > >
> > > > > > > > Kind regards,
> > > > > > > > Emmanouil
> > > > > > > >
> > > > > > > > [1] https://cloud.google.com/pubsub/docs/overview
> > > > > > > >
> > > > > > > > *Emmanouil Gkatziouras*
> > > > > > > > https://egkatzioura.com/ |
> > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > > > > > > https://github.com/gkatzioura
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, 11 Oct 2019 at 21:45, Denis Magda <[hidden email]
> >
> > > > wrote:
> > > > > > > >
> > > > > > > > > Hello Emmanouil,
> > > > > > > > >
> > > > > > > > > Are you referring to any specific implementation tailored
> > for a
> > > > > > cloud?
> > > > > > > If
> > > > > > > > > I'm not mistaken you were mentioning AWS/GCE. Is there any
> > > Github
> > > > > > > source
> > > > > > > > > code?
> > > > > > > > >
> > > > > > > > > Here is a list of existing integrations with streaming and
> > > > pub/sub
> > > > > > > > > frameworks:
> > > > > > > > > https://apacheignite-mix.readme.io/docs/overview
> > > > > > > > >
> > > > > > > > > with the source code in Ignite master:
> > > > > > > > > https://github.com/apache/ignite/tree/master/modules
> > > > > > > > >
> > > > > > > > > Check out those resources to see what are the most suitable
> > > > Ignite
> > > > > > APIs
> > > > > > > > for
> > > > > > > > > such integrations.
> > > > > > > > >
> > > > > > > > > @Saikat Maitra <[hidden email]> , @Roman <
> > > > > [hidden email]
> > > > > > >
> > > > > > > ,
> > > > > > > > as
> > > > > > > > > streaming technologies contributors/maintainers, could you
> > > please
> > > > > > join
> > > > > > > > this
> > > > > > > > > thread?
> > > > > > > > >
> > > > > > > > > -
> > > > > > > > > Denis
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Oct 10, 2019 at 7:58 AM Emmanouil Gkatziouras <
> > > > > > > > > [hidden email]>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hello everyone,
> > > > > > > > > >
> > > > > > > > > > I started using Ignite lately. Part of my work involves
> > > > Pub/Sub,
> > > > > > > thus I
> > > > > > > > > > created a pub/sub streamer.
> > > > > > > > > > Pub/Sub is pretty close to Kafka.
> > > > > > > > > > Here's how the streamer works. Every node creates an
> > instance
> > > > of
> > > > > a
> > > > > > > > > Pub/Sub
> > > > > > > > > > client.
> > > > > > > > > > The clients subscribe to the same topic using the same
> > > > > subscription
> > > > > > > > name.
> > > > > > > > > > A message is being sent and one of the nodes will receive
> > the
> > > > > > > message,
> > > > > > > > > > process it and then forward it to Ignite using the
> > extractor
> > > > > > > provided.
> > > > > > > > > >
> > > > > > > > > > Any thoughts?
> > > > > > > > > >
> > > > > > > > > > *Emmanouil Gkatziouras*
> > > > > > > > > > https://egkatzioura.com/ |
> > > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > > > > > > > > https://github.com/gkatzioura
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Saikat Maitra]

Hello Denis,

Sure, sounds good. I will create a separate discussion thread to get
community feedback on whether we
should join the Bahir or kick-off "Ignite Extensions" project.

Regards,
Saikat

On Thu, Oct 17, 2019 at 2:21 PM Denis Magda <[hidden email]> wrote:

> Folks,
>
> The concept of Apache Bahir is precisely what we need! Saikat, thanks for
> doing the research. Why I believe the * idea* fits us well:
>
>    - All integrations can be stored in separate Github repositories and
>    have their dev lifecycles. We've not obliged to couple all the
> integrations
>    in a single repo. For instance, Spark can be located in a dedicated repo
>    while streaming integrations might be in a single one. It's up to us.
>    - All the repositories and integrations belong to ASF. We're not dumping
>    them on Github but rather keep supporting and developing in accordance
> with
>    ASF vision and practices.
>    - There is a way to reward contributors via committership and PMC
>    membership. You won't get this if a project is just one of the millions
> of
>    Github projects.
>
> Saikat, as Ignite PMC member, could you please kick-off a separate dev-list
> discussion to involve more community members and referring to this thread?
> I think the primary question the community needs to answer whether we
> should join the Bahir or kick-off "Ignite Extensions" project?
>
> -
> Denis
>
>
> On Thu, Oct 17, 2019 at 2:54 AM Alexey Zinoviev <[hidden email]>
> wrote:
>
> > Maybe we could move all our Streaming Integrations there, but what is
> about
> > maintaining and committer permissions to the new repositories?
> > I see the list of the committers and PMC members there
> > https://bahir.apache.org/community-members/
> >
> > Could somebody from Ignite community be added to this list as a
> > PMC/committer to maintain Ignite integrations?
> > If yes, I'd happy to join this project and collaborate with these guys
> > together
> > If no, and we should start from the zero level with external PRs - hmmm,
> > it's better to have our own external repository with ApacheBahirr
> ideology.
> >
> > Also, I agree, that all connectors could be moved there (in ApacheBahirr
> > like repository), but not all modules from the page
> >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> > because
> > they use different parts of core modules and could be developed with
> > different velocity.
> >
> > In reality, before creation of new repositories we need wide discussion
> > based on the document mentioned above.
> >
> > P.S. Of course, we could experiment in the next release 2.8 with one or
> two
> > integrations like twitter or ZeroMQ
> > Also, I'd like Denis Magda idea for the separate repositories in Apache
> > Github to maintain them by anybody.
> >
> >
> > чт, 17 окт. 2019 г. в 05:02, Saikat Maitra <[hidden email]>:
> >
> > > Hi Denis, Emmanouil
> > >
> > > Thank you for your email. I think creating a separate integration
> project
> > > in github and also proposing it as an apache incubator project will be
> a
> > > good move. The other separate integration modules
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> > > can
> > > be moved to this new apache incubator project.
> > >
> > > There are similar integration available for Flink and Spark in Apache
> > Bahir
> > > https://bahir.apache.org/
> > >
> > > Do you think we should reach out to Apache Bahir community with a
> > proposal
> > > or we should plan to create a separate Apache Incubator project?
> > >
> > > Regards,
> > > Saikat
> > >
> > >
> > >
> > > On Wed, Oct 16, 2019 at 6:21 AM Emmanouil Gkatziouras <
> > > [hidden email]>
> > > wrote:
> > >
> > > > Hi all!
> > > >
> > > > Based on the discussions most probably this is going to be on another
> > > > GitHub repo. Therefore I will prepare a standalone project with the
> > > Pub/Sub
> > > > module and once the repository is created a pull request shall be
> > issued.
> > > > If there is anything else I can do in order to facilitate this please
> > let
> > > > me know.
> > > >
> > > > If a ticket is created for this issue, I shall be able to create a
> pull
> > > > request and thus a build will take place on ignite's CI.
> > > > If everyone is aligned with adding a Pub/Sub implementation may I
> > create
> > > a
> > > > ticket on JIRA?
> > > >
> > > > My next question has to do on how Ignite configures the build jobs in
> > > Team
> > > > City. The unit tests for my implementation need a local Pub/Sub
> server
> > up
> > > > and running (they provide one for testing purposes). Will this take
> > > effect
> > > > in a form of a build script inside my implementation or is it
> something
> > > > that should be configured on Team City?
> > > >
> > > > Kind regards,
> > > > Emmanouil
> > > >
> > > >
> > > > *Emmanouil Gkatziouras*
> > > > https://egkatzioura.com/ |
> > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > > https://github.com/gkatzioura
> > > >
> > > >
> > > > On Tue, 15 Oct 2019 at 23:27, Denis Magda <[hidden email]> wrote:
> > > >
> > > > > Emmanouil, Saikat,
> > > > >
> > > > > After contemplating on this, let me propose another way with this
> > > > > particular integration that is aligned with our modularization
> > > endeavor.
> > > > > The modularization [1] defines the Ignite core, Ignite modules, and
> > > > > independent integrations. The first two are to be located in Ignite
> > > > > repositories and to be tested/updated by the community for every
> > > release.
> > > > > The modules are a subset of existing integrations that are adopted
> > > widely
> > > > > and require community attention.
> > > > >
> > > > > As for the independent integrations [1], those are supposed to be
> > > located
> > > > > in a separate GitHub repo. I would advise us to start implementing
> > this
> > > > > concept with the Pub/Sub integration. Even though it will be
> located
> > > in a
> > > > > separate Github repo, it will be listed among officially available
> > > > > integrations on the Ignite website featuring Emmanouil as a
> > > contributor.
> > > > >
> > > > > What do you think?
> > > > >
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> > > > >
> > > > >
> > > > >
> > > > > -
> > > > > Denis
> > > > >
> > > > >
> > > > > On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra <
> > [hidden email]
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Emmanouil,
> > > > > >
> > > > > > The changes looks good to me. I wanted to check if you are also
> > able
> > > to
> > > > > > validate that the once data is added to Ignite cluster you are
> also
> > > > able
> > > > > to
> > > > > > access it using another Ignite client or using rest endpoints.
> > > > > >
> > > > > > We use Teamcity for CI process, the details are as follows
> > > > > >
> > > > > > Trigger validation of those test suites that have been affected
> by
> > > your
> > > > > > changes on TeamCity <http://ci.ignite.apache.org/>: <
> > > > > http://204.14.53.153/
> > > > > > >
> > > > > >
> > > > > >    - Locate a test suite you have to check, press button named
> > "..."
> > > > that
> > > > > >    is located on the left of "Run" button. "Run custom build"
> > dialog
> > > > will
> > > > > >    appear;
> > > > > >    - Go to "Changes" tab and choose
> > "pull/<pull-request-number>/head"
> > > > in
> > > > > >    "Build branch" dropdown list;
> > > > > >    - Press "Run build" button and monitor tests results.
> > > > > >
> > > > > >
> > > > > >
> > > > > > @Roman [hidden email] - can you also please take a look?
> > > > > >
> > > > > > Regards,
> > > > > > Saikat
> > > > > >
> > > > > > On Sun, Oct 13, 2019 at 6:20 PM Emmanouil Gkatziouras <
> > > > > > [hidden email]>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi @Saikat Maitra
> > > > > > >
> > > > > > > I have my implementation on an Ignite fork on my personal
> GitHub
> > > > > account
> > > > > > > [1]
> > > > > > > It is based on the Kafka implementation which is of a polling
> > > nature.
> > > > > > > GCP does not provide a service like Kafka. Instead it provides
> us
> > > > with
> > > > > > > Pub/Sub which is just like Kafka.
> > > > > > > Therefore using the Kafka streamer as a guideline made more
> sense
> > > to
> > > > > me.
> > > > > > >
> > > > > > > Apart from the implementation I have also added a unit test.
> > > > > > > It definitely needs some polishing and a double check on my
> side
> > to
> > > > > make
> > > > > > > the review smooth.
> > > > > > > For the tests a local Pub/Sub [2] server has to be setup
> however
> > I
> > > am
> > > > > not
> > > > > > > sure which is the best way to do so with regards to your CI/CD.
> > > > > > >
> > > > > > > [1]
> > > https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub
> > > > > > > [2] https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/
> > > > > > >
> > > > > > > *Emmanouil Gkatziouras*
> > > > > > > https://egkatzioura.com/ |
> > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > > > > > https://github.com/gkatzioura
> > > > > > >
> > > > > > >
> > > > > > > On Sat, 12 Oct 2019 at 02:11, Saikat Maitra <
> > > [hidden email]
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello Emmanouil,
> > > > > > > >
> > > > > > > > Can you please share more info about the implementation?
> > > > > > > >
> > > > > > > > We have similar implementation for data streamers with Kafka
> > and
> > > > > > RocketMQ
> > > > > > > >
> > > > > > > > https://apacheignite-mix.readme.io/docs/kafka-streamer
> > > > > > > >
> > > > > > > > https://apacheignite-mix.readme.io/docs/rocketmq-streamer
> > > > > > > >
> > > > > > > > Please review and share if you have any questions.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Saikat
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Oct 11, 2019 at 4:56 PM Emmanouil Gkatziouras <
> > > > > > > > [hidden email]>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > By mentioning Pub/Sub I mean Google Cloud's Pub/Sub [1]
> > > > > > > > > Yes it is indeed cloud specific. It is the messaging
> service
> > > > > provided
> > > > > > > by
> > > > > > > > > GCP.
> > > > > > > > > I have made the integration with it, but I haven't
> published
> > it
> > > > yet
> > > > > > (I
> > > > > > > am
> > > > > > > > > waiting for the discussion/ticket and proceed with the
> > > > > contribution).
> > > > > > > > >
> > > > > > > > > Kind regards,
> > > > > > > > > Emmanouil
> > > > > > > > >
> > > > > > > > > [1] https://cloud.google.com/pubsub/docs/overview
> > > > > > > > >
> > > > > > > > > *Emmanouil Gkatziouras*
> > > > > > > > > https://egkatzioura.com/ |
> > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > > > > > > > https://github.com/gkatzioura
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, 11 Oct 2019 at 21:45, Denis Magda <
> [hidden email]
> > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hello Emmanouil,
> > > > > > > > > >
> > > > > > > > > > Are you referring to any specific implementation tailored
> > > for a
> > > > > > > cloud?
> > > > > > > > If
> > > > > > > > > > I'm not mistaken you were mentioning AWS/GCE. Is there
> any
> > > > Github
> > > > > > > > source
> > > > > > > > > > code?
> > > > > > > > > >
> > > > > > > > > > Here is a list of existing integrations with streaming
> and
> > > > > pub/sub
> > > > > > > > > > frameworks:
> > > > > > > > > > https://apacheignite-mix.readme.io/docs/overview
> > > > > > > > > >
> > > > > > > > > > with the source code in Ignite master:
> > > > > > > > > > https://github.com/apache/ignite/tree/master/modules
> > > > > > > > > >
> > > > > > > > > > Check out those resources to see what are the most
> suitable
> > > > > Ignite
> > > > > > > APIs
> > > > > > > > > for
> > > > > > > > > > such integrations.
> > > > > > > > > >
> > > > > > > > > > @Saikat Maitra <[hidden email]> , @Roman <
> > > > > > [hidden email]
> > > > > > > >
> > > > > > > > ,
> > > > > > > > > as
> > > > > > > > > > streaming technologies contributors/maintainers, could
> you
> > > > please
> > > > > > > join
> > > > > > > > > this
> > > > > > > > > > thread?
> > > > > > > > > >
> > > > > > > > > > -
> > > > > > > > > > Denis
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Oct 10, 2019 at 7:58 AM Emmanouil Gkatziouras <
> > > > > > > > > > [hidden email]>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hello everyone,
> > > > > > > > > > >
> > > > > > > > > > > I started using Ignite lately. Part of my work involves
> > > > > Pub/Sub,
> > > > > > > > thus I
> > > > > > > > > > > created a pub/sub streamer.
> > > > > > > > > > > Pub/Sub is pretty close to Kafka.
> > > > > > > > > > > Here's how the streamer works. Every node creates an
> > > instance
> > > > > of
> > > > > > a
> > > > > > > > > > Pub/Sub
> > > > > > > > > > > client.
> > > > > > > > > > > The clients subscribe to the same topic using the same
> > > > > > subscription
> > > > > > > > > name.
> > > > > > > > > > > A message is being sent and one of the nodes will
> receive
> > > the
> > > > > > > > message,
> > > > > > > > > > > process it and then forward it to Ignite using the
> > > extractor
> > > > > > > > provided.
> > > > > > > > > > >
> > > > > > > > > > > Any thoughts?
> > > > > > > > > > >
> > > > > > > > > > > *Emmanouil Gkatziouras*
> > > > > > > > > > > https://egkatzioura.com/ |
> > > > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > > > > > > > > > https://github.com/gkatzioura
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Saikat Maitra]

Hello,

As discussed I have created following discussion threads on our migration
proposals for Apache Ignite extensions:

http://apache-ignite-users.70518.x6.nabble.com/DISCUSS-Proposal-for-Ignite-Extensions-as-a-separate-Bahir-module-or-Incubator-project-td29829.html

https://www.mail-archive.com/dev@.../msg02703.html

I will share update as I hear more information.

Regards,
Saikat



On Fri, Oct 18, 2019 at 9:03 PM Saikat Maitra <[hidden email]>
wrote:

> Hello Denis,
>
> Sure, sounds good. I will create a separate discussion thread to get
> community feedback on whether we
> should join the Bahir or kick-off "Ignite Extensions" project.
>
> Regards,
> Saikat
>
> On Thu, Oct 17, 2019 at 2:21 PM Denis Magda <[hidden email]> wrote:
>
>> Folks,
>>
>> The concept of Apache Bahir is precisely what we need! Saikat, thanks for
>> doing the research. Why I believe the * idea* fits us well:
>>
>>    - All integrations can be stored in separate Github repositories and
>>    have their dev lifecycles. We've not obliged to couple all the
>> integrations
>>    in a single repo. For instance, Spark can be located in a dedicated
>> repo
>>    while streaming integrations might be in a single one. It's up to us.
>>    - All the repositories and integrations belong to ASF. We're not
>> dumping
>>    them on Github but rather keep supporting and developing in accordance
>> with
>>    ASF vision and practices.
>>    - There is a way to reward contributors via committership and PMC
>>    membership. You won't get this if a project is just one of the
>> millions of
>>    Github projects.
>>
>> Saikat, as Ignite PMC member, could you please kick-off a separate
>> dev-list
>> discussion to involve more community members and referring to this thread?
>> I think the primary question the community needs to answer whether we
>> should join the Bahir or kick-off "Ignite Extensions" project?
>>
>> -
>> Denis
>>
>>
>> On Thu, Oct 17, 2019 at 2:54 AM Alexey Zinoviev <[hidden email]>
>> wrote:
>>
>> > Maybe we could move all our Streaming Integrations there, but what is
>> about
>> > maintaining and committer permissions to the new repositories?
>> > I see the list of the committers and PMC members there
>> > https://bahir.apache.org/community-members/
>> >
>> > Could somebody from Ignite community be added to this list as a
>> > PMC/committer to maintain Ignite integrations?
>> > If yes, I'd happy to join this project and collaborate with these guys
>> > together
>> > If no, and we should start from the zero level with external PRs - hmmm,
>> > it's better to have our own external repository with ApacheBahirr
>> ideology.
>> >
>> > Also, I agree, that all connectors could be moved there (in ApacheBahirr
>> > like repository), but not all modules from the page
>> >
>> >
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
>> > because
>> > they use different parts of core modules and could be developed with
>> > different velocity.
>> >
>> > In reality, before creation of new repositories we need wide discussion
>> > based on the document mentioned above.
>> >
>> > P.S. Of course, we could experiment in the next release 2.8 with one or
>> two
>> > integrations like twitter or ZeroMQ
>> > Also, I'd like Denis Magda idea for the separate repositories in Apache
>> > Github to maintain them by anybody.
>> >
>> >
>> > чт, 17 окт. 2019 г. в 05:02, Saikat Maitra <[hidden email]>:
>> >
>> > > Hi Denis, Emmanouil
>> > >
>> > > Thank you for your email. I think creating a separate integration
>> project
>> > > in github and also proposing it as an apache incubator project will
>> be a
>> > > good move. The other separate integration modules
>> > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
>> > > can
>> > > be moved to this new apache incubator project.
>> > >
>> > > There are similar integration available for Flink and Spark in Apache
>> > Bahir
>> > > https://bahir.apache.org/
>> > >
>> > > Do you think we should reach out to Apache Bahir community with a
>> > proposal
>> > > or we should plan to create a separate Apache Incubator project?
>> > >
>> > > Regards,
>> > > Saikat
>> > >
>> > >
>> > >
>> > > On Wed, Oct 16, 2019 at 6:21 AM Emmanouil Gkatziouras <
>> > > [hidden email]>
>> > > wrote:
>> > >
>> > > > Hi all!
>> > > >
>> > > > Based on the discussions most probably this is going to be on
>> another
>> > > > GitHub repo. Therefore I will prepare a standalone project with the
>> > > Pub/Sub
>> > > > module and once the repository is created a pull request shall be
>> > issued.
>> > > > If there is anything else I can do in order to facilitate this
>> please
>> > let
>> > > > me know.
>> > > >
>> > > > If a ticket is created for this issue, I shall be able to create a
>> pull
>> > > > request and thus a build will take place on ignite's CI.
>> > > > If everyone is aligned with adding a Pub/Sub implementation may I
>> > create
>> > > a
>> > > > ticket on JIRA?
>> > > >
>> > > > My next question has to do on how Ignite configures the build jobs
>> in
>> > > Team
>> > > > City. The unit tests for my implementation need a local Pub/Sub
>> server
>> > up
>> > > > and running (they provide one for testing purposes). Will this take
>> > > effect
>> > > > in a form of a build script inside my implementation or is it
>> something
>> > > > that should be configured on Team City?
>> > > >
>> > > > Kind regards,
>> > > > Emmanouil
>> > > >
>> > > >
>> > > > *Emmanouil Gkatziouras*
>> > > > https://egkatzioura.com/ |
>> > > > https://www.linkedin.com/in/gkatziourasemmanouil/
>> > > > https://github.com/gkatzioura
>> > > >
>> > > >
>> > > > On Tue, 15 Oct 2019 at 23:27, Denis Magda <[hidden email]>
>> wrote:
>> > > >
>> > > > > Emmanouil, Saikat,
>> > > > >
>> > > > > After contemplating on this, let me propose another way with this
>> > > > > particular integration that is aligned with our modularization
>> > > endeavor.
>> > > > > The modularization [1] defines the Ignite core, Ignite modules,
>> and
>> > > > > independent integrations. The first two are to be located in
>> Ignite
>> > > > > repositories and to be tested/updated by the community for every
>> > > release.
>> > > > > The modules are a subset of existing integrations that are adopted
>> > > widely
>> > > > > and require community attention.
>> > > > >
>> > > > > As for the independent integrations [1], those are supposed to be
>> > > located
>> > > > > in a separate GitHub repo. I would advise us to start implementing
>> > this
>> > > > > concept with the Pub/Sub integration. Even though it will be
>> located
>> > > in a
>> > > > > separate Github repo, it will be listed among officially available
>> > > > > integrations on the Ignite website featuring Emmanouil as a
>> > > contributor.
>> > > > >
>> > > > > What do you think?
>> > > > >
>> > > > >
>> > > > > [1]
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
>> > > > >
>> > > > >
>> > > > >
>> > > > > -
>> > > > > Denis
>> > > > >
>> > > > >
>> > > > > On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra <
>> > [hidden email]
>> > > >
>> > > > > wrote:
>> > > > >
>> > > > > > Hi Emmanouil,
>> > > > > >
>> > > > > > The changes looks good to me. I wanted to check if you are also
>> > able
>> > > to
>> > > > > > validate that the once data is added to Ignite cluster you are
>> also
>> > > > able
>> > > > > to
>> > > > > > access it using another Ignite client or using rest endpoints.
>> > > > > >
>> > > > > > We use Teamcity for CI process, the details are as follows
>> > > > > >
>> > > > > > Trigger validation of those test suites that have been affected
>> by
>> > > your
>> > > > > > changes on TeamCity <http://ci.ignite.apache.org/>: <
>> > > > > http://204.14.53.153/
>> > > > > > >
>> > > > > >
>> > > > > >    - Locate a test suite you have to check, press button named
>> > "..."
>> > > > that
>> > > > > >    is located on the left of "Run" button. "Run custom build"
>> > dialog
>> > > > will
>> > > > > >    appear;
>> > > > > >    - Go to "Changes" tab and choose
>> > "pull/<pull-request-number>/head"
>> > > > in
>> > > > > >    "Build branch" dropdown list;
>> > > > > >    - Press "Run build" button and monitor tests results.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > @Roman [hidden email] - can you also please take a look?
>> > > > > >
>> > > > > > Regards,
>> > > > > > Saikat
>> > > > > >
>> > > > > > On Sun, Oct 13, 2019 at 6:20 PM Emmanouil Gkatziouras <
>> > > > > > [hidden email]>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi @Saikat Maitra
>> > > > > > >
>> > > > > > > I have my implementation on an Ignite fork on my personal
>> GitHub
>> > > > > account
>> > > > > > > [1]
>> > > > > > > It is based on the Kafka implementation which is of a polling
>> > > nature.
>> > > > > > > GCP does not provide a service like Kafka. Instead it
>> provides us
>> > > > with
>> > > > > > > Pub/Sub which is just like Kafka.
>> > > > > > > Therefore using the Kafka streamer as a guideline made more
>> sense
>> > > to
>> > > > > me.
>> > > > > > >
>> > > > > > > Apart from the implementation I have also added a unit test.
>> > > > > > > It definitely needs some polishing and a double check on my
>> side
>> > to
>> > > > > make
>> > > > > > > the review smooth.
>> > > > > > > For the tests a local Pub/Sub [2] server has to be setup
>> however
>> > I
>> > > am
>> > > > > not
>> > > > > > > sure which is the best way to do so with regards to your
>> CI/CD.
>> > > > > > >
>> > > > > > > [1]
>> > > https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub
>> > > > > > > [2]
>> https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/
>> > > > > > >
>> > > > > > > *Emmanouil Gkatziouras*
>> > > > > > > https://egkatzioura.com/ |
>> > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
>> > > > > > > https://github.com/gkatzioura
>> > > > > > >
>> > > > > > >
>> > > > > > > On Sat, 12 Oct 2019 at 02:11, Saikat Maitra <
>> > > [hidden email]
>> > > > >
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > Hello Emmanouil,
>> > > > > > > >
>> > > > > > > > Can you please share more info about the implementation?
>> > > > > > > >
>> > > > > > > > We have similar implementation for data streamers with Kafka
>> > and
>> > > > > > RocketMQ
>> > > > > > > >
>> > > > > > > > https://apacheignite-mix.readme.io/docs/kafka-streamer
>> > > > > > > >
>> > > > > > > > https://apacheignite-mix.readme.io/docs/rocketmq-streamer
>> > > > > > > >
>> > > > > > > > Please review and share if you have any questions.
>> > > > > > > >
>> > > > > > > > Regards,
>> > > > > > > > Saikat
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Fri, Oct 11, 2019 at 4:56 PM Emmanouil Gkatziouras <
>> > > > > > > > [hidden email]>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Hi all,
>> > > > > > > > >
>> > > > > > > > > By mentioning Pub/Sub I mean Google Cloud's Pub/Sub [1]
>> > > > > > > > > Yes it is indeed cloud specific. It is the messaging
>> service
>> > > > > provided
>> > > > > > > by
>> > > > > > > > > GCP.
>> > > > > > > > > I have made the integration with it, but I haven't
>> published
>> > it
>> > > > yet
>> > > > > > (I
>> > > > > > > am
>> > > > > > > > > waiting for the discussion/ticket and proceed with the
>> > > > > contribution).
>> > > > > > > > >
>> > > > > > > > > Kind regards,
>> > > > > > > > > Emmanouil
>> > > > > > > > >
>> > > > > > > > > [1] https://cloud.google.com/pubsub/docs/overview
>> > > > > > > > >
>> > > > > > > > > *Emmanouil Gkatziouras*
>> > > > > > > > > https://egkatzioura.com/ |
>> > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
>> > > > > > > > > https://github.com/gkatzioura
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Fri, 11 Oct 2019 at 21:45, Denis Magda <
>> [hidden email]
>> > >
>> > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Hello Emmanouil,
>> > > > > > > > > >
>> > > > > > > > > > Are you referring to any specific implementation
>> tailored
>> > > for a
>> > > > > > > cloud?
>> > > > > > > > If
>> > > > > > > > > > I'm not mistaken you were mentioning AWS/GCE. Is there
>> any
>> > > > Github
>> > > > > > > > source
>> > > > > > > > > > code?
>> > > > > > > > > >
>> > > > > > > > > > Here is a list of existing integrations with streaming
>> and
>> > > > > pub/sub
>> > > > > > > > > > frameworks:
>> > > > > > > > > > https://apacheignite-mix.readme.io/docs/overview
>> > > > > > > > > >
>> > > > > > > > > > with the source code in Ignite master:
>> > > > > > > > > > https://github.com/apache/ignite/tree/master/modules
>> > > > > > > > > >
>> > > > > > > > > > Check out those resources to see what are the most
>> suitable
>> > > > > Ignite
>> > > > > > > APIs
>> > > > > > > > > for
>> > > > > > > > > > such integrations.
>> > > > > > > > > >
>> > > > > > > > > > @Saikat Maitra <[hidden email]> , @Roman <
>> > > > > > [hidden email]
>> > > > > > > >
>> > > > > > > > ,
>> > > > > > > > > as
>> > > > > > > > > > streaming technologies contributors/maintainers, could
>> you
>> > > > please
>> > > > > > > join
>> > > > > > > > > this
>> > > > > > > > > > thread?
>> > > > > > > > > >
>> > > > > > > > > > -
>> > > > > > > > > > Denis
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On Thu, Oct 10, 2019 at 7:58 AM Emmanouil Gkatziouras <
>> > > > > > > > > > [hidden email]>
>> > > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Hello everyone,
>> > > > > > > > > > >
>> > > > > > > > > > > I started using Ignite lately. Part of my work
>> involves
>> > > > > Pub/Sub,
>> > > > > > > > thus I
>> > > > > > > > > > > created a pub/sub streamer.
>> > > > > > > > > > > Pub/Sub is pretty close to Kafka.
>> > > > > > > > > > > Here's how the streamer works. Every node creates an
>> > > instance
>> > > > > of
>> > > > > > a
>> > > > > > > > > > Pub/Sub
>> > > > > > > > > > > client.
>> > > > > > > > > > > The clients subscribe to the same topic using the same
>> > > > > > subscription
>> > > > > > > > > name.
>> > > > > > > > > > > A message is being sent and one of the nodes will
>> receive
>> > > the
>> > > > > > > > message,
>> > > > > > > > > > > process it and then forward it to Ignite using the
>> > > extractor
>> > > > > > > > provided.
>> > > > > > > > > > >
>> > > > > > > > > > > Any thoughts?
>> > > > > > > > > > >
>> > > > > > > > > > > *Emmanouil Gkatziouras*
>> > > > > > > > > > > https://egkatzioura.com/ |
>> > > > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
>> > > > > > > > > > > https://github.com/gkatzioura
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

[Saikat Maitra]

Hello Emmanouil,

As discussed earlier, I discussed with Apache Bahir community and they are
interested to have Apache Ignite extensions as part of Apache Bahir project.

Can you please share Pub/Sub streamer implementation details with Apache
Bahir community and request for feedback.

https://bahir.apache.org/

I have also requested for contributor access in Jira for Apache Bahir
project so that I can create issues and assign to myself.

I can help with code reviews as well.

Here is the email thread for reference
https://www.mail-archive.com/dev@.../msg02703.html

Regards,
Saikat







On Mon, Oct 21, 2019 at 7:54 PM Saikat Maitra <[hidden email]>
wrote:

> Hello,
>
> As discussed I have created following discussion threads on our migration
> proposals for Apache Ignite extensions:
>
>
> http://apache-ignite-users.70518.x6.nabble.com/DISCUSS-Proposal-for-Ignite-Extensions-as-a-separate-Bahir-module-or-Incubator-project-td29829.html
>
> https://www.mail-archive.com/dev@.../msg02703.html
>
> I will share update as I hear more information.
>
> Regards,
> Saikat
>
>
>
> On Fri, Oct 18, 2019 at 9:03 PM Saikat Maitra <[hidden email]>
> wrote:
>
>> Hello Denis,
>>
>> Sure, sounds good. I will create a separate discussion thread to get
>> community feedback on whether we
>> should join the Bahir or kick-off "Ignite Extensions" project.
>>
>> Regards,
>> Saikat
>>
>> On Thu, Oct 17, 2019 at 2:21 PM Denis Magda <[hidden email]> wrote:
>>
>>> Folks,
>>>
>>> The concept of Apache Bahir is precisely what we need! Saikat, thanks for
>>> doing the research. Why I believe the * idea* fits us well:
>>>
>>>    - All integrations can be stored in separate Github repositories and
>>>    have their dev lifecycles. We've not obliged to couple all the
>>> integrations
>>>    in a single repo. For instance, Spark can be located in a dedicated
>>> repo
>>>    while streaming integrations might be in a single one. It's up to us.
>>>    - All the repositories and integrations belong to ASF. We're not
>>> dumping
>>>    them on Github but rather keep supporting and developing in
>>> accordance with
>>>    ASF vision and practices.
>>>    - There is a way to reward contributors via committership and PMC
>>>    membership. You won't get this if a project is just one of the
>>> millions of
>>>    Github projects.
>>>
>>> Saikat, as Ignite PMC member, could you please kick-off a separate
>>> dev-list
>>> discussion to involve more community members and referring to this
>>> thread?
>>> I think the primary question the community needs to answer whether we
>>> should join the Bahir or kick-off "Ignite Extensions" project?
>>>
>>> -
>>> Denis
>>>
>>>
>>> On Thu, Oct 17, 2019 at 2:54 AM Alexey Zinoviev <[hidden email]>
>>> wrote:
>>>
>>> > Maybe we could move all our Streaming Integrations there, but what is
>>> about
>>> > maintaining and committer permissions to the new repositories?
>>> > I see the list of the committers and PMC members there
>>> > https://bahir.apache.org/community-members/
>>> >
>>> > Could somebody from Ignite community be added to this list as a
>>> > PMC/committer to maintain Ignite integrations?
>>> > If yes, I'd happy to join this project and collaborate with these guys
>>> > together
>>> > If no, and we should start from the zero level with external PRs -
>>> hmmm,
>>> > it's better to have our own external repository with ApacheBahirr
>>> ideology.
>>> >
>>> > Also, I agree, that all connectors could be moved there (in
>>> ApacheBahirr
>>> > like repository), but not all modules from the page
>>> >
>>> >
>>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
>>> > because
>>> > they use different parts of core modules and could be developed with
>>> > different velocity.
>>> >
>>> > In reality, before creation of new repositories we need wide discussion
>>> > based on the document mentioned above.
>>> >
>>> > P.S. Of course, we could experiment in the next release 2.8 with one
>>> or two
>>> > integrations like twitter or ZeroMQ
>>> > Also, I'd like Denis Magda idea for the separate repositories in Apache
>>> > Github to maintain them by anybody.
>>> >
>>> >
>>> > чт, 17 окт. 2019 г. в 05:02, Saikat Maitra <[hidden email]>:
>>> >
>>> > > Hi Denis, Emmanouil
>>> > >
>>> > > Thank you for your email. I think creating a separate integration
>>> project
>>> > > in github and also proposing it as an apache incubator project will
>>> be a
>>> > > good move. The other separate integration modules
>>> > >
>>> > >
>>> >
>>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
>>> > > can
>>> > > be moved to this new apache incubator project.
>>> > >
>>> > > There are similar integration available for Flink and Spark in Apache
>>> > Bahir
>>> > > https://bahir.apache.org/
>>> > >
>>> > > Do you think we should reach out to Apache Bahir community with a
>>> > proposal
>>> > > or we should plan to create a separate Apache Incubator project?
>>> > >
>>> > > Regards,
>>> > > Saikat
>>> > >
>>> > >
>>> > >
>>> > > On Wed, Oct 16, 2019 at 6:21 AM Emmanouil Gkatziouras <
>>> > > [hidden email]>
>>> > > wrote:
>>> > >
>>> > > > Hi all!
>>> > > >
>>> > > > Based on the discussions most probably this is going to be on
>>> another
>>> > > > GitHub repo. Therefore I will prepare a standalone project with the
>>> > > Pub/Sub
>>> > > > module and once the repository is created a pull request shall be
>>> > issued.
>>> > > > If there is anything else I can do in order to facilitate this
>>> please
>>> > let
>>> > > > me know.
>>> > > >
>>> > > > If a ticket is created for this issue, I shall be able to create a
>>> pull
>>> > > > request and thus a build will take place on ignite's CI.
>>> > > > If everyone is aligned with adding a Pub/Sub implementation may I
>>> > create
>>> > > a
>>> > > > ticket on JIRA?
>>> > > >
>>> > > > My next question has to do on how Ignite configures the build jobs
>>> in
>>> > > Team
>>> > > > City. The unit tests for my implementation need a local Pub/Sub
>>> server
>>> > up
>>> > > > and running (they provide one for testing purposes). Will this take
>>> > > effect
>>> > > > in a form of a build script inside my implementation or is it
>>> something
>>> > > > that should be configured on Team City?
>>> > > >
>>> > > > Kind regards,
>>> > > > Emmanouil
>>> > > >
>>> > > >
>>> > > > *Emmanouil Gkatziouras*
>>> > > > https://egkatzioura.com/ |
>>> > > > https://www.linkedin.com/in/gkatziourasemmanouil/
>>> > > > https://github.com/gkatzioura
>>> > > >
>>> > > >
>>> > > > On Tue, 15 Oct 2019 at 23:27, Denis Magda <[hidden email]>
>>> wrote:
>>> > > >
>>> > > > > Emmanouil, Saikat,
>>> > > > >
>>> > > > > After contemplating on this, let me propose another way with this
>>> > > > > particular integration that is aligned with our modularization
>>> > > endeavor.
>>> > > > > The modularization [1] defines the Ignite core, Ignite modules,
>>> and
>>> > > > > independent integrations. The first two are to be located in
>>> Ignite
>>> > > > > repositories and to be tested/updated by the community for every
>>> > > release.
>>> > > > > The modules are a subset of existing integrations that are
>>> adopted
>>> > > widely
>>> > > > > and require community attention.
>>> > > > >
>>> > > > > As for the independent integrations [1], those are supposed to be
>>> > > located
>>> > > > > in a separate GitHub repo. I would advise us to start
>>> implementing
>>> > this
>>> > > > > concept with the Pub/Sub integration. Even though it will be
>>> located
>>> > > in a
>>> > > > > separate Github repo, it will be listed among officially
>>> available
>>> > > > > integrations on the Ignite website featuring Emmanouil as a
>>> > > contributor.
>>> > > > >
>>> > > > > What do you think?
>>> > > > >
>>> > > > >
>>> > > > > [1]
>>> > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > -
>>> > > > > Denis
>>> > > > >
>>> > > > >
>>> > > > > On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra <
>>> > [hidden email]
>>> > > >
>>> > > > > wrote:
>>> > > > >
>>> > > > > > Hi Emmanouil,
>>> > > > > >
>>> > > > > > The changes looks good to me. I wanted to check if you are also
>>> > able
>>> > > to
>>> > > > > > validate that the once data is added to Ignite cluster you are
>>> also
>>> > > > able
>>> > > > > to
>>> > > > > > access it using another Ignite client or using rest endpoints.
>>> > > > > >
>>> > > > > > We use Teamcity for CI process, the details are as follows
>>> > > > > >
>>> > > > > > Trigger validation of those test suites that have been
>>> affected by
>>> > > your
>>> > > > > > changes on TeamCity <http://ci.ignite.apache.org/>: <
>>> > > > > http://204.14.53.153/
>>> > > > > > >
>>> > > > > >
>>> > > > > >    - Locate a test suite you have to check, press button named
>>> > "..."
>>> > > > that
>>> > > > > >    is located on the left of "Run" button. "Run custom build"
>>> > dialog
>>> > > > will
>>> > > > > >    appear;
>>> > > > > >    - Go to "Changes" tab and choose
>>> > "pull/<pull-request-number>/head"
>>> > > > in
>>> > > > > >    "Build branch" dropdown list;
>>> > > > > >    - Press "Run build" button and monitor tests results.
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > > @Roman [hidden email] - can you also please take a look?
>>> > > > > >
>>> > > > > > Regards,
>>> > > > > > Saikat
>>> > > > > >
>>> > > > > > On Sun, Oct 13, 2019 at 6:20 PM Emmanouil Gkatziouras <
>>> > > > > > [hidden email]>
>>> > > > > > wrote:
>>> > > > > >
>>> > > > > > > Hi @Saikat Maitra
>>> > > > > > >
>>> > > > > > > I have my implementation on an Ignite fork on my personal
>>> GitHub
>>> > > > > account
>>> > > > > > > [1]
>>> > > > > > > It is based on the Kafka implementation which is of a polling
>>> > > nature.
>>> > > > > > > GCP does not provide a service like Kafka. Instead it
>>> provides us
>>> > > > with
>>> > > > > > > Pub/Sub which is just like Kafka.
>>> > > > > > > Therefore using the Kafka streamer as a guideline made more
>>> sense
>>> > > to
>>> > > > > me.
>>> > > > > > >
>>> > > > > > > Apart from the implementation I have also added a unit test.
>>> > > > > > > It definitely needs some polishing and a double check on my
>>> side
>>> > to
>>> > > > > make
>>> > > > > > > the review smooth.
>>> > > > > > > For the tests a local Pub/Sub [2] server has to be setup
>>> however
>>> > I
>>> > > am
>>> > > > > not
>>> > > > > > > sure which is the best way to do so with regards to your
>>> CI/CD.
>>> > > > > > >
>>> > > > > > > [1]
>>> > > https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub
>>> > > > > > > [2]
>>> https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/
>>> > > > > > >
>>> > > > > > > *Emmanouil Gkatziouras*
>>> > > > > > > https://egkatzioura.com/ |
>>> > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
>>> > > > > > > https://github.com/gkatzioura
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > On Sat, 12 Oct 2019 at 02:11, Saikat Maitra <
>>> > > [hidden email]
>>> > > > >
>>> > > > > > > wrote:
>>> > > > > > >
>>> > > > > > > > Hello Emmanouil,
>>> > > > > > > >
>>> > > > > > > > Can you please share more info about the implementation?
>>> > > > > > > >
>>> > > > > > > > We have similar implementation for data streamers with
>>> Kafka
>>> > and
>>> > > > > > RocketMQ
>>> > > > > > > >
>>> > > > > > > > https://apacheignite-mix.readme.io/docs/kafka-streamer
>>> > > > > > > >
>>> > > > > > > > https://apacheignite-mix.readme.io/docs/rocketmq-streamer
>>> > > > > > > >
>>> > > > > > > > Please review and share if you have any questions.
>>> > > > > > > >
>>> > > > > > > > Regards,
>>> > > > > > > > Saikat
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > > On Fri, Oct 11, 2019 at 4:56 PM Emmanouil Gkatziouras <
>>> > > > > > > > [hidden email]>
>>> > > > > > > > wrote:
>>> > > > > > > >
>>> > > > > > > > > Hi all,
>>> > > > > > > > >
>>> > > > > > > > > By mentioning Pub/Sub I mean Google Cloud's Pub/Sub [1]
>>> > > > > > > > > Yes it is indeed cloud specific. It is the messaging
>>> service
>>> > > > > provided
>>> > > > > > > by
>>> > > > > > > > > GCP.
>>> > > > > > > > > I have made the integration with it, but I haven't
>>> published
>>> > it
>>> > > > yet
>>> > > > > > (I
>>> > > > > > > am
>>> > > > > > > > > waiting for the discussion/ticket and proceed with the
>>> > > > > contribution).
>>> > > > > > > > >
>>> > > > > > > > > Kind regards,
>>> > > > > > > > > Emmanouil
>>> > > > > > > > >
>>> > > > > > > > > [1] https://cloud.google.com/pubsub/docs/overview
>>> > > > > > > > >
>>> > > > > > > > > *Emmanouil Gkatziouras*
>>> > > > > > > > > https://egkatzioura.com/ |
>>> > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
>>> > > > > > > > > https://github.com/gkatzioura
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > > On Fri, 11 Oct 2019 at 21:45, Denis Magda <
>>> [hidden email]
>>> > >
>>> > > > > wrote:
>>> > > > > > > > >
>>> > > > > > > > > > Hello Emmanouil,
>>> > > > > > > > > >
>>> > > > > > > > > > Are you referring to any specific implementation
>>> tailored
>>> > > for a
>>> > > > > > > cloud?
>>> > > > > > > > If
>>> > > > > > > > > > I'm not mistaken you were mentioning AWS/GCE. Is there
>>> any
>>> > > > Github
>>> > > > > > > > source
>>> > > > > > > > > > code?
>>> > > > > > > > > >
>>> > > > > > > > > > Here is a list of existing integrations with streaming
>>> and
>>> > > > > pub/sub
>>> > > > > > > > > > frameworks:
>>> > > > > > > > > > https://apacheignite-mix.readme.io/docs/overview
>>> > > > > > > > > >
>>> > > > > > > > > > with the source code in Ignite master:
>>> > > > > > > > > > https://github.com/apache/ignite/tree/master/modules
>>> > > > > > > > > >
>>> > > > > > > > > > Check out those resources to see what are the most
>>> suitable
>>> > > > > Ignite
>>> > > > > > > APIs
>>> > > > > > > > > for
>>> > > > > > > > > > such integrations.
>>> > > > > > > > > >
>>> > > > > > > > > > @Saikat Maitra <[hidden email]> , @Roman <
>>> > > > > > [hidden email]
>>> > > > > > > >
>>> > > > > > > > ,
>>> > > > > > > > > as
>>> > > > > > > > > > streaming technologies contributors/maintainers, could
>>> you
>>> > > > please
>>> > > > > > > join
>>> > > > > > > > > this
>>> > > > > > > > > > thread?
>>> > > > > > > > > >
>>> > > > > > > > > > -
>>> > > > > > > > > > Denis
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > > On Thu, Oct 10, 2019 at 7:58 AM Emmanouil Gkatziouras <
>>> > > > > > > > > > [hidden email]>
>>> > > > > > > > > > wrote:
>>> > > > > > > > > >
>>> > > > > > > > > > > Hello everyone,
>>> > > > > > > > > > >
>>> > > > > > > > > > > I started using Ignite lately. Part of my work
>>> involves
>>> > > > > Pub/Sub,
>>> > > > > > > > thus I
>>> > > > > > > > > > > created a pub/sub streamer.
>>> > > > > > > > > > > Pub/Sub is pretty close to Kafka.
>>> > > > > > > > > > > Here's how the streamer works. Every node creates an
>>> > > instance
>>> > > > > of
>>> > > > > > a
>>> > > > > > > > > > Pub/Sub
>>> > > > > > > > > > > client.
>>> > > > > > > > > > > The clients subscribe to the same topic using the
>>> same
>>> > > > > > subscription
>>> > > > > > > > > name.
>>> > > > > > > > > > > A message is being sent and one of the nodes will
>>> receive
>>> > > the
>>> > > > > > > > message,
>>> > > > > > > > > > > process it and then forward it to Ignite using the
>>> > > extractor
>>> > > > > > > > provided.
>>> > > > > > > > > > >
>>> > > > > > > > > > > Any thoughts?
>>> > > > > > > > > > >
>>> > > > > > > > > > > *Emmanouil Gkatziouras*
>>> > > > > > > > > > > https://egkatzioura.com/ |
>>> > > > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
>>> > > > > > > > > > > https://github.com/gkatzioura
>>> > > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>>

[dmagda]

Saikat, thanks for checking with Bahir!

However, let’s run this idea through the rest of the community before
getting down to the implementation. As an alternate option, we might adopt
their approach for Ignite extensions but keep those within Ignite community
in separate repositories. Would you mind starting a separate discussion
with Ignite dev?

Denis

On Tuesday, October 22, 2019, Saikat Maitra <[hidden email]> wrote:

> Hello Emmanouil,
>
> As discussed earlier, I discussed with Apache Bahir community and they are
> interested to have Apache Ignite extensions as part of Apache Bahir
> project.
>
> Can you please share Pub/Sub streamer implementation details with Apache
> Bahir community and request for feedback.
>
> https://bahir.apache.org/
>
> I have also requested for contributor access in Jira for Apache Bahir
> project so that I can create issues and assign to myself.
>
> I can help with code reviews as well.
>
> Here is the email thread for reference
> https://www.mail-archive.com/dev@.../msg02703.html
>
> Regards,
> Saikat
>
>
>
>
>
>
>
> On Mon, Oct 21, 2019 at 7:54 PM Saikat Maitra <[hidden email]>
> wrote:
>
> > Hello,
> >
> > As discussed I have created following discussion threads on our migration
> > proposals for Apache Ignite extensions:
> >
> >
> > http://apache-ignite-users.70518.x6.nabble.com/DISCUSS-
> Proposal-for-Ignite-Extensions-as-a-separate-Bahir-module-or-Incubator-
> project-td29829.html
> >
> > https://www.mail-archive.com/dev@.../msg02703.html
> >
> > I will share update as I hear more information.
> >
> > Regards,
> > Saikat
> >
> >
> >
> > On Fri, Oct 18, 2019 at 9:03 PM Saikat Maitra <[hidden email]>
> > wrote:
> >
> >> Hello Denis,
> >>
> >> Sure, sounds good. I will create a separate discussion thread to get
> >> community feedback on whether we
> >> should join the Bahir or kick-off "Ignite Extensions" project.
> >>
> >> Regards,
> >> Saikat
> >>
> >> On Thu, Oct 17, 2019 at 2:21 PM Denis Magda <[hidden email]> wrote:
> >>
> >>> Folks,
> >>>
> >>> The concept of Apache Bahir is precisely what we need! Saikat, thanks
> for
> >>> doing the research. Why I believe the * idea* fits us well:
> >>>
> >>>    - All integrations can be stored in separate Github repositories and
> >>>    have their dev lifecycles. We've not obliged to couple all the
> >>> integrations
> >>>    in a single repo. For instance, Spark can be located in a dedicated
> >>> repo
> >>>    while streaming integrations might be in a single one. It's up to
> us.
> >>>    - All the repositories and integrations belong to ASF. We're not
> >>> dumping
> >>>    them on Github but rather keep supporting and developing in
> >>> accordance with
> >>>    ASF vision and practices.
> >>>    - There is a way to reward contributors via committership and PMC
> >>>    membership. You won't get this if a project is just one of the
> >>> millions of
> >>>    Github projects.
> >>>
> >>> Saikat, as Ignite PMC member, could you please kick-off a separate
> >>> dev-list
> >>> discussion to involve more community members and referring to this
> >>> thread?
> >>> I think the primary question the community needs to answer whether we
> >>> should join the Bahir or kick-off "Ignite Extensions" project?
> >>>
> >>> -
> >>> Denis
> >>>
> >>>
> >>> On Thu, Oct 17, 2019 at 2:54 AM Alexey Zinoviev <
> [hidden email]>
> >>> wrote:
> >>>
> >>> > Maybe we could move all our Streaming Integrations there, but what is
> >>> about
> >>> > maintaining and committer permissions to the new repositories?
> >>> > I see the list of the committers and PMC members there
> >>> > https://bahir.apache.org/community-members/
> >>> >
> >>> > Could somebody from Ignite community be added to this list as a
> >>> > PMC/committer to maintain Ignite integrations?
> >>> > If yes, I'd happy to join this project and collaborate with these
> guys
> >>> > together
> >>> > If no, and we should start from the zero level with external PRs -
> >>> hmmm,
> >>> > it's better to have our own external repository with ApacheBahirr
> >>> ideology.
> >>> >
> >>> > Also, I agree, that all connectors could be moved there (in
> >>> ApacheBahirr
> >>> > like repository), but not all modules from the page
> >>> >
> >>> >
> >>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> 36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> >>> > because
> >>> > they use different parts of core modules and could be developed with
> >>> > different velocity.
> >>> >
> >>> > In reality, before creation of new repositories we need wide
> discussion
> >>> > based on the document mentioned above.
> >>> >
> >>> > P.S. Of course, we could experiment in the next release 2.8 with one
> >>> or two
> >>> > integrations like twitter or ZeroMQ
> >>> > Also, I'd like Denis Magda idea for the separate repositories in
> Apache
> >>> > Github to maintain them by anybody.
> >>> >
> >>> >
> >>> > чт, 17 окт. 2019 г. в 05:02, Saikat Maitra <[hidden email]
> >:
> >>> >
> >>> > > Hi Denis, Emmanouil
> >>> > >
> >>> > > Thank you for your email. I think creating a separate integration
> >>> project
> >>> > > in github and also proposing it as an apache incubator project will
> >>> be a
> >>> > > good move. The other separate integration modules
> >>> > >
> >>> > >
> >>> >
> >>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> 36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> >>> > > can
> >>> > > be moved to this new apache incubator project.
> >>> > >
> >>> > > There are similar integration available for Flink and Spark in
> Apache
> >>> > Bahir
> >>> > > https://bahir.apache.org/
> >>> > >
> >>> > > Do you think we should reach out to Apache Bahir community with a
> >>> > proposal
> >>> > > or we should plan to create a separate Apache Incubator project?
> >>> > >
> >>> > > Regards,
> >>> > > Saikat
> >>> > >
> >>> > >
> >>> > >
> >>> > > On Wed, Oct 16, 2019 at 6:21 AM Emmanouil Gkatziouras <
> >>> > > [hidden email]>
> >>> > > wrote:
> >>> > >
> >>> > > > Hi all!
> >>> > > >
> >>> > > > Based on the discussions most probably this is going to be on
> >>> another
> >>> > > > GitHub repo. Therefore I will prepare a standalone project with
> the
> >>> > > Pub/Sub
> >>> > > > module and once the repository is created a pull request shall be
> >>> > issued.
> >>> > > > If there is anything else I can do in order to facilitate this
> >>> please
> >>> > let
> >>> > > > me know.
> >>> > > >
> >>> > > > If a ticket is created for this issue, I shall be able to create
> a
> >>> pull
> >>> > > > request and thus a build will take place on ignite's CI.
> >>> > > > If everyone is aligned with adding a Pub/Sub implementation may I
> >>> > create
> >>> > > a
> >>> > > > ticket on JIRA?
> >>> > > >
> >>> > > > My next question has to do on how Ignite configures the build
> jobs
> >>> in
> >>> > > Team
> >>> > > > City. The unit tests for my implementation need a local Pub/Sub
> >>> server
> >>> > up
> >>> > > > and running (they provide one for testing purposes). Will this
> take
> >>> > > effect
> >>> > > > in a form of a build script inside my implementation or is it
> >>> something
> >>> > > > that should be configured on Team City?
> >>> > > >
> >>> > > > Kind regards,
> >>> > > > Emmanouil
> >>> > > >
> >>> > > >
> >>> > > > *Emmanouil Gkatziouras*
> >>> > > > https://egkatzioura.com/ |
> >>> > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> >>> > > > https://github.com/gkatzioura
> >>> > > >
> >>> > > >
> >>> > > > On Tue, 15 Oct 2019 at 23:27, Denis Magda <[hidden email]>
> >>> wrote:
> >>> > > >
> >>> > > > > Emmanouil, Saikat,
> >>> > > > >
> >>> > > > > After contemplating on this, let me propose another way with
> this
> >>> > > > > particular integration that is aligned with our modularization
> >>> > > endeavor.
> >>> > > > > The modularization [1] defines the Ignite core, Ignite modules,
> >>> and
> >>> > > > > independent integrations. The first two are to be located in
> >>> Ignite
> >>> > > > > repositories and to be tested/updated by the community for
> every
> >>> > > release.
> >>> > > > > The modules are a subset of existing integrations that are
> >>> adopted
> >>> > > widely
> >>> > > > > and require community attention.
> >>> > > > >
> >>> > > > > As for the independent integrations [1], those are supposed to
> be
> >>> > > located
> >>> > > > > in a separate GitHub repo. I would advise us to start
> >>> implementing
> >>> > this
> >>> > > > > concept with the Pub/Sub integration. Even though it will be
> >>> located
> >>> > > in a
> >>> > > > > separate Github repo, it will be listed among officially
> >>> available
> >>> > > > > integrations on the Ignite website featuring Emmanouil as a
> >>> > > contributor.
> >>> > > > >
> >>> > > > > What do you think?
> >>> > > > >
> >>> > > > >
> >>> > > > > [1]
> >>> > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> 36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> >>> > > > >
> >>> > > > >
> >>> > > > >
> >>> > > > > -
> >>> > > > > Denis
> >>> > > > >
> >>> > > > >
> >>> > > > > On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra <
> >>> > [hidden email]
> >>> > > >
> >>> > > > > wrote:
> >>> > > > >
> >>> > > > > > Hi Emmanouil,
> >>> > > > > >
> >>> > > > > > The changes looks good to me. I wanted to check if you are
> also
> >>> > able
> >>> > > to
> >>> > > > > > validate that the once data is added to Ignite cluster you
> are
> >>> also
> >>> > > > able
> >>> > > > > to
> >>> > > > > > access it using another Ignite client or using rest
> endpoints.
> >>> > > > > >
> >>> > > > > > We use Teamcity for CI process, the details are as follows
> >>> > > > > >
> >>> > > > > > Trigger validation of those test suites that have been
> >>> affected by
> >>> > > your
> >>> > > > > > changes on TeamCity <http://ci.ignite.apache.org/>: <
> >>> > > > > http://204.14.53.153/
> >>> > > > > > >
> >>> > > > > >
> >>> > > > > >    - Locate a test suite you have to check, press button
> named
> >>> > "..."
> >>> > > > that
> >>> > > > > >    is located on the left of "Run" button. "Run custom build"
> >>> > dialog
> >>> > > > will
> >>> > > > > >    appear;
> >>> > > > > >    - Go to "Changes" tab and choose
> >>> > "pull/<pull-request-number>/head"
> >>> > > > in
> >>> > > > > >    "Build branch" dropdown list;
> >>> > > > > >    - Press "Run build" button and monitor tests results.
> >>> > > > > >
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > @Roman [hidden email] - can you also please take a look?
> >>> > > > > >
> >>> > > > > > Regards,
> >>> > > > > > Saikat
> >>> > > > > >
> >>> > > > > > On Sun, Oct 13, 2019 at 6:20 PM Emmanouil Gkatziouras <
> >>> > > > > > [hidden email]>
> >>> > > > > > wrote:
> >>> > > > > >
> >>> > > > > > > Hi @Saikat Maitra
> >>> > > > > > >
> >>> > > > > > > I have my implementation on an Ignite fork on my personal
> >>> GitHub
> >>> > > > > account
> >>> > > > > > > [1]
> >>> > > > > > > It is based on the Kafka implementation which is of a
> polling
> >>> > > nature.
> >>> > > > > > > GCP does not provide a service like Kafka. Instead it
> >>> provides us
> >>> > > > with
> >>> > > > > > > Pub/Sub which is just like Kafka.
> >>> > > > > > > Therefore using the Kafka streamer as a guideline made more
> >>> sense
> >>> > > to
> >>> > > > > me.
> >>> > > > > > >
> >>> > > > > > > Apart from the implementation I have also added a unit
> test.
> >>> > > > > > > It definitely needs some polishing and a double check on my
> >>> side
> >>> > to
> >>> > > > > make
> >>> > > > > > > the review smooth.
> >>> > > > > > > For the tests a local Pub/Sub [2] server has to be setup
> >>> however
> >>> > I
> >>> > > am
> >>> > > > > not
> >>> > > > > > > sure which is the best way to do so with regards to your
> >>> CI/CD.
> >>> > > > > > >
> >>> > > > > > > [1]
> >>> > > https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub
> >>> > > > > > > [2]
> >>> https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/
> >>> > > > > > >
> >>> > > > > > > *Emmanouil Gkatziouras*
> >>> > > > > > > https://egkatzioura.com/ |
> >>> > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> >>> > > > > > > https://github.com/gkatzioura
> >>> > > > > > >
> >>> > > > > > >
> >>> > > > > > > On Sat, 12 Oct 2019 at 02:11, Saikat Maitra <
> >>> > > [hidden email]
> >>> > > > >
> >>> > > > > > > wrote:
> >>> > > > > > >
> >>> > > > > > > > Hello Emmanouil,
> >>> > > > > > > >
> >>> > > > > > > > Can you please share more info about the implementation?
> >>> > > > > > > >
> >>> > > > > > > > We have similar implementation for data streamers with
> >>> Kafka
> >>> > and
> >>> > > > > > RocketMQ
> >>> > > > > > > >
> >>> > > > > > > > https://apacheignite-mix.readme.io/docs/kafka-streamer
> >>> > > > > > > >
> >>> > > > > > > > https://apacheignite-mix.readme.io/docs/rocketmq-
> streamer
> >>> > > > > > > >
> >>> > > > > > > > Please review and share if you have any questions.
> >>> > > > > > > >
> >>> > > > > > > > Regards,
> >>> > > > > > > > Saikat
> >>> > > > > > > >
> >>> > > > > > > >
> >>> > > > > > > >
> >>> > > > > > > > On Fri, Oct 11, 2019 at 4:56 PM Emmanouil Gkatziouras <
> >>> > > > > > > > [hidden email]>
> >>> > > > > > > > wrote:
> >>> > > > > > > >
> >>> > > > > > > > > Hi all,
> >>> > > > > > > > >
> >>> > > > > > > > > By mentioning Pub/Sub I mean Google Cloud's Pub/Sub [1]
> >>> > > > > > > > > Yes it is indeed cloud specific. It is the messaging
> >>> service
> >>> > > > > provided
> >>> > > > > > > by
> >>> > > > > > > > > GCP.
> >>> > > > > > > > > I have made the integration with it, but I haven't
> >>> published
> >>> > it
> >>> > > > yet
> >>> > > > > > (I
> >>> > > > > > > am
> >>> > > > > > > > > waiting for the discussion/ticket and proceed with the
> >>> > > > > contribution).
> >>> > > > > > > > >
> >>> > > > > > > > > Kind regards,
> >>> > > > > > > > > Emmanouil
> >>> > > > > > > > >
> >>> > > > > > > > > [1] https://cloud.google.com/pubsub/docs/overview
> >>> > > > > > > > >
> >>> > > > > > > > > *Emmanouil Gkatziouras*
> >>> > > > > > > > > https://egkatzioura.com/ |
> >>> > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> >>> > > > > > > > > https://github.com/gkatzioura
> >>> > > > > > > > >
> >>> > > > > > > > >
> >>> > > > > > > > > On Fri, 11 Oct 2019 at 21:45, Denis Magda <
> >>> [hidden email]
> >>> > >
> >>> > > > > wrote:
> >>> > > > > > > > >
> >>> > > > > > > > > > Hello Emmanouil,
> >>> > > > > > > > > >
> >>> > > > > > > > > > Are you referring to any specific implementation
> >>> tailored
> >>> > > for a
> >>> > > > > > > cloud?
> >>> > > > > > > > If
> >>> > > > > > > > > > I'm not mistaken you were mentioning AWS/GCE. Is
> there
> >>> any
> >>> > > > Github
> >>> > > > > > > > source
> >>> > > > > > > > > > code?
> >>> > > > > > > > > >
> >>> > > > > > > > > > Here is a list of existing integrations with
> streaming
> >>> and
> >>> > > > > pub/sub
> >>> > > > > > > > > > frameworks:
> >>> > > > > > > > > > https://apacheignite-mix.readme.io/docs/overview
> >>> > > > > > > > > >
> >>> > > > > > > > > > with the source code in Ignite master:
> >>> > > > > > > > > > https://github.com/apache/ignite/tree/master/modules
> >>> > > > > > > > > >
> >>> > > > > > > > > > Check out those resources to see what are the most
> >>> suitable
> >>> > > > > Ignite
> >>> > > > > > > APIs
> >>> > > > > > > > > for
> >>> > > > > > > > > > such integrations.
> >>> > > > > > > > > >
> >>> > > > > > > > > > @Saikat Maitra <[hidden email]> , @Roman <
> >>> > > > > > [hidden email]
> >>> > > > > > > >
> >>> > > > > > > > ,
> >>> > > > > > > > > as
> >>> > > > > > > > > > streaming technologies contributors/maintainers,
> could
> >>> you
> >>> > > > please
> >>> > > > > > > join
> >>> > > > > > > > > this
> >>> > > > > > > > > > thread?
> >>> > > > > > > > > >
> >>> > > > > > > > > > -
> >>> > > > > > > > > > Denis
> >>> > > > > > > > > >
> >>> > > > > > > > > >
> >>> > > > > > > > > > On Thu, Oct 10, 2019 at 7:58 AM Emmanouil
> Gkatziouras <
> >>> > > > > > > > > > [hidden email]>
> >>> > > > > > > > > > wrote:
> >>> > > > > > > > > >
> >>> > > > > > > > > > > Hello everyone,
> >>> > > > > > > > > > >
> >>> > > > > > > > > > > I started using Ignite lately. Part of my work
> >>> involves
> >>> > > > > Pub/Sub,
> >>> > > > > > > > thus I
> >>> > > > > > > > > > > created a pub/sub streamer.
> >>> > > > > > > > > > > Pub/Sub is pretty close to Kafka.
> >>> > > > > > > > > > > Here's how the streamer works. Every node creates
> an
> >>> > > instance
> >>> > > > > of
> >>> > > > > > a
> >>> > > > > > > > > > Pub/Sub
> >>> > > > > > > > > > > client.
> >>> > > > > > > > > > > The clients subscribe to the same topic using the
> >>> same
> >>> > > > > > subscription
> >>> > > > > > > > > name.
> >>> > > > > > > > > > > A message is being sent and one of the nodes will
> >>> receive
> >>> > > the
> >>> > > > > > > > message,
> >>> > > > > > > > > > > process it and then forward it to Ignite using the
> >>> > > extractor
> >>> > > > > > > > provided.
> >>> > > > > > > > > > >
> >>> > > > > > > > > > > Any thoughts?
> >>> > > > > > > > > > >
> >>> > > > > > > > > > > *Emmanouil Gkatziouras*
> >>> > > > > > > > > > > https://egkatzioura.com/ |
> >>> > > > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> >>> > > > > > > > > > > https://github.com/gkatzioura
> >>> > > > > > > > > > >
> >>> > > > > > > > > >
> >>> > > > > > > > >
> >>> > > > > > > >
> >>> > > > > > >
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> >>
>

--
-
Denis

[Saikat Maitra]

Hi Denis,

Thank you for your response. Yes, I have created the following discussion
thread for Ignite extensions migration.

http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSS-Proposal-for-Ignite-Extensions-as-a-separate-Bahir-module-or-Incubator-project-td44064.html

Regards,
Saikat

On Tue, Oct 22, 2019 at 8:17 PM Denis Magda <[hidden email]> wrote:

> Saikat, thanks for checking with Bahir!
>
> However, let’s run this idea through the rest of the community before
> getting down to the implementation. As an alternate option, we might adopt
> their approach for Ignite extensions but keep those within Ignite community
> in separate repositories. Would you mind starting a separate discussion
> with Ignite dev?
>
> Denis
>
> On Tuesday, October 22, 2019, Saikat Maitra <[hidden email]>
> wrote:
>
> > Hello Emmanouil,
> >
> > As discussed earlier, I discussed with Apache Bahir community and they
> are
> > interested to have Apache Ignite extensions as part of Apache Bahir
> > project.
> >
> > Can you please share Pub/Sub streamer implementation details with Apache
> > Bahir community and request for feedback.
> >
> > https://bahir.apache.org/
> >
> > I have also requested for contributor access in Jira for Apache Bahir
> > project so that I can create issues and assign to myself.
> >
> > I can help with code reviews as well.
> >
> > Here is the email thread for reference
> > https://www.mail-archive.com/dev@.../msg02703.html
> >
> > Regards,
> > Saikat
> >
> >
> >
> >
> >
> >
> >
> > On Mon, Oct 21, 2019 at 7:54 PM Saikat Maitra <[hidden email]>
> > wrote:
> >
> > > Hello,
> > >
> > > As discussed I have created following discussion threads on our
> migration
> > > proposals for Apache Ignite extensions:
> > >
> > >
> > > http://apache-ignite-users.70518.x6.nabble.com/DISCUSS-
> > Proposal-for-Ignite-Extensions-as-a-separate-Bahir-module-or-Incubator-
> > project-td29829.html
> > >
> > > https://www.mail-archive.com/dev@.../msg02703.html
> > >
> > > I will share update as I hear more information.
> > >
> > > Regards,
> > > Saikat
> > >
> > >
> > >
> > > On Fri, Oct 18, 2019 at 9:03 PM Saikat Maitra <[hidden email]
> >
> > > wrote:
> > >
> > >> Hello Denis,
> > >>
> > >> Sure, sounds good. I will create a separate discussion thread to get
> > >> community feedback on whether we
> > >> should join the Bahir or kick-off "Ignite Extensions" project.
> > >>
> > >> Regards,
> > >> Saikat
> > >>
> > >> On Thu, Oct 17, 2019 at 2:21 PM Denis Magda <[hidden email]>
> wrote:
> > >>
> > >>> Folks,
> > >>>
> > >>> The concept of Apache Bahir is precisely what we need! Saikat, thanks
> > for
> > >>> doing the research. Why I believe the * idea* fits us well:
> > >>>
> > >>>    - All integrations can be stored in separate Github repositories
> and
> > >>>    have their dev lifecycles. We've not obliged to couple all the
> > >>> integrations
> > >>>    in a single repo. For instance, Spark can be located in a
> dedicated
> > >>> repo
> > >>>    while streaming integrations might be in a single one. It's up to
> > us.
> > >>>    - All the repositories and integrations belong to ASF. We're not
> > >>> dumping
> > >>>    them on Github but rather keep supporting and developing in
> > >>> accordance with
> > >>>    ASF vision and practices.
> > >>>    - There is a way to reward contributors via committership and PMC
> > >>>    membership. You won't get this if a project is just one of the
> > >>> millions of
> > >>>    Github projects.
> > >>>
> > >>> Saikat, as Ignite PMC member, could you please kick-off a separate
> > >>> dev-list
> > >>> discussion to involve more community members and referring to this
> > >>> thread?
> > >>> I think the primary question the community needs to answer whether we
> > >>> should join the Bahir or kick-off "Ignite Extensions" project?
> > >>>
> > >>> -
> > >>> Denis
> > >>>
> > >>>
> > >>> On Thu, Oct 17, 2019 at 2:54 AM Alexey Zinoviev <
> > [hidden email]>
> > >>> wrote:
> > >>>
> > >>> > Maybe we could move all our Streaming Integrations there, but what
> is
> > >>> about
> > >>> > maintaining and committer permissions to the new repositories?
> > >>> > I see the list of the committers and PMC members there
> > >>> > https://bahir.apache.org/community-members/
> > >>> >
> > >>> > Could somebody from Ignite community be added to this list as a
> > >>> > PMC/committer to maintain Ignite integrations?
> > >>> > If yes, I'd happy to join this project and collaborate with these
> > guys
> > >>> > together
> > >>> > If no, and we should start from the zero level with external PRs -
> > >>> hmmm,
> > >>> > it's better to have our own external repository with ApacheBahirr
> > >>> ideology.
> > >>> >
> > >>> > Also, I agree, that all connectors could be moved there (in
> > >>> ApacheBahirr
> > >>> > like repository), but not all modules from the page
> > >>> >
> > >>> >
> > >>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> > 36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> > >>> > because
> > >>> > they use different parts of core modules and could be developed
> with
> > >>> > different velocity.
> > >>> >
> > >>> > In reality, before creation of new repositories we need wide
> > discussion
> > >>> > based on the document mentioned above.
> > >>> >
> > >>> > P.S. Of course, we could experiment in the next release 2.8 with
> one
> > >>> or two
> > >>> > integrations like twitter or ZeroMQ
> > >>> > Also, I'd like Denis Magda idea for the separate repositories in
> > Apache
> > >>> > Github to maintain them by anybody.
> > >>> >
> > >>> >
> > >>> > чт, 17 окт. 2019 г. в 05:02, Saikat Maitra <
> [hidden email]
> > >:
> > >>> >
> > >>> > > Hi Denis, Emmanouil
> > >>> > >
> > >>> > > Thank you for your email. I think creating a separate integration
> > >>> project
> > >>> > > in github and also proposing it as an apache incubator project
> will
> > >>> be a
> > >>> > > good move. The other separate integration modules
> > >>> > >
> > >>> > >
> > >>> >
> > >>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> > 36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> > >>> > > can
> > >>> > > be moved to this new apache incubator project.
> > >>> > >
> > >>> > > There are similar integration available for Flink and Spark in
> > Apache
> > >>> > Bahir
> > >>> > > https://bahir.apache.org/
> > >>> > >
> > >>> > > Do you think we should reach out to Apache Bahir community with a
> > >>> > proposal
> > >>> > > or we should plan to create a separate Apache Incubator project?
> > >>> > >
> > >>> > > Regards,
> > >>> > > Saikat
> > >>> > >
> > >>> > >
> > >>> > >
> > >>> > > On Wed, Oct 16, 2019 at 6:21 AM Emmanouil Gkatziouras <
> > >>> > > [hidden email]>
> > >>> > > wrote:
> > >>> > >
> > >>> > > > Hi all!
> > >>> > > >
> > >>> > > > Based on the discussions most probably this is going to be on
> > >>> another
> > >>> > > > GitHub repo. Therefore I will prepare a standalone project with
> > the
> > >>> > > Pub/Sub
> > >>> > > > module and once the repository is created a pull request shall
> be
> > >>> > issued.
> > >>> > > > If there is anything else I can do in order to facilitate this
> > >>> please
> > >>> > let
> > >>> > > > me know.
> > >>> > > >
> > >>> > > > If a ticket is created for this issue, I shall be able to
> create
> > a
> > >>> pull
> > >>> > > > request and thus a build will take place on ignite's CI.
> > >>> > > > If everyone is aligned with adding a Pub/Sub implementation
> may I
> > >>> > create
> > >>> > > a
> > >>> > > > ticket on JIRA?
> > >>> > > >
> > >>> > > > My next question has to do on how Ignite configures the build
> > jobs
> > >>> in
> > >>> > > Team
> > >>> > > > City. The unit tests for my implementation need a local Pub/Sub
> > >>> server
> > >>> > up
> > >>> > > > and running (they provide one for testing purposes). Will this
> > take
> > >>> > > effect
> > >>> > > > in a form of a build script inside my implementation or is it
> > >>> something
> > >>> > > > that should be configured on Team City?
> > >>> > > >
> > >>> > > > Kind regards,
> > >>> > > > Emmanouil
> > >>> > > >
> > >>> > > >
> > >>> > > > *Emmanouil Gkatziouras*
> > >>> > > > https://egkatzioura.com/ |
> > >>> > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > >>> > > > https://github.com/gkatzioura
> > >>> > > >
> > >>> > > >
> > >>> > > > On Tue, 15 Oct 2019 at 23:27, Denis Magda <[hidden email]>
> > >>> wrote:
> > >>> > > >
> > >>> > > > > Emmanouil, Saikat,
> > >>> > > > >
> > >>> > > > > After contemplating on this, let me propose another way with
> > this
> > >>> > > > > particular integration that is aligned with our
> modularization
> > >>> > > endeavor.
> > >>> > > > > The modularization [1] defines the Ignite core, Ignite
> modules,
> > >>> and
> > >>> > > > > independent integrations. The first two are to be located in
> > >>> Ignite
> > >>> > > > > repositories and to be tested/updated by the community for
> > every
> > >>> > > release.
> > >>> > > > > The modules are a subset of existing integrations that are
> > >>> adopted
> > >>> > > widely
> > >>> > > > > and require community attention.
> > >>> > > > >
> > >>> > > > > As for the independent integrations [1], those are supposed
> to
> > be
> > >>> > > located
> > >>> > > > > in a separate GitHub repo. I would advise us to start
> > >>> implementing
> > >>> > this
> > >>> > > > > concept with the Pub/Sub integration. Even though it will be
> > >>> located
> > >>> > > in a
> > >>> > > > > separate Github repo, it will be listed among officially
> > >>> available
> > >>> > > > > integrations on the Ignite website featuring Emmanouil as a
> > >>> > > contributor.
> > >>> > > > >
> > >>> > > > > What do you think?
> > >>> > > > >
> > >>> > > > >
> > >>> > > > > [1]
> > >>> > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> > 36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> > >>> > > > >
> > >>> > > > >
> > >>> > > > >
> > >>> > > > > -
> > >>> > > > > Denis
> > >>> > > > >
> > >>> > > > >
> > >>> > > > > On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra <
> > >>> > [hidden email]
> > >>> > > >
> > >>> > > > > wrote:
> > >>> > > > >
> > >>> > > > > > Hi Emmanouil,
> > >>> > > > > >
> > >>> > > > > > The changes looks good to me. I wanted to check if you are
> > also
> > >>> > able
> > >>> > > to
> > >>> > > > > > validate that the once data is added to Ignite cluster you
> > are
> > >>> also
> > >>> > > > able
> > >>> > > > > to
> > >>> > > > > > access it using another Ignite client or using rest
> > endpoints.
> > >>> > > > > >
> > >>> > > > > > We use Teamcity for CI process, the details are as follows
> > >>> > > > > >
> > >>> > > > > > Trigger validation of those test suites that have been
> > >>> affected by
> > >>> > > your
> > >>> > > > > > changes on TeamCity <http://ci.ignite.apache.org/>: <
> > >>> > > > > http://204.14.53.153/
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > > >    - Locate a test suite you have to check, press button
> > named
> > >>> > "..."
> > >>> > > > that
> > >>> > > > > >    is located on the left of "Run" button. "Run custom
> build"
> > >>> > dialog
> > >>> > > > will
> > >>> > > > > >    appear;
> > >>> > > > > >    - Go to "Changes" tab and choose
> > >>> > "pull/<pull-request-number>/head"
> > >>> > > > in
> > >>> > > > > >    "Build branch" dropdown list;
> > >>> > > > > >    - Press "Run build" button and monitor tests results.
> > >>> > > > > >
> > >>> > > > > >
> > >>> > > > > >
> > >>> > > > > > @Roman [hidden email] - can you also please take a
> look?
> > >>> > > > > >
> > >>> > > > > > Regards,
> > >>> > > > > > Saikat
> > >>> > > > > >
> > >>> > > > > > On Sun, Oct 13, 2019 at 6:20 PM Emmanouil Gkatziouras <
> > >>> > > > > > [hidden email]>
> > >>> > > > > > wrote:
> > >>> > > > > >
> > >>> > > > > > > Hi @Saikat Maitra
> > >>> > > > > > >
> > >>> > > > > > > I have my implementation on an Ignite fork on my personal
> > >>> GitHub
> > >>> > > > > account
> > >>> > > > > > > [1]
> > >>> > > > > > > It is based on the Kafka implementation which is of a
> > polling
> > >>> > > nature.
> > >>> > > > > > > GCP does not provide a service like Kafka. Instead it
> > >>> provides us
> > >>> > > > with
> > >>> > > > > > > Pub/Sub which is just like Kafka.
> > >>> > > > > > > Therefore using the Kafka streamer as a guideline made
> more
> > >>> sense
> > >>> > > to
> > >>> > > > > me.
> > >>> > > > > > >
> > >>> > > > > > > Apart from the implementation I have also added a unit
> > test.
> > >>> > > > > > > It definitely needs some polishing and a double check on
> my
> > >>> side
> > >>> > to
> > >>> > > > > make
> > >>> > > > > > > the review smooth.
> > >>> > > > > > > For the tests a local Pub/Sub [2] server has to be setup
> > >>> however
> > >>> > I
> > >>> > > am
> > >>> > > > > not
> > >>> > > > > > > sure which is the best way to do so with regards to your
> > >>> CI/CD.
> > >>> > > > > > >
> > >>> > > > > > > [1]
> > >>> > > https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub
> > >>> > > > > > > [2]
> > >>> https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/
> > >>> > > > > > >
> > >>> > > > > > > *Emmanouil Gkatziouras*
> > >>> > > > > > > https://egkatzioura.com/ |
> > >>> > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > >>> > > > > > > https://github.com/gkatzioura
> > >>> > > > > > >
> > >>> > > > > > >
> > >>> > > > > > > On Sat, 12 Oct 2019 at 02:11, Saikat Maitra <
> > >>> > > [hidden email]
> > >>> > > > >
> > >>> > > > > > > wrote:
> > >>> > > > > > >
> > >>> > > > > > > > Hello Emmanouil,
> > >>> > > > > > > >
> > >>> > > > > > > > Can you please share more info about the
> implementation?
> > >>> > > > > > > >
> > >>> > > > > > > > We have similar implementation for data streamers with
> > >>> Kafka
> > >>> > and
> > >>> > > > > > RocketMQ
> > >>> > > > > > > >
> > >>> > > > > > > > https://apacheignite-mix.readme.io/docs/kafka-streamer
> > >>> > > > > > > >
> > >>> > > > > > > > https://apacheignite-mix.readme.io/docs/rocketmq-
> > streamer
> > >>> > > > > > > >
> > >>> > > > > > > > Please review and share if you have any questions.
> > >>> > > > > > > >
> > >>> > > > > > > > Regards,
> > >>> > > > > > > > Saikat
> > >>> > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > > > On Fri, Oct 11, 2019 at 4:56 PM Emmanouil Gkatziouras <
> > >>> > > > > > > > [hidden email]>
> > >>> > > > > > > > wrote:
> > >>> > > > > > > >
> > >>> > > > > > > > > Hi all,
> > >>> > > > > > > > >
> > >>> > > > > > > > > By mentioning Pub/Sub I mean Google Cloud's Pub/Sub
> [1]
> > >>> > > > > > > > > Yes it is indeed cloud specific. It is the messaging
> > >>> service
> > >>> > > > > provided
> > >>> > > > > > > by
> > >>> > > > > > > > > GCP.
> > >>> > > > > > > > > I have made the integration with it, but I haven't
> > >>> published
> > >>> > it
> > >>> > > > yet
> > >>> > > > > > (I
> > >>> > > > > > > am
> > >>> > > > > > > > > waiting for the discussion/ticket and proceed with
> the
> > >>> > > > > contribution).
> > >>> > > > > > > > >
> > >>> > > > > > > > > Kind regards,
> > >>> > > > > > > > > Emmanouil
> > >>> > > > > > > > >
> > >>> > > > > > > > > [1] https://cloud.google.com/pubsub/docs/overview
> > >>> > > > > > > > >
> > >>> > > > > > > > > *Emmanouil Gkatziouras*
> > >>> > > > > > > > > https://egkatzioura.com/ |
> > >>> > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > >>> > > > > > > > > https://github.com/gkatzioura
> > >>> > > > > > > > >
> > >>> > > > > > > > >
> > >>> > > > > > > > > On Fri, 11 Oct 2019 at 21:45, Denis Magda <
> > >>> [hidden email]
> > >>> > >
> > >>> > > > > wrote:
> > >>> > > > > > > > >
> > >>> > > > > > > > > > Hello Emmanouil,
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > Are you referring to any specific implementation
> > >>> tailored
> > >>> > > for a
> > >>> > > > > > > cloud?
> > >>> > > > > > > > If
> > >>> > > > > > > > > > I'm not mistaken you were mentioning AWS/GCE. Is
> > there
> > >>> any
> > >>> > > > Github
> > >>> > > > > > > > source
> > >>> > > > > > > > > > code?
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > Here is a list of existing integrations with
> > streaming
> > >>> and
> > >>> > > > > pub/sub
> > >>> > > > > > > > > > frameworks:
> > >>> > > > > > > > > > https://apacheignite-mix.readme.io/docs/overview
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > with the source code in Ignite master:
> > >>> > > > > > > > > >
> https://github.com/apache/ignite/tree/master/modules
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > Check out those resources to see what are the most
> > >>> suitable
> > >>> > > > > Ignite
> > >>> > > > > > > APIs
> > >>> > > > > > > > > for
> > >>> > > > > > > > > > such integrations.
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > @Saikat Maitra <[hidden email]> , @Roman
> <
> > >>> > > > > > [hidden email]
> > >>> > > > > > > >
> > >>> > > > > > > > ,
> > >>> > > > > > > > > as
> > >>> > > > > > > > > > streaming technologies contributors/maintainers,
> > could
> > >>> you
> > >>> > > > please
> > >>> > > > > > > join
> > >>> > > > > > > > > this
> > >>> > > > > > > > > > thread?
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > -
> > >>> > > > > > > > > > Denis
> > >>> > > > > > > > > >
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > On Thu, Oct 10, 2019 at 7:58 AM Emmanouil
> > Gkatziouras <
> > >>> > > > > > > > > > [hidden email]>
> > >>> > > > > > > > > > wrote:
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > > Hello everyone,
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > > I started using Ignite lately. Part of my work
> > >>> involves
> > >>> > > > > Pub/Sub,
> > >>> > > > > > > > thus I
> > >>> > > > > > > > > > > created a pub/sub streamer.
> > >>> > > > > > > > > > > Pub/Sub is pretty close to Kafka.
> > >>> > > > > > > > > > > Here's how the streamer works. Every node creates
> > an
> > >>> > > instance
> > >>> > > > > of
> > >>> > > > > > a
> > >>> > > > > > > > > > Pub/Sub
> > >>> > > > > > > > > > > client.
> > >>> > > > > > > > > > > The clients subscribe to the same topic using the
> > >>> same
> > >>> > > > > > subscription
> > >>> > > > > > > > > name.
> > >>> > > > > > > > > > > A message is being sent and one of the nodes will
> > >>> receive
> > >>> > > the
> > >>> > > > > > > > message,
> > >>> > > > > > > > > > > process it and then forward it to Ignite using
> the
> > >>> > > extractor
> > >>> > > > > > > > provided.
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > > Any thoughts?
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > > *Emmanouil Gkatziouras*
> > >>> > > > > > > > > > > https://egkatzioura.com/ |
> > >>> > > > > > > > > > >
> https://www.linkedin.com/in/gkatziourasemmanouil/
> > >>> > > > > > > > > > > https://github.com/gkatzioura
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > >
> > >>> > > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> > >>
> >
>
>
> --
> -
> Denis
>

[Ilya Kasnacheev]

Hello!

My take on this: we want to offload some of integrations, but please commit
and release them to another project first, and only then we will decide to
remove them from our repository.

Please also make sure they are available with same artifact name and API.
If this has to be changed, let's wait for AI 3.0

Regards,
--
Ilya Kasnacheev


ср, 23 окт. 2019 г. в 03:23, Saikat Maitra <[hidden email]>:

> Hello Emmanouil,
>
> As discussed earlier, I discussed with Apache Bahir community and they are
> interested to have Apache Ignite extensions as part of Apache Bahir
> project.
>
> Can you please share Pub/Sub streamer implementation details with Apache
> Bahir community and request for feedback.
>
> https://bahir.apache.org/
>
> I have also requested for contributor access in Jira for Apache Bahir
> project so that I can create issues and assign to myself.
>
> I can help with code reviews as well.
>
> Here is the email thread for reference
> https://www.mail-archive.com/dev@.../msg02703.html
>
> Regards,
> Saikat
>
>
>
>
>
>
>
> On Mon, Oct 21, 2019 at 7:54 PM Saikat Maitra <[hidden email]>
> wrote:
>
> > Hello,
> >
> > As discussed I have created following discussion threads on our migration
> > proposals for Apache Ignite extensions:
> >
> >
> >
> http://apache-ignite-users.70518.x6.nabble.com/DISCUSS-Proposal-for-Ignite-Extensions-as-a-separate-Bahir-module-or-Incubator-project-td29829.html
> >
> > https://www.mail-archive.com/dev@.../msg02703.html
> >
> > I will share update as I hear more information.
> >
> > Regards,
> > Saikat
> >
> >
> >
> > On Fri, Oct 18, 2019 at 9:03 PM Saikat Maitra <[hidden email]>
> > wrote:
> >
> >> Hello Denis,
> >>
> >> Sure, sounds good. I will create a separate discussion thread to get
> >> community feedback on whether we
> >> should join the Bahir or kick-off "Ignite Extensions" project.
> >>
> >> Regards,
> >> Saikat
> >>
> >> On Thu, Oct 17, 2019 at 2:21 PM Denis Magda <[hidden email]> wrote:
> >>
> >>> Folks,
> >>>
> >>> The concept of Apache Bahir is precisely what we need! Saikat, thanks
> for
> >>> doing the research. Why I believe the * idea* fits us well:
> >>>
> >>>    - All integrations can be stored in separate Github repositories and
> >>>    have their dev lifecycles. We've not obliged to couple all the
> >>> integrations
> >>>    in a single repo. For instance, Spark can be located in a dedicated
> >>> repo
> >>>    while streaming integrations might be in a single one. It's up to
> us.
> >>>    - All the repositories and integrations belong to ASF. We're not
> >>> dumping
> >>>    them on Github but rather keep supporting and developing in
> >>> accordance with
> >>>    ASF vision and practices.
> >>>    - There is a way to reward contributors via committership and PMC
> >>>    membership. You won't get this if a project is just one of the
> >>> millions of
> >>>    Github projects.
> >>>
> >>> Saikat, as Ignite PMC member, could you please kick-off a separate
> >>> dev-list
> >>> discussion to involve more community members and referring to this
> >>> thread?
> >>> I think the primary question the community needs to answer whether we
> >>> should join the Bahir or kick-off "Ignite Extensions" project?
> >>>
> >>> -
> >>> Denis
> >>>
> >>>
> >>> On Thu, Oct 17, 2019 at 2:54 AM Alexey Zinoviev <
> [hidden email]>
> >>> wrote:
> >>>
> >>> > Maybe we could move all our Streaming Integrations there, but what is
> >>> about
> >>> > maintaining and committer permissions to the new repositories?
> >>> > I see the list of the committers and PMC members there
> >>> > https://bahir.apache.org/community-members/
> >>> >
> >>> > Could somebody from Ignite community be added to this list as a
> >>> > PMC/committer to maintain Ignite integrations?
> >>> > If yes, I'd happy to join this project and collaborate with these
> guys
> >>> > together
> >>> > If no, and we should start from the zero level with external PRs -
> >>> hmmm,
> >>> > it's better to have our own external repository with ApacheBahirr
> >>> ideology.
> >>> >
> >>> > Also, I agree, that all connectors could be moved there (in
> >>> ApacheBahirr
> >>> > like repository), but not all modules from the page
> >>> >
> >>> >
> >>>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> >>> > because
> >>> > they use different parts of core modules and could be developed with
> >>> > different velocity.
> >>> >
> >>> > In reality, before creation of new repositories we need wide
> discussion
> >>> > based on the document mentioned above.
> >>> >
> >>> > P.S. Of course, we could experiment in the next release 2.8 with one
> >>> or two
> >>> > integrations like twitter or ZeroMQ
> >>> > Also, I'd like Denis Magda idea for the separate repositories in
> Apache
> >>> > Github to maintain them by anybody.
> >>> >
> >>> >
> >>> > чт, 17 окт. 2019 г. в 05:02, Saikat Maitra <[hidden email]
> >:
> >>> >
> >>> > > Hi Denis, Emmanouil
> >>> > >
> >>> > > Thank you for your email. I think creating a separate integration
> >>> project
> >>> > > in github and also proposing it as an apache incubator project will
> >>> be a
> >>> > > good move. The other separate integration modules
> >>> > >
> >>> > >
> >>> >
> >>>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> >>> > > can
> >>> > > be moved to this new apache incubator project.
> >>> > >
> >>> > > There are similar integration available for Flink and Spark in
> Apache
> >>> > Bahir
> >>> > > https://bahir.apache.org/
> >>> > >
> >>> > > Do you think we should reach out to Apache Bahir community with a
> >>> > proposal
> >>> > > or we should plan to create a separate Apache Incubator project?
> >>> > >
> >>> > > Regards,
> >>> > > Saikat
> >>> > >
> >>> > >
> >>> > >
> >>> > > On Wed, Oct 16, 2019 at 6:21 AM Emmanouil Gkatziouras <
> >>> > > [hidden email]>
> >>> > > wrote:
> >>> > >
> >>> > > > Hi all!
> >>> > > >
> >>> > > > Based on the discussions most probably this is going to be on
> >>> another
> >>> > > > GitHub repo. Therefore I will prepare a standalone project with
> the
> >>> > > Pub/Sub
> >>> > > > module and once the repository is created a pull request shall be
> >>> > issued.
> >>> > > > If there is anything else I can do in order to facilitate this
> >>> please
> >>> > let
> >>> > > > me know.
> >>> > > >
> >>> > > > If a ticket is created for this issue, I shall be able to create
> a
> >>> pull
> >>> > > > request and thus a build will take place on ignite's CI.
> >>> > > > If everyone is aligned with adding a Pub/Sub implementation may I
> >>> > create
> >>> > > a
> >>> > > > ticket on JIRA?
> >>> > > >
> >>> > > > My next question has to do on how Ignite configures the build
> jobs
> >>> in
> >>> > > Team
> >>> > > > City. The unit tests for my implementation need a local Pub/Sub
> >>> server
> >>> > up
> >>> > > > and running (they provide one for testing purposes). Will this
> take
> >>> > > effect
> >>> > > > in a form of a build script inside my implementation or is it
> >>> something
> >>> > > > that should be configured on Team City?
> >>> > > >
> >>> > > > Kind regards,
> >>> > > > Emmanouil
> >>> > > >
> >>> > > >
> >>> > > > *Emmanouil Gkatziouras*
> >>> > > > https://egkatzioura.com/ |
> >>> > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> >>> > > > https://github.com/gkatzioura
> >>> > > >
> >>> > > >
> >>> > > > On Tue, 15 Oct 2019 at 23:27, Denis Magda <[hidden email]>
> >>> wrote:
> >>> > > >
> >>> > > > > Emmanouil, Saikat,
> >>> > > > >
> >>> > > > > After contemplating on this, let me propose another way with
> this
> >>> > > > > particular integration that is aligned with our modularization
> >>> > > endeavor.
> >>> > > > > The modularization [1] defines the Ignite core, Ignite modules,
> >>> and
> >>> > > > > independent integrations. The first two are to be located in
> >>> Ignite
> >>> > > > > repositories and to be tested/updated by the community for
> every
> >>> > > release.
> >>> > > > > The modules are a subset of existing integrations that are
> >>> adopted
> >>> > > widely
> >>> > > > > and require community attention.
> >>> > > > >
> >>> > > > > As for the independent integrations [1], those are supposed to
> be
> >>> > > located
> >>> > > > > in a separate GitHub repo. I would advise us to start
> >>> implementing
> >>> > this
> >>> > > > > concept with the Pub/Sub integration. Even though it will be
> >>> located
> >>> > > in a
> >>> > > > > separate Github repo, it will be listed among officially
> >>> available
> >>> > > > > integrations on the Ignite website featuring Emmanouil as a
> >>> > > contributor.
> >>> > > > >
> >>> > > > > What do you think?
> >>> > > > >
> >>> > > > >
> >>> > > > > [1]
> >>> > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> >>> > > > >
> >>> > > > >
> >>> > > > >
> >>> > > > > -
> >>> > > > > Denis
> >>> > > > >
> >>> > > > >
> >>> > > > > On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra <
> >>> > [hidden email]
> >>> > > >
> >>> > > > > wrote:
> >>> > > > >
> >>> > > > > > Hi Emmanouil,
> >>> > > > > >
> >>> > > > > > The changes looks good to me. I wanted to check if you are
> also
> >>> > able
> >>> > > to
> >>> > > > > > validate that the once data is added to Ignite cluster you
> are
> >>> also
> >>> > > > able
> >>> > > > > to
> >>> > > > > > access it using another Ignite client or using rest
> endpoints.
> >>> > > > > >
> >>> > > > > > We use Teamcity for CI process, the details are as follows
> >>> > > > > >
> >>> > > > > > Trigger validation of those test suites that have been
> >>> affected by
> >>> > > your
> >>> > > > > > changes on TeamCity <http://ci.ignite.apache.org/>: <
> >>> > > > > http://204.14.53.153/
> >>> > > > > > >
> >>> > > > > >
> >>> > > > > >    - Locate a test suite you have to check, press button
> named
> >>> > "..."
> >>> > > > that
> >>> > > > > >    is located on the left of "Run" button. "Run custom build"
> >>> > dialog
> >>> > > > will
> >>> > > > > >    appear;
> >>> > > > > >    - Go to "Changes" tab and choose
> >>> > "pull/<pull-request-number>/head"
> >>> > > > in
> >>> > > > > >    "Build branch" dropdown list;
> >>> > > > > >    - Press "Run build" button and monitor tests results.
> >>> > > > > >
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > @Roman [hidden email] - can you also please take a look?
> >>> > > > > >
> >>> > > > > > Regards,
> >>> > > > > > Saikat
> >>> > > > > >
> >>> > > > > > On Sun, Oct 13, 2019 at 6:20 PM Emmanouil Gkatziouras <
> >>> > > > > > [hidden email]>
> >>> > > > > > wrote:
> >>> > > > > >
> >>> > > > > > > Hi @Saikat Maitra
> >>> > > > > > >
> >>> > > > > > > I have my implementation on an Ignite fork on my personal
> >>> GitHub
> >>> > > > > account
> >>> > > > > > > [1]
> >>> > > > > > > It is based on the Kafka implementation which is of a
> polling
> >>> > > nature.
> >>> > > > > > > GCP does not provide a service like Kafka. Instead it
> >>> provides us
> >>> > > > with
> >>> > > > > > > Pub/Sub which is just like Kafka.
> >>> > > > > > > Therefore using the Kafka streamer as a guideline made more
> >>> sense
> >>> > > to
> >>> > > > > me.
> >>> > > > > > >
> >>> > > > > > > Apart from the implementation I have also added a unit
> test.
> >>> > > > > > > It definitely needs some polishing and a double check on my
> >>> side
> >>> > to
> >>> > > > > make
> >>> > > > > > > the review smooth.
> >>> > > > > > > For the tests a local Pub/Sub [2] server has to be setup
> >>> however
> >>> > I
> >>> > > am
> >>> > > > > not
> >>> > > > > > > sure which is the best way to do so with regards to your
> >>> CI/CD.
> >>> > > > > > >
> >>> > > > > > > [1]
> >>> > > https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub
> >>> > > > > > > [2]
> >>> https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/
> >>> > > > > > >
> >>> > > > > > > *Emmanouil Gkatziouras*
> >>> > > > > > > https://egkatzioura.com/ |
> >>> > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> >>> > > > > > > https://github.com/gkatzioura
> >>> > > > > > >
> >>> > > > > > >
> >>> > > > > > > On Sat, 12 Oct 2019 at 02:11, Saikat Maitra <
> >>> > > [hidden email]
> >>> > > > >
> >>> > > > > > > wrote:
> >>> > > > > > >
> >>> > > > > > > > Hello Emmanouil,
> >>> > > > > > > >
> >>> > > > > > > > Can you please share more info about the implementation?
> >>> > > > > > > >
> >>> > > > > > > > We have similar implementation for data streamers with
> >>> Kafka
> >>> > and
> >>> > > > > > RocketMQ
> >>> > > > > > > >
> >>> > > > > > > > https://apacheignite-mix.readme.io/docs/kafka-streamer
> >>> > > > > > > >
> >>> > > > > > > >
> https://apacheignite-mix.readme.io/docs/rocketmq-streamer
> >>> > > > > > > >
> >>> > > > > > > > Please review and share if you have any questions.
> >>> > > > > > > >
> >>> > > > > > > > Regards,
> >>> > > > > > > > Saikat
> >>> > > > > > > >
> >>> > > > > > > >
> >>> > > > > > > >
> >>> > > > > > > > On Fri, Oct 11, 2019 at 4:56 PM Emmanouil Gkatziouras <
> >>> > > > > > > > [hidden email]>
> >>> > > > > > > > wrote:
> >>> > > > > > > >
> >>> > > > > > > > > Hi all,
> >>> > > > > > > > >
> >>> > > > > > > > > By mentioning Pub/Sub I mean Google Cloud's Pub/Sub [1]
> >>> > > > > > > > > Yes it is indeed cloud specific. It is the messaging
> >>> service
> >>> > > > > provided
> >>> > > > > > > by
> >>> > > > > > > > > GCP.
> >>> > > > > > > > > I have made the integration with it, but I haven't
> >>> published
> >>> > it
> >>> > > > yet
> >>> > > > > > (I
> >>> > > > > > > am
> >>> > > > > > > > > waiting for the discussion/ticket and proceed with the
> >>> > > > > contribution).
> >>> > > > > > > > >
> >>> > > > > > > > > Kind regards,
> >>> > > > > > > > > Emmanouil
> >>> > > > > > > > >
> >>> > > > > > > > > [1] https://cloud.google.com/pubsub/docs/overview
> >>> > > > > > > > >
> >>> > > > > > > > > *Emmanouil Gkatziouras*
> >>> > > > > > > > > https://egkatzioura.com/ |
> >>> > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> >>> > > > > > > > > https://github.com/gkatzioura
> >>> > > > > > > > >
> >>> > > > > > > > >
> >>> > > > > > > > > On Fri, 11 Oct 2019 at 21:45, Denis Magda <
> >>> [hidden email]
> >>> > >
> >>> > > > > wrote:
> >>> > > > > > > > >
> >>> > > > > > > > > > Hello Emmanouil,
> >>> > > > > > > > > >
> >>> > > > > > > > > > Are you referring to any specific implementation
> >>> tailored
> >>> > > for a
> >>> > > > > > > cloud?
> >>> > > > > > > > If
> >>> > > > > > > > > > I'm not mistaken you were mentioning AWS/GCE. Is
> there
> >>> any
> >>> > > > Github
> >>> > > > > > > > source
> >>> > > > > > > > > > code?
> >>> > > > > > > > > >
> >>> > > > > > > > > > Here is a list of existing integrations with
> streaming
> >>> and
> >>> > > > > pub/sub
> >>> > > > > > > > > > frameworks:
> >>> > > > > > > > > > https://apacheignite-mix.readme.io/docs/overview
> >>> > > > > > > > > >
> >>> > > > > > > > > > with the source code in Ignite master:
> >>> > > > > > > > > > https://github.com/apache/ignite/tree/master/modules
> >>> > > > > > > > > >
> >>> > > > > > > > > > Check out those resources to see what are the most
> >>> suitable
> >>> > > > > Ignite
> >>> > > > > > > APIs
> >>> > > > > > > > > for
> >>> > > > > > > > > > such integrations.
> >>> > > > > > > > > >
> >>> > > > > > > > > > @Saikat Maitra <[hidden email]> , @Roman <
> >>> > > > > > [hidden email]
> >>> > > > > > > >
> >>> > > > > > > > ,
> >>> > > > > > > > > as
> >>> > > > > > > > > > streaming technologies contributors/maintainers,
> could
> >>> you
> >>> > > > please
> >>> > > > > > > join
> >>> > > > > > > > > this
> >>> > > > > > > > > > thread?
> >>> > > > > > > > > >
> >>> > > > > > > > > > -
> >>> > > > > > > > > > Denis
> >>> > > > > > > > > >
> >>> > > > > > > > > >
> >>> > > > > > > > > > On Thu, Oct 10, 2019 at 7:58 AM Emmanouil
> Gkatziouras <
> >>> > > > > > > > > > [hidden email]>
> >>> > > > > > > > > > wrote:
> >>> > > > > > > > > >
> >>> > > > > > > > > > > Hello everyone,
> >>> > > > > > > > > > >
> >>> > > > > > > > > > > I started using Ignite lately. Part of my work
> >>> involves
> >>> > > > > Pub/Sub,
> >>> > > > > > > > thus I
> >>> > > > > > > > > > > created a pub/sub streamer.
> >>> > > > > > > > > > > Pub/Sub is pretty close to Kafka.
> >>> > > > > > > > > > > Here's how the streamer works. Every node creates
> an
> >>> > > instance
> >>> > > > > of
> >>> > > > > > a
> >>> > > > > > > > > > Pub/Sub
> >>> > > > > > > > > > > client.
> >>> > > > > > > > > > > The clients subscribe to the same topic using the
> >>> same
> >>> > > > > > subscription
> >>> > > > > > > > > name.
> >>> > > > > > > > > > > A message is being sent and one of the nodes will
> >>> receive
> >>> > > the
> >>> > > > > > > > message,
> >>> > > > > > > > > > > process it and then forward it to Ignite using the
> >>> > > extractor
> >>> > > > > > > > provided.
> >>> > > > > > > > > > >
> >>> > > > > > > > > > > Any thoughts?
> >>> > > > > > > > > > >
> >>> > > > > > > > > > > *Emmanouil Gkatziouras*
> >>> > > > > > > > > > > https://egkatzioura.com/ |
> >>> > > > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> >>> > > > > > > > > > > https://github.com/gkatzioura
> >>> > > > > > > > > > >
> >>> > > > > > > > > >
> >>> > > > > > > > >
> >>> > > > > > > >
> >>> > > > > > >
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> >>
>

[Saikat Maitra]

Hello Ilya,

Thank you for your email. Yes, before we remove any integration we will
move them to another repository and release.

We are thinking we will start with Pub/Sub Streamer implementation and
continue to move other integration as mentioned in the following doc.

https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations

Regards,
Saikat


On Thu, Oct 24, 2019 at 11:00 AM Ilya Kasnacheev <[hidden email]>
wrote:

> Hello!
>
> My take on this: we want to offload some of integrations, but please commit
> and release them to another project first, and only then we will decide to
> remove them from our repository.
>
> Please also make sure they are available with same artifact name and API.
> If this has to be changed, let's wait for AI 3.0
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 23 окт. 2019 г. в 03:23, Saikat Maitra <[hidden email]>:
>
> > Hello Emmanouil,
> >
> > As discussed earlier, I discussed with Apache Bahir community and they
> are
> > interested to have Apache Ignite extensions as part of Apache Bahir
> > project.
> >
> > Can you please share Pub/Sub streamer implementation details with Apache
> > Bahir community and request for feedback.
> >
> > https://bahir.apache.org/
> >
> > I have also requested for contributor access in Jira for Apache Bahir
> > project so that I can create issues and assign to myself.
> >
> > I can help with code reviews as well.
> >
> > Here is the email thread for reference
> > https://www.mail-archive.com/dev@.../msg02703.html
> >
> > Regards,
> > Saikat
> >
> >
> >
> >
> >
> >
> >
> > On Mon, Oct 21, 2019 at 7:54 PM Saikat Maitra <[hidden email]>
> > wrote:
> >
> > > Hello,
> > >
> > > As discussed I have created following discussion threads on our
> migration
> > > proposals for Apache Ignite extensions:
> > >
> > >
> > >
> >
> http://apache-ignite-users.70518.x6.nabble.com/DISCUSS-Proposal-for-Ignite-Extensions-as-a-separate-Bahir-module-or-Incubator-project-td29829.html
> > >
> > > https://www.mail-archive.com/dev@.../msg02703.html
> > >
> > > I will share update as I hear more information.
> > >
> > > Regards,
> > > Saikat
> > >
> > >
> > >
> > > On Fri, Oct 18, 2019 at 9:03 PM Saikat Maitra <[hidden email]
> >
> > > wrote:
> > >
> > >> Hello Denis,
> > >>
> > >> Sure, sounds good. I will create a separate discussion thread to get
> > >> community feedback on whether we
> > >> should join the Bahir or kick-off "Ignite Extensions" project.
> > >>
> > >> Regards,
> > >> Saikat
> > >>
> > >> On Thu, Oct 17, 2019 at 2:21 PM Denis Magda <[hidden email]>
> wrote:
> > >>
> > >>> Folks,
> > >>>
> > >>> The concept of Apache Bahir is precisely what we need! Saikat, thanks
> > for
> > >>> doing the research. Why I believe the * idea* fits us well:
> > >>>
> > >>>    - All integrations can be stored in separate Github repositories
> and
> > >>>    have their dev lifecycles. We've not obliged to couple all the
> > >>> integrations
> > >>>    in a single repo. For instance, Spark can be located in a
> dedicated
> > >>> repo
> > >>>    while streaming integrations might be in a single one. It's up to
> > us.
> > >>>    - All the repositories and integrations belong to ASF. We're not
> > >>> dumping
> > >>>    them on Github but rather keep supporting and developing in
> > >>> accordance with
> > >>>    ASF vision and practices.
> > >>>    - There is a way to reward contributors via committership and PMC
> > >>>    membership. You won't get this if a project is just one of the
> > >>> millions of
> > >>>    Github projects.
> > >>>
> > >>> Saikat, as Ignite PMC member, could you please kick-off a separate
> > >>> dev-list
> > >>> discussion to involve more community members and referring to this
> > >>> thread?
> > >>> I think the primary question the community needs to answer whether we
> > >>> should join the Bahir or kick-off "Ignite Extensions" project?
> > >>>
> > >>> -
> > >>> Denis
> > >>>
> > >>>
> > >>> On Thu, Oct 17, 2019 at 2:54 AM Alexey Zinoviev <
> > [hidden email]>
> > >>> wrote:
> > >>>
> > >>> > Maybe we could move all our Streaming Integrations there, but what
> is
> > >>> about
> > >>> > maintaining and committer permissions to the new repositories?
> > >>> > I see the list of the committers and PMC members there
> > >>> > https://bahir.apache.org/community-members/
> > >>> >
> > >>> > Could somebody from Ignite community be added to this list as a
> > >>> > PMC/committer to maintain Ignite integrations?
> > >>> > If yes, I'd happy to join this project and collaborate with these
> > guys
> > >>> > together
> > >>> > If no, and we should start from the zero level with external PRs -
> > >>> hmmm,
> > >>> > it's better to have our own external repository with ApacheBahirr
> > >>> ideology.
> > >>> >
> > >>> > Also, I agree, that all connectors could be moved there (in
> > >>> ApacheBahirr
> > >>> > like repository), but not all modules from the page
> > >>> >
> > >>> >
> > >>>
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> > >>> > because
> > >>> > they use different parts of core modules and could be developed
> with
> > >>> > different velocity.
> > >>> >
> > >>> > In reality, before creation of new repositories we need wide
> > discussion
> > >>> > based on the document mentioned above.
> > >>> >
> > >>> > P.S. Of course, we could experiment in the next release 2.8 with
> one
> > >>> or two
> > >>> > integrations like twitter or ZeroMQ
> > >>> > Also, I'd like Denis Magda idea for the separate repositories in
> > Apache
> > >>> > Github to maintain them by anybody.
> > >>> >
> > >>> >
> > >>> > чт, 17 окт. 2019 г. в 05:02, Saikat Maitra <
> [hidden email]
> > >:
> > >>> >
> > >>> > > Hi Denis, Emmanouil
> > >>> > >
> > >>> > > Thank you for your email. I think creating a separate integration
> > >>> project
> > >>> > > in github and also proposing it as an apache incubator project
> will
> > >>> be a
> > >>> > > good move. The other separate integration modules
> > >>> > >
> > >>> > >
> > >>> >
> > >>>
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> > >>> > > can
> > >>> > > be moved to this new apache incubator project.
> > >>> > >
> > >>> > > There are similar integration available for Flink and Spark in
> > Apache
> > >>> > Bahir
> > >>> > > https://bahir.apache.org/
> > >>> > >
> > >>> > > Do you think we should reach out to Apache Bahir community with a
> > >>> > proposal
> > >>> > > or we should plan to create a separate Apache Incubator project?
> > >>> > >
> > >>> > > Regards,
> > >>> > > Saikat
> > >>> > >
> > >>> > >
> > >>> > >
> > >>> > > On Wed, Oct 16, 2019 at 6:21 AM Emmanouil Gkatziouras <
> > >>> > > [hidden email]>
> > >>> > > wrote:
> > >>> > >
> > >>> > > > Hi all!
> > >>> > > >
> > >>> > > > Based on the discussions most probably this is going to be on
> > >>> another
> > >>> > > > GitHub repo. Therefore I will prepare a standalone project with
> > the
> > >>> > > Pub/Sub
> > >>> > > > module and once the repository is created a pull request shall
> be
> > >>> > issued.
> > >>> > > > If there is anything else I can do in order to facilitate this
> > >>> please
> > >>> > let
> > >>> > > > me know.
> > >>> > > >
> > >>> > > > If a ticket is created for this issue, I shall be able to
> create
> > a
> > >>> pull
> > >>> > > > request and thus a build will take place on ignite's CI.
> > >>> > > > If everyone is aligned with adding a Pub/Sub implementation
> may I
> > >>> > create
> > >>> > > a
> > >>> > > > ticket on JIRA?
> > >>> > > >
> > >>> > > > My next question has to do on how Ignite configures the build
> > jobs
> > >>> in
> > >>> > > Team
> > >>> > > > City. The unit tests for my implementation need a local Pub/Sub
> > >>> server
> > >>> > up
> > >>> > > > and running (they provide one for testing purposes). Will this
> > take
> > >>> > > effect
> > >>> > > > in a form of a build script inside my implementation or is it
> > >>> something
> > >>> > > > that should be configured on Team City?
> > >>> > > >
> > >>> > > > Kind regards,
> > >>> > > > Emmanouil
> > >>> > > >
> > >>> > > >
> > >>> > > > *Emmanouil Gkatziouras*
> > >>> > > > https://egkatzioura.com/ |
> > >>> > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > >>> > > > https://github.com/gkatzioura
> > >>> > > >
> > >>> > > >
> > >>> > > > On Tue, 15 Oct 2019 at 23:27, Denis Magda <[hidden email]>
> > >>> wrote:
> > >>> > > >
> > >>> > > > > Emmanouil, Saikat,
> > >>> > > > >
> > >>> > > > > After contemplating on this, let me propose another way with
> > this
> > >>> > > > > particular integration that is aligned with our
> modularization
> > >>> > > endeavor.
> > >>> > > > > The modularization [1] defines the Ignite core, Ignite
> modules,
> > >>> and
> > >>> > > > > independent integrations. The first two are to be located in
> > >>> Ignite
> > >>> > > > > repositories and to be tested/updated by the community for
> > every
> > >>> > > release.
> > >>> > > > > The modules are a subset of existing integrations that are
> > >>> adopted
> > >>> > > widely
> > >>> > > > > and require community attention.
> > >>> > > > >
> > >>> > > > > As for the independent integrations [1], those are supposed
> to
> > be
> > >>> > > located
> > >>> > > > > in a separate GitHub repo. I would advise us to start
> > >>> implementing
> > >>> > this
> > >>> > > > > concept with the Pub/Sub integration. Even though it will be
> > >>> located
> > >>> > > in a
> > >>> > > > > separate Github repo, it will be listed among officially
> > >>> available
> > >>> > > > > integrations on the Ignite website featuring Emmanouil as a
> > >>> > > contributor.
> > >>> > > > >
> > >>> > > > > What do you think?
> > >>> > > > >
> > >>> > > > >
> > >>> > > > > [1]
> > >>> > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> > >>> > > > >
> > >>> > > > >
> > >>> > > > >
> > >>> > > > > -
> > >>> > > > > Denis
> > >>> > > > >
> > >>> > > > >
> > >>> > > > > On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra <
> > >>> > [hidden email]
> > >>> > > >
> > >>> > > > > wrote:
> > >>> > > > >
> > >>> > > > > > Hi Emmanouil,
> > >>> > > > > >
> > >>> > > > > > The changes looks good to me. I wanted to check if you are
> > also
> > >>> > able
> > >>> > > to
> > >>> > > > > > validate that the once data is added to Ignite cluster you
> > are
> > >>> also
> > >>> > > > able
> > >>> > > > > to
> > >>> > > > > > access it using another Ignite client or using rest
> > endpoints.
> > >>> > > > > >
> > >>> > > > > > We use Teamcity for CI process, the details are as follows
> > >>> > > > > >
> > >>> > > > > > Trigger validation of those test suites that have been
> > >>> affected by
> > >>> > > your
> > >>> > > > > > changes on TeamCity <http://ci.ignite.apache.org/>: <
> > >>> > > > > http://204.14.53.153/
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > > >    - Locate a test suite you have to check, press button
> > named
> > >>> > "..."
> > >>> > > > that
> > >>> > > > > >    is located on the left of "Run" button. "Run custom
> build"
> > >>> > dialog
> > >>> > > > will
> > >>> > > > > >    appear;
> > >>> > > > > >    - Go to "Changes" tab and choose
> > >>> > "pull/<pull-request-number>/head"
> > >>> > > > in
> > >>> > > > > >    "Build branch" dropdown list;
> > >>> > > > > >    - Press "Run build" button and monitor tests results.
> > >>> > > > > >
> > >>> > > > > >
> > >>> > > > > >
> > >>> > > > > > @Roman [hidden email] - can you also please take a
> look?
> > >>> > > > > >
> > >>> > > > > > Regards,
> > >>> > > > > > Saikat
> > >>> > > > > >
> > >>> > > > > > On Sun, Oct 13, 2019 at 6:20 PM Emmanouil Gkatziouras <
> > >>> > > > > > [hidden email]>
> > >>> > > > > > wrote:
> > >>> > > > > >
> > >>> > > > > > > Hi @Saikat Maitra
> > >>> > > > > > >
> > >>> > > > > > > I have my implementation on an Ignite fork on my personal
> > >>> GitHub
> > >>> > > > > account
> > >>> > > > > > > [1]
> > >>> > > > > > > It is based on the Kafka implementation which is of a
> > polling
> > >>> > > nature.
> > >>> > > > > > > GCP does not provide a service like Kafka. Instead it
> > >>> provides us
> > >>> > > > with
> > >>> > > > > > > Pub/Sub which is just like Kafka.
> > >>> > > > > > > Therefore using the Kafka streamer as a guideline made
> more
> > >>> sense
> > >>> > > to
> > >>> > > > > me.
> > >>> > > > > > >
> > >>> > > > > > > Apart from the implementation I have also added a unit
> > test.
> > >>> > > > > > > It definitely needs some polishing and a double check on
> my
> > >>> side
> > >>> > to
> > >>> > > > > make
> > >>> > > > > > > the review smooth.
> > >>> > > > > > > For the tests a local Pub/Sub [2] server has to be setup
> > >>> however
> > >>> > I
> > >>> > > am
> > >>> > > > > not
> > >>> > > > > > > sure which is the best way to do so with regards to your
> > >>> CI/CD.
> > >>> > > > > > >
> > >>> > > > > > > [1]
> > >>> > > https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub
> > >>> > > > > > > [2]
> > >>> https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/
> > >>> > > > > > >
> > >>> > > > > > > *Emmanouil Gkatziouras*
> > >>> > > > > > > https://egkatzioura.com/ |
> > >>> > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > >>> > > > > > > https://github.com/gkatzioura
> > >>> > > > > > >
> > >>> > > > > > >
> > >>> > > > > > > On Sat, 12 Oct 2019 at 02:11, Saikat Maitra <
> > >>> > > [hidden email]
> > >>> > > > >
> > >>> > > > > > > wrote:
> > >>> > > > > > >
> > >>> > > > > > > > Hello Emmanouil,
> > >>> > > > > > > >
> > >>> > > > > > > > Can you please share more info about the
> implementation?
> > >>> > > > > > > >
> > >>> > > > > > > > We have similar implementation for data streamers with
> > >>> Kafka
> > >>> > and
> > >>> > > > > > RocketMQ
> > >>> > > > > > > >
> > >>> > > > > > > > https://apacheignite-mix.readme.io/docs/kafka-streamer
> > >>> > > > > > > >
> > >>> > > > > > > >
> > https://apacheignite-mix.readme.io/docs/rocketmq-streamer
> > >>> > > > > > > >
> > >>> > > > > > > > Please review and share if you have any questions.
> > >>> > > > > > > >
> > >>> > > > > > > > Regards,
> > >>> > > > > > > > Saikat
> > >>> > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > > > On Fri, Oct 11, 2019 at 4:56 PM Emmanouil Gkatziouras <
> > >>> > > > > > > > [hidden email]>
> > >>> > > > > > > > wrote:
> > >>> > > > > > > >
> > >>> > > > > > > > > Hi all,
> > >>> > > > > > > > >
> > >>> > > > > > > > > By mentioning Pub/Sub I mean Google Cloud's Pub/Sub
> [1]
> > >>> > > > > > > > > Yes it is indeed cloud specific. It is the messaging
> > >>> service
> > >>> > > > > provided
> > >>> > > > > > > by
> > >>> > > > > > > > > GCP.
> > >>> > > > > > > > > I have made the integration with it, but I haven't
> > >>> published
> > >>> > it
> > >>> > > > yet
> > >>> > > > > > (I
> > >>> > > > > > > am
> > >>> > > > > > > > > waiting for the discussion/ticket and proceed with
> the
> > >>> > > > > contribution).
> > >>> > > > > > > > >
> > >>> > > > > > > > > Kind regards,
> > >>> > > > > > > > > Emmanouil
> > >>> > > > > > > > >
> > >>> > > > > > > > > [1] https://cloud.google.com/pubsub/docs/overview
> > >>> > > > > > > > >
> > >>> > > > > > > > > *Emmanouil Gkatziouras*
> > >>> > > > > > > > > https://egkatzioura.com/ |
> > >>> > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > >>> > > > > > > > > https://github.com/gkatzioura
> > >>> > > > > > > > >
> > >>> > > > > > > > >
> > >>> > > > > > > > > On Fri, 11 Oct 2019 at 21:45, Denis Magda <
> > >>> [hidden email]
> > >>> > >
> > >>> > > > > wrote:
> > >>> > > > > > > > >
> > >>> > > > > > > > > > Hello Emmanouil,
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > Are you referring to any specific implementation
> > >>> tailored
> > >>> > > for a
> > >>> > > > > > > cloud?
> > >>> > > > > > > > If
> > >>> > > > > > > > > > I'm not mistaken you were mentioning AWS/GCE. Is
> > there
> > >>> any
> > >>> > > > Github
> > >>> > > > > > > > source
> > >>> > > > > > > > > > code?
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > Here is a list of existing integrations with
> > streaming
> > >>> and
> > >>> > > > > pub/sub
> > >>> > > > > > > > > > frameworks:
> > >>> > > > > > > > > > https://apacheignite-mix.readme.io/docs/overview
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > with the source code in Ignite master:
> > >>> > > > > > > > > >
> https://github.com/apache/ignite/tree/master/modules
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > Check out those resources to see what are the most
> > >>> suitable
> > >>> > > > > Ignite
> > >>> > > > > > > APIs
> > >>> > > > > > > > > for
> > >>> > > > > > > > > > such integrations.
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > @Saikat Maitra <[hidden email]> , @Roman
> <
> > >>> > > > > > [hidden email]
> > >>> > > > > > > >
> > >>> > > > > > > > ,
> > >>> > > > > > > > > as
> > >>> > > > > > > > > > streaming technologies contributors/maintainers,
> > could
> > >>> you
> > >>> > > > please
> > >>> > > > > > > join
> > >>> > > > > > > > > this
> > >>> > > > > > > > > > thread?
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > -
> > >>> > > > > > > > > > Denis
> > >>> > > > > > > > > >
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > On Thu, Oct 10, 2019 at 7:58 AM Emmanouil
> > Gkatziouras <
> > >>> > > > > > > > > > [hidden email]>
> > >>> > > > > > > > > > wrote:
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > > Hello everyone,
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > > I started using Ignite lately. Part of my work
> > >>> involves
> > >>> > > > > Pub/Sub,
> > >>> > > > > > > > thus I
> > >>> > > > > > > > > > > created a pub/sub streamer.
> > >>> > > > > > > > > > > Pub/Sub is pretty close to Kafka.
> > >>> > > > > > > > > > > Here's how the streamer works. Every node creates
> > an
> > >>> > > instance
> > >>> > > > > of
> > >>> > > > > > a
> > >>> > > > > > > > > > Pub/Sub
> > >>> > > > > > > > > > > client.
> > >>> > > > > > > > > > > The clients subscribe to the same topic using the
> > >>> same
> > >>> > > > > > subscription
> > >>> > > > > > > > > name.
> > >>> > > > > > > > > > > A message is being sent and one of the nodes will
> > >>> receive
> > >>> > > the
> > >>> > > > > > > > message,
> > >>> > > > > > > > > > > process it and then forward it to Ignite using
> the
> > >>> > > extractor
> > >>> > > > > > > > provided.
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > > Any thoughts?
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > > *Emmanouil Gkatziouras*
> > >>> > > > > > > > > > > https://egkatzioura.com/ |
> > >>> > > > > > > > > > >
> https://www.linkedin.com/in/gkatziourasemmanouil/
> > >>> > > > > > > > > > > https://github.com/gkatzioura
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > >
> > >>> > > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> > >>
> >
>

[Emmanouil Gkatziouras]

Hi all!

This is an update. The integration is tested with mocks based on the
original Ignite repo [1].
I am currently waiting for the first commits on the new integration project
to take place in order to create my pull request accordingly to the initial
commits.
In order to simulate the Pub/Sub server in a test I used a test server
implementation [2]. Unfortunately they don't have this util uploaded in a
maven repo, thus it needs to be hosted on a maven repo (Apache License
2.0). Is there a repo for utilities that Ignite uses for the test/build?
I have also created a ticket [3]

Kind regards

[1] https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub
[2] https://github.com/GoogleCloudPlatform/kafka-pubsub-emulator
[3] https://issues.apache.org/jira/browse/IGNITE-12262

*Emmanouil Gkatziouras*
https://egkatzioura.com/ | https://www.linkedin.com/in/gkatziourasemmanouil/
https://github.com/gkatzioura


On Fri, 25 Oct 2019 at 03:35, Saikat Maitra <[hidden email]> wrote:

> Hello Ilya,
>
> Thank you for your email. Yes, before we remove any integration we will
> move them to another repository and release.
>
> We are thinking we will start with Pub/Sub Streamer implementation and
> continue to move other integration as mentioned in the following doc.
>
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
>
> Regards,
> Saikat
>
>
> On Thu, Oct 24, 2019 at 11:00 AM Ilya Kasnacheev <
> [hidden email]>
> wrote:
>
> > Hello!
> >
> > My take on this: we want to offload some of integrations, but please
> commit
> > and release them to another project first, and only then we will decide
> to
> > remove them from our repository.
> >
> > Please also make sure they are available with same artifact name and API.
> > If this has to be changed, let's wait for AI 3.0
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > ср, 23 окт. 2019 г. в 03:23, Saikat Maitra <[hidden email]>:
> >
> > > Hello Emmanouil,
> > >
> > > As discussed earlier, I discussed with Apache Bahir community and they
> > are
> > > interested to have Apache Ignite extensions as part of Apache Bahir
> > > project.
> > >
> > > Can you please share Pub/Sub streamer implementation details with
> Apache
> > > Bahir community and request for feedback.
> > >
> > > https://bahir.apache.org/
> > >
> > > I have also requested for contributor access in Jira for Apache Bahir
> > > project so that I can create issues and assign to myself.
> > >
> > > I can help with code reviews as well.
> > >
> > > Here is the email thread for reference
> > > https://www.mail-archive.com/dev@.../msg02703.html
> > >
> > > Regards,
> > > Saikat
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Oct 21, 2019 at 7:54 PM Saikat Maitra <[hidden email]
> >
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > As discussed I have created following discussion threads on our
> > migration
> > > > proposals for Apache Ignite extensions:
> > > >
> > > >
> > > >
> > >
> >
> http://apache-ignite-users.70518.x6.nabble.com/DISCUSS-Proposal-for-Ignite-Extensions-as-a-separate-Bahir-module-or-Incubator-project-td29829.html
> > > >
> > > > https://www.mail-archive.com/dev@.../msg02703.html
> > > >
> > > > I will share update as I hear more information.
> > > >
> > > > Regards,
> > > > Saikat
> > > >
> > > >
> > > >
> > > > On Fri, Oct 18, 2019 at 9:03 PM Saikat Maitra <
> [hidden email]
> > >
> > > > wrote:
> > > >
> > > >> Hello Denis,
> > > >>
> > > >> Sure, sounds good. I will create a separate discussion thread to get
> > > >> community feedback on whether we
> > > >> should join the Bahir or kick-off "Ignite Extensions" project.
> > > >>
> > > >> Regards,
> > > >> Saikat
> > > >>
> > > >> On Thu, Oct 17, 2019 at 2:21 PM Denis Magda <[hidden email]>
> > wrote:
> > > >>
> > > >>> Folks,
> > > >>>
> > > >>> The concept of Apache Bahir is precisely what we need! Saikat,
> thanks
> > > for
> > > >>> doing the research. Why I believe the * idea* fits us well:
> > > >>>
> > > >>>    - All integrations can be stored in separate Github repositories
> > and
> > > >>>    have their dev lifecycles. We've not obliged to couple all the
> > > >>> integrations
> > > >>>    in a single repo. For instance, Spark can be located in a
> > dedicated
> > > >>> repo
> > > >>>    while streaming integrations might be in a single one. It's up
> to
> > > us.
> > > >>>    - All the repositories and integrations belong to ASF. We're not
> > > >>> dumping
> > > >>>    them on Github but rather keep supporting and developing in
> > > >>> accordance with
> > > >>>    ASF vision and practices.
> > > >>>    - There is a way to reward contributors via committership and
> PMC
> > > >>>    membership. You won't get this if a project is just one of the
> > > >>> millions of
> > > >>>    Github projects.
> > > >>>
> > > >>> Saikat, as Ignite PMC member, could you please kick-off a separate
> > > >>> dev-list
> > > >>> discussion to involve more community members and referring to this
> > > >>> thread?
> > > >>> I think the primary question the community needs to answer whether
> we
> > > >>> should join the Bahir or kick-off "Ignite Extensions" project?
> > > >>>
> > > >>> -
> > > >>> Denis
> > > >>>
> > > >>>
> > > >>> On Thu, Oct 17, 2019 at 2:54 AM Alexey Zinoviev <
> > > [hidden email]>
> > > >>> wrote:
> > > >>>
> > > >>> > Maybe we could move all our Streaming Integrations there, but
> what
> > is
> > > >>> about
> > > >>> > maintaining and committer permissions to the new repositories?
> > > >>> > I see the list of the committers and PMC members there
> > > >>> > https://bahir.apache.org/community-members/
> > > >>> >
> > > >>> > Could somebody from Ignite community be added to this list as a
> > > >>> > PMC/committer to maintain Ignite integrations?
> > > >>> > If yes, I'd happy to join this project and collaborate with these
> > > guys
> > > >>> > together
> > > >>> > If no, and we should start from the zero level with external PRs
> -
> > > >>> hmmm,
> > > >>> > it's better to have our own external repository with ApacheBahirr
> > > >>> ideology.
> > > >>> >
> > > >>> > Also, I agree, that all connectors could be moved there (in
> > > >>> ApacheBahirr
> > > >>> > like repository), but not all modules from the page
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> > > >>> > because
> > > >>> > they use different parts of core modules and could be developed
> > with
> > > >>> > different velocity.
> > > >>> >
> > > >>> > In reality, before creation of new repositories we need wide
> > > discussion
> > > >>> > based on the document mentioned above.
> > > >>> >
> > > >>> > P.S. Of course, we could experiment in the next release 2.8 with
> > one
> > > >>> or two
> > > >>> > integrations like twitter or ZeroMQ
> > > >>> > Also, I'd like Denis Magda idea for the separate repositories in
> > > Apache
> > > >>> > Github to maintain them by anybody.
> > > >>> >
> > > >>> >
> > > >>> > чт, 17 окт. 2019 г. в 05:02, Saikat Maitra <
> > [hidden email]
> > > >:
> > > >>> >
> > > >>> > > Hi Denis, Emmanouil
> > > >>> > >
> > > >>> > > Thank you for your email. I think creating a separate
> integration
> > > >>> project
> > > >>> > > in github and also proposing it as an apache incubator project
> > will
> > > >>> be a
> > > >>> > > good move. The other separate integration modules
> > > >>> > >
> > > >>> > >
> > > >>> >
> > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> > > >>> > > can
> > > >>> > > be moved to this new apache incubator project.
> > > >>> > >
> > > >>> > > There are similar integration available for Flink and Spark in
> > > Apache
> > > >>> > Bahir
> > > >>> > > https://bahir.apache.org/
> > > >>> > >
> > > >>> > > Do you think we should reach out to Apache Bahir community
> with a
> > > >>> > proposal
> > > >>> > > or we should plan to create a separate Apache Incubator
> project?
> > > >>> > >
> > > >>> > > Regards,
> > > >>> > > Saikat
> > > >>> > >
> > > >>> > >
> > > >>> > >
> > > >>> > > On Wed, Oct 16, 2019 at 6:21 AM Emmanouil Gkatziouras <
> > > >>> > > [hidden email]>
> > > >>> > > wrote:
> > > >>> > >
> > > >>> > > > Hi all!
> > > >>> > > >
> > > >>> > > > Based on the discussions most probably this is going to be on
> > > >>> another
> > > >>> > > > GitHub repo. Therefore I will prepare a standalone project
> with
> > > the
> > > >>> > > Pub/Sub
> > > >>> > > > module and once the repository is created a pull request
> shall
> > be
> > > >>> > issued.
> > > >>> > > > If there is anything else I can do in order to facilitate
> this
> > > >>> please
> > > >>> > let
> > > >>> > > > me know.
> > > >>> > > >
> > > >>> > > > If a ticket is created for this issue, I shall be able to
> > create
> > > a
> > > >>> pull
> > > >>> > > > request and thus a build will take place on ignite's CI.
> > > >>> > > > If everyone is aligned with adding a Pub/Sub implementation
> > may I
> > > >>> > create
> > > >>> > > a
> > > >>> > > > ticket on JIRA?
> > > >>> > > >
> > > >>> > > > My next question has to do on how Ignite configures the build
> > > jobs
> > > >>> in
> > > >>> > > Team
> > > >>> > > > City. The unit tests for my implementation need a local
> Pub/Sub
> > > >>> server
> > > >>> > up
> > > >>> > > > and running (they provide one for testing purposes). Will
> this
> > > take
> > > >>> > > effect
> > > >>> > > > in a form of a build script inside my implementation or is it
> > > >>> something
> > > >>> > > > that should be configured on Team City?
> > > >>> > > >
> > > >>> > > > Kind regards,
> > > >>> > > > Emmanouil
> > > >>> > > >
> > > >>> > > >
> > > >>> > > > *Emmanouil Gkatziouras*
> > > >>> > > > https://egkatzioura.com/ |
> > > >>> > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > >>> > > > https://github.com/gkatzioura
> > > >>> > > >
> > > >>> > > >
> > > >>> > > > On Tue, 15 Oct 2019 at 23:27, Denis Magda <[hidden email]
> >
> > > >>> wrote:
> > > >>> > > >
> > > >>> > > > > Emmanouil, Saikat,
> > > >>> > > > >
> > > >>> > > > > After contemplating on this, let me propose another way
> with
> > > this
> > > >>> > > > > particular integration that is aligned with our
> > modularization
> > > >>> > > endeavor.
> > > >>> > > > > The modularization [1] defines the Ignite core, Ignite
> > modules,
> > > >>> and
> > > >>> > > > > independent integrations. The first two are to be located
> in
> > > >>> Ignite
> > > >>> > > > > repositories and to be tested/updated by the community for
> > > every
> > > >>> > > release.
> > > >>> > > > > The modules are a subset of existing integrations that are
> > > >>> adopted
> > > >>> > > widely
> > > >>> > > > > and require community attention.
> > > >>> > > > >
> > > >>> > > > > As for the independent integrations [1], those are supposed
> > to
> > > be
> > > >>> > > located
> > > >>> > > > > in a separate GitHub repo. I would advise us to start
> > > >>> implementing
> > > >>> > this
> > > >>> > > > > concept with the Pub/Sub integration. Even though it will
> be
> > > >>> located
> > > >>> > > in a
> > > >>> > > > > separate Github repo, it will be listed among officially
> > > >>> available
> > > >>> > > > > integrations on the Ignite website featuring Emmanouil as a
> > > >>> > > contributor.
> > > >>> > > > >
> > > >>> > > > > What do you think?
> > > >>> > > > >
> > > >>> > > > >
> > > >>> > > > > [1]
> > > >>> > > > >
> > > >>> > > > >
> > > >>> > > >
> > > >>> > >
> > > >>> >
> > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> > > >>> > > > >
> > > >>> > > > >
> > > >>> > > > >
> > > >>> > > > > -
> > > >>> > > > > Denis
> > > >>> > > > >
> > > >>> > > > >
> > > >>> > > > > On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra <
> > > >>> > [hidden email]
> > > >>> > > >
> > > >>> > > > > wrote:
> > > >>> > > > >
> > > >>> > > > > > Hi Emmanouil,
> > > >>> > > > > >
> > > >>> > > > > > The changes looks good to me. I wanted to check if you
> are
> > > also
> > > >>> > able
> > > >>> > > to
> > > >>> > > > > > validate that the once data is added to Ignite cluster
> you
> > > are
> > > >>> also
> > > >>> > > > able
> > > >>> > > > > to
> > > >>> > > > > > access it using another Ignite client or using rest
> > > endpoints.
> > > >>> > > > > >
> > > >>> > > > > > We use Teamcity for CI process, the details are as
> follows
> > > >>> > > > > >
> > > >>> > > > > > Trigger validation of those test suites that have been
> > > >>> affected by
> > > >>> > > your
> > > >>> > > > > > changes on TeamCity <http://ci.ignite.apache.org/>: <
> > > >>> > > > > http://204.14.53.153/
> > > >>> > > > > > >
> > > >>> > > > > >
> > > >>> > > > > >    - Locate a test suite you have to check, press button
> > > named
> > > >>> > "..."
> > > >>> > > > that
> > > >>> > > > > >    is located on the left of "Run" button. "Run custom
> > build"
> > > >>> > dialog
> > > >>> > > > will
> > > >>> > > > > >    appear;
> > > >>> > > > > >    - Go to "Changes" tab and choose
> > > >>> > "pull/<pull-request-number>/head"
> > > >>> > > > in
> > > >>> > > > > >    "Build branch" dropdown list;
> > > >>> > > > > >    - Press "Run build" button and monitor tests results.
> > > >>> > > > > >
> > > >>> > > > > >
> > > >>> > > > > >
> > > >>> > > > > > @Roman [hidden email] - can you also please take a
> > look?
> > > >>> > > > > >
> > > >>> > > > > > Regards,
> > > >>> > > > > > Saikat
> > > >>> > > > > >
> > > >>> > > > > > On Sun, Oct 13, 2019 at 6:20 PM Emmanouil Gkatziouras <
> > > >>> > > > > > [hidden email]>
> > > >>> > > > > > wrote:
> > > >>> > > > > >
> > > >>> > > > > > > Hi @Saikat Maitra
> > > >>> > > > > > >
> > > >>> > > > > > > I have my implementation on an Ignite fork on my
> personal
> > > >>> GitHub
> > > >>> > > > > account
> > > >>> > > > > > > [1]
> > > >>> > > > > > > It is based on the Kafka implementation which is of a
> > > polling
> > > >>> > > nature.
> > > >>> > > > > > > GCP does not provide a service like Kafka. Instead it
> > > >>> provides us
> > > >>> > > > with
> > > >>> > > > > > > Pub/Sub which is just like Kafka.
> > > >>> > > > > > > Therefore using the Kafka streamer as a guideline made
> > more
> > > >>> sense
> > > >>> > > to
> > > >>> > > > > me.
> > > >>> > > > > > >
> > > >>> > > > > > > Apart from the implementation I have also added a unit
> > > test.
> > > >>> > > > > > > It definitely needs some polishing and a double check
> on
> > my
> > > >>> side
> > > >>> > to
> > > >>> > > > > make
> > > >>> > > > > > > the review smooth.
> > > >>> > > > > > > For the tests a local Pub/Sub [2] server has to be
> setup
> > > >>> however
> > > >>> > I
> > > >>> > > am
> > > >>> > > > > not
> > > >>> > > > > > > sure which is the best way to do so with regards to
> your
> > > >>> CI/CD.
> > > >>> > > > > > >
> > > >>> > > > > > > [1]
> > > >>> > >
> https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub
> > > >>> > > > > > > [2]
> > > >>> https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/
> > > >>> > > > > > >
> > > >>> > > > > > > *Emmanouil Gkatziouras*
> > > >>> > > > > > > https://egkatzioura.com/ |
> > > >>> > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > >>> > > > > > > https://github.com/gkatzioura
> > > >>> > > > > > >
> > > >>> > > > > > >
> > > >>> > > > > > > On Sat, 12 Oct 2019 at 02:11, Saikat Maitra <
> > > >>> > > [hidden email]
> > > >>> > > > >
> > > >>> > > > > > > wrote:
> > > >>> > > > > > >
> > > >>> > > > > > > > Hello Emmanouil,
> > > >>> > > > > > > >
> > > >>> > > > > > > > Can you please share more info about the
> > implementation?
> > > >>> > > > > > > >
> > > >>> > > > > > > > We have similar implementation for data streamers
> with
> > > >>> Kafka
> > > >>> > and
> > > >>> > > > > > RocketMQ
> > > >>> > > > > > > >
> > > >>> > > > > > > >
> https://apacheignite-mix.readme.io/docs/kafka-streamer
> > > >>> > > > > > > >
> > > >>> > > > > > > >
> > > https://apacheignite-mix.readme.io/docs/rocketmq-streamer
> > > >>> > > > > > > >
> > > >>> > > > > > > > Please review and share if you have any questions.
> > > >>> > > > > > > >
> > > >>> > > > > > > > Regards,
> > > >>> > > > > > > > Saikat
> > > >>> > > > > > > >
> > > >>> > > > > > > >
> > > >>> > > > > > > >
> > > >>> > > > > > > > On Fri, Oct 11, 2019 at 4:56 PM Emmanouil
> Gkatziouras <
> > > >>> > > > > > > > [hidden email]>
> > > >>> > > > > > > > wrote:
> > > >>> > > > > > > >
> > > >>> > > > > > > > > Hi all,
> > > >>> > > > > > > > >
> > > >>> > > > > > > > > By mentioning Pub/Sub I mean Google Cloud's Pub/Sub
> > [1]
> > > >>> > > > > > > > > Yes it is indeed cloud specific. It is the
> messaging
> > > >>> service
> > > >>> > > > > provided
> > > >>> > > > > > > by
> > > >>> > > > > > > > > GCP.
> > > >>> > > > > > > > > I have made the integration with it, but I haven't
> > > >>> published
> > > >>> > it
> > > >>> > > > yet
> > > >>> > > > > > (I
> > > >>> > > > > > > am
> > > >>> > > > > > > > > waiting for the discussion/ticket and proceed with
> > the
> > > >>> > > > > contribution).
> > > >>> > > > > > > > >
> > > >>> > > > > > > > > Kind regards,
> > > >>> > > > > > > > > Emmanouil
> > > >>> > > > > > > > >
> > > >>> > > > > > > > > [1] https://cloud.google.com/pubsub/docs/overview
> > > >>> > > > > > > > >
> > > >>> > > > > > > > > *Emmanouil Gkatziouras*
> > > >>> > > > > > > > > https://egkatzioura.com/ |
> > > >>> > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > >>> > > > > > > > > https://github.com/gkatzioura
> > > >>> > > > > > > > >
> > > >>> > > > > > > > >
> > > >>> > > > > > > > > On Fri, 11 Oct 2019 at 21:45, Denis Magda <
> > > >>> [hidden email]
> > > >>> > >
> > > >>> > > > > wrote:
> > > >>> > > > > > > > >
> > > >>> > > > > > > > > > Hello Emmanouil,
> > > >>> > > > > > > > > >
> > > >>> > > > > > > > > > Are you referring to any specific implementation
> > > >>> tailored
> > > >>> > > for a
> > > >>> > > > > > > cloud?
> > > >>> > > > > > > > If
> > > >>> > > > > > > > > > I'm not mistaken you were mentioning AWS/GCE. Is
> > > there
> > > >>> any
> > > >>> > > > Github
> > > >>> > > > > > > > source
> > > >>> > > > > > > > > > code?
> > > >>> > > > > > > > > >
> > > >>> > > > > > > > > > Here is a list of existing integrations with
> > > streaming
> > > >>> and
> > > >>> > > > > pub/sub
> > > >>> > > > > > > > > > frameworks:
> > > >>> > > > > > > > > > https://apacheignite-mix.readme.io/docs/overview
> > > >>> > > > > > > > > >
> > > >>> > > > > > > > > > with the source code in Ignite master:
> > > >>> > > > > > > > > >
> > https://github.com/apache/ignite/tree/master/modules
> > > >>> > > > > > > > > >
> > > >>> > > > > > > > > > Check out those resources to see what are the
> most
> > > >>> suitable
> > > >>> > > > > Ignite
> > > >>> > > > > > > APIs
> > > >>> > > > > > > > > for
> > > >>> > > > > > > > > > such integrations.
> > > >>> > > > > > > > > >
> > > >>> > > > > > > > > > @Saikat Maitra <[hidden email]> ,
> @Roman
> > <
> > > >>> > > > > > [hidden email]
> > > >>> > > > > > > >
> > > >>> > > > > > > > ,
> > > >>> > > > > > > > > as
> > > >>> > > > > > > > > > streaming technologies contributors/maintainers,
> > > could
> > > >>> you
> > > >>> > > > please
> > > >>> > > > > > > join
> > > >>> > > > > > > > > this
> > > >>> > > > > > > > > > thread?
> > > >>> > > > > > > > > >
> > > >>> > > > > > > > > > -
> > > >>> > > > > > > > > > Denis
> > > >>> > > > > > > > > >
> > > >>> > > > > > > > > >
> > > >>> > > > > > > > > > On Thu, Oct 10, 2019 at 7:58 AM Emmanouil
> > > Gkatziouras <
> > > >>> > > > > > > > > > [hidden email]>
> > > >>> > > > > > > > > > wrote:
> > > >>> > > > > > > > > >
> > > >>> > > > > > > > > > > Hello everyone,
> > > >>> > > > > > > > > > >
> > > >>> > > > > > > > > > > I started using Ignite lately. Part of my work
> > > >>> involves
> > > >>> > > > > Pub/Sub,
> > > >>> > > > > > > > thus I
> > > >>> > > > > > > > > > > created a pub/sub streamer.
> > > >>> > > > > > > > > > > Pub/Sub is pretty close to Kafka.
> > > >>> > > > > > > > > > > Here's how the streamer works. Every node
> creates
> > > an
> > > >>> > > instance
> > > >>> > > > > of
> > > >>> > > > > > a
> > > >>> > > > > > > > > > Pub/Sub
> > > >>> > > > > > > > > > > client.
> > > >>> > > > > > > > > > > The clients subscribe to the same topic using
> the
> > > >>> same
> > > >>> > > > > > subscription
> > > >>> > > > > > > > > name.
> > > >>> > > > > > > > > > > A message is being sent and one of the nodes
> will
> > > >>> receive
> > > >>> > > the
> > > >>> > > > > > > > message,
> > > >>> > > > > > > > > > > process it and then forward it to Ignite using
> > the
> > > >>> > > extractor
> > > >>> > > > > > > > provided.
> > > >>> > > > > > > > > > >
> > > >>> > > > > > > > > > > Any thoughts?
> > > >>> > > > > > > > > > >
> > > >>> > > > > > > > > > > *Emmanouil Gkatziouras*
> > > >>> > > > > > > > > > > https://egkatzioura.com/ |
> > > >>> > > > > > > > > > >
> > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > >>> > > > > > > > > > > https://github.com/gkatzioura
> > > >>> > > > > > > > > > >
> > > >>> > > > > > > > > >
> > > >>> > > > > > > > >
> > > >>> > > > > > > >
> > > >>> > > > > > >
> > > >>> > > > > >
> > > >>> > > > >
> > > >>> > > >
> > > >>> > >
> > > >>> >
> > > >>>
> > > >>
> > >
> >
>