Integration with Apache Apex (incubating)

[Thomas Weise]

Dear Ignite community,

Apache Apex (http://apex.incubator.apache.org/) is an in-memory stream
processing and analytics platform, built for scalability, low-latency
processing, high availability and operability. It has a pipeline processing
architecture can be used for real-time and batch processing. Apex runs
natively on YARN and HDFS.

Typical Apex applications integrate with external systems: ingestion, state
saving, lookup data for processing, land compute results for downstream
consumers etc. The Malhar library has a wide range of connectors and
compute functions readily available to assemble Apex applications.

I wanted to see if there is interest to develop Ignite integrations as part
of Malhar. Some ideas:

   - Checkpointing (
   https://www.datatorrent.com/blog/blog-introduction-to-checkpoint/) could
   use the IGFS (possibly with Ignite native integration of the storage agent
   interface).
   - Compute results could be stored in Ignite (example dimension
   computation
   https://www.datatorrent.com/blog/blog-dimensions-computation-aggregate-navigator-part-1-intro/
    )
   - Enrichment/lookup could be implemented with Ignite as storage layer.

Thanks,
Thomas

[dsetrakyan]

Hi Thomas,

Welcome to the Ignite community! Thanks for suggesting various ways to
integrate with Apache Apex. Give me a couple of days to look at it deeper
and respond.

D.

On Sun, Feb 21, 2016 at 10:09 AM, Thomas Weise <[hidden email]> wrote:

> Dear Ignite community,
>
> Apache Apex (http://apex.incubator.apache.org/) is an in-memory stream
> processing and analytics platform, built for scalability, low-latency
> processing, high availability and operability. It has a pipeline processing
> architecture can be used for real-time and batch processing. Apex runs
> natively on YARN and HDFS.
>
> Typical Apex applications integrate with external systems: ingestion, state
> saving, lookup data for processing, land compute results for downstream
> consumers etc. The Malhar library has a wide range of connectors and
> compute functions readily available to assemble Apex applications.
>
> I wanted to see if there is interest to develop Ignite integrations as part
> of Malhar. Some ideas:
>
>    - Checkpointing (
>    https://www.datatorrent.com/blog/blog-introduction-to-checkpoint/)
> could
>    use the IGFS (possibly with Ignite native integration of the storage
> agent
>    interface).
>    - Compute results could be stored in Ignite (example dimension
>    computation
>
> https://www.datatorrent.com/blog/blog-dimensions-computation-aggregate-navigator-part-1-intro/
>     )
>    - Enrichment/lookup could be implemented with Ignite as storage layer.
>
> Thanks,
> Thomas
>

[sbiradar1975]

Hello D,
I landed on this thread while searching for the Apache Ignite integration with Apex. Is there an update on this topic somewhere that I might missing?  

Thanks!

[Vladisav Jelisavcic]

Hi,

I just recently started working on this,
you can track progress here [1] and of course here [2],
we also started a separate thread here [3].

Best regards,
Vladisav

[1] https://issues.apache.org/jira/browse/IGNITE-3131
[2] https://issues.apache.org/jira/browse/APEXMALHAR-2091
[3]
https://mail-archives.apache.org/mod_mbox/ignite-dev/201605.mbox/%3CCA+5xAo3jOuFk-NDbNrfDw-BckDC9ufRVmYW3x3FOZ=B771GQAg@...%3E


On Tue, May 24, 2016 at 1:33 AM, sbiradar1975 <[hidden email]> wrote:

> Hello D,
> I landed on this thread while searching for the Apache Ignite integration
> with Apex. Is there an update on this topic somewhere that I might missing?
>
> Thanks!
>
>
>
> --
> View this message in context:
> http://apache-ignite-developers.2346864.n4.nabble.com/Integration-with-Apache-Apex-incubating-tp7535p9069.html
> Sent from the Apache Ignite Developers mailing list archive at Nabble.com.
>