Using IgniteContext

[Konstantin Boudnik-2]

Guys,

I am running a simple example to check the latest Spark acceleration module in
Ignite and am hitting some strange stuff (which musta be my configuration
issue, but I can not figure out what is this).

This is how I run spark-shell in my docker container (spark-master and worker
are started in the same container)

% spark-shell --packages org.apache.ignite:ignite-spark:1.3.3-p2-SNAPSHOT --repositories https://repository.apache.org/content/groups/snapshots --master spark://ignite.docker:7077

then I execute this:

scala>
import org.apache.ignite.spark._
import org.apache.ignite.configuration._
val ic = new IgniteContext[Integer, Integer](sc, "/bigtop/spark-ignite-config.xml")

15/08/18 23:26:52 INFO XmlBeanDefinitionReader: Loading XML bean definitions from URL [file:/bigtop/spark-ignite-config.xml]
15/08/18 23:26:52 INFO GenericApplicationContext: Refreshing org.springframework.context.support.GenericApplicationContext@7f85c5d1: startup date [Tue Aug 18 2
3:26:52 UTC 2015]; root of context hierarchy
15/08/18 23:26:53 INFO IgniteKernal:

>>>    __________  ________________
>>>   /  _/ ___/ |/ /  _/_  __/ __/
>>>  _/ // (7 7    // /  / / / _/
>>> /___/\___/_/|_/___/ /_/ /___/
>>>
>>> ver. 1.3.3-p2-SNAPSHOT#20150814-sha1:9300bbc7
>>> 2015 Copyright(C) Apache Software Foundation
>>>
>>> Ignite documentation: http://ignite.incubator.apache.org

15/08/18 23:26:53 INFO IgniteKernal: Config URL: n/a
15/08/18 23:26:53 INFO IgniteKernal: Daemon mode: off
15/08/18 23:26:53 INFO IgniteKernal: OS: Linux 3.13.0-61-generic amd64
15/08/18 23:26:53 INFO IgniteKernal: OS user: root
15/08/18 23:26:53 INFO IgniteKernal: Language runtime: Scala ver. 2.10.4
15/08/18 23:26:53 INFO IgniteKernal: VM information: Java(TM) SE Runtime Environment 1.7.0_60-b19 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 24.60-b0
9
15/08/18 23:26:53 INFO IgniteKernal: VM total memory: 0.48GB
15/08/18 23:26:53 INFO IgniteKernal: Remote Management [restart: off, REST: on, JMX (remote: off)]
15/08/18 23:26:53 INFO IgniteKernal: IGNITE_HOME=null
15/08/18 23:26:53 INFO IgniteKernal: VM arguments: [-XX:MaxPermSize=128m, -Dscala.usejavacp=true, -Xms512m, -Xmx512m]
15/08/18 23:26:53 INFO IgniteKernal: Configured caches ['ignite-marshaller-sys-cache', 'ignite-sys-cache', 'ignite-atomics-sys-cache', 'SharedNumbers']
15/08/18 23:26:53 INFO IgniteKernal: Non-loopback local IPs: 172.17.0.1, fe80:0:0:0:42:acff:fe11:1%4
15/08/18 23:26:53 INFO IgniteKernal: Enabled local MACs: 0242AC110001
15/08/18 23:26:53 INFO IgnitePluginProcessor: Configured plugins:
15/08/18 23:26:53 INFO IgnitePluginProcessor:   ^-- None
15/08/18 23:26:53 INFO IgnitePluginProcessor:
15/08/18 23:26:53 INFO TcpCommunicationSpi: IPC shared memory server endpoint started [port=48100, tokDir=/tmp/ignite/work/ipc/shmem/9fa0e60d-8d5a-4606-9797-4dc86c683a32-10416]
15/08/18 23:26:53 INFO TcpCommunicationSpi: Successfully bound shared memory communication to TCP port [port=48100, locHost=0.0.0.0/0.0.0.0]
15/08/18 23:26:53 INFO TcpCommunicationSpi: Successfully bound to TCP port [port=47100, locHost=0.0.0.0/0.0.0.0]
15/08/18 23:26:53 WARN NoopCheckpointSpi: Checkpoints are disabled (to enable configure any GridCheckpointSpi implementation)
15/08/18 23:26:53 WARN GridCollisionManager: Collision resolution is disabled (all jobs will be activated upon arrival).
15/08/18 23:26:53 WARN NoopSwapSpaceSpi: Swap space is disabled. To enable use FileSwapSpaceSpi.
15/08/18 23:26:53 INFO IgniteKernal: Security status [authentication=off, communication encryption=off]
15/08/18 23:26:53 INFO GridTcpRestProtocol: Command protocol successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0, port=11211]
15/08/18 23:26:53 WARN TcpDiscoveryMulticastIpFinder: TcpDiscoveryMulticastIpFinder has no pre-configured addresses (it is recommended in production to specify at least one address in TcpDiscoveryMulticastIpFinder.getAddresses() configuration property)
15/08/18 23:26:53 WARN TcpDiscoverySpi: No addresses registered in the IP finder (will retry in 2000ms): TcpDiscoveryMulticastIpFinder [mcastGrp=228.1.2.4, mcastPort=47400, resWaitTime=500, addrReqAttempts=2, locAddr=null, super=TcpDiscoveryVmIpFinder [addrs=[], super=TcpDiscoveryIpFinderAdapter [shared=true]]]
[23:27:03] New version is available at ....

and after that the process is just hanging there without any progress. The spark-ignite-config is below

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="
       http://www.springframework.org/schema/beans
       http://www.springframework.org/schema/beans/spring-beans.xsd">
    <!--
        Alter configuration below as needed.
    -->
    <bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
      <property name="cacheConfiguration">
        <list>
            <!-- Partitioned cache example configuration (Atomic mode). -->
            <bean class="org.apache.ignite.configuration.CacheConfiguration">
                <property name="name" value="SharedNumbers"/>
                <property name="affinity">
                  <bean class="org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction">
                    <property name="partitions" value="32"/>
                  </bean>
                </property>
                <property name="indexedTypes">
                    <array>
                        <value>java.lang.Integer</value>
                        <value>java.lang.Integer</value>
                    </array>
                </property>
            </bean>
        </list>
      </property>
    </bean>
</beans>

What shall I do at this point? Thanks in advance!
  Cos

[Alexey Goncharuk]

Cos,

You have created IgniteContext in client mode (it is a default mode), so
IgniteContext waits until at least one server node is started. I thought we
fixed the message for a client that cannot find server nodes, but
apparently the message is not clear yet. To fix this, just start
additionally an ignite node with ignite.sh and the same config.

Also, let me know how I can improve the documentation about Igntie-Spark
integration - you are one of the first persons trying it by yourself, so
your feedback is priceless :)

2015-08-18 16:42 GMT-07:00 Konstantin Boudnik <[hidden email]>:

> Guys,
>
> I am running a simple example to check the latest Spark acceleration
> module in
> Ignite and am hitting some strange stuff (which musta be my configuration
> issue, but I can not figure out what is this).
>
> This is how I run spark-shell in my docker container (spark-master and
> worker
> are started in the same container)
>
> % spark-shell --packages org.apache.ignite:ignite-spark:1.3.3-p2-SNAPSHOT
> --repositories https://repository.apache.org/content/groups/snapshots
> --master spark://ignite.docker:7077
>
> then I execute this:
>
> scala>
> import org.apache.ignite.spark._
> import org.apache.ignite.configuration._
> val ic = new IgniteContext[Integer, Integer](sc,
> "/bigtop/spark-ignite-config.xml")
>
> 15/08/18 23:26:52 INFO XmlBeanDefinitionReader: Loading XML bean
> definitions from URL [file:/bigtop/spark-ignite-config.xml]
> 15/08/18 23:26:52 INFO GenericApplicationContext: Refreshing
> org.springframework.context.support.GenericApplicationContext@7f85c5d1:
> startup date [Tue Aug 18 2
> 3:26:52 UTC 2015]; root of context hierarchy
> 15/08/18 23:26:53 INFO IgniteKernal:
>
> >>>    __________  ________________
> >>>   /  _/ ___/ |/ /  _/_  __/ __/
> >>>  _/ // (7 7    // /  / / / _/
> >>> /___/\___/_/|_/___/ /_/ /___/
> >>>
> >>> ver. 1.3.3-p2-SNAPSHOT#20150814-sha1:9300bbc7
> >>> 2015 Copyright(C) Apache Software Foundation
> >>>
> >>> Ignite documentation: http://ignite.incubator.apache.org
>
> 15/08/18 23:26:53 INFO IgniteKernal: Config URL: n/a
> 15/08/18 23:26:53 INFO IgniteKernal: Daemon mode: off
> 15/08/18 23:26:53 INFO IgniteKernal: OS: Linux 3.13.0-61-generic amd64
> 15/08/18 23:26:53 INFO IgniteKernal: OS user: root
> 15/08/18 23:26:53 INFO IgniteKernal: Language runtime: Scala ver. 2.10.4
> 15/08/18 23:26:53 INFO IgniteKernal: VM information: Java(TM) SE Runtime
> Environment 1.7.0_60-b19 Oracle Corporation Java HotSpot(TM) 64-Bit Server
> VM 24.60-b0
> 9
> 15/08/18 23:26:53 INFO IgniteKernal: VM total memory: 0.48GB
> 15/08/18 23:26:53 INFO IgniteKernal: Remote Management [restart: off,
> REST: on, JMX (remote: off)]
> 15/08/18 23:26:53 INFO IgniteKernal: IGNITE_HOME=null
> 15/08/18 23:26:53 INFO IgniteKernal: VM arguments: [-XX:MaxPermSize=128m,
> -Dscala.usejavacp=true, -Xms512m, -Xmx512m]
> 15/08/18 23:26:53 INFO IgniteKernal: Configured caches
> ['ignite-marshaller-sys-cache', 'ignite-sys-cache',
> 'ignite-atomics-sys-cache', 'SharedNumbers']
> 15/08/18 23:26:53 INFO IgniteKernal: Non-loopback local IPs: 172.17.0.1,
> fe80:0:0:0:42:acff:fe11:1%4
> 15/08/18 23:26:53 INFO IgniteKernal: Enabled local MACs: 0242AC110001
> 15/08/18 23:26:53 INFO IgnitePluginProcessor: Configured plugins:
> 15/08/18 23:26:53 INFO IgnitePluginProcessor:   ^-- None
> 15/08/18 23:26:53 INFO IgnitePluginProcessor:
> 15/08/18 23:26:53 INFO TcpCommunicationSpi: IPC shared memory server
> endpoint started [port=48100,
> tokDir=/tmp/ignite/work/ipc/shmem/9fa0e60d-8d5a-4606-9797-4dc86c683a32-10416]
> 15/08/18 23:26:53 INFO TcpCommunicationSpi: Successfully bound shared
> memory communication to TCP port [port=48100, locHost=0.0.0.0/0.0.0.0]
> 15/08/18 23:26:53 INFO TcpCommunicationSpi: Successfully bound to TCP port
> [port=47100, locHost=0.0.0.0/0.0.0.0]
> 15/08/18 23:26:53 WARN NoopCheckpointSpi: Checkpoints are disabled (to
> enable configure any GridCheckpointSpi implementation)
> 15/08/18 23:26:53 WARN GridCollisionManager: Collision resolution is
> disabled (all jobs will be activated upon arrival).
> 15/08/18 23:26:53 WARN NoopSwapSpaceSpi: Swap space is disabled. To enable
> use FileSwapSpaceSpi.
> 15/08/18 23:26:53 INFO IgniteKernal: Security status [authentication=off,
> communication encryption=off]
> 15/08/18 23:26:53 INFO GridTcpRestProtocol: Command protocol successfully
> started [name=TCP binary, host=0.0.0.0/0.0.0.0, port=11211]
> 15/08/18 23:26:53 WARN TcpDiscoveryMulticastIpFinder:
> TcpDiscoveryMulticastIpFinder has no pre-configured addresses (it is
> recommended in production to specify at least one address in
> TcpDiscoveryMulticastIpFinder.getAddresses() configuration property)
> 15/08/18 23:26:53 WARN TcpDiscoverySpi: No addresses registered in the IP
> finder (will retry in 2000ms): TcpDiscoveryMulticastIpFinder
> [mcastGrp=228.1.2.4, mcastPort=47400, resWaitTime=500, addrReqAttempts=2,
> locAddr=null, super=TcpDiscoveryVmIpFinder [addrs=[],
> super=TcpDiscoveryIpFinderAdapter [shared=true]]]
> [23:27:03] New version is available at ....
>
> and after that the process is just hanging there without any progress. The
> spark-ignite-config is below
>
> <beans xmlns="http://www.springframework.org/schema/beans"
>        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>        xsi:schemaLocation="
>        http://www.springframework.org/schema/beans
>        http://www.springframework.org/schema/beans/spring-beans.xsd">
>     <!--
>         Alter configuration below as needed.
>     -->
>     <bean id="grid.cfg"
> class="org.apache.ignite.configuration.IgniteConfiguration">
>       <property name="cacheConfiguration">
>         <list>
>             <!-- Partitioned cache example configuration (Atomic mode). -->
>             <bean
> class="org.apache.ignite.configuration.CacheConfiguration">
>                 <property name="name" value="SharedNumbers"/>
>                 <property name="affinity">
>                   <bean
> class="org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction">
>                     <property name="partitions" value="32"/>
>                   </bean>
>                 </property>
>                 <property name="indexedTypes">
>                     <array>
>                         <value>java.lang.Integer</value>
>                         <value>java.lang.Integer</value>
>                     </array>
>                 </property>
>             </bean>
>         </list>
>       </property>
>     </bean>
> </beans>
>
> What shall I do at this point? Thanks in advance!
>   Cos
>
>
>

[Konstantin Boudnik-2]

Thanks Alexey - that helped somewhat! and let us to figure out the following.

For the benefit of the others here's what we have discovered during the skype-debug:

 1. server and client node process should be started under the same uid. In my
    case I was starting server node as root and then executor was exec'ed by a
    spark-worker. The latter made the effective uid of the executor's owner to
    be spark.
 2. When bin/ignite.sh starts not from Linux service it uses default value for
    work_dir == $IGNITE_HOME/work On Linux systems we are setting work_dir to
    /var/run/ignite-hadoop/work belonging to ignite:ignite with effective
    permissions 777

Because of 2. in 1. case executor couldn't write into server's node
work directory with root-only write permissions. As you've figured out: some
exceptions weren't propagating all the way up which made the debugging to be a
royal PITA.

The right way to fix it, I believe, is to make bin/ignite.sh to always check
for /etc/default/ignite-hadoop and source it if found. This way, we'll get the
same environment no matter if the node started manually or via linux service.
What do you think?

We have stepped on another issue, where ignie-indexing libs weren't included
into the package. While it is perfectly legit for Hadoop accelerator, for
Spark accelerator these are essential for SQL querying. I have opened
BIGTOP-1969 to address that.

Thanks for your help tonight!
  Cos

On Tue, Aug 18, 2015 at 07:44PM, Alexey Goncharuk wrote:

> Cos,
>
> You have created IgniteContext in client mode (it is a default mode), so
> IgniteContext waits until at least one server node is started. I thought we
> fixed the message for a client that cannot find server nodes, but
> apparently the message is not clear yet. To fix this, just start
> additionally an ignite node with ignite.sh and the same config.
>
> Also, let me know how I can improve the documentation about Igntie-Spark
> integration - you are one of the first persons trying it by yourself, so
> your feedback is priceless :)
>
> 2015-08-18 16:42 GMT-07:00 Konstantin Boudnik <[hidden email]>:
>
> > Guys,
> >
> > I am running a simple example to check the latest Spark acceleration
> > module in
> > Ignite and am hitting some strange stuff (which musta be my configuration
> > issue, but I can not figure out what is this).
> >
> > This is how I run spark-shell in my docker container (spark-master and
> > worker
> > are started in the same container)
> >
> > % spark-shell --packages org.apache.ignite:ignite-spark:1.3.3-p2-SNAPSHOT
> > --repositories https://repository.apache.org/content/groups/snapshots
> > --master spark://ignite.docker:7077
> >
> > then I execute this:
> >
> > scala>
> > import org.apache.ignite.spark._
> > import org.apache.ignite.configuration._
> > val ic = new IgniteContext[Integer, Integer](sc,
> > "/bigtop/spark-ignite-config.xml")
> >
> > 15/08/18 23:26:52 INFO XmlBeanDefinitionReader: Loading XML bean
> > definitions from URL [file:/bigtop/spark-ignite-config.xml]
> > 15/08/18 23:26:52 INFO GenericApplicationContext: Refreshing
> > org.springframework.context.support.GenericApplicationContext@7f85c5d1:
> > startup date [Tue Aug 18 2
> > 3:26:52 UTC 2015]; root of context hierarchy
> > 15/08/18 23:26:53 INFO IgniteKernal:
> >
> > >>>    __________  ________________
> > >>>   /  _/ ___/ |/ /  _/_  __/ __/
> > >>>  _/ // (7 7    // /  / / / _/
> > >>> /___/\___/_/|_/___/ /_/ /___/
> > >>>
> > >>> ver. 1.3.3-p2-SNAPSHOT#20150814-sha1:9300bbc7
> > >>> 2015 Copyright(C) Apache Software Foundation
> > >>>
> > >>> Ignite documentation: http://ignite.incubator.apache.org
> >
> > 15/08/18 23:26:53 INFO IgniteKernal: Config URL: n/a
> > 15/08/18 23:26:53 INFO IgniteKernal: Daemon mode: off
> > 15/08/18 23:26:53 INFO IgniteKernal: OS: Linux 3.13.0-61-generic amd64
> > 15/08/18 23:26:53 INFO IgniteKernal: OS user: root
> > 15/08/18 23:26:53 INFO IgniteKernal: Language runtime: Scala ver. 2.10.4
> > 15/08/18 23:26:53 INFO IgniteKernal: VM information: Java(TM) SE Runtime
> > Environment 1.7.0_60-b19 Oracle Corporation Java HotSpot(TM) 64-Bit Server
> > VM 24.60-b0
> > 9
> > 15/08/18 23:26:53 INFO IgniteKernal: VM total memory: 0.48GB
> > 15/08/18 23:26:53 INFO IgniteKernal: Remote Management [restart: off,
> > REST: on, JMX (remote: off)]
> > 15/08/18 23:26:53 INFO IgniteKernal: IGNITE_HOME=null
> > 15/08/18 23:26:53 INFO IgniteKernal: VM arguments: [-XX:MaxPermSize=128m,
> > -Dscala.usejavacp=true, -Xms512m, -Xmx512m]
> > 15/08/18 23:26:53 INFO IgniteKernal: Configured caches
> > ['ignite-marshaller-sys-cache', 'ignite-sys-cache',
> > 'ignite-atomics-sys-cache', 'SharedNumbers']
> > 15/08/18 23:26:53 INFO IgniteKernal: Non-loopback local IPs: 172.17.0.1,
> > fe80:0:0:0:42:acff:fe11:1%4
> > 15/08/18 23:26:53 INFO IgniteKernal: Enabled local MACs: 0242AC110001
> > 15/08/18 23:26:53 INFO IgnitePluginProcessor: Configured plugins:
> > 15/08/18 23:26:53 INFO IgnitePluginProcessor:   ^-- None
> > 15/08/18 23:26:53 INFO IgnitePluginProcessor:
> > 15/08/18 23:26:53 INFO TcpCommunicationSpi: IPC shared memory server
> > endpoint started [port=48100,
> > tokDir=/tmp/ignite/work/ipc/shmem/9fa0e60d-8d5a-4606-9797-4dc86c683a32-10416]
> > 15/08/18 23:26:53 INFO TcpCommunicationSpi: Successfully bound shared
> > memory communication to TCP port [port=48100, locHost=0.0.0.0/0.0.0.0]
> > 15/08/18 23:26:53 INFO TcpCommunicationSpi: Successfully bound to TCP port
> > [port=47100, locHost=0.0.0.0/0.0.0.0]
> > 15/08/18 23:26:53 WARN NoopCheckpointSpi: Checkpoints are disabled (to
> > enable configure any GridCheckpointSpi implementation)
> > 15/08/18 23:26:53 WARN GridCollisionManager: Collision resolution is
> > disabled (all jobs will be activated upon arrival).
> > 15/08/18 23:26:53 WARN NoopSwapSpaceSpi: Swap space is disabled. To enable
> > use FileSwapSpaceSpi.
> > 15/08/18 23:26:53 INFO IgniteKernal: Security status [authentication=off,
> > communication encryption=off]
> > 15/08/18 23:26:53 INFO GridTcpRestProtocol: Command protocol successfully
> > started [name=TCP binary, host=0.0.0.0/0.0.0.0, port=11211]
> > 15/08/18 23:26:53 WARN TcpDiscoveryMulticastIpFinder:
> > TcpDiscoveryMulticastIpFinder has no pre-configured addresses (it is
> > recommended in production to specify at least one address in
> > TcpDiscoveryMulticastIpFinder.getAddresses() configuration property)
> > 15/08/18 23:26:53 WARN TcpDiscoverySpi: No addresses registered in the IP
> > finder (will retry in 2000ms): TcpDiscoveryMulticastIpFinder
> > [mcastGrp=228.1.2.4, mcastPort=47400, resWaitTime=500, addrReqAttempts=2,
> > locAddr=null, super=TcpDiscoveryVmIpFinder [addrs=[],
> > super=TcpDiscoveryIpFinderAdapter [shared=true]]]
> > [23:27:03] New version is available at ....
> >
> > and after that the process is just hanging there without any progress. The
> > spark-ignite-config is below
> >
> > <beans xmlns="http://www.springframework.org/schema/beans"
> >        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> >        xsi:schemaLocation="
> >        http://www.springframework.org/schema/beans
> >        http://www.springframework.org/schema/beans/spring-beans.xsd">
> >     <!--
> >         Alter configuration below as needed.
> >     -->
> >     <bean id="grid.cfg"
> > class="org.apache.ignite.configuration.IgniteConfiguration">
> >       <property name="cacheConfiguration">
> >         <list>
> >             <!-- Partitioned cache example configuration (Atomic mode). -->
> >             <bean
> > class="org.apache.ignite.configuration.CacheConfiguration">
> >                 <property name="name" value="SharedNumbers"/>
> >                 <property name="affinity">
> >                   <bean
> > class="org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction">
> >                     <property name="partitions" value="32"/>
> >                   </bean>
> >                 </property>
> >                 <property name="indexedTypes">
> >                     <array>
> >                         <value>java.lang.Integer</value>
> >                         <value>java.lang.Integer</value>
> >                     </array>
> >                 </property>
> >             </bean>
> >         </list>
> >       </property>
> >     </bean>
> > </beans>
> >
> > What shall I do at this point? Thanks in advance!
> >   Cos
> >
> >
> >

[Konstantin Boudnik-2]

Ah, here's the catch: when we build ignite for Bigtop (eg only hadoop
accelerator with -Dignite.edition=hadoop) the ignite-indexing is simply not in
the the release assembly. As the result - it doesn't make it into the package.

This seems like something that needs to be addressed on the Ignite side,
right?

Cos

On Wed, Aug 19, 2015 at 01:01AM, Konstantin Boudnik wrote:

> Thanks Alexey - that helped somewhat! and let us to figure out the following.
>
> For the benefit of the others here's what we have discovered during the skype-debug:
>
>  1. server and client node process should be started under the same uid. In my
>     case I was starting server node as root and then executor was exec'ed by a
>     spark-worker. The latter made the effective uid of the executor's owner to
>     be spark.
>  2. When bin/ignite.sh starts not from Linux service it uses default value for
>     work_dir == $IGNITE_HOME/work On Linux systems we are setting work_dir to
>     /var/run/ignite-hadoop/work belonging to ignite:ignite with effective
>     permissions 777
>
> Because of 2. in 1. case executor couldn't write into server's node
> work directory with root-only write permissions. As you've figured out: some
> exceptions weren't propagating all the way up which made the debugging to be a
> royal PITA.
>
> The right way to fix it, I believe, is to make bin/ignite.sh to always check
> for /etc/default/ignite-hadoop and source it if found. This way, we'll get the
> same environment no matter if the node started manually or via linux service.
> What do you think?
>
> We have stepped on another issue, where ignie-indexing libs weren't included
> into the package. While it is perfectly legit for Hadoop accelerator, for
> Spark accelerator these are essential for SQL querying. I have opened
> BIGTOP-1969 to address that.
>
> Thanks for your help tonight!
>   Cos
>
> On Tue, Aug 18, 2015 at 07:44PM, Alexey Goncharuk wrote:
> > Cos,
> >
> > You have created IgniteContext in client mode (it is a default mode), so
> > IgniteContext waits until at least one server node is started. I thought we
> > fixed the message for a client that cannot find server nodes, but
> > apparently the message is not clear yet. To fix this, just start
> > additionally an ignite node with ignite.sh and the same config.
> >
> > Also, let me know how I can improve the documentation about Igntie-Spark
> > integration - you are one of the first persons trying it by yourself, so
> > your feedback is priceless :)
> >
> > 2015-08-18 16:42 GMT-07:00 Konstantin Boudnik <[hidden email]>:
> >
> > > Guys,
> > >
> > > I am running a simple example to check the latest Spark acceleration
> > > module in
> > > Ignite and am hitting some strange stuff (which musta be my configuration
> > > issue, but I can not figure out what is this).
> > >
> > > This is how I run spark-shell in my docker container (spark-master and
> > > worker
> > > are started in the same container)
> > >
> > > % spark-shell --packages org.apache.ignite:ignite-spark:1.3.3-p2-SNAPSHOT
> > > --repositories https://repository.apache.org/content/groups/snapshots
> > > --master spark://ignite.docker:7077
> > >
> > > then I execute this:
> > >
> > > scala>
> > > import org.apache.ignite.spark._
> > > import org.apache.ignite.configuration._
> > > val ic = new IgniteContext[Integer, Integer](sc,
> > > "/bigtop/spark-ignite-config.xml")
> > >
> > > 15/08/18 23:26:52 INFO XmlBeanDefinitionReader: Loading XML bean
> > > definitions from URL [file:/bigtop/spark-ignite-config.xml]
> > > 15/08/18 23:26:52 INFO GenericApplicationContext: Refreshing
> > > org.springframework.context.support.GenericApplicationContext@7f85c5d1:
> > > startup date [Tue Aug 18 2
> > > 3:26:52 UTC 2015]; root of context hierarchy
> > > 15/08/18 23:26:53 INFO IgniteKernal:
> > >
> > > >>>    __________  ________________
> > > >>>   /  _/ ___/ |/ /  _/_  __/ __/
> > > >>>  _/ // (7 7    // /  / / / _/
> > > >>> /___/\___/_/|_/___/ /_/ /___/
> > > >>>
> > > >>> ver. 1.3.3-p2-SNAPSHOT#20150814-sha1:9300bbc7
> > > >>> 2015 Copyright(C) Apache Software Foundation
> > > >>>
> > > >>> Ignite documentation: http://ignite.incubator.apache.org
> > >
> > > 15/08/18 23:26:53 INFO IgniteKernal: Config URL: n/a
> > > 15/08/18 23:26:53 INFO IgniteKernal: Daemon mode: off
> > > 15/08/18 23:26:53 INFO IgniteKernal: OS: Linux 3.13.0-61-generic amd64
> > > 15/08/18 23:26:53 INFO IgniteKernal: OS user: root
> > > 15/08/18 23:26:53 INFO IgniteKernal: Language runtime: Scala ver. 2.10.4
> > > 15/08/18 23:26:53 INFO IgniteKernal: VM information: Java(TM) SE Runtime
> > > Environment 1.7.0_60-b19 Oracle Corporation Java HotSpot(TM) 64-Bit Server
> > > VM 24.60-b0
> > > 9
> > > 15/08/18 23:26:53 INFO IgniteKernal: VM total memory: 0.48GB
> > > 15/08/18 23:26:53 INFO IgniteKernal: Remote Management [restart: off,
> > > REST: on, JMX (remote: off)]
> > > 15/08/18 23:26:53 INFO IgniteKernal: IGNITE_HOME=null
> > > 15/08/18 23:26:53 INFO IgniteKernal: VM arguments: [-XX:MaxPermSize=128m,
> > > -Dscala.usejavacp=true, -Xms512m, -Xmx512m]
> > > 15/08/18 23:26:53 INFO IgniteKernal: Configured caches
> > > ['ignite-marshaller-sys-cache', 'ignite-sys-cache',
> > > 'ignite-atomics-sys-cache', 'SharedNumbers']
> > > 15/08/18 23:26:53 INFO IgniteKernal: Non-loopback local IPs: 172.17.0.1,
> > > fe80:0:0:0:42:acff:fe11:1%4
> > > 15/08/18 23:26:53 INFO IgniteKernal: Enabled local MACs: 0242AC110001
> > > 15/08/18 23:26:53 INFO IgnitePluginProcessor: Configured plugins:
> > > 15/08/18 23:26:53 INFO IgnitePluginProcessor:   ^-- None
> > > 15/08/18 23:26:53 INFO IgnitePluginProcessor:
> > > 15/08/18 23:26:53 INFO TcpCommunicationSpi: IPC shared memory server
> > > endpoint started [port=48100,
> > > tokDir=/tmp/ignite/work/ipc/shmem/9fa0e60d-8d5a-4606-9797-4dc86c683a32-10416]
> > > 15/08/18 23:26:53 INFO TcpCommunicationSpi: Successfully bound shared
> > > memory communication to TCP port [port=48100, locHost=0.0.0.0/0.0.0.0]
> > > 15/08/18 23:26:53 INFO TcpCommunicationSpi: Successfully bound to TCP port
> > > [port=47100, locHost=0.0.0.0/0.0.0.0]
> > > 15/08/18 23:26:53 WARN NoopCheckpointSpi: Checkpoints are disabled (to
> > > enable configure any GridCheckpointSpi implementation)
> > > 15/08/18 23:26:53 WARN GridCollisionManager: Collision resolution is
> > > disabled (all jobs will be activated upon arrival).
> > > 15/08/18 23:26:53 WARN NoopSwapSpaceSpi: Swap space is disabled. To enable
> > > use FileSwapSpaceSpi.
> > > 15/08/18 23:26:53 INFO IgniteKernal: Security status [authentication=off,
> > > communication encryption=off]
> > > 15/08/18 23:26:53 INFO GridTcpRestProtocol: Command protocol successfully
> > > started [name=TCP binary, host=0.0.0.0/0.0.0.0, port=11211]
> > > 15/08/18 23:26:53 WARN TcpDiscoveryMulticastIpFinder:
> > > TcpDiscoveryMulticastIpFinder has no pre-configured addresses (it is
> > > recommended in production to specify at least one address in
> > > TcpDiscoveryMulticastIpFinder.getAddresses() configuration property)
> > > 15/08/18 23:26:53 WARN TcpDiscoverySpi: No addresses registered in the IP
> > > finder (will retry in 2000ms): TcpDiscoveryMulticastIpFinder
> > > [mcastGrp=228.1.2.4, mcastPort=47400, resWaitTime=500, addrReqAttempts=2,
> > > locAddr=null, super=TcpDiscoveryVmIpFinder [addrs=[],
> > > super=TcpDiscoveryIpFinderAdapter [shared=true]]]
> > > [23:27:03] New version is available at ....
> > >
> > > and after that the process is just hanging there without any progress. The
> > > spark-ignite-config is below
> > >
> > > <beans xmlns="http://www.springframework.org/schema/beans"
> > >        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> > >        xsi:schemaLocation="
> > >        http://www.springframework.org/schema/beans
> > >        http://www.springframework.org/schema/beans/spring-beans.xsd">
> > >     <!--
> > >         Alter configuration below as needed.
> > >     -->
> > >     <bean id="grid.cfg"
> > > class="org.apache.ignite.configuration.IgniteConfiguration">
> > >       <property name="cacheConfiguration">
> > >         <list>
> > >             <!-- Partitioned cache example configuration (Atomic mode). -->
> > >             <bean
> > > class="org.apache.ignite.configuration.CacheConfiguration">
> > >                 <property name="name" value="SharedNumbers"/>
> > >                 <property name="affinity">
> > >                   <bean
> > > class="org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction">
> > >                     <property name="partitions" value="32"/>
> > >                   </bean>
> > >                 </property>
> > >                 <property name="indexedTypes">
> > >                     <array>
> > >                         <value>java.lang.Integer</value>
> > >                         <value>java.lang.Integer</value>
> > >                     </array>
> > >                 </property>
> > >             </bean>
> > >         </list>
> > >       </property>
> > >     </bean>
> > > </beans>
> > >
> > > What shall I do at this point? Thanks in advance!
> > >   Cos
> > >
> > >
> > >

[Konstantin Boudnik-2]

Ok, this

diff --git assembly/dependencies-hadoop.xml assembly/dependencies-hadoop.xml
index 4638643..43e8c24 100644
--- assembly/dependencies-hadoop.xml
+++ assembly/dependencies-hadoop.xml
@@ -32,6 +32,7 @@
     <moduleSets>
         <moduleSet>
             <includes>
+                <include>org.apache.ignite:ignite-indexing</include>
                 <include>org.apache.ignite:ignite-spring</include>
                 <include>org.apache.ignite:ignite-log4j</include>
             </includes>

does the trick and now the packages include the indexing. Are you guys ok
with the fix? If so I will open a JIRA and patch the master tomorrow.
Or shall the fix be conceptually different? Like instead of making
ignite-indexing a dependency of ignite-hadoop we should make it a part of the
base or something? Thoughts!

Cos

On Wed, Aug 19, 2015 at 01:27AM, Konstantin Boudnik wrote:

> Ah, here's the catch: when we build ignite for Bigtop (eg only hadoop
> accelerator with -Dignite.edition=hadoop) the ignite-indexing is simply not in
> the the release assembly. As the result - it doesn't make it into the package.
>
> This seems like something that needs to be addressed on the Ignite side,
> right?
>
> Cos
>
> On Wed, Aug 19, 2015 at 01:01AM, Konstantin Boudnik wrote:
> > Thanks Alexey - that helped somewhat! and let us to figure out the following.
> >
> > For the benefit of the others here's what we have discovered during the skype-debug:
> >
> >  1. server and client node process should be started under the same uid. In my
> >     case I was starting server node as root and then executor was exec'ed by a
> >     spark-worker. The latter made the effective uid of the executor's owner to
> >     be spark.
> >  2. When bin/ignite.sh starts not from Linux service it uses default value for
> >     work_dir == $IGNITE_HOME/work On Linux systems we are setting work_dir to
> >     /var/run/ignite-hadoop/work belonging to ignite:ignite with effective
> >     permissions 777
> >
> > Because of 2. in 1. case executor couldn't write into server's node
> > work directory with root-only write permissions. As you've figured out: some
> > exceptions weren't propagating all the way up which made the debugging to be a
> > royal PITA.
> >
> > The right way to fix it, I believe, is to make bin/ignite.sh to always check
> > for /etc/default/ignite-hadoop and source it if found. This way, we'll get the
> > same environment no matter if the node started manually or via linux service.
> > What do you think?
> >
> > We have stepped on another issue, where ignie-indexing libs weren't included
> > into the package. While it is perfectly legit for Hadoop accelerator, for
> > Spark accelerator these are essential for SQL querying. I have opened
> > BIGTOP-1969 to address that.
> >
> > Thanks for your help tonight!
> >   Cos
> >
> > On Tue, Aug 18, 2015 at 07:44PM, Alexey Goncharuk wrote:
> > > Cos,
> > >
> > > You have created IgniteContext in client mode (it is a default mode), so
> > > IgniteContext waits until at least one server node is started. I thought we
> > > fixed the message for a client that cannot find server nodes, but
> > > apparently the message is not clear yet. To fix this, just start
> > > additionally an ignite node with ignite.sh and the same config.
> > >
> > > Also, let me know how I can improve the documentation about Igntie-Spark
> > > integration - you are one of the first persons trying it by yourself, so
> > > your feedback is priceless :)
> > >
> > > 2015-08-18 16:42 GMT-07:00 Konstantin Boudnik <[hidden email]>:
> > >
> > > > Guys,
> > > >
> > > > I am running a simple example to check the latest Spark acceleration
> > > > module in
> > > > Ignite and am hitting some strange stuff (which musta be my configuration
> > > > issue, but I can not figure out what is this).
> > > >
> > > > This is how I run spark-shell in my docker container (spark-master and
> > > > worker
> > > > are started in the same container)
> > > >
> > > > % spark-shell --packages org.apache.ignite:ignite-spark:1.3.3-p2-SNAPSHOT
> > > > --repositories https://repository.apache.org/content/groups/snapshots
> > > > --master spark://ignite.docker:7077
> > > >
> > > > then I execute this:
> > > >
> > > > scala>
> > > > import org.apache.ignite.spark._
> > > > import org.apache.ignite.configuration._
> > > > val ic = new IgniteContext[Integer, Integer](sc,
> > > > "/bigtop/spark-ignite-config.xml")
> > > >
> > > > 15/08/18 23:26:52 INFO XmlBeanDefinitionReader: Loading XML bean
> > > > definitions from URL [file:/bigtop/spark-ignite-config.xml]
> > > > 15/08/18 23:26:52 INFO GenericApplicationContext: Refreshing
> > > > org.springframework.context.support.GenericApplicationContext@7f85c5d1:
> > > > startup date [Tue Aug 18 2
> > > > 3:26:52 UTC 2015]; root of context hierarchy
> > > > 15/08/18 23:26:53 INFO IgniteKernal:
> > > >
> > > > >>>    __________  ________________
> > > > >>>   /  _/ ___/ |/ /  _/_  __/ __/
> > > > >>>  _/ // (7 7    // /  / / / _/
> > > > >>> /___/\___/_/|_/___/ /_/ /___/
> > > > >>>
> > > > >>> ver. 1.3.3-p2-SNAPSHOT#20150814-sha1:9300bbc7
> > > > >>> 2015 Copyright(C) Apache Software Foundation
> > > > >>>
> > > > >>> Ignite documentation: http://ignite.incubator.apache.org
> > > >
> > > > 15/08/18 23:26:53 INFO IgniteKernal: Config URL: n/a
> > > > 15/08/18 23:26:53 INFO IgniteKernal: Daemon mode: off
> > > > 15/08/18 23:26:53 INFO IgniteKernal: OS: Linux 3.13.0-61-generic amd64
> > > > 15/08/18 23:26:53 INFO IgniteKernal: OS user: root
> > > > 15/08/18 23:26:53 INFO IgniteKernal: Language runtime: Scala ver. 2.10.4
> > > > 15/08/18 23:26:53 INFO IgniteKernal: VM information: Java(TM) SE Runtime
> > > > Environment 1.7.0_60-b19 Oracle Corporation Java HotSpot(TM) 64-Bit Server
> > > > VM 24.60-b0
> > > > 9
> > > > 15/08/18 23:26:53 INFO IgniteKernal: VM total memory: 0.48GB
> > > > 15/08/18 23:26:53 INFO IgniteKernal: Remote Management [restart: off,
> > > > REST: on, JMX (remote: off)]
> > > > 15/08/18 23:26:53 INFO IgniteKernal: IGNITE_HOME=null
> > > > 15/08/18 23:26:53 INFO IgniteKernal: VM arguments: [-XX:MaxPermSize=128m,
> > > > -Dscala.usejavacp=true, -Xms512m, -Xmx512m]
> > > > 15/08/18 23:26:53 INFO IgniteKernal: Configured caches
> > > > ['ignite-marshaller-sys-cache', 'ignite-sys-cache',
> > > > 'ignite-atomics-sys-cache', 'SharedNumbers']
> > > > 15/08/18 23:26:53 INFO IgniteKernal: Non-loopback local IPs: 172.17.0.1,
> > > > fe80:0:0:0:42:acff:fe11:1%4
> > > > 15/08/18 23:26:53 INFO IgniteKernal: Enabled local MACs: 0242AC110001
> > > > 15/08/18 23:26:53 INFO IgnitePluginProcessor: Configured plugins:
> > > > 15/08/18 23:26:53 INFO IgnitePluginProcessor:   ^-- None
> > > > 15/08/18 23:26:53 INFO IgnitePluginProcessor:
> > > > 15/08/18 23:26:53 INFO TcpCommunicationSpi: IPC shared memory server
> > > > endpoint started [port=48100,
> > > > tokDir=/tmp/ignite/work/ipc/shmem/9fa0e60d-8d5a-4606-9797-4dc86c683a32-10416]
> > > > 15/08/18 23:26:53 INFO TcpCommunicationSpi: Successfully bound shared
> > > > memory communication to TCP port [port=48100, locHost=0.0.0.0/0.0.0.0]
> > > > 15/08/18 23:26:53 INFO TcpCommunicationSpi: Successfully bound to TCP port
> > > > [port=47100, locHost=0.0.0.0/0.0.0.0]
> > > > 15/08/18 23:26:53 WARN NoopCheckpointSpi: Checkpoints are disabled (to
> > > > enable configure any GridCheckpointSpi implementation)
> > > > 15/08/18 23:26:53 WARN GridCollisionManager: Collision resolution is
> > > > disabled (all jobs will be activated upon arrival).
> > > > 15/08/18 23:26:53 WARN NoopSwapSpaceSpi: Swap space is disabled. To enable
> > > > use FileSwapSpaceSpi.
> > > > 15/08/18 23:26:53 INFO IgniteKernal: Security status [authentication=off,
> > > > communication encryption=off]
> > > > 15/08/18 23:26:53 INFO GridTcpRestProtocol: Command protocol successfully
> > > > started [name=TCP binary, host=0.0.0.0/0.0.0.0, port=11211]
> > > > 15/08/18 23:26:53 WARN TcpDiscoveryMulticastIpFinder:
> > > > TcpDiscoveryMulticastIpFinder has no pre-configured addresses (it is
> > > > recommended in production to specify at least one address in
> > > > TcpDiscoveryMulticastIpFinder.getAddresses() configuration property)
> > > > 15/08/18 23:26:53 WARN TcpDiscoverySpi: No addresses registered in the IP
> > > > finder (will retry in 2000ms): TcpDiscoveryMulticastIpFinder
> > > > [mcastGrp=228.1.2.4, mcastPort=47400, resWaitTime=500, addrReqAttempts=2,
> > > > locAddr=null, super=TcpDiscoveryVmIpFinder [addrs=[],
> > > > super=TcpDiscoveryIpFinderAdapter [shared=true]]]
> > > > [23:27:03] New version is available at ....
> > > >
> > > > and after that the process is just hanging there without any progress. The
> > > > spark-ignite-config is below
> > > >
> > > > <beans xmlns="http://www.springframework.org/schema/beans"
> > > >        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> > > >        xsi:schemaLocation="
> > > >        http://www.springframework.org/schema/beans
> > > >        http://www.springframework.org/schema/beans/spring-beans.xsd">
> > > >     <!--
> > > >         Alter configuration below as needed.
> > > >     -->
> > > >     <bean id="grid.cfg"
> > > > class="org.apache.ignite.configuration.IgniteConfiguration">
> > > >       <property name="cacheConfiguration">
> > > >         <list>
> > > >             <!-- Partitioned cache example configuration (Atomic mode). -->
> > > >             <bean
> > > > class="org.apache.ignite.configuration.CacheConfiguration">
> > > >                 <property name="name" value="SharedNumbers"/>
> > > >                 <property name="affinity">
> > > >                   <bean
> > > > class="org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction">
> > > >                     <property name="partitions" value="32"/>
> > > >                   </bean>
> > > >                 </property>
> > > >                 <property name="indexedTypes">
> > > >                     <array>
> > > >                         <value>java.lang.Integer</value>
> > > >                         <value>java.lang.Integer</value>
> > > >                     </array>
> > > >                 </property>
> > > >             </bean>
> > > >         </list>
> > > >       </property>
> > > >     </bean>
> > > > </beans>
> > > >
> > > > What shall I do at this point? Thanks in advance!
> > > >   Cos
> > > >
> > > >
> > > >