Pluggable SecondaryFileSystemProvider?

[Ivan V.]

Hi, dev,
one of Ignite users discovered that IGFS does not correctly work with
secondary HDFS file system if that HDFS is "kerberised". The issue is
described in https://issues.apache.org/jira/browse/IGNITE-2195 (This may
also be close to this issue:
http://apache-ignite-users.70518.x6.nabble.com/Exception-in-Kerberos-Yarn-cluster-td1950.html
). The user suggests a fix that creates SecondaryFileSystem using some
proxy UserGroupInformation, and that workaround works for him.
However, we're not sure this fix is generic enough and will work well in
all possible configurations.
So, there is an idea to make SecondaryFileSystemProvider pluggable to allow
users to write own code for specific configurations. An example can be seen
in this pull request:
https://github.com/apache/ignite/pull/350 .
Pros, cons, other thoughts?
Thanks in advance.
--
ivan

[Ivan V.]

To summarize suggested change in short: we add constructor
org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem#IgniteHadoopIgfsSecondaryFileSystem(java.lang.String,
org.apache.ignite.internal.processors.hadoop.SecondaryFileSystemProvider) ,
allowing to plug an arbitrary SecondaryFileSystemProvider bean when
constructing IgniteHadoopIgfsSecondaryFileSystem , e.g:

                       <bean
class="org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem">
                            <constructor-arg name="userName" value="ivan"/>
                            <constructor-arg name="provider">
                                <bean class="
*com.foo.MySuperSecondaryFileSystemProvider*">
                                        <constructor-arg name="secUri"
value="hdfs://localhost:9000"/>
                                        <constructor-arg
name="secConfPath"><null/></constructor-arg>
                                </bean>
                            </constructor-arg>
                        </bean>


On Thu, Dec 17, 2015 at 10:14 PM, Ivan V. <[hidden email]> wrote:

> Hi, dev,
> one of Ignite users discovered that IGFS does not correctly work with
> secondary HDFS file system if that HDFS is "kerberised". The issue is
> described in https://issues.apache.org/jira/browse/IGNITE-2195 (This may
> also be close to this issue:
> http://apache-ignite-users.70518.x6.nabble.com/Exception-in-Kerberos-Yarn-cluster-td1950.html
> ). The user suggests a fix that creates SecondaryFileSystem using some
> proxy UserGroupInformation, and that workaround works for him.
> However, we're not sure this fix is generic enough and will work well in
> all possible configurations.
> So, there is an idea to make SecondaryFileSystemProvider pluggable to
> allow users to write own code for specific configurations. An example can
> be seen in this pull request:
> https://github.com/apache/ignite/pull/350 .
> Pros, cons, other thoughts?
> Thanks in advance.
> --
> ivan
>

[Vladimir Ozerov]

+ 1 for idea in general, but I have several concerns:

1) SecondaryFileSystemProvider is internal class and cannot be exposed. We
need some other interface here.
2) I would avoid setting properties through constructor because it is
error-prone when Spring is used. Instead, I would create setters as in
almost all other components.
3) Ideally we should not break existing interfaces.

To summarize, it could looks as follows:

class IgniteHadoopIgfsSecondaryFileSystem {
    /** These properties already exist. */
    void setUri(String fileSystemUri);
    void setConfig(String pathToConfig);
    void setUserName(String userName);

    /** This is a new property. */
    void setFileSystemFactory(HadoopFileSystemFactory factory);
}

interface HadoopFileSystemFactory {
    FileSystem create(String uri, String configPath, String userName);
}

Vladimir.

On Thu, Dec 17, 2015 at 10:34 PM, Ivan V. <[hidden email]> wrote:

> To summarize suggested change in short: we add constructor
>
> org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem#IgniteHadoopIgfsSecondaryFileSystem(java.lang.String,
> org.apache.ignite.internal.processors.hadoop.SecondaryFileSystemProvider) ,
> allowing to plug an arbitrary SecondaryFileSystemProvider bean when
> constructing IgniteHadoopIgfsSecondaryFileSystem , e.g:
>
>                        <bean
> class="org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem">
>                             <constructor-arg name="userName" value="ivan"/>
>                             <constructor-arg name="provider">
>                                 <bean class="
> *com.foo.MySuperSecondaryFileSystemProvider*">
>                                         <constructor-arg name="secUri"
> value="hdfs://localhost:9000"/>
>                                         <constructor-arg
> name="secConfPath"><null/></constructor-arg>
>                                 </bean>
>                             </constructor-arg>
>                         </bean>
>
>
> On Thu, Dec 17, 2015 at 10:14 PM, Ivan V. <[hidden email]>
> wrote:
>
> > Hi, dev,
> > one of Ignite users discovered that IGFS does not correctly work with
> > secondary HDFS file system if that HDFS is "kerberised". The issue is
> > described in https://issues.apache.org/jira/browse/IGNITE-2195 (This may
> > also be close to this issue:
> >
> http://apache-ignite-users.70518.x6.nabble.com/Exception-in-Kerberos-Yarn-cluster-td1950.html
> > ). The user suggests a fix that creates SecondaryFileSystem using some
> > proxy UserGroupInformation, and that workaround works for him.
> > However, we're not sure this fix is generic enough and will work well in
> > all possible configurations.
> > So, there is an idea to make SecondaryFileSystemProvider pluggable to
> > allow users to write own code for specific configurations. An example can
> > be seen in this pull request:
> > https://github.com/apache/ignite/pull/350 .
> > Pros, cons, other thoughts?
> > Thanks in advance.
> > --
> > ivan
> >
>

[dsetrakyan]

+1 on Vladimir’s suggestion. Let’s remember to document it whenever we make
the change.

I also think that we should have an out-of-the-box implementation of the
file-system-factory with a fix for Kerberos.

D.

On Thu, Dec 17, 2015 at 12:01 PM, Vladimir Ozerov <[hidden email]>
wrote:

> + 1 for idea in general, but I have several concerns:
>
> 1) SecondaryFileSystemProvider is internal class and cannot be exposed. We
> need some other interface here.
> 2) I would avoid setting properties through constructor because it is
> error-prone when Spring is used. Instead, I would create setters as in
> almost all other components.
> 3) Ideally we should not break existing interfaces.
>
> To summarize, it could looks as follows:
>
> class IgniteHadoopIgfsSecondaryFileSystem {
>     /** These properties already exist. */
>     void setUri(String fileSystemUri);
>     void setConfig(String pathToConfig);
>     void setUserName(String userName);
>
>     /** This is a new property. */
>     void setFileSystemFactory(HadoopFileSystemFactory factory);
> }
>
> interface HadoopFileSystemFactory {
>     FileSystem create(String uri, String configPath, String userName);
> }
>
> Vladimir.
>
> On Thu, Dec 17, 2015 at 10:34 PM, Ivan V. <[hidden email]>
> wrote:
>
> > To summarize suggested change in short: we add constructor
> >
> >
> org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem#IgniteHadoopIgfsSecondaryFileSystem(java.lang.String,
> >
> org.apache.ignite.internal.processors.hadoop.SecondaryFileSystemProvider) ,
> > allowing to plug an arbitrary SecondaryFileSystemProvider bean when
> > constructing IgniteHadoopIgfsSecondaryFileSystem , e.g:
> >
> >                        <bean
> > class="org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem">
> >                             <constructor-arg name="userName"
> value="ivan"/>
> >                             <constructor-arg name="provider">
> >                                 <bean class="
> > *com.foo.MySuperSecondaryFileSystemProvider*">
> >                                         <constructor-arg name="secUri"
> > value="hdfs://localhost:9000"/>
> >                                         <constructor-arg
> > name="secConfPath"><null/></constructor-arg>
> >                                 </bean>
> >                             </constructor-arg>
> >                         </bean>
> >
> >
> > On Thu, Dec 17, 2015 at 10:14 PM, Ivan V. <[hidden email]>
> > wrote:
> >
> > > Hi, dev,
> > > one of Ignite users discovered that IGFS does not correctly work with
> > > secondary HDFS file system if that HDFS is "kerberised". The issue is
> > > described in https://issues.apache.org/jira/browse/IGNITE-2195 (This
> may
> > > also be close to this issue:
> > >
> >
> http://apache-ignite-users.70518.x6.nabble.com/Exception-in-Kerberos-Yarn-cluster-td1950.html
> > > ). The user suggests a fix that creates SecondaryFileSystem using some
> > > proxy UserGroupInformation, and that workaround works for him.
> > > However, we're not sure this fix is generic enough and will work well
> in
> > > all possible configurations.
> > > So, there is an idea to make SecondaryFileSystemProvider pluggable to
> > > allow users to write own code for specific configurations. An example
> can
> > > be seen in this pull request:
> > > https://github.com/apache/ignite/pull/350 .
> > > Pros, cons, other thoughts?
> > > Thanks in advance.
> > > --
> > > ivan
> > >
> >
>