Apache Ignite Developers - Legacy Mail Archive

Hadoop and hive not working after setting ignite igfs.

Classic

List

Threaded

5 messages Options

pawanpawar

Hadoop and hive not working after setting ignite igfs.

Hi team I am getting error after configuring ignite with hadoop and hive. when I try to run any hadoop cmd it show me this error.

pawanpawar@cloud:~$ hadoop fs -ls -R /
-ls: Fatal internal error
java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.ignite.hadoop.fs.v2.IgniteHadoopFileSystem.<init>()
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2595)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:354)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325)
at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:224)
at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:207)
at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:190)
at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
Caused by: java.lang.NoSuchMethodException: org.apache.ignite.hadoop.fs.v2.IgniteHadoopFileSystem.<init>()
at java.lang.Class.getConstructor0(Class.java:3082)
at java.lang.Class.getDeclaredConstructor(Class.java:2178)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
... 17 more

and when I try to start hiveserver2 it gives me same error.

I have setup the HADOOP_CLASSPATH and other configuration as shown in tutorial https://apacheignite-fs.readme.io/v1.5.0

Vladimir Ozerov

Re: Hadoop and hive not working after setting ignite igfs.

Hi,

Please double-check that you set correct class name in *core-site.xml*. It
seems to me that you should change "v2" to "v1":
org.apache.ignite.hadoop.fs.*v1*.IgniteHadoopFileSystem

Vladimir.

On Thu, Mar 31, 2016 at 1:22 AM, pawanpawar <[hidden email]> wrote:

> Hi team I am getting error after configuring ignite with hadoop and hive.
> when I try to run any hadoop cmd it show me this error.
>
> pawanpawar@cloud:~$ hadoop fs -ls -R /
> -ls: Fatal internal error
> java.lang.RuntimeException: java.lang.NoSuchMethodException:
> org.apache.ignite.hadoop.fs.v2.IgniteHadoopFileSystem.<init>()
> at
>
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2595)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:354)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
> at
> org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325)
> at
> org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:224)
> at
> org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:207)
> at
> org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:190)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
> Caused by: java.lang.NoSuchMethodException:
> org.apache.ignite.hadoop.fs.v2.IgniteHadoopFileSystem.<init>()
> at java.lang.Class.getConstructor0(Class.java:3082)
> at java.lang.Class.getDeclaredConstructor(Class.java:2178)
> at
>
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
> ... 17 more
>
>
> and when I try to start hiveserver2 it gives me same error.
>
> I have setup the HADOOP_CLASSPATH and other configuration as shown in
> tutorial https://apacheignite-fs.readme.io/v1.5.0
>
>
>
> --
> View this message in context:
> http://apache-ignite-developers.2346864.n4.nabble.com/Hadoop-and-hive-not-working-after-setting-ignite-igfs-tp8178.html
> Sent from the Apache Ignite Developers mailing list archive at Nabble.com.
>

pawanpawar

Re: Hadoop and hive not working after setting ignite igfs.

Thanks for your reply, I am using Hadoop 2, so I used "v2". Is this for hadoop version?

On Thu, Mar 31, 2016 at 11:11 AM, Vladimir Ozerov [via Apache Ignite Developers] <[hidden email]> wrote:

Hi,

Please double-check that you set correct class name in *core-site.xml*. It
seems to me that you should change "v2" to "v1":
org.apache.ignite.hadoop.fs.*v1*.IgniteHadoopFileSystem

Vladimir.

On Thu, Mar 31, 2016 at 1:22 AM, pawanpawar <[hidden email]> wrote:

> Hi team I am getting error after configuring ignite with hadoop and hive.
> when I try to run any hadoop cmd it show me this error.
>
> pawanpawar@cloud:~$ hadoop fs -ls -R /
> -ls: Fatal internal error
> java.lang.RuntimeException: java.lang.NoSuchMethodException:
> org.apache.ignite.hadoop.fs.v2.IgniteHadoopFileSystem.<init>()
> at
>
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2595)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:354)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
> at
> org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325)
> at
> org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:224)
> at
> org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:207)
> at
> org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:190)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
> Caused by: java.lang.NoSuchMethodException:
> org.apache.ignite.hadoop.fs.v2.IgniteHadoopFileSystem.<init>()
> at java.lang.Class.getConstructor0(Class.java:3082)
> at java.lang.Class.getDeclaredConstructor(Class.java:2178)
> at
>
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
> ... 17 more
>
>
> and when I try to start hiveserver2 it gives me same error.
>
> I have setup the HADOOP_CLASSPATH and other configuration as shown in
> tutorial https://apacheignite-fs.readme.io/v1.5.0
>
>
>
> --
> View this message in context:
> http://apache-ignite-developers.2346864.n4.nabble.com/Hadoop-and-hive-not-working-after-setting-ignite-igfs-tp8178.html
> Sent from the Apache Ignite Developers mailing list archive at Nabble.com.
>

If you reply to this email, your message will be added to the discussion below:
http://apache-ignite-developers.2346864.n4.nabble.com/Hadoop-and-hive-not-working-after-setting-ignite-igfs-tp8178p8179.html

To unsubscribe from Hadoop and hive not working after setting ignite igfs., click here.
NAML

Thanks & Regards

Pawan Pawar

Mobile: +91 9993585256

Email: [hidden email]

Skype: pawarskype

Vladimir Ozerov

Re: Hadoop and hive not working after setting ignite igfs.

"v1" or "v2" doesn't stand for Hadoop 1.x and Hadoop 2.x. Instead, this is
about either old "FileSystem" API or new "AbstractFileSystem" API. In your
stack trace I see that you use "FileSystem", so you should try setting "v1".

Vladimir.

On Thu, Mar 31, 2016 at 8:44 AM, pawanpawar <[hidden email]> wrote:

> Thanks for your reply, I am using Hadoop 2, so I used "v2". Is this for
> hadoop version?
>
> On Thu, Mar 31, 2016 at 11:11 AM, Vladimir Ozerov [via Apache Ignite
> Developers] <[hidden email]> wrote:
>
> > Hi,
> >
> > Please double-check that you set correct class name in *core-site.xml*.
> It
> > seems to me that you should change "v2" to "v1":
> > org.apache.ignite.hadoop.fs.*v1*.IgniteHadoopFileSystem
> >
> > Vladimir.
> >
> > On Thu, Mar 31, 2016 at 1:22 AM, pawanpawar <[hidden email]
> > <http:///user/SendEmail.jtp?type=node&node=8179&i=0>> wrote:
> >
> > > Hi team I am getting error after configuring ignite with hadoop and
> > hive.
> > > when I try to run any hadoop cmd it show me this error.
> > >
> > > pawanpawar@cloud:~$ hadoop fs -ls -R /
> > > -ls: Fatal internal error
> > > java.lang.RuntimeException: java.lang.NoSuchMethodException:
> > > org.apache.ignite.hadoop.fs.v2.IgniteHadoopFileSystem.<init>()
> > > at
> > >
> > >
> >
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
> >
> > > at
> > > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2595)
> > > at
> > org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
> > > at
> > > org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
> > > at
> > org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
> > > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
> > > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
> > > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:354)
> > > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
> > > at
> > > org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325)
> > > at
> > > org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:224)
> > > at
> > > org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:207)
> > > at
> > >
> org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:190)
> > > at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
> > > at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
> > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> > > at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
> > > Caused by: java.lang.NoSuchMethodException:
> > > org.apache.ignite.hadoop.fs.v2.IgniteHadoopFileSystem.<init>()
> > > at java.lang.Class.getConstructor0(Class.java:3082)
> > > at java.lang.Class.getDeclaredConstructor(Class.java:2178)
> > > at
> > >
> > >
> >
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
> >
> > > ... 17 more
> > >
> > >
> > > and when I try to start hiveserver2 it gives me same error.
> > >
> > > I have setup the HADOOP_CLASSPATH and other configuration as shown in
> > > tutorial https://apacheignite-fs.readme.io/v1.5.0
> > >
> > >
> > >
> > > --
> > > View this message in context:
> > >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/Hadoop-and-hive-not-working-after-setting-ignite-igfs-tp8178.html
> > > Sent from the Apache Ignite Developers mailing list archive at
> > Nabble.com.
> > >
> >
> >
> > ------------------------------
> > If you reply to this email, your message will be added to the discussion
> > below:
> >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/Hadoop-and-hive-not-working-after-setting-ignite-igfs-tp8178p8179.html
> > To unsubscribe from Hadoop and hive not working after setting ignite
> > igfs., click here
> > <
> >
> > .
> > NAML
> > <
> http://apache-ignite-developers.2346864.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
> >
> >
>
>
>
> --
> *Thanks & Regards*
> *Pawan Pawar*
> *Mobile: +91 9993585256*
> *Email: [hidden email] <[hidden email]>*
> *Skype: pawarskype*
>
>
>
>
> --
> View this message in context:
> http://apache-ignite-developers.2346864.n4.nabble.com/Hadoop-and-hive-not-working-after-setting-ignite-igfs-tp8178p8180.html
> Sent from the Apache Ignite Developers mailing list archive at Nabble.com.
>

pawanpawar

Re: Hadoop and hive not working after setting ignite igfs.

Thanks I changed as you suggest me to do. but now when I try to run the hadoop cmd it show me different error.

pawanpawar@cloud:~$ hadoop fs -ls -R /

ls: Generic IGFS error occurred.

I checked my hadoop log file it show me nothing. but when I try to start my hadoop history server it show me this error in its log file.

2016-03-31 13:04:34,324 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer: registered UNIX signal handlers for [TERM, HUP, INT]

2016-03-31 13:04:34,995 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: JobHistory Init

2016-03-31 13:04:35,735 INFO org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [igfs://[hidden email]]

2016-03-31 13:04:35,806 INFO org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Error creating done directory: [igfs://igfs@192.168.1.5/tmp/hadoop-yarn/staging/history/done]

org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Error creating done directory: [igfs://igfs@192.168.1.5/tmp/hadoop-yarn/staging/history/done]

at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.tryCreatingHistoryDirs(HistoryFileManager.java:587)

at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.createHistoryDirs(HistoryFileManager.java:533)

at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.serviceInit(HistoryFileManager.java:501)

at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

at org.apache.hadoop.mapreduce.v2.hs.JobHistory.serviceInit(JobHistory.java:94)

at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)

at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.serviceInit(JobHistoryServer.java:142)

at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.launchJobHistoryServer(JobHistoryServer.java:219)

at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.main(JobHistoryServer.java:229)

Caused by: java.io.IOException: Generic IGFS error occurred.

at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsUtils.cast(HadoopIgfsUtils.java:132)

at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper.withReconnectHandling(HadoopIgfsWrapper.java:327)

at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper.info(HadoopIgfsWrapper.java:120)

at org.apache.ignite.hadoop.fs.v2.IgniteHadoopFileSystem.getFileStatus(IgniteHadoopFileSystem.java:802)

at org.apache.hadoop.fs.FileContext$14.next(FileContext.java:1120)

at org.apache.hadoop.fs.FileContext$14.next(FileContext.java:1116)

at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)

at org.apache.hadoop.fs.FileContext.getFileStatus(FileContext.java:1116)

at org.apache.hadoop.fs.FileContext$Util.exists(FileContext.java:1581)

at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.mkdir(HistoryFileManager.java:640)

at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.tryCreatingHistoryDirs(HistoryFileManager.java:570)

... 10 more

Caused by: class org.apache.ignite.IgniteCheckedException: Generic IGFS error occurred.

at org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7005)

at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:166)

at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:115)

at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsOutProc.info(HadoopIgfsOutProc.java:209)

at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper$2.apply(HadoopIgfsWrapper.java:123)

at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper$2.apply(HadoopIgfsWrapper.java:120)

at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper.withReconnectHandling(HadoopIgfsWrapper.java:310)

... 19 more

Caused by: class org.apache.ignite.igfs.IgfsException: Generic IGFS error occurred.

at org.apache.ignite.internal.igfs.common.IgfsControlResponse.throwError(IgfsControlResponse.java:294)

at org.apache.ignite.internal.igfs.common.IgfsControlResponse.throwError(IgfsControlResponse.java:303)

at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsOutProc$1.apply(HadoopIgfsOutProc.java:509)

at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsOutProc$1.apply(HadoopIgfsOutProc.java:503)

at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:54)

at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:28)

at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:262)

at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListeners(GridFutureAdapter.java:250)

at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:380)

at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:346)

at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:323)

at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsIpcIo$ReaderThread.run(HadoopIgfsIpcIo.java:575)

2016-03-31 13:04:35,809 INFO org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.hs.JobHistory failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Error creating done directory: [igfs://igfs@192.168.1.5/tmp/hadoop-yarn/staging/history/done]