Evans Ye created IGNITE-909:
-------------------------------
Summary: Support HadoopExternalSplit in HadoopV2Context to get pig jobs to function
Key: IGNITE-909
URL:
https://issues.apache.org/jira/browse/IGNITE-909 Project: Ignite
Issue Type: Task
Components: hadoop
Affects Versions: sprint-4
Reporter: Evans Ye
Currently ignite does not support to run pig jobs on top of it. A test script like the following:
{code}
grunt> A = load '/passwd';
grunt> B = foreach A generate $0;
grunt> dump B;
{code}
can lead to a failure stack trace showing below:
{code}
[13:33:49,455][ERROR][Hadoop-task-16820899-f25d-40ee-acc7-7ec0b86f4c27_10-MAP-0-0-#165%null%][HadoopRunnableTask] Task execution failed.
class org.apache.ignite.IgniteCheckedException: class org.apache.ignite.IgniteCheckedException: null
at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2MapTask.run0(HadoopV2MapTask.java:102)
at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2Task.run(HadoopV2Task.java:50)
at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2TaskContext.run(HadoopV2TaskContext.java:193)
at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.runTask(HadoopRunnableTask.java:176)
at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:120)
at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:36)
at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService$2.body(HadoopExecutorService.java:183)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:107)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.UnsupportedOperationException
at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2Context.getInputSplit(HadoopV2Context.java:93)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.getInputSplit(WrappedMapper.java:76)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.setup(PigGenericMapBase.java:202)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2MapTask.run0(HadoopV2MapTask.java:84)
... 8 more
{code}
Implementing the TODO section in HadoopV2Context should get pig jobs to function.
{code}
@Override public InputSplit getInputSplit() {
if (inputSplit == null) {
HadoopInputSplit split = ctx.taskInfo().inputSplit();
if (split == null)
return null;
if (split instanceof HadoopFileBlock) {
HadoopFileBlock fileBlock = (HadoopFileBlock)split;
inputSplit = new FileSplit(new Path(fileBlock.file()), fileBlock.start(), fileBlock.length(), null);
}
else if (split instanceof HadoopExternalSplit)
throw new UnsupportedOperationException(); // TODO
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Locked
