Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected

[Prithiraj]

Hello Stephen,

I tried to execute your program in Amazon EMR, using AMI 3.3.1 (Hadoop 2.4.0)

Also I checked with different libraries of hadoop-core/hadoop-client.

On executing:

hadoop jar cc-warc-examples-0.1-SNAPSHOT-jar-with-dependencies.jar org.commoncrawl.examples.mapreduce.WATServerType

I get the following recurring error:
Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected

With mapred (old api) I don't get the error. But I will need features from MapReduce library.

Let me know what you recommend or if you have any workaround for that.

Thanks,
Prith

[AaroC357]

Prith,

I am also experiencing this issue. I was wondering if you have come across a solution?

I do not have a lot of experience in this area, but my best guess is that one of the jar files was compiled under Hadoop 1, and is not compatible with Hadoop 2. I say this because TaskAttemptContext was changed from a Class to Interface in the respective versions of Hadoop.

Additionally, looking at the dependencies in the maven repository, my best guess is that the webarchive-commons jar is the culprit. Not sure, and I'm not sure how to work around this.

Any assistance would be much appreciated.

Thanks,
Aaron

[Prithiraj]

HI Aaron,

I have used the following dependencies and things worked good for me. Please update the pom.xml with the following, and let me know if you come across any issues.

  <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk</artifactId>
            <version>1.0.002</version>
        </dependency>
        <dependency>
            <groupId>commons-codec</groupId>
            <artifactId>commons-codec</artifactId>
            <version>1.4</version>
        </dependency>
        <dependency>
            <groupId>commons-io</groupId>
            <artifactId>commons-io</artifactId>
            <version>2.4</version>
        </dependency>
        <dependency>
            <groupId>commons-logging</groupId>
            <artifactId>commons-logging</artifactId>
            <version>1.1.3</version>
        </dependency>
        <dependency>
            <groupId>commons-httpclient</groupId>
            <artifactId>commons-httpclient</artifactId>
            <version>3.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpcore</artifactId>
            <version>4.2.5</version>
        </dependency>
        <dependency>
            <groupId>org.codehaus.jackson</groupId>
            <artifactId>jackson-core-asl</artifactId>
            <version>1.9.10</version>
        </dependency>
        <dependency>
            <groupId>org.codehaus.jackson</groupId>
            <artifactId>jackson-mapper-asl</artifactId>
            <version>1.9.10</version>
        </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-client</artifactId>
        <version>2.6.0</version>
    </dependency>
    <dependency>
        <groupId>org.netpreserve.commons</groupId>
        <artifactId>webarchive-commons</artifactId>
        <version>1.1.4</version>
    </dependency>
  </dependencies>

I exactly don't remember the details as this was a library issue and not a conceptual stuff. But there were version incompatibilities, which didn't let me do the stuff in earlier attempts.

Thanks,
Prith

[AaroC357]

Thanks for the info. I tried adding the dependencies you listed to my pom but still no joy.

I'm trying to do a small subset of the example given before scaling up to a job on AWS. Specifically, I've created a Maven project in Eclipse where I am solely trying to perform a word count on an individual WET file. I'm trying to run the job in HortonWorks Sandbox HDFS 2.7.1.2.3. Can't figure out what I'm missing.

Anyway, thanks for responding so promptly, much appreciated.

Aaron

[AaroC357]

Did you have a lot of warnings regarding overlapping classes between jars upon build?