Skip to content

Issue when trying to build #38

Closed
Closed
@phungleson

Description

@phungleson
Contributor

Hey guys,

I got this error when trying to build my project

[error] missing or invalid dependency detected while loading class file 'package.class'.
[error] Could not access term streaming in package org.apache.spark,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'package.class' was compiled against an incompatible version of org.apache.spark.
[error] missing or invalid dependency detected while loading class file 'package.class'.
[error] Could not access type DStream in value org.apache.spark.dstream,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'package.class' was compiled against an incompatible version of org.apache.spark.dstream.
[error] two errors found

My simple.sbt looks like this and I use sbt assembly

version := "1.0"
scalaVersion := "2.11.8"

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.0" % "provided"
libraryDependencies += "org.apache.hadoop" % "hadoop-aws" % "2.7.3"
libraryDependencies += "com.amazonaws" % "aws-java-sdk" % "1.7.4"
libraryDependencies += "com.github.benfradet" %% "spark-kafka-0-10-writer" % "0.2.0"

Thanks if you can give me some help.

Cheers,

Activity

BenFradet

BenFradet commented on Feb 1, 2017

@BenFradet
Owner

You need to depend on spark-streaming also:

libraryDependencies += "org.apache.spark" %% "spark-streaming" % "2.0.0" % "provided"
phungleson

phungleson commented on Feb 1, 2017

@phungleson
ContributorAuthor

Awesome, it works!

Thanks mate!

BenFradet

BenFradet commented on Feb 1, 2017

@BenFradet
Owner

No problem 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @phungleson@BenFradet

        Issue actions

          Issue when trying to build · Issue #38 · BenFradet/spark-kafka-writer