Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue when trying to build #38

Closed
phungleson opened this issue Feb 1, 2017 · 3 comments
Closed

Issue when trying to build #38

phungleson opened this issue Feb 1, 2017 · 3 comments
Labels

Comments

@phungleson
Copy link
Contributor

Hey guys,

I got this error when trying to build my project

[error] missing or invalid dependency detected while loading class file 'package.class'.
[error] Could not access term streaming in package org.apache.spark,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'package.class' was compiled against an incompatible version of org.apache.spark.
[error] missing or invalid dependency detected while loading class file 'package.class'.
[error] Could not access type DStream in value org.apache.spark.dstream,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'package.class' was compiled against an incompatible version of org.apache.spark.dstream.
[error] two errors found

My simple.sbt looks like this and I use sbt assembly

version := "1.0"
scalaVersion := "2.11.8"

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.0" % "provided"
libraryDependencies += "org.apache.hadoop" % "hadoop-aws" % "2.7.3"
libraryDependencies += "com.amazonaws" % "aws-java-sdk" % "1.7.4"
libraryDependencies += "com.github.benfradet" %% "spark-kafka-0-10-writer" % "0.2.0"

Thanks if you can give me some help.

Cheers,

@BenFradet
Copy link
Owner

You need to depend on spark-streaming also:

libraryDependencies += "org.apache.spark" %% "spark-streaming" % "2.0.0" % "provided"

@phungleson
Copy link
Contributor Author

Awesome, it works!

Thanks mate!

@BenFradet
Copy link
Owner

No problem 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants