Closed
Description
Hey guys,
I got this error when trying to build my project
[error] missing or invalid dependency detected while loading class file 'package.class'.
[error] Could not access term streaming in package org.apache.spark,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'package.class' was compiled against an incompatible version of org.apache.spark.
[error] missing or invalid dependency detected while loading class file 'package.class'.
[error] Could not access type DStream in value org.apache.spark.dstream,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'package.class' was compiled against an incompatible version of org.apache.spark.dstream.
[error] two errors found
My simple.sbt
looks like this and I use sbt assembly
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.0" % "provided"
libraryDependencies += "org.apache.hadoop" % "hadoop-aws" % "2.7.3"
libraryDependencies += "com.amazonaws" % "aws-java-sdk" % "1.7.4"
libraryDependencies += "com.github.benfradet" %% "spark-kafka-0-10-writer" % "0.2.0"
Thanks if you can give me some help.
Cheers,
Activity
BenFradet commentedon Feb 1, 2017
You need to depend on spark-streaming also:
phungleson commentedon Feb 1, 2017
Awesome, it works!
Thanks mate!
BenFradet commentedon Feb 1, 2017
No problem 👍