Skip to content

allwefantasy/spark-submitter-console

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
 
 
 
 
 
 
 
 

Repository files navigation

Spark Submitter Console

Spark Submitter Console is a tinny but useful web console in which you can

  1. Uploading jar
  2. Submitting jar to Yarn cluster
  3. Monitoring/Manager your applications:

How to setup

  1. Clone and package:

    mvn clean package -DskipTests -Pshade
    
  2. find the spark_jobs_2017-08-20.sql in resources directory and import to MySQL.

  3. Create /tmp/jdbc.properties like this:

    url=jdbc:mysql://127.0.0.1:3306/spark_jobs?useUnicode=true&characterEncoding=utf8
    userName=xxxx
    password=xxx
    
  4. Find the jar and start it:

java -cp .:[your jar path] tech.mlsql.spark.submitter.SparkSubmitterConsoleApp \
  -yarnUrl http://127.0.0.1 \
  -jdbcPath /tmp/jdbc.properties 

Limitation for 0.1.0

  1. Only support Yarn Cluster
  2. No Auth available. We recommend you to placed it behind Nginx which can provide the auth.
  3. No high availability design

About

A web application for submitting spark application

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published