Only "Spark Developer"

Job Title: Lead Spark developerDuration: Long Term Job Description:We are looking for a Spark developer who knows how to fully exploit the potential of our Spark cluster.You will clean, transform, and analyze vast amounts of raw data from various systems using Spark to provide ready-to-use data to our feature developers and business analysts.Responsibilities:Create Scala/Spark jobs for data transformation and aggregationProduce unit tests for Spark transformations and helper methodsWrite Scala doc-style documentation with all codeDesign data processing pipelinesRequired Skills: Scala (with a focus on the functional programming paradigm)Scalatest, JUnit, Mockito, Embedded CassandraApache Spark 2.xApache Spark RDD APIApache Spark SQL Data Frame APIApache Spark MLlib APIApache Spark GraphX APIApache Spark Streaming APISpark query tuning and performance optimizationSQL database integration Microsoft, Oracle, Postgres, and/or MySQLExperience working with HDFS, S3, Cassandra and/or DynamoDBDeep understanding of distributed systems (e.g. CAP theorem, partitioning, replication, consistency, and consensus)This involves both ad-hoc requests as well as data pipelines that are embedded in our production environment.

Key Skills

• SQL