Big Data Developer

Optomi in partnership with a leader within the financial services industry is seeking an Big Data Developer. This developer will work on ingesting, storing, validating and disseminating after transforming data in a consumable format for business intelligence teams and data analysts to get deeper business insight from the data.

Responsibilities of the right candidate: Design and develop ETL pipeline for collecting, validating and transforming data according to the specificationDevelop automated unit tests, functional tests and performance testsMaintain optimal data pipeline architectureDesign ETL jobs for optimal execution in AWS cloud environmentReduce processing time and cost of ETL workloadsParticipate in peer reviews and design/code review meetingsProvide support for production support operations teamImplement data quality checks.Identify areas where machine learning can be used to identify data anomalies

Experience of the right candidate: 4+ years of experience in programming language Java or Scala4+ years of experience in ETL projects3+ years of experience in big data projects2+ years of experience with API development (REST API’s) and testing (Rest Assured)Strong experience in big data technologies like AWS EMR, AWS EKS, Apache SparkStrong experience with serverless technologies like AWS Dynamo DB, AWS LambdaStrong experience in processing with JSON filesStrong and creative analytical and problem-solving skills

Key Skills

• AWS
• Java
• AWS Lambda