Big Data Developer

Job Title: Big Data EngineerLocation: 100% RemoteDuration: Long Term Contract Job Overview:The Database/Big Data Engineer will be responsible for analyzing requirements, prototyping data analysis solutions, collecting, preparing data for modeling and productionizing the data preparation pipelines on AWS. Candidates need to have strong capabilities in large data warehouses using relational database, performance tuning, UNIX scripting, as well as DBA tasks in Postgres. The complexity driving this project is the domain understanding and volume of financial data in the petabytes of disparate data sets on an integrated platform providing interactive analytics to the users. Job Responsibilities:• Analyze highly complex business requirements; generate technical specifications to design or redesign complex software components and applications• Act as an expert technical resource for data modeling, simulation and analysis efforts• Leverage industry Java/Python and SQL best practices to design, test, implement and support a multi-tier Java/Python solution.• Ensure quality security and compliance requirements are met for supported area• Prepare Technical Specification documents that explain the technical solutions and list the technologies to be used while implementing the solution.• Test driven development is the approach• Optimize and tune data structures, stored procedures, and queries on big data technologies.• Perform Postgres upgrades across all environments and monitor AWS resources to ensure resources are optimally used and recommend solutions to reduce AWS cost.• Ability to effectively use AWS managed services (step function, EMR (hive/spark), Lambda, Glue, and/or Athena.• Excellent problem solving and debugging skills• Work with teams in SCRUM process and be flexible/thrive in an evolving environment Qualifications:• Bachelor’s degree in computer science, Information Systems or related discipline with at least five (5) years of related experience, or equivalent training and / or work experience.• Substantial experience with at least one relational database system/product (preferably Postgres RDS)• Demonstrated technical expertise in database technologies/concepts which resulted in deployment of enterprise quality solutions.• Knowledge of industry leading data warehousing, tuning, enterprise reporting, data profiling, data movement, monitoring and diagnostic tools and approaches.• Strong hands-on SQL, Python, EMR, Spark SQL, performance tuning experiences in the past 3 years• Hands on experience in following the iterative and agile SDLC• Experience in implementing design patterns and ability to adapt to changing patterns quickly

Key Skills

• AWS Lambda