Senior Kafka Developer

Job Role: Senior Kafka Developer Type: Remote Contract Job Location: Remote from Eu

Job Description :

• Advance data streaming development experience using any of the tool’s e.g Apache Kafka, Confluent Kafka, Flume, Storm, MQ • Understanding of ETL solution development in real-time • Fluent in Java or Scala • Fluent in writing SQL scripts and dealing with RDBMS systems • Unix commands and scripting • Experience of DevOps tools (GIT, Jenkins basics) is must • In addition to this it would be bonus if you know Azure Cloud r9eal-time services

As a Senior Kafka Developer, you will be responsible for architecting, implementing, and managing real-time event streaming systems using Confluent Kafka while integrating these streams with Microsoft Azure DataLake for secure storage and processing. You will work closely with other engineers, architects, project managers, and stakeholders to create robust and efficient data pipeline solutions supporting business objectives. This role requires strong technical competency along with excellent collaboration skills and the ability to thrive in fast-paced agile environments.Responsibilities:

Lead the conceptualization, architecture, and development of complex, large-scale data ingestion pipelines based on Confluent Kafka, utilizing best practices around schema management, security, resilience, and observability.Ensure seamless integration between Confluent Kafka clusters and Azure DataLake by leveraging appropriate APIs, SDKs, and toolsets provided by both ecosystems.Define and implement strategies for partitioning, replication, compression, encryption, and retention policies according to organizational guidelines and industry regulations.Optimize resource utilization, minimize latency, and enhance overall system performance via fine-tuning configurations, scaling techniques, batch sizes, parallelism settings, etc.Create comprehensive documentation covering solution designs, configuration guides, user manuals, test cases, and operational procedures.Work collaboratively with multi-disciplinary teams during sprint cycles, participating in planning sessions, daily standups, backlog groomings, demos, and retrospectives.Continuously monitor and troubleshoot issues affecting data integrity, delivery timeliness, or platform availability; proactively address bottlenecks, errors, or failures through root cause analysis and remediation plans.Contribute towards building CI/CD pipelines, automation frameworks, testing scenarios, and DevOps processes to streamline software development lifecycle activities.Maintain current awareness of emerging trends, tools, libraries, protocols, and standards relevant to Kafka, Azure DataLake, cloud computing, big data, IoT, AI/ML, and cybersecurity.Demonstrated proficiency in administering and operating Confluent Kafka clusters at scale, preferably in Linux environment(s).Solid background in cloud computing, specifically Microsoft Azure, with emphasis on PaaS components like Azure Event Hubs, Functions, Stream Analytics, Databricks, Synapse, Cosmos DB, Key Vault, Logic Apps, API Management, and Power BI.Comprehensive understanding of data warehousing concepts, ETL patterns, dimensional modeling, fact / lookup tables, star / snowflake schemas, materialized views, indexes, sharding, and partitioning strategies.Practical familiarity with RESTful web services, gRPC, HTTP/2, AMQP, Webhooks, GraphQL, OAuth2, JWT, OpenAPI Specification, RAML, WSDL, SOAP UI, Postman, SwaggerHub, and ReadyAPI.Proficient in programming languages such as Java, Scala, Go, .NET Core, Node.js, Python, Rust, Ruby, PHP, Perl, or SWIFT.Skilled in shell scripting (Bash, Zsh), object-oriented design principles, microservices architecture, domain-driven design, containers (Docker, Kubernetes), serverless functions, message brokers, load balancers, reverse proxies, service meshes, and API gateways.Well versed in Agile/Scrum project management methodology, Git source control, pull requests, code reviews, pair programming, trunk-based development, feature flags, A/B testing, canary releases, blue/green deployments, chaos engineering, ITIL incident management, change advisory boards, postmortem analyses, runbooks, SLAs, OKRs, NFRs, and QoS metrics.

Key Skills

• DevOps
• Java
• SQL
• Jenkins
• Kafka