Lead Data Engineer

ABOUT THE JOB:

• Lead the development and ongoing management of the company's cloud-based data lake and common data platforms (CDP), ensuring they are scalable, secure, and optimized for business needs.
• Oversee data ingestion processes from a variety of on-premise and cloud-based sources, utilizing batch and real-time methods for collecting and integrating data.
• Utilize Azure Data Factory, Databricks, and other relevant tools to transform raw data according to business logic, ensuring it is stored and processed efficiently in multiple layers (bronze, silver, gold).
• Implement and manage real-time data ingestion using Confluent Cloud (Kafka) and similar technologies to support business intelligence and decision-making.
• Ensure data privacy protocols are followed by masking Personally Identifiable Information (PII) and safeguarding sensitive data throughout its lifecycle.
• Set up and maintain CI/CD pipelines to automate data workflows using Azure DevOps, ensuring smooth and efficient deployments.
• Continuously monitor and optimize ETL jobs, data pipelines, and cloud resources to improve system performance, minimize errors, and reduce costs.
• Work closely with the data science team to prepare clean, stable, and high-quality data for AI/ML model development and other predictive analytics initiatives.
• Lead the development of interactive Power BI dashboards and reports to deliver insights into business performance, sales behaviors, fraud detection, and more.
• Establish efficient and secure methods for sharing data across teams using platforms like SharePoint and Databricks Unity Catalog.
• Mentor junior data engineers, foster a collaborative team environment, and drive best practices in data engineering.