Data Engineer

1. Data pipeline development and optimization

· Design, build, and maintain efficient, scalable ETL/ELT pipelines for structured and unstructured data.

· Implement modular, reusable Python-based processing scripts and manage job orchestration using Power Automate, Linux cron, or other scheduling frameworks.

· Continuously monitor and optimize pipeline performance for speed and resource utilization.

2. Cloud infrastructure and cost management

· Deploy and manage data infrastructure leveraging AWS services (e.g. Lambda, Glue, S3, RDS, Athena).

· Ensure system uptime, data accessibility, and high availability across environments.

· Proactively optimize storage and compute costs through monitoring, automation, and right-sizing of resources.

3. Automation, CI/CD & DevOps practices

· Automate deployment workflows using GitLab CI/CD pipelines.

· Maintain a clean, testable codebase with proper version control and documentation.

· Introduce best practices for environment provisioning, rollback, and automated testing.

4. Data quality, security & governance

· Implement robust data validation, logging, and monitoring practices across pipelines.

· Ensure data privacy, integrity, and lifecycle compliance in collaboration with IT security policy.

· Maintain documentation including data dictionaries, flow diagrams, and operational runbooks.

5. Cross team technical support & enablement

· Collaborate with IT, software, and business system teams to integrate backend data solutions.

· Ensure that platform components (pipelines, storage, compute) support the broader application and reporting ecosystem.

· Provide technical guidance on data ingestion, transformation, and storage patterns.

Required Competencies

·Strong programming skills in Python, with experience building modular and reusable components for data workflows

·Proficient in SQLfor data manipulation and performance tuning across relational databases (PostgreSQL, MySQL)

·Hands-on experience with AWS servicessuch as Lambda, Glue, S3, RDS, and Athena, with a focus on cost and performance optimization

·Fluent in CI/CD and automation frameworks, particularly using GitLab pipelines, Power Automate, and scripting on Linux/Windows environments

·Solid understanding of data pipeline orchestration(e.g., cron, Airflow, Power Automate) and containerization (e.g., Docker)

·Knowledge of data governance, privacy, and quality control, including validation, monitoring, and alerting mechanisms

·Familiar with version control, infrastructure documentation, and maintaining data dictionaries or runbooks

·Demonstrated ability to collaborate with cross-functional technical teams, supporting data infrastructure integration and maintenance

Qualification and Experience Required

·Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related technical field

·Professional certification in data engineering, cloud architecture, or AWS (e.g., AWS Certified Data Analytics or Solutions Architect) is an advantage

·Minimum 3 years of hands-on experience in a data engineering or backend data infrastructure role

·Proven track record in building and maintaining production-grade ETL/ELT pipelines using Python and SQL

·Demonstrated experience managing cloud-based infrastructure, with a strong understanding of cost optimization, monitoring, and scalability (preferably on AWS)

·Experience working with DevOps practices, CI/CD pipelines (e.g., GitLab), and automation tools for deployment and workflow orchestration

·Familiarity with data quality, security, and governance best practices

·Experience in a multi-business or regional group environment is a plus