Job Summary

We are looking for a highly skilled Data Engineer with strong expertise in ETL/ELT development, cloud-based data platforms, and modern data engineering practices. The ideal candidate will have hands-on experience with Azure Databricks, Snowflake, Spark, Python, and infrastructure automation. Experience working with AI/ML and Large Language Models (LLMs) will be a strong advantage.

The candidate will be responsible for designing, building, and maintaining scalable data pipelines, optimizing data processing frameworks, and enabling advanced analytics and AI-driven solutions across the organization.

Required Skills

• Data Engineering and ETL/ELT Processes
• Python
• Apache Spark (PySpark, Spark SQL)
• Azure Databricks (Unity Catalog)
• Snowflake
• Azure Functions
• Azure Service Bus
• SQL
• Terraform
• CI/CD Tools and Processes
• GitHub Actions
• Git
• Artifactory
• SonarQube
• AI/ML and Large Language Models (LLM)

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines for enterprise data platforms.
• Build and optimize data processing solutions using Python, PySpark, and Spark SQL.
• Develop and manage data solutions on Azure Databricks and Snowflake.
• Implement and maintain Unity Catalog-based governance and security frameworks.
• Create serverless integrations and event-driven workflows using Azure Functions and Azure Service Bus.
• Develop efficient SQL queries, data models, and performance optimization strategies.
• Automate infrastructure deployment and management using Terraform.
• Implement CI/CD pipelines and DevOps best practices using GitHub Actions, Git, Artifactory, and SonarQube.
• Collaborate with data architects, business analysts, and cross-functional teams to deliver data-driven solutions.
• Support AI/ML initiatives by preparing and engineering datasets for predictive analytics and LLM-based applications.
• Ensure data quality, security, governance, and operational excellence across data platforms.

Required Qualifications

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.
• 7–8 years of hands-on experience in Data Engineering and ETL/ELT development.
• Strong expertise in Python, Spark, and SQL.
• Experience working with Azure Databricks and Snowflake in enterprise environments.
• Hands-on experience with Terraform and Infrastructure as Code (IaC).
• Strong understanding of DevOps and CI/CD methodologies.
• Experience with cloud-native services and distributed data processing systems.
• Excellent analytical, troubleshooting, and communication skills.

Preferred Qualifications

• Experience with AI/ML model deployment and MLOps practices.
• Hands-on exposure to Generative AI and Large Language Model (LLM) implementations.
• Knowledge of data governance, data security, and metadata management.
• Azure certifications or Snowflake certifications are a plus.

Key Search Keywords

ETL, ELT, Data Engineering, Python, PySpark, Spark SQL, Azure Databricks, Unity Catalog, Snowflake, Azure Functions, Azure Service Bus, SQL, Terraform, GitHub Actions, Git, Artifactory, SonarQube, CI/CD, AI/ML, Generative AI, LLM.

Pay: ₹510,442.41 - ₹1,869,415.67 per year

Work Location: Hybrid remote in Chittoor, Andhra Pradesh