Senior Associate & Manager - Databricks | Azure | PySpark PwC India

About PwC India

Job Description

You are a highly skilled and experienced Senior Data Engineer with over 6 years of hands-on experience in Data Engineering. You have a strong background in Databricks, SQL, PySpark, and Microsoft Azure. Your expertise includes building and optimizing data pipelines and architectures, transforming raw data into usable formats, and collaborating with data scientists, analysts, and stakeholders to support data-driven decision-making. Your key responsibilities include designing, building, and maintaining scalable data pipelines using Apache Spark and PySpark in Databricks, developing and managing data integration and ETL workflows in Azure Data Factory and Databricks, optimizing Spark jobs, writing complex SQL queries, implementing data governance practices, collaborating with cross-functional teams, monitoring and optimizing data platforms hosted on Azure, and providing technical leadership and mentoring to junior team members. You possess strong expertise in Databricks, PySpark, and distributed computing using Apache Spark. You have expert-level knowledge of SQL for data analysis and optimization. Your experience includes working with Azure Cloud Services such as Azure Data Factory, Azure Data Lake Storage, Azure Synapse, and Azure SQL DB. You are familiar with CI/CD pipelines, version control, and DevOps best practices in the Azure environment. Additionally, you have a solid understanding of data modeling, data warehousing, and data governance practices. Preferred qualifications include being a Databricks Certified Developer or holding Microsoft Azure certification(s), experience with real-time data streaming, familiarity with scripting languages for automation, experience in Agile/Scrum environments, and knowledge of the insurance domain. You have strong analytical and problem-solving skills, excellent communication skills, and experience in client-facing roles.,