Data Engineer PRODAT IT SOLUTIONS (INDIA) PVT. LTD

About PRODAT IT SOLUTIONS (INDIA) PVT. LTD

Job Description

The Data Engineer role at Prodat IT Solutions requires a highly skilled professional with over 5 years of experience in big data engineering and a strong background in Microsoft Fabric. As a Data Engineer, you will be responsible for designing, developing, and optimizing end-to-end data pipelines using PySpark notebooks within the Fabric Lakehouse environment. You will work closely with cross-functional teams to ensure data quality, implement data validation frameworks, and optimize distributed Spark workloads for performance and cost-efficiency. Key Responsibilities: - Design, develop, and optimize end-to-end data pipelines within the Fabric Lakehouse environment. - Architect and maintain Delta Lake structures with efficient partitioning and versioning in OneLake. - Implement robust data validation, testing, and monitoring frameworks to ensure data quality and integrity. - Collaborate with Data Architects, Analysts, and AI Engineers to deliver high-quality datasets for reporting tools. - Build and manage CI/CD pipelines for Fabric notebooks, dataflows, and other assets using Azure DevOps or GitHub. - Ensure compliance with data governance and security standards using tools like Microsoft Purview. - Provide technical leadership, conduct code reviews, and mentor junior engineers. - Contribute to architecture discussions, solution design, and documentation of data workflows. Required Qualifications: - Bachelors or Masters degree in Computer Science, Data Engineering, Information Systems, or a related field. - 5+ years of experience in big data engineering, with at least 2 years of hands-on experience with PySpark. - Proven experience with Microsoft Fabric, Lakehouse architecture, Delta Lake, and OneLake. - Strong SQL and Spark optimization skills. - Experience with Power BI, DirectLake, and semantic model design. - Familiarity with medallion architecture (bronze, silver, gold layers) and data warehousing principles. - Experience implementing CI/CD pipelines and version control for data workflows. - Strong understanding of data testing frameworks and monitoring solutions. - Excellent communication and problem-solving skills. Preferred Qualifications: - Certifications in Microsoft Fabric, Azure Data Engineering, or Databricks. - Experience with Microsoft Purview for data governance and lineage. - Familiarity with other modern data platforms like Azure Synapse, Databricks, or Snowflake. - Exposure to AI/ML workloads integrated into data pipelines. Join Prodat IT Solutions to be part of a data-driven culture where you can work with cutting-edge technologies, collaborate with talented professionals, and contribute to impactful projects that shape the future of data and analytics. To apply, please send your updated resume to Rakes.k@prodatit.com / 9606018324. This is a full-time position with benefits including health insurance, Provident Fund, and work from home options. Kindly mention your Notice Period when applying. Work Location: In person,