Data Engineer Spark And Delta Lake People Prime Worldwide
People Prime Worldwide
Office Location
Full Time
Experience: 6 - 6 years required
Pay:
Salary Information not included
Type: Full Time
Location: Hyderabad
Skills: spark, Python, Scala, Java, ETL, Data Quality, Data validation, Git, containerization, Data Integration, Datastage, Prophecy, Informatica, Ab Initio, RDDs, Dataframes, datasets, SparkSQL, Spark Streaming, Delta Lake, Data Ingestion, Data Warehouse, data modelling, cicd
About People Prime Worldwide
Job Description
About Client :- Our client is a French multinational information technology (IT) services and consulting company, headquartered in Paris, France. Founded in 1967, It has been a leader in business transformation for over 50 years, leveraging technology to address a wide range of business needs, from strategy and design to managing operations. The company is committed to unleashing human energy through technology for an inclusive and sustainable future, helping organizations accelerate their transition to a digital and sustainable world. They provide a variety of services, including consulting, technology, professional, and outsourcing services. Job Details:- location : Gurugram/Noida/Chennai and Bangalore Mode Of Work : Hybrid Notice Period : Immediate Joiners Experience : 6-8 yrs Type Of Hire : Contract to Hire JOB DESCRIPTION: Understanding of Spark core concepts like RDDs, DataFrames, DataSets, SparkSQL and Spark Streaming. Experience with Spark optimization techniques. Deep knowledge of Delta Lake features like time travel, schema evolution, data partitioning. Ability to design and implement data pipelines using Spark and Delta Lake as the data storage layer. Proficiency in Python/Scala/Java for Spark development and integrate with ETL process. Knowledge of data ingestion techniques from various sources (flat files, CSV, API, database) Understanding of data quality best practices and data validation techniques. Other Skills: Understanding of data warehouse concepts, data modelling techniques. Expertise in Git for code management. Familiarity with CI/CD pipelines and containerization technologies. Nice to have experience using data integration tools like DataStage/Prophecy/Informatica/Ab Initio",