Data Engineer - Pyspark Veracity Software Inc
Veracity Software Inc
Office Location
Full Time
Experience: 3 - 3 years required
Pay:
Salary Information not included
Type: Full Time
Location: Maharashtra
Skills: Python, emr, Athena, Git, SQL, Redshift, aws glue, IAM role, CloudFormation, Apache Airflow, pyspark, Open Metadata, Data Lakehouse, S3, ETL Pipeline, Aws Lambda
About Veracity Software Inc
Job Description
The ideal candidate for this position should have advanced proficiency in Python, with a solid understanding of inheritance and classes. Additionally, the candidate should be well-versed in EMR, Athena, Redshift, AWS Glue, IAM roles, CloudFormation (CFT is optional), Apache Airflow, Git, SQL, Py-Spark, Open Metadata, and Data Lakehouse. Experience with metadata management is highly desirable, particularly with AWS Services such as S3. The candidate should possess the following key skills: - Creation of ETL Pipelines - Deploying code in EMR - Querying in Athena - Creating Airflow Dags for scheduling ETL pipelines - Knowledge of AWS Lambda and ability to create Lambda functions This role is for an individual contributor, and as such, the candidate is expected to autonomously manage client communication and proactively resolve technical issues without external assistance.,