Senior Data Engineer (Exp 5+, AWS, Python, Pyspark) Atyeti Inc

About Atyeti Inc

Job Description

Job Title: Senior Data Engineer Shift Timing: From 11:00 AM onwards Experience: (5+ in Python and PySpark) Education: Bachelors degree in Computer Science, Engineering, Statistics, Mathematics, or a related field (Masters preferred) Key Responsibilities: Design, build, and maintain scalable and secure data pipelines using AWS Glue and Python. Develop and manage ETL/ELT processes for seamless data integration between cloud-native systems (e.g., Snowflake) and SaaS platforms. Monitor and maintain data pipelines using CloudWatch and implement alerting via SNS. Ensure data governance, compliance, and security best practices across the data lifecycle. Collaborate with cross-functional teams including product owners, data analysts, and engineering teams. Maintain and enhance data environments including data lakes, data warehouses, and distributed systems. Implement version control and CI/CD pipelines using GitHub and GitHub Actions. Work in an Agile environment and contribute to continuous improvement initiatives. Mandatory Skills: 10+ years of experience in data engineering with strong expertise in AWS Glue (Crawler, Data Catalog) 5+ years of hands-on experience with Python and PySpark Strong ETL development and support experience using AWS services (Glue, Lambda, SNS, S3, Athena, Secret Manager) Proficient in writing and optimizing complex SQL queries (Advanced SQL, PL/SQL) Experience with CI/CD pipelines using GitHub Actions Good understanding of monitoring tools like CloudWatch Experience in Agile methodology and working in cross-functional teams Preferred Skills: Experience with Snowflake, including internal/external tables, stages, and data masking policies Familiarity with BI tools like Power BI and Tableau Knowledge of Infrastructure as Code tools: Terraform and CloudFormation Exposure to Jira and GitHub for project and source code management Understanding of secure data practices and compliance frameworks.,