Sr Data Engineer

As a Data Engineer, you will play a crucial role in designing, implementing, and maintaining robust data pipelines on the Databricks platform. Your primary responsibility will be to ensure the smooth flow of data by collaborating with data scientists, analysts, and stakeholders to guarantee data availability and integrity for analytics and reporting purposes. Your key responsibilities will include designing and developing data pipelines using Databricks and Apache Spark to create scalable and efficient solutions. You will be responsible for integrating data from various sources such as databases, APIs, and external datasets, focusing on maintaining data quality and consistency throughout the process. In addition, you will be tasked with developing and optimizing ETL processes to support data analytics and business intelligence needs. Performance optimization will be a critical aspect of your role, where you will work on tuning Spark jobs and managing resource allocation to ensure optimal data processing efficiency. Collaboration will be key in your role as you will closely work with data scientists, analysts, and stakeholders to understand data requirements and provide effective solutions. Implementing data validation and cleansing procedures will be essential to guarantee data accuracy and reliability. Documentation plays a vital role in maintaining transparency and knowledge sharing within the team. Therefore, you will be responsible for documenting data pipelines, processes, and architecture to facilitate maintenance and enhance knowledge sharing among team members. Monitoring and troubleshooting data pipelines will also be part of your responsibilities. You will be required to monitor pipeline performance and reliability, as well as troubleshoot any issues that may arise to ensure the smooth functioning of the data infrastructure.,