Big Data Analyst

Tata Consultancy Services is an Indian multinational information technology services and consulting company headquartered in Mumbai, Maharashtra, India with its largest campus located in Chennai, Tamil Nadu, India. As of February 2021, TCS is the largest IT services company in the world by market capitalisation.

Summary

The Big Data Expert will be responsible for building, maintaining data pipelines and data products to ingest, process large volume of structured / unstructured data from various sources. The Big Data Expert will work on analyzing the data needs, migrating the data into an Enterprise data lake, build data products and reports. The role requires experience with building real time and batch based ETL pipelines with strong understanding of big data technologies and distributed processing frameworks with.

Skill Needs

o   Expertise working with large scale distributed systems (Hadoop, Spark, Hive). 

o   Strong understanding of the big data cluster, and its architecture

o   Experience building and optimizing big data ETL pipelines.

o   Advanced programming skills with Python, Java, Scala.

o   Good knowledge of spark internals and performance tuning of spark jobs.

o   Strong SQL skills and is comfortable operating with relational data models and structure. 

o   Experience with messaging systems like Kafka.

o   Experience with No SQL databases, DashDB, etc.

o   Expertise with Continuous Integration/Continuous Delivery workflows and supporting applications.

o   Exposure to cloud environments and architectures. (Preferably AWS)

o   Ability to work collaboratively with other teams. 

o   Experience with containerization using tools such as Docker will be added advantage. 

o   Strong knowledge of Linux and Bash.  Can interact with the OS at the command line and create shell scripts to automate workflows. 

o   Advanced understanding of software development and collaboration, including experience with tools such as Git.

o   Excellent written and verbal communication skills, comfortable presenting in front of non-technical audiences.

o   Experience is orchestration tools like Airflow, Jenkins

o   Experience in Apache Ni-Fi process

Essential Responsibilities include but not limited to:

• Design and develop ETL workflows to migrate data from varied data sources including SQL Server, Netezza, Kafka etc. in batch and real-time.
• Develop checks and balances to ensure integrity of the ingested data.
• Design and Develop Spark jobs as per requirements for data processing needs.
• Work with Analysts and Data Scientists to assist them in building scalable data products.
• Designs systems, alerts, and dashboards to monitor data products in production

Salary budget- 2.5 times of work experience and in exceptional case it will be 3 times of work experience.