UN

Pyspark Engineer

FeaturedUrgent Hiring
Uniweb CreationVerified
4.1(2,671 Reviews)
Hyderabad, Telangana
FULL TIME
5-10 Years
7,00,000 - 15,00,000 P.A.

Fast-Track Direct Hiring100% Free Apply

Employer is actively reviewing applications today. Shortlisted candidates receive direct callbacks within 24 hours.

Immediate Joiners Preferred
Posted 06/11/2026
PySparkSpark Streaming+5 more

Job Description

Role Summary

We are seeking a PySpark Engineer to design, build, and optimise Spark applications and ETL data pipelines for a complex enterprise data lake. The role covers batch and streaming workloads, Delta Lake implementation, Spark performance tuning and cluster sizing, and CI/CD automation for PySpark. The engineer will operate highly available Spark clusters with monitoring, work with Agile application development teams on data strategy and dataflows, and act as a technical specialist with responsibility for guiding a small team.

Key Responsibilities

·     Design and develop Spark applications using PySpark for enterprise-scale data processing.

·     Build and maintain ETL data pipelines feeding a complex data lake implementation.

·     Implement Delta Lake (delta.io) for enterprise-grade data storage and reliability.

·     Perform optimisation and performance tuning of Spark applications.

·     Develop Spark Streaming and Structured Streaming workloads.

·     Build and set up CI/CD pipelines for PySpark deployments.

·     Write and execute test cases for Spark applications, including performance tests.

·     Size Spark clusters and manage resources across Spark Standalone, YARN, and Kubernetes cluster managers.

·     Set up and operate highly available Spark clusters with operational monitoring.

·     Work with Agile application development teams to implement data strategies, build dataflows, and define conceptual data models.

·     Forecast environment requirements based on anticipated demand from multiple application development teams.

·     Create short-term plans to deliver environments supporting sprint-based development.

·     Provide technical guidance and manage a small team of technical specialists.

Primary Skills (Must Have)

·     Experience designing and developing Spark applications using PySpark.

·      Hands-on experience building and maintaining ETL data pipelines.

·     Expertise in Python development.

·     Proficiency in writing SQL scripts.

·     Spark Streaming and Structured Streaming knowledge is mandatory.

·     Experience with Delta Lake (delta.io) for enterprise-grade implementation.

·     Optimisation and performance tuning of Spark applications.

·     Experience with different cluster managers:

o    Spark Standalone

o    YARN

o    Kubernetes

·     Spark cluster sizing and resource management for a complex data lake implementation.

·     Experience setting up and operating highly available Spark clusters with operational monitoring.

·     Experience building and setting up CI/CD pipelines for PySpark.

·     Experience writing and executing test cases for Spark applications, including performance tests.

·     Knowledge of Big Data on Cloud, preferably GCP services such as Dataproc and GCS.

·     Strong communication skills and the ability to plan and prioritise own time effectively.

·     Ability to manage a small team as technical specialists.

Secondary Skills (Nice to Have)

·     Java or Scala development experience.

·     AWS or Azure cloud platform experience.

·     Exposure to workflow orchestration tools such as Airflow or Cloud Composer.

·     Familiarity with data governance, lineage, and cataloguing practices.

·     Observability tooling for Spark workloads – metrics, logging, and alerting.

GCP Data Engineer certification is an advantage

Interested Candidate, Please connect on 88377021169

Benefits & Perks

Health Insurance
Work From Office
Paid Time Off
Learning & Development
Performance Bonus
On-site Cafeteria
SPONSORED / ADVERTISEMENTPagaar India Partners

Apply for this Job

Create your profile and apply in seconds.

100% Free to Apply
Direct Apply to Company
No Middleman

Share this Job

Help friends apply

Company Overview

UN

Uniweb Creation

4.1(2,671)
Company Office
IT
10,000+ Employees
Hyderabad, Telangana
View Company Profile

Salary Insights

Estimated salary for this role in Hyderabad

₹ 12 LPA
Min
₹ 18 LPA
Average
₹ 24 LPA
Max

Get Job Alerts on Mobile

Instant notifications on WhatsApp & Phone Screen.

SPONSORED / ADVERTISEMENTPagaar India Partners

Similar Jobs You Might Like

View All
NO

Data Scientist Gen AI / Agentic AI

Not to disclose as per our agreement with client

9,00,000 - 10,00,000 P.A.
Pune
3-5 Years
Data Scientist Gen AI / Agentic AISoftware Engineering
06/11/2026Apply Now →
SP

Software

SpocitServiceBPO

7,00,000 - 9,00,000 P.A.
Dilsukhnagar
3-5 Years
UI
06/11/2026Apply Now →
NO

Staff Engineer

NovoTree Minds Consulting Private Limited

15,00,000 - 25,00,000 P.A.
Bengaluru
5-10 Years
Javaspring boot
06/11/2026Apply Now →
PR

Marketing Trainee

programming.com

3,00,000 - 5,00,000 P.A.
sector 8B
Fresher
 Assist in digital marketing campaigns.  Create engaging content for social media and marketing platforms.  Conduct market research and competitor analysis.  Support email marketing and lead generation activities.  Coordinate with internal teams for marketing initiatives.  Prepare campaign reports and performance insights.
06/11/2026Apply Now →

Pyspark Engineer

₹7,00,000 - ₹15,00,000 P.A.