Analytics / Data

Senior Data Engineer

NTT DATA India

  • Bengaluru, Karnataka
  • Onsite

Posted 3 Nov 2026

About this role

Role Summary NTT DATA is seeking a highly experienced Senior Data Engineer / Technical Lead for a large-scale banking data modernization program involving migration of approximately 2 PB of historical data from Cloudera Hadoop/Hive to Apache Iceberg. The role requires a strong hands-on engineer who can lead the engineering team while continuing to design, develop, troubleshoot and optimize the solution. The candidate will own technical execution of the migration framework, Spark/PySpark engineering, Iceberg implementation, data modelling/schema mapping, Cyber/TaaS integration, migration controls, reconciliation, performance tuning and technical delivery of migration waves. This is not a coordination-only Technical Lead role. The candidate is expected to contribute to code, conduct design/code reviews, troubleshoot complex production issues and mentor Data Engineers. Experience 10+ years of Data Engineering experience, including:

  • 5+ years hands-on Apache Spark/PySpark.
  • Strong Hadoop/Hive experience.
  • Experience leading Data Engineering teams.
  • Large-scale data migration/data modernization experience.
  • Strong data modelling and source-to-target mapping experience. Multi-hundred-TB/PB-scale migration experience is strongly preferred.

Key Responsibilities

Technical Leadership

  • Lead a team of Data Engineers through assessment, development, migration, validation and production execution.
  • Translate architecture and migration standards into detailed engineering designs and reusable implementation patterns.
  • Own technical estimates, engineering backlog, work allocation and technical delivery.
  • Conduct design and code reviews and enforce engineering standards.
  • Mentor Data Engineers and provide hands-on technical guidance.
  • Act as the primary engineering escalation point for complex migration and production issues.
  • Partner with Data Architect, Technical PM, security, infrastructure and PNC technical teams.
  • Lead root-cause analysis and ensure fixes are incorporated into reusable framework patterns rather than implemented as one-off solutions. Hadoop-to-Iceberg Migration
  • Lead technical execution of approximately 2 PB of Hadoop/Hive historical-data migration to Apache Iceberg.
  • Harden and extend the reusable, configuration-driven migration framework developed during the POC.
  • Analyze Hive/HDFS metadata, tables, partitions, file layouts and migration complexity.
  • Implement migration strategies including bulk/CTAS, partition-wise and chunk-wise processing.
  • Design execution approaches for small, medium, large and very-large tables.
  • Optimize parallel migration across tables and partitions.
  • Implement checkpointing, restart/recovery, idempotency and safe retries.
  • Lead technical execution and troubleshooting of migration waves. Data Modelling & Schema Engineering
  • Analyze existing Hive data models and translate them into approved Apache Iceberg physical models.
  • Develop and review conceptual/logical/physical data models where required, with primary emphasis on physical modelling for migration.
  • Perform source-to-target data mapping and schema/data-type mapping.
  • Define and implement table structures consistent with approved Iceberg standards.
  • Analyze: o entities and relationships, o primary/foreign-key relationships, o data types, o partition keys, o business keys, o reference/cross-reference structures.
  • Identify schema incompatibilities and recommend remediation.
  • Work with the Data Architect on partitioning, file-layout and model-design decisions.
  • Ensure migrated models preserve required business relationships and downstream compatibility.
  • Support future Bronze/Silver/Gold modelling requirements where applicable. Important: enterprise modelling standards remain owned by the Data Architect; the Senior DE/Technical Lead is responsible for detailed modelling and implementation within those standards. Spark/PySpark Engineering & Performance
  • Develop and review production-grade Python/PySpark components.
  • Tune Spark jobs for PB-scale migration.
  • Diagnose Spark stages, executors, shuffle, skew, memory and I/O bottlenecks.
  • Optimize table/partition/chunk parallelism.
  • Tune workloads to meet migration-throughput objectives.
  • Identify bottlenecks across: Hadoop Read → Spark → TaaS/Tokenization → Iceberg Write → Reconciliation
  • Coordinate technical remediation with infrastructure/storage/network teams where bottlenecks are outside the migration framework. Apache Iceberg Engineering
  • Implement approved Iceberg table standards.
  • Develop/configure Iceberg tables and Spark→Iceberg processing.
  • Implement: o partitioning, o schema evolution, o snapshots, o file sizing, o compaction, o metadata maintenance, o small-file remediation.
  • Troubleshoot Iceberg catalog/write/commit issues.
  • Optimize Iceberg table performance and storage characteristics. Cyber-3 / Tokenization
  • Implement PNC-approved TaaS/Protegrity tokenization and retokenization patterns.
  • Integrate tokenization into Spark migration pipelines.
  • Implement batching, retry, timeout and recovery handling.
  • Troubleshoot tokenization failures/performance bottlenecks.
  • Measure impact of tokenization on end-to-end migration throughput.
  • Support Cyber/tokenization validation based on PNC-defined rules. Migration Controls & Reconciliation
  • Lead implementation of PNC Data Movement Controls, including: o checkpointing, o idempotency, o control totals, o restart/recovery, o retry, o audit logging, o exception handling.
  • Build reusable source-to-target reconciliation capabilities covering: o row counts, o schema validation, o partition completeness, o control totals, o aggregates, o hash/checksum where appropriate, o null/duplicate validation, o tokenization validation.
  • Isolate discrepancies at table/partition level.
  • Enable selective reprocessing rather than full-table reruns.
  • Produce technical evidence required for migration acceptance. Mandatory Skills Strong hands-on expertise in:
  • Python
  • PySpark
  • Apache Spark / Spark SQL
  • Hadoop / HDFS
  • Hive / HiveQL / Hive Metastore
  • SQL
  • Apache Iceberg
  • Data modelling
  • Source-to-target mapping
  • Parquet
  • Linux
  • Git
  • Large-volume ETL/data migration
  • Data validation/reconciliation
  • Spark performance tuning
  • Distributed-system troubleshooting. Data Modelling Strong experience with:
  • logical and physical data modelling,
  • source-to-target mapping,
  • schema/data-type mapping,
  • entity relationships,
  • keys and referential relationships,
  • dimensional/relational modelling concepts,
  • large-scale analytical data models. Technical Leadership Demonstrated experience:
  • leading Data Engineering teams,
  • technical estimation/planning,
  • detailed solution design,
  • code/design reviews,
  • mentoring engineers,
  • production troubleshooting,
  • root-cause analysis,
  • communicating technical issues with architects/customer teams. Preferred Skills
  • Apache Polaris/Nessie/Glue.
  • OpenShift/Kubernetes/Spark Operator.
  • S3-compatible object storage.
  • NetApp StorageGRID/ONTAP.
  • TaaS/Protegrity or equivalent tokenization technology.
  • Trino.
  • PostgreSQL.
  • Control-M/Airflow.
  • CI/CD for Spark/Python applications.
  • Banking/financial-services experience.
  • PB-scale data migration.
  • Bronze/Silver/Gold modelling and implementation.
  • HiveQL→Spark SQL modernization.

Before you apply

Company Jobs India lists this vacancy from the employer's official careers website. Apply only on that website. Genuine employers do not ask for a fee, deposit, OTP or bank details to give you an interview or an offer letter.

Skills

Software & It

Role details

Work mode
Onsite
Job functions
Analytics / Data
Application method
Continue to the employer’s application website

Listed from NTT DATA India’s official careers website · last checked 17 Nov 2026.

Employer

NTT DATA India

View the company profile and its current opportunities.

Company profile