Analytics / Data

Data Engineer

NTT DATA India

  • Bengaluru, Karnataka
  • Onsite

Posted 16 Nov 2026

About this role

"Position Overview We are seeking an experienced Databricks Data Engineer to design and enhance enterprise document-processing and data-extraction solutions.

The role will focus on ingesting structured and unstructured content, extracting business and policy information using LLM-based approaches, improving extraction accuracy, and making validated information available to downstream applications through an ontology-driven data layer. The engineer will join a small, hands-on team working on document intelligence solutions supporting brokerage and underwriting workflows.

A major focus will be redesigning existing pipelines to improve accuracy, performance, scalability, and maintainability while supporting a broader migration from Microsoft Fabric to Databricks.

Key Responsibilities

  • Design, build, and optimize Databricks data pipelines for document ingestion, classification, extraction, validation, and downstream consumption.
  • Process information from multiple document and communication formats, including: o PDF documents o Emails and attachments o Excel/XLSX files o Insurance applications o Statements of Values (SOVs) o Other structured and unstructured documents
  • Improve existing extraction pipelines where LLMs produce false positives, incorrect metadata, or inaccurate field values.
  • Develop validation and quality-control approaches that improve the reliability of extracted information.
  • Implement document classification and content-aware extraction strategies.
  • Handle source-specific extraction rules, such as extracting particular attributes from an email rather than from attached documents.
  • Design pipelines that transform extracted information into consistent, structured JSON outputs for downstream applications.
  • Contribute to the design and implementation of a custom ontology layer that represents key insurance business entities and relationships.
  • Map extracted information into the ontology so applications can consume standardized business concepts rather than raw document fields.
  • Address performance challenges associated with processing large or complex Excel/XLSX and SOV files.
  • Optimize pipeline scalability and concurrency for production workloads.
  • Work with PostgreSQL/Lakebase-based persistence for final structured outputs.
  • Collaborate closely with brokerage, underwriting, AI, and data engineering teams to connect extracted application data with downstream underwriting processes.

Required Qualifications

  • Strong hands-on experience with Databricks and production data engineering.
  • Strong proficiency in Python, PySpark, and SQL.
  • Experience building scalable ingestion and transformation pipelines on the Databricks platform.
  • Experience processing unstructured and semi-structured data, particularly PDFs, emails, Excel files, JSON, and business documents.
  • Experience integrating LLMs or AI models into data-processing pipelines.
  • Understanding of prompt-based and structured data extraction from documents.
  • Experience implementing data validation, reconciliation, or quality controls for AI-generated outputs.
  • Strong understanding of data modeling, metadata, schemas, entity relationships, and semantic/ontology concepts.
  • Experience designing APIs, JSON structures, or data products for downstream application consumption.
  • Experience troubleshooting performance, scalability, and concurrency issues in production data pipelines.
  • Experience working with cloud-based data architectures. Minimum 7+ years of experience in data engineering, software development, or production data pipeline development. Minimum 4+ years of hands-on experience with Databricks and scalable data-processing pipelines."

Before you apply

Company Jobs India lists this vacancy from the employer's official careers website. Apply only on that website. Genuine employers do not ask for a fee, deposit, OTP or bank details to give you an interview or an offer letter.

Skills

Software & ItExcel

Role details

Work mode
Onsite
Job functions
Analytics / Data
Application method
Continue to the employer’s application website

Listed from NTT DATA India’s official careers website · last checked 17 Nov 2026.

Employer

NTT DATA India

View the company profile and its current opportunities.

Company profile