I've been acquainted with Deepanshu for a solid 12 years. Throughout this time, I've encountered few professionals with his remarkable knack for continuous learning and self-improvement. Right from the beginning, I recognized his diligence and sincerity. His capacity to remain composed in challenging situations truly sets him apart. Without a doubt, he would be an invaluable asset to any organization, and I take great pleasure in recommending him as a true professional.
Deepanshu Singh
Data Engineer.
I design and tune ETL/ELT pipelines and Medallion-architecture (Bronze/Silver/Gold) data platforms that move raw, messy data from source systems to governed, trustworthy tables — for BFSI, healthcare, logistics, and manufacturing clients.
ENGINEERING
(LIC HFL PLATFORM)
(UP FROM 92%)
(90 → 15 MIN)
SUPPORTED
REDUCED
Experience
Six client engagements across BFSI, healthcare, logistics, manufacturing, and relocation services — each one a pipeline shipped to production.
GHG — Carbon Emission CalculatorClient: Purolator Inc. · Shipping & Logistics
- Built Glue ETL jobs to ingest raw data from S3 input to operating buckets in Parquet, applied carbon-calculator transformations, and wrote final outputs to S3 in CSV.
- Loaded calculator results into RDS PostgreSQL via Lambda-invoked stored procedures; partitioned the operating bucket to improve performance and cut scan costs.
- Orchestrated the end-to-end pipeline using Step Functions, EventBridge, and CloudWatch monitoring.
Data Engineering & AnalyticsClient: LIC Housing Finance Limited · BFSI
- Managed a ₹3 lakh crore loan portfolio through massive, scalable, and robust data pipelines — architected an enterprise-scale platform processing 5+ TB of housing loan, customer, repayment, and financial data daily for near real-time reporting.
- Designed and built ETL/ELT pipelines using Python, PySpark, SQL, and Airflow, cutting data ingestion time by 65% and improving data availability for business users.
- Built a centralized Data Warehouse and Data Lake consolidating Loan Origination Systems, Core Lending Systems, CRM, Collections, and Finance applications.
- Developed automated pipelines for loan disbursement, EMI repayment, foreclosure, delinquency, and collections analytics.
- Optimized Spark jobs and SQL workloads, cutting batch runtime from 8 hours to under 2.5 hours for faster regulatory and management reporting.
- Implemented an end-to-end data quality and reconciliation framework, raising data accuracy from 92% to 99.8%.
SDP & CDP — Supplier & Customer Data PlatformsClient: Dover Corporation · Manufacturing
- Ingested data from ERP, SAP, SFTP, APIs, Oracle, IBM DB2, and Blob Storage into an Azure SQL landing schema using ADF, in full and incremental modes; archived daily to Parquet for backtracking.
- Resolved 35% of duplicates, nulls, and datatype issues using Azure Data Flows and stored procedures; implemented upsert logic in the staging schema.
- Designed dim/fact tables and views consumed by analytics teams for Power BI semantic models; configured Logic Apps alerts on pipeline failure.
Aires Data Engineering & AnalyticsClient: Aires · Relocation Services
- Implemented Medallion architecture; ingested 715 Oracle tables into Bronze Delta Lake with parameterized full/incremental loads.
- Cut ingestion time from 90 to 15 minutes for 4 large tables by tuning JDBC reads (partitionColumn, numPartitions, lowerBound, upperBound).
- Promoted 173 curated tables to Silver with Change Data Feed; built 29 Redshift Spectrum materialized views in Gold for fast retrieval.
- Applied Z-Ordering on high-cardinality joins and partition pruning; orchestrated pipelines with Glue Workflows and Crawlers.
HEDIS Certification & Storyboard MigrationClient: EXL Services Pvt Ltd · Healthcare
- Built ADF pipelines moving data from SQL Server to ADLS Gen2; updated scripts for 2024 HEDIS measurement-year requirements and automated NCQA sample-deck ingestion.
- Migrated an on-prem distribution to Azure Cloud and SQL Warehouse; tuned SQL queries to improve retrieval performance by 20%, contributing to a 30% portfolio uplift for institutions.
- Generated and uploaded certification outputs to the NCQA portal; coordinated with NCQA to resolve dataset discrepancies.
Lending Club — ETL Pipeline
Clients: Arman Financial Services, ShareIndia, Chaitanya India, Cashtree, Prayas, Midland, Namra, Prayatna, Saksham Gram — NBFC / Microfinance
- ›Built finance-domain pipelines on AWS using PySpark; optimized lending workflows for microfinance institutions, cutting processing time by 20%.
- ›Collaborated with business analysts to implement RBI-mandated CDF, CIC, and TUDF reports for regulatory and credit bureau submissions on a daily/monthly basis.
- ›Led data migration for Cashtree, Saksham Gram, Namra Finance, Prayas Finance, VFS, and Arman Financial Services, ensuring data integrity and compliance.
- ›Resolved 75% of duplicates, nulls, and datatype issues during cleansing for downstream ML and analytics teams.
- ›Implemented a Delta Lakehouse architecture supporting updates, deletes, and historical versioning — improving data accuracy by 25% across IL and JLG lending modules.
- ›Developed and maintained full-stack web applications across front-end and back-end components.
- ›Designed relational database schemas, wrote optimized queries, and supported application data needs.
- ›Collaborated with cross-functional teams to deliver features on schedule — the foundation that led into data engineering.
Technical Skills
The stack behind every pipeline in the build log above.
Cloud Platforms
Big Data & ETL
Data Warehousing
Programming & Query
Databases
BI & Tools
Core Concepts
Certifications
Verified on the Databricks credential wallet — click any card to view the digital badge.
Databricks Certified Data Engineer Associate
Databricks Accredited — Databricks Fundamentals
Databricks Accredited — AWS Databricks Platform Architect
Snowflake — Hands-On Essentials: Data Warehousing Workshop
Snowflake — Hands-On Essentials: Collaboration, Marketplace & Cost Estimation Workshop
Snowflake — Hands-On Essentials: Data Application Builders Workshop
Snowflake — Hands-On Essentials: Data Lake Workshop
Snowflake — Hands-On Essentials: Data Engineering Workshop
DBT Labs — DBT Fundamentals
Databricks Accredited - Azure Databricks Platform Architect
Partner Training - Advantages of Azure Databricks & Microsoft Fabric
Partner Training - Cloud Native SPARK Migration (Verified)
Partner Training - Cloud Native SPARK Migration Pre-sales
Partner Training - Advantages of the Lakehouse
Databricks Accredited - AI Agent Fundamentals
Databricks Accredited - Get Started with AI Agents on Databricks
Databricks Accredited - Generative AI Fundamentals
Partner Training - Financial Services Industry Specialization for Gen AI & LLM
Partner Training - Advantages of Gen AI with Data Intelligence
Partner Training - Gen AI & LLM on Databricks
Partner Training - Gen AI & LLM on Databricks (Verified)
Partner Training - Sales Ready
Partner Training - (Industry) HLS Knowledge Badge
Databricks Accredited - Data Ingestion with Lakeflow Connect
Databricks Accredited - Deploy Workloads with Lakeflow Jobs
Databricks Accredited - Build Data Pipelines with Lakeflow Spark Declarative Pipelines
Databricks Accredited - Automated Deployment with Declarative Automation Bundles
Partner Training - Azure Databricks Foundations
Partner Training - Healthcare & Life Sciences Industry Specialization for Gen AI & LLM
Partner Training - ISV Fundamentals
Partner Training - Lakebridge for SQL Source Migration (Verified)
Confluent Apache Kafka Fundamentals Accreditation
Confluent Certified Data Streaming Engineer Foundations
Databricks certified Data Engineer Professional
LinkedIn Recommendations
What managers and colleagues have written on LinkedIn — swipe or use the arrows to read more.
Deepanshu is one of most intelligent, diligent and hard working resource. he believes in learning, unlearning and relearning. Whenever a problem statement is given to him , he does extreme research and development on that.
I highly recommend Deepanshu as a skilled and knowledgeable database admin. He has a deep understanding of database architecture and is able to manage and optimize large, complex databases with ease. His attention to detail and problem-solving skills are remarkable, and they consistently deliver high-quality work.
Its great to have a co worker like Deepanshu . He has very good knowledge of database and a keen learner . The best thing about him is that he is very dedicated towards his work .
It was great working with Deepanshu. He is a talented and keen learner. He is also very dedicated and determined in whatever he does.
Deepanshu's work ethics are really good. From understanding requirements to development, he knows how to manage. Under pressure he shines bright.
Working with you was a great experience. Keep Shining!
I have known Deepanshu for 12 years now. Apparently, I haven't come across many professionals such as Deepanshu who have this innate ability to learn and upskill themself every day with great passion. From our early days, I sensed that he was a diligent and sincere chap. His ability to stay calm and composed in the toughest of periods made him stand out. I have no doubt that he will be a potential human resource for any organization he works for and it's my pleasure to recommend him as a professional.