Sheet 01 — Profile

Deepanshu Singh
Data Engineer.

AWS · Azure · Databricks / 6+ yrs experience / Delhi, India

I design and tune ETL/ELT pipelines and Medallion-architecture (Bronze/Silver/Gold) data platforms that move raw, messy data from source systems to governed, trustworthy tables — for BFSI, healthcare, logistics, and manufacturing clients.

Medallion architecture pipeline: sources into Bronze, Silver, Gold layers, out to reporting S3 / ADLS Gen2 Oracle / SAP / ERP SFTP / APIs / DB2 BRONZE raw · full & CDC SILVER cleansed · CDF GOLD curated · views Redshift / Athena Power BI Reports DRAWN BY: D. SINGH PROJECT: DATA PLATFORM SCALE: 1:1 REV: 6 STATUS: IN PRODUCTION SHEET 01/01
6+
YEARS IN DATA
ENGINEERING
5+ TB
PROCESSED DAILY
(LIC HFL PLATFORM)
99.8%
DATA ACCURACY
(UP FROM 92%)
6x
FASTER INGESTION
(90 → 15 MIN)
₹3L Cr
LOAN PORTFOLIO
SUPPORTED
8h→2.5h
BATCH RUNTIME
REDUCED
Sheet 02 — Build Log

Experience

Six client engagements across BFSI, healthcare, logistics, manufacturing, and relocation services — each one a pipeline shipped to production.

R Systems International Limited
Data Engineer
Apr 2024 — Jul 2026
GHG — Carbon Emission CalculatorClient: Purolator Inc. · Shipping & Logistics
AWS GluePySparkStep FunctionsLambdaS3AthenaRDS PostgreSQLTerraformEventBridgeCloudWatch
  • Built Glue ETL jobs to ingest raw data from S3 input to operating buckets in Parquet, applied carbon-calculator transformations, and wrote final outputs to S3 in CSV.
  • Loaded calculator results into RDS PostgreSQL via Lambda-invoked stored procedures; partitioned the operating bucket to improve performance and cut scan costs.
  • Orchestrated the end-to-end pipeline using Step Functions, EventBridge, and CloudWatch monitoring.
Data Engineering & AnalyticsClient: LIC Housing Finance Limited · BFSI
AWS GluePySparkStep FunctionsLambdaS3AthenaDelta LakeRDS PostgreSQLAirflowEventBridgeCloudWatch
  • Managed a ₹3 lakh crore loan portfolio through massive, scalable, and robust data pipelines — architected an enterprise-scale platform processing 5+ TB of housing loan, customer, repayment, and financial data daily for near real-time reporting.
  • Designed and built ETL/ELT pipelines using Python, PySpark, SQL, and Airflow, cutting data ingestion time by 65% and improving data availability for business users.
  • Built a centralized Data Warehouse and Data Lake consolidating Loan Origination Systems, Core Lending Systems, CRM, Collections, and Finance applications.
  • Developed automated pipelines for loan disbursement, EMI repayment, foreclosure, delinquency, and collections analytics.
  • Optimized Spark jobs and SQL workloads, cutting batch runtime from 8 hours to under 2.5 hours for faster regulatory and management reporting.
  • Implemented an end-to-end data quality and reconciliation framework, raising data accuracy from 92% to 99.8%.
SDP & CDP — Supplier & Customer Data PlatformsClient: Dover Corporation · Manufacturing
Azure Data FactoryAzure SQLBlob StoragePySparkDatabricksAzure DevOpsPower BI
  • Ingested data from ERP, SAP, SFTP, APIs, Oracle, IBM DB2, and Blob Storage into an Azure SQL landing schema using ADF, in full and incremental modes; archived daily to Parquet for backtracking.
  • Resolved 35% of duplicates, nulls, and datatype issues using Azure Data Flows and stored procedures; implemented upsert logic in the staging schema.
  • Designed dim/fact tables and views consumed by analytics teams for Power BI semantic models; configured Logic Apps alerts on pipeline failure.
Aires Data Engineering & AnalyticsClient: Aires · Relocation Services
AWS GluePySparkS3Delta LakeAmazon RedshiftRedshift SpectrumGlue CrawlerGlue Workflows
  • Implemented Medallion architecture; ingested 715 Oracle tables into Bronze Delta Lake with parameterized full/incremental loads.
  • Cut ingestion time from 90 to 15 minutes for 4 large tables by tuning JDBC reads (partitionColumn, numPartitions, lowerBound, upperBound).
  • Promoted 173 curated tables to Silver with Change Data Feed; built 29 Redshift Spectrum materialized views in Gold for fast retrieval.
  • Applied Z-Ordering on high-cardinality joins and partition pruning; orchestrated pipelines with Glue Workflows and Crawlers.
HEDIS Certification & Storyboard MigrationClient: EXL Services Pvt Ltd · Healthcare
Azure DatabricksPySparkAzure Data FactoryADLS Gen2Azure SQL Warehouse
  • Built ADF pipelines moving data from SQL Server to ADLS Gen2; updated scripts for 2024 HEDIS measurement-year requirements and automated NCQA sample-deck ingestion.
  • Migrated an on-prem distribution to Azure Cloud and SQL Warehouse; tuned SQL queries to improve retrieval performance by 20%, contributing to a 30% portfolio uplift for institutions.
  • Generated and uploaded certification outputs to the NCQA portal; coordinated with NCQA to resolve dataset discrepancies.
Twinline Business Solutions Pvt Ltd
Senior Database Developer
Aug 2021 — Mar 2024

Lending Club — ETL Pipeline

Clients: Arman Financial Services, ShareIndia, Chaitanya India, Cashtree, Prayas, Midland, Namra, Prayatna, Saksham Gram — NBFC / Microfinance

AWS GlueS3DatabricksPySparkAmazon RedshiftDelta Lake
  • ›Built finance-domain pipelines on AWS using PySpark; optimized lending workflows for microfinance institutions, cutting processing time by 20%.
  • ›Collaborated with business analysts to implement RBI-mandated CDF, CIC, and TUDF reports for regulatory and credit bureau submissions on a daily/monthly basis.
  • ›Led data migration for Cashtree, Saksham Gram, Namra Finance, Prayas Finance, VFS, and Arman Financial Services, ensuring data integrity and compliance.
  • ›Resolved 75% of duplicates, nulls, and datatype issues during cleansing for downstream ML and analytics teams.
  • ›Implemented a Delta Lakehouse architecture supporting updates, deletes, and historical versioning — improving data accuracy by 25% across IL and JLG lending modules.
Digidex Labs Pvt Ltd
Full Stack Developer
Nov 2019 — Apr 2021
  • ›Developed and maintained full-stack web applications across front-end and back-end components.
  • ›Designed relational database schemas, wrote optimized queries, and supported application data needs.
  • ›Collaborated with cross-functional teams to deliver features on schedule — the foundation that led into data engineering.
Sheet 03 — Component Legend

Technical Skills

The stack behind every pipeline in the build log above.

Cloud Platforms

AWS GlueS3LambdaStep FunctionsEventBridgeCloudWatchAthenaRedshiftRDSAzure Data FactoryDatabricksAzure SQLADLS Gen2Blob StorageLogic AppsAzure DevOps

Big Data & ETL

Apache SparkPySparkDatabricksDelta LakeMedallion ArchitectureAWS Glue WorkflowsAzure Data Factory

Data Warehousing

Amazon RedshiftRedshift SpectrumAzure SQL WarehouseSnowflake (POC)

Programming & Query

PythonPySparkSQLT-SQLPL/SQL

Databases

MSSQLOraclePostgreSQLIBM DB2

BI & Tools

Power BIGitHubAzure DevOpsTerraform

Core Concepts

Data Modeling (Dim/Fact)Performance TuningPartitioningZ-OrderingCDCQuery OptimizationData Quality
Sheet 04 — Approvals

Certifications

Verified on the Databricks credential wallet — click any card to view the digital badge.

Databricks Certified Data Engineer Associate badge

Databricks Certified Data Engineer Associate

Valid Feb 2025 – Feb 2027
Databricks Accredited — Databricks Fundamentals badge

Databricks Accredited — Databricks Fundamentals

Valid Jan 2025 – Jan 2026
Databricks Accredited AWS Databricks Platform Architect badge

Databricks Accredited — AWS Databricks Platform Architect

Valid Feb 2025 – Feb 2027
Snowflake — Hands-On Essentials: Data Warehousing Workshop badge

Snowflake — Hands-On Essentials: Data Warehousing Workshop

Valid from Apr 2025
Snowflake — Hands-On Essentials: Collaboration, Marketplace & Cost Estimation Workshop badge

Snowflake — Hands-On Essentials: Collaboration, Marketplace & Cost Estimation Workshop

Valid from Apr 2025
Snowflake — Hands-On Essentials: Data Application Builders Workshop badge

Snowflake — Hands-On Essentials: Data Application Builders Workshop

Valid from Apr 2025
Snowflake — Hands-On Essentials: Data Lake Workshop badge

Snowflake — Hands-On Essentials: Data Lake Workshop

Valid from Apr 2025
Snowflake — Hands-On Essentials: Data Engineering Workshop badge

Snowflake — Hands-On Essentials: Data Engineering Workshop

Valid from May 2025
DBT Labs — DBT Fundamentals badge

DBT Labs — DBT Fundamentals

Valid from May 2025
Databricks Accredited - Azure Databricks Platform Architect badge

Databricks Accredited - Azure Databricks Platform Architect

Valid from May 2025
Databricks - Advantages of Azure Databricks & Microsoft Fabric badge

Partner Training - Advantages of Azure Databricks & Microsoft Fabric

Valid Sep 2025 – Sep 2027
Databricks - Cloud Native SPARK Migration (Verified) badge

Partner Training - Cloud Native SPARK Migration (Verified)

Valid Sep 2025 – Sep 2027
Databricks Cloud Native SPARK Migration Pre-sales badge

Partner Training - Cloud Native SPARK Migration Pre-sales

Valid Sep 2025 – Sep 2027
Databricks Advantages of the Lakehouse badge

Partner Training - Advantages of the Lakehouse

Valid Sep 2025 – Sep 2027
Databricks Accredited - AI Agent Fundamentals badge

Databricks Accredited - AI Agent Fundamentals

Valid from Sep 2025
Databricks Accredited - Get Started with AI Agents on Databricks badge

Databricks Accredited - Get Started with AI Agents on Databricks

Valid from Sep 2025
Databricks Accredited - Generative AI Fundamentals badge

Databricks Accredited - Generative AI Fundamentals

Valid from Sep 2025
Databricks Financial Services Industry Specialization for Gen AI & LLM badge

Partner Training - Financial Services Industry Specialization for Gen AI & LLM

Valid Oct 2025 – Oct 2027
Databricks Advantages of Gen AI with Data Intelligence badge

Partner Training - Advantages of Gen AI with Data Intelligence

Valid Oct 2025 – Oct 2027
Databricks Gen AI & LLM on Databricks badge

Partner Training - Gen AI & LLM on Databricks

Valid Oct 2025 – Oct 2027
Databricks Gen AI & LLM on Databricks (Verified)
           badge

Partner Training - Gen AI & LLM on Databricks (Verified)

Valid Oct 2025 – Oct 2027
Databricks Sales Ready badge

Partner Training - Sales Ready

Valid Oct 2025 – Oct 2027
Databricks (Industry) HLS Knowledge badge

Partner Training - (Industry) HLS Knowledge Badge

Valid May 2026 – May 2028
Databricks Accredited - Data Ingestion with Lakeflow Connect badge

Databricks Accredited - Data Ingestion with Lakeflow Connect

Valid from May 2026
Databricks Accredited - Deploy Workloads with Lakeflow Jobs badge

Databricks Accredited - Deploy Workloads with Lakeflow Jobs

Valid from May 2026
Databricks Accredited - Build Data Pipelines with Lakeflow Spark Declarative Pipelines badge

Databricks Accredited - Build Data Pipelines with Lakeflow Spark Declarative Pipelines

Valid from May 2026
Databricks Accredited - Automated Deployment with Declarative Automation Bundles badge

Databricks Accredited - Automated Deployment with Declarative Automation Bundles

Valid from Jun 2026
Databricks Azure Databricks Foundations badge

Partner Training - Azure Databricks Foundations

Valid from Jun 2026
Databricks Healthcare & Life Sciences Industry Specialization for Gen AI & LLM badge

Partner Training - Healthcare & Life Sciences Industry Specialization for Gen AI & LLM

Valid Jun 2026 – Jun 2028
Databricks ISV Fundamentals badge

Partner Training - ISV Fundamentals

Valid Jun 2026 – Jun 2028
Databricks Lakebridge for SQL Source Migration (Verified) badge

Partner Training - Lakebridge for SQL Source Migration (Verified)

Valid Jun 2026 – Jun 2027
Confluent Apache Kafka Fundamentals Accreditation badge

Confluent Apache Kafka Fundamentals Accreditation

Valid Aug 2026 – Aug 2028
Confluent Certified Data Streaming Engineer Foundations badge

Confluent Certified Data Streaming Engineer Foundations

Valid Aug 2026 – Aug 2028
Databricks certified Data Engineer Professional badge

Databricks certified Data Engineer Professional

Valid Aug 2026 – Aug 2028
Sheet 05 — Endorsements

LinkedIn Recommendations

What managers and colleagues have written on LinkedIn — swipe or use the arrows to read more.

“

I've been acquainted with Deepanshu for a solid 12 years. Throughout this time, I've encountered few professionals with his remarkable knack for continuous learning and self-improvement. Right from the beginning, I recognized his diligence and sincerity. His capacity to remain composed in challenging situations truly sets him apart. Without a doubt, he would be an invaluable asset to any organization, and I take great pleasure in recommending him as a true professional.

“

Deepanshu is one of most intelligent, diligent and hard working resource. he believes in learning, unlearning and relearning. Whenever a problem statement is given to him , he does extreme research and development on that.

“

I highly recommend Deepanshu as a skilled and knowledgeable database admin. He has a deep understanding of database architecture and is able to manage and optimize large, complex databases with ease. His attention to detail and problem-solving skills are remarkable, and they consistently deliver high-quality work.

“

Its great to have a co worker like Deepanshu . He has very good knowledge of database and a keen learner . The best thing about him is that he is very dedicated towards his work .

“

It was great working with Deepanshu. He is a talented and keen learner. He is also very dedicated and determined in whatever he does.

“

Deepanshu's work ethics are really good. From understanding requirements to development, he knows how to manage. Under pressure he shines bright.
Working with you was a great experience. Keep Shining!

“

I have known Deepanshu for 12 years now. Apparently, I haven't come across many professionals such as Deepanshu who have this innate ability to learn and upskill themself every day with great passion. From our early days, I sensed that he was a diligent and sincere chap. His ability to stay calm and composed in the toughest of periods made him stand out. I have no doubt that he will be a potential human resource for any organization he works for and it's my pleasure to recommend him as a professional.

Sheet 06 — Foundation

Education

Masters in Computer Applications (MCA)

Guru Gobind Singh Indraprastha University (GGSIPU)
2020

Bachelors of Computer Applications (BCA)

Guru Gobind Singh Indraprastha University (GGSIPU)
2017