Our client is seeking an experienced, hands-on
Senior Data Engineer
to help build and scale the enterprise data foundation supporting Analytics, AI, and self-service decision-making across the company.
This role requires more than just moving data between systems. The successful candidate will act as a data product architect, transforming complex, messy operational data into trusted, reusable, business-ready data products. These products will power enterprise analytics, Tableau reporting,
Databricks Genie / Genie Code , machine learning, and emerging GenAI use cases across Manufacturing, Supply Chain, Sales, Finance, and HR.
You will play a pivotal role in shifting the organization away from one-off, siloed pipelines toward a highly governed, centralized single-source-of-truth platform built entirely on
Databricks and AWS .
Key Responsibilities: 1. Data Engineering & Lakehouse Development
Design, develop, test, and maintain scalable ETL/ELT pipelines using
Databricks, Spark, SQL, Python, and AWS .
Build and maintain lakehouse solutions leveraging
Delta Lake
and
Medallion Architecture (Bronze, Silver, Gold) .
Integrate structured and semi-structured data from enterprise applications, manufacturing systems, databases, APIs, and cloud operational sources.
Optimize Databricks workloads, SQL queries, Delta tables, compute utilization, and end-to-end pipeline performance.
Implement orchestration, monitoring, error handling, data reconciliation, and automated recovery for production data pipelines.
Design scalable enterprise data models (dimensional modeling, star schemas, fact/dimension structures) supporting cross-functional analytics and standardized KPIs.
Transform raw operational data into curated, business-ready data products that can be reused across dashboards, analytics, AI, and ML applications.
Collaborate closely with business and analytics teams to translate complex business definitions into technically robust data models.
3. Analytics AI & Self-Service Enablement
Build AI-ready and Genie-ready data products
that allow business users to securely analyze enterprise data using Databricks Genie and Genie Code.
Structure data, metadata, relationships, business definitions, and semantic/contextual layers to maximize the accuracy of conversational AI analytics.
Enable engineers, analysts, and business users to perform governed self-service analysis without creating uncontrolled copies of data.
Partner with data science teams to prepare feature-ready datasets for predictive analytics, machine learning, and MLOps workloads.
4. Data Governance, Quality & DataOps
Implement data governance, cataloging, lineage, metadata management, and role-based access controls using
Databricks Unity Catalog .
Build automated data-quality checks for completeness, accuracy, consistency, freshness, and cross-system reconciliation.
Apply Git-based development, code reviews, testing, version control, and automated
CI/CD processes
for Databricks assets.
Focus heavily on developing reusable engineering frameworks and templates rather than building hard-coded, one-off pipelines.
#J-18808-Ljbffr
Lead/Senior Data Engineer employer: Celebal Technologies
Join a forward-thinking company that values innovation and collaboration, where as a Databricks Architect, you'll have the opportunity to work with cutting-edge technologies in a dynamic environment. Our commitment to employee growth is reflected in our continuous learning programs and supportive work culture, ensuring you thrive both personally and professionally. Located in a vibrant tech hub, we offer competitive benefits and a unique chance to shape the future of data architecture while enjoying a balanced work-life experience.
