Databricks Engineer

About the Role We are looking for a skilled

Databricks Engineer

todesign, develop, and maintain scalable data engineering solutions using the

DatabricksLakehouse Platform .

The ideal candidate will have strong hands‑on experiencewith

Databricks, Apache Spark, Python, SQL, Delta Lake, and cloud dataplatforms , with the ability to build reliable and high‑performance datapipelines.

Key Responsibilities

Design, develop, and maintain scalable data pipelines using

Databricks and Apache Spark .

Develop ETL/ELT pipelines using

PySpark, Python, and SQL .

Build and maintain

Delta Lake

tables and data processing workflows.

Work with Databricks

Lakehouse architecture

and related data engineering components.

Develop batch and, where required, near-real-time data processing solutions.

Ingest and transform data from databases, APIs, files, and other data sources.

Implement data cleansing, transformation, validation, and quality checks.

Optimise Spark jobs and Databricks workloads for performance and cost efficiency.

Work with cloud storage and data services across

Azure, AWS, or GCP .

Implement data security, access controls, and governance within the data platform.

Collaborate with Data Architects, Data Scientists, BI Developers, and business stakeholders.

Troubleshoot data pipeline failures and resolve performance and data-quality issues.

Develop and maintain technical documentation for data pipelines and solutions.

Participate in code reviews, testing, deployment, and production support.

Follow Agile development practices and contribute to continuous improvement.

Required Skills & Experience

3–5 years of experience in

Data Engineering .

Strong hands‑on experience with

Databricks .

Strong knowledge of:

Apache Spark / PySpark

Python

SQL

Delta Lake

ETL/ELT concepts

Experience developing and managing data pipelines.

Good understanding of data warehousing and data modelling concepts.

Experience working with cloud platforms, preferably

Microsoft Azure .

Experience with cloud storage such as

Azure Data Lake Storage (ADLS) , Amazon S3, or Google Cloud Storage.

Experience with relational and/or NoSQL databases.

Good understanding of data quality, validation, and governance.

Familiarity with Git and CI/CD practices.

Strong troubleshooting and analytical skills.

Good to Have

Databricks Certified Data Engineer Associate/Professional

certification.

Experience with

Azure Data Factory .

Experience with

Microsoft Fabric .

Knowledge of

Unity Catalog

and Databricks governance.

Experience with Databricks Workflows and job orchestration.

Experience with streaming technologies such as

Kafka

or Structured Streaming.

Experience with Power BI or other BI platforms.

Exposure to Machine Learning workflows on Databricks.

Experience with Terraform or Infrastructure as Code.

Knowledge of DevOps and CI/CD pipelines.

Candidate Profile The ideal candidate should be a hands‑on

Databricks/Data Engineer

capable of independently developing data pipelines, troubleshooting production issues, optimising Spark workloads, and collaborating with technical and business teams.

#J-18808-Ljbffr