Senior Data Engineer (Databricks & AI-Enabled Data Platforms) (4811)

Job description Come work for a large global financial and insurance products company! This is your chance !!

Start a successful career in a renowned company in the international market! Great opportunity!

Global insurance and asset management company seeks a responsible, organized, dynamic and team-oriented person.

Main responsibilities We are seeking a highly skilled Senior Data Engineer to design, build, and optimize modern data platforms thatenable advanced analytics, AI/ML, and business intelligence solutions. The ideal candidate will possess deepexpertise in Databricks, cloud-based data engineering, data architecture, and modern software engineeringpractices.

This role will provide technical leadership, establish engineering standards, and drive thedevelopment of scalable, secure, and reliable data solutions supporting enterprise-wide initiatives.

Key Responsibilities

Data Platform Engineering:

Design, develop, and maintain scalable data pipelines using Databricks, Spark, and cloud-native technologies;

Build and optimize batch and real-time data ingestion frameworks;

Develop reusable engineering patterns, frameworks, and accelerators for enterprise data solutions;

Implement data quality, observability, lineage, and governance capabilities across the data ecosystem;

Support large-scale data modernization and cloud migration initiatives.

Databricks Engineering:

Design and implement Delta Lake architectures and Medallion data models;

Build optimized ETL/ELT pipelines using Databricks notebooks, workflows, and Delta Live Tables;

Establish performance tuning, cost optimization, and workload management best practices;

Leverage Unity Catalog for data governance, security, and metadata management;

Implement CI/CD pipelines and Infrastructure-as-Code for Databricks deployments.

Cloud Data Engineering:

Design cloud-native data architectures across Azure and AWS environments;

Integrate enterprise data sources including SQL Server, Oracle, APIs, SaaS platforms, and streamingplatforms;

Build resilient and secure data services supporting high availability and disaster recovery requirements;

Partner with infrastructure and security teams to implement cloud security controls and compliancestandards.

AI and Advanced Analytics Enablement:

Build and optimize data products supporting AI, machine learning, and generative AI workloads;

Develop feature engineering pipelines and curated datasets for model training and inference;

Support vector databases, semantic search, RAG (Retrieval-Augmented Generation), and LLM-powered applications;

Collaborate with Data Scientists and AI Engineers to operationalize machine learning solutions;

Implement MLOps and DataOps practices to support scalable AI delivery.

Technical Leadership:

Serve as a technical lead for complex data engineering initiatives;

Define architecture standards, engineering best practices, and development guidelines;

Mentor junior and mid-level engineers;

Conduct architecture reviews and provide technical recommendations;

Drive adoption of automation, observability, and platform engineering principles.

Requirements and skills Required Qualifications

Education:

Bachelor's degree in Computer Science, Information Technology, Engineering, or related discipline;

Master's degree preferred.

Experience

8+ years of experience in data engineering, data warehousing, or data platform development;

5+ years of hands-on experience with Databricks and Apache Spark;

Experience designing and implementing enterprise-scale cloud data platforms;

Experience leading technical initiatives and mentoring engineering teams.

Technical Skills

Data Engineering:

Databricks;

Apache Spark (PySpark, Spark SQL);

Delta Lake;

Delta Live Tables;

Unity Catalog;

Data Warehousing and Data Modeling.

Programming:

Python;

SQL;

Scala (preferred);

PowerShell or scripting automation experience.

Cloud Platforms:

Microsoft Azure;

AWS;

Cloud-native storage and compute services.

Databases:

SQL Server;

Oracle;

PostgreSQL;

Snowflake (preferred).

Data Integration:

Azure Data Factory;

Databricks Workflows;

Kafka/Event Streaming;

REST APIs.

DevOps & Automation:

GitHub;

Azure DevOps;

CI/CD pipelines;

Terraform;

Infrastructure as Code.

Observability:

Monitoring and alerting solutions;

Data quality frameworks;

Performance optimization;

Logging and troubleshooting.

AI & Emerging Technologies:

Machine Learning data pipelines;

Feature Stores;

MLOps;

Generative AI concepts;

Vector databases;

Retrieval-Augmented Generation (RAG);

AI governance and responsible AI principles.

Preferred Qualifications

Databricks Certified Data Engineer Professional;

Azure Data Engineer Associate certification;

AWS Data Analytics certification;

Experience supporting regulated industries such as Insurance, Financial Services, or Healthcare;

Experience implementing enterprise data governance programs;

Experience integrating AI solutions into production environments.

Key Competencies

Strategic thinking and problem solving;

Technical leadership;

Architecture design;

Stakeholder management;

Communication and presentation skills;

Continuous improvement mindset;

lnnovation and AI adoption;

Risk and compliance awareness.

Success Measures

Delivery of scalable, resilient, and secure data solutions;

Adoption of enterprise engineering standards and best practices;

Improvement in platform performance, reliability, and cost efficiency;

Accelerated delivery of analytics and AI capabilities;

Reduction in operational support effort through automation and observability;

Successful mentoring and development of engineering talent.

Additional information Modelo de contratação:

PJ.

Forma de atuação:

Híbrido(3x por semana presencial no escritório de Pinheiros/SP).

#J-18808-Ljbffr