Cloud Data Platform Engineer

Koud is looking for a Cloud Data Platform Engineer to work with one of our clients.The position focuses on Databricks platform engineering, cloud infrastructure, automation, security, governance, and operational reliability across Azure and/or AWS environments. The professional will be responsible for designing, implementing, administering, and optimizing scalable and secure data platforms, supporting data engineering, analytics, machine learning, and artificial intelligence initiatives.

Key Responsibilities:

Design, implement, and administer Databricks workspaces, clusters, and platform components across Azure and/or AWS environments;

Build and maintain scalable, secure, highly available, and reliable Databricks environments;

Define platform standards, architectures, governance frameworks, and operational best practices;

Configure and administer Unity Catalog, Delta Lake, workspace governance, and identity and access controls;

Implement platform lifecycle management, upgrades, capacity planning, and optimization;

Provision and manage cloud infrastructure using Infrastructure as Code tools, particularly Terraform;

Automate platform deployments, configuration management, and operational processes;

Design secure networks, private connectivity, and integration patterns across cloud environments;

Implement backup, disaster recovery, high availability, and business continuity solutions;

Monitor platform health, availability, performance, and capacity;

Develop observability solutions for infrastructure and data workloads, including monitoring, logging, and auditing;

Identify and troubleshoot issues related to the platform, networking, infrastructure, and workloads;

Conduct root cause analyses and implement corrective actions;

Implement enterprise security controls, RBAC, least-privilege access models, secrets management, encryption, and key management;

Support compliance requirements and security audits;

Develop operational runbooks, support processes, and reusable automation standards;

Work closely with Data Engineering, Data Science, AI Engineering, and Cloud Infrastructure teams;

Provide technical guidance on platform architecture, operational excellence, and cloud engineering best practices;

Support the technical development of other professionals and contribute to the continuous evolution of the platform.

Requirements:

Bachelor's degree in Computer Science, Information Technology, Engineering, or related fields, or equivalent practical experience;

Strong experience in Cloud Engineering, Platform Engineering, Infrastructure Engineering, or related areas;

Strong hands-on experience implementing and administering Databricks environments;

Strong experience with enterprise cloud platforms, particularly Azure and/or AWS;

Experience designing secure, scalable, and highly available cloud solutions;

Solid knowledge of Databricks administration, including workspace management, cluster configuration and optimization, workflows, and job scheduling;

Strong hands-on experience with Unity Catalog and Delta Lake;

Strong experience with Infrastructure as Code, particularly Terraform;

Experience with CI/CD pipelines using Azure DevOps and/or GitHub Actions;

Knowledge of Python, PowerShell, and/or Bash for automation;

Knowledge of cloud networking, including VNET/VPC design and Private Endpoints;

Experience with RBAC, identity federation, secrets management, encryption, and key management;

Experience with monitoring and observability tools such as Azure Monitor, CloudWatch, and Log Analytics;

Strong problem-solving skills and the ability to work independently in complex enterprise environments;

Good communication skills and the ability to collaborate with different technical teams and stakeholders;

Fluent English.

Preferred:

Databricks Certified Platform Administrator certification;

Databricks Certified Data Engineer Professional certification;

Azure Administrator Associate or Azure Solutions Architect certification;

AWS Solutions Architect certification;

Experience implementing enterprise AI and Machine Learning platforms;

Experience in regulated environments, particularly in the Insurance or Financial Services sectors;

Experience with AI and Machine Learning capabilities within the Databricks ecosystem.

Additional Information:

Hybrid position, with 3 days per week working on-site in Pinheiros, São Paulo.

#J-18808-Ljbffr