Cloud Data Architect - Python, SQL, GCP
October 7
Description
• Schedule: Monday – Friday (09:00 AM - 06:00 PM AEST) • As a Cloud Data Architect, you will work closely with data engineers and analytics teams to ensure that the infrastructure supports complex data workflows, pipelines, and analytics needs, optimizing for performance, security, and cost-efficiency. • Design and Architect Cloud Infrastructure: Lead the design of GCP-based infrastructure to support data pipelines, machine learning, and analytics workloads, ensuring scalability and reliability. • Install and Manage Apache Airflow on Kubernetes: Set up and maintain Airflow for orchestrating data workflows in a Kubernetes environment, ensuring seamless scheduling and execution of DAGs. • Provision and Manage Databricks Environments: Set up Databricks clusters and integrations with GCP services, ensuring efficient use of resources for data processing and analytics. • Implement Infrastructure as Code (IaC): Use tools like Terraform or Cloud Deployment Manager to automate the provisioning and management of cloud infrastructure. • Optimize Cloud Data Services: Utilize GCP’s data products, such as BigQuery, Dataflow, Pub/Sub, and Cloud Storage, to build scalable data architectures. • Collaborate with Data Engineering Teams: Work closely with data engineers to design efficient data pipelines, optimize data workflows, and support data integration and processing at scale. • Ensure Security and Compliance: Implement best practices for securing data infrastructure, including IAM policies, VPC configurations, and data encryption. • Performance Monitoring and Optimization: Monitor and optimize the performance of cloud infrastructure, ensuring that it meets the needs of data pipelines and analytical workloads. • Cost Management: Implement cost-effective solutions and continuously optimize cloud infrastructure to reduce operational costs without compromising performance. • Documentation and Best Practices: Maintain detailed documentation of the cloud architecture and establish best practices for data infrastructure management.
Requirements
• Experience with machine learning infrastructure and deploying ML models on cloud platforms. • Minimum of 3 years of experience in a similar role. 3+ years of experience in Python programming & SQL. • Experience with CI/CD pipelines for infrastructure and data workflows. • Strong understanding of data modelling, design, ETL processes, and data warehousing concepts. • Experience with data visualization tools (e.g., Tableau, Power BI, Looker Studio) and proficiency in dashboard development and reporting. • Certification in Google Cloud (e.g., Professional Cloud Architect, Professional Data Engineer). Databricks certification will be advantageous as well. • Familiarity with machine learning and AI concepts. • Stakeholder management. • Knowledge of best practices in data security and compliance. • Docker. • Kubernetes.
Apply Now