What you will be doing:
Are you a highly skilled Senior Data Platform Engineer with a passion for cutting-edge technology and a desire to make a real impact? Join our innovative team at ComplyAdvantage, where you'll design, manage, monitor, and optimise a diverse array of data systems running on AWS and GCP. Your expertise will ensure the seamless operation of SQL, NoSQL, Kafka, and Spark-based data layers, supporting our global client base.
Key Responsibilities:
- Innovative Design and Maintenance: Develop and sustain data layer orchestration systems, managing self-managed and PaaS data layer technologies, and ensure data consistency while designing databases.
- Performance and Availability: Oversee data layer components on AWS and GCP, guaranteeing top performance and reliability. Conduct regular performance tuning and troubleshoot issues.
- Backup and Cost Management: Configure and deploy robust backup systems for databases, Kafka, and Spark, ensuring data integrity and enabling disaster recovery. Implement cost management strategies to optimise resource utilisation.
- Documentation and Collaboration: Maintain thorough documentation for all Data Infrastructure engineering processes. Work closely with various teams to integrate data sources and support application requirements, adhering to best practices.
- Capacity and Project Planning: Monitor and manage data infrastructure capacity, planning for future growth and scalability. Collaborate with stakeholders to define and prioritise engineering projects, setting realistic timelines and expectations.
- Technical Support: Provide guidance and support to engineering team members on database and data layer architecture and technologies. Collaborate with Software Engineers to develop resilient data sharding strategies.
Our Tech Stack:
- Cloud-Based Infrastructure: Fully cloud-based with a Kubernetes-focused tech stack. Compute workloads run in Kubernetes clusters across multiple regions.
- Infrastructure Management: Heavy use of Terraform and Helm, adhering to GitOps paradigms for managing cloud infrastructure and Kubernetes applications.
- Core Technologies: Extensive use of Kafka, distributed PostgreSQL and Cassandra QL, Elasticsearch, and Databricks/Spark. Development of inter-cloud failover options to support multi-cloud plans.
- Wide Array of Applications: Teams build and release containerised applications for low latency APIs, machine learning models, and data processing pipelines.
About You:
- Experience: Significant expertise with AWS and/or GCP core services, data systems (Apache Kafka, Apache Spark, Elasticsearch, PostgreSQL, Cassandra), and Kubernetes. Experience codifying infrastructure in Terraform and/or Helm charts.
- Cost and Performance: Awareness of cost and performance considerations in designing data systems architectures. Proficiency with logging, monitoring, and tracing tools.
- CI/CD and Security: Familiarity with CI tooling, test pipelines, and asset generation (e.g., Docker images, Helm charts). Understanding of security considerations in data systems.
- Data Modelling: Experience with data modelling, sharding, and indexing strategies for large-scale databases.
Nice to Have:
- Experience running distributed databases within Kubernetes.
- Familiarity with distributed SQL and NoSQL databases such as Yugabyte, Cockroach, Spanner, HBase, or CouchDB.
Education:
- BSc/BA degree in computer science, engineering, or related discipline OR equivalent experience in required skills
What’s in it for you?
- Equity as we want you to have a part of what we are building
- Unlimited Time Off Policy- A work-life balance and focus on our well-being are critical to keeping us performing at our best
- We embrace a hybrid approach that requires employees to be in the office for two days a week. We strongly believe that this approach fosters collaboration and enables the building of meaningful relationships
- You will also get a new starter budget to kit out your home office
- Opportunity to work on innovative projects with smart-minded people keen to share their knowledge and continuously improve
- Annual learning budget (prorated based on start date) to drive your performance and career development
