Overview:
Join our forward-thinking data team to drive the development of cutting-edge data solutions for Generative AI applications. We leverage data to empower our customers, fuel our own innovation, and support groundbreaking research. As a Senior Data Engineer, you will take a leadership role in designing, optimizing, and scaling our data infrastructure, ensuring that our solutions are not only robust but also capable of meeting the challenges of tomorrow.
Your Responsibilities:
Lead the design and implementation of scalable data architectures to handle large-scale datasets (terabytes of text) with a focus on storage, versioning, and documentation best practices.
Architect, develop, and oversee the maintenance of web services that enable efficient consumption of harvested data.
Partner with researchers, software engineers, and leadership to continuously refine data collection methodologies and identify new data opportunities.
Strategize and prepare large datasets for diverse Machine Learning use cases, with an emphasis on Generative AI.
Build, optimize, and automate advanced preprocessing pipelines tailored to specific applications, ensuring high performance and reliability.
Ensure data services are robust, scalable, and meet the needs of cross-functional teams developing new products on top of our data infrastructure.
Mentor and guide junior data engineers, fostering a culture of knowledge sharing and continuous improvement.
Your Profile:
You have 7+ years of experience as a Data Engineer, with a proven track record of architecting large-scale data systems.
You are an expert in Python and proficient in at least one other programming language (e.g., Java, Scala, or Golang).
You possess a deep understanding of distributed systems, with a demonstrated ability to design and manage efficient data pipelines in both cloud and on-prem environments.
You have a strong software engineering background, with a focus on writing clean, maintainable, and well-documented code.
You excel at data wrangling, including advanced techniques for extracting, transforming, cleaning, and standardizing data from multiple sources.
You bring expertise in Generative AI use cases and understand the pivotal role of data in developing cutting-edge AI solutions.
You have experience in driving projects, influencing stakeholders, and aligning data strategies with broader business goals.
Nice to Have:
Experience working in multi cloud environments (e.g., GCP, Azure, AWS) as well as on-premise data solutions.
Background in Machine Learning or Data Science, with a particular focus on applying data engineering principles to AI research.
Proficiency in Golang and an interest in adopting new technologies.
Familiarity with Kubernetes for container orchestration and managing scalable deployments.
