Staff Software Engineer(MLOps)

Toast creates technology to help restaurants and local businesses succeed in a digital world, helping business owners operate, increase sales, engage customers, and keep employees happy. Now, more than ever, the Toast team is committed to our customers. We’re taking steps to help restaurants navigate these unprecedented times with technology, resources, and community. Our focus is on building the restaurant platform that helps restaurants adapt, take control, and get back to what they do best: building the businesses they love. And because our technology is purpose-built for restaurants, by restaurant people, restaurants can trust that we’ll deliver on their needs for today while investing in experiences that will power their restaurant of the future. Toast is looking for a Staff Machine Learning Engineer to serve as a technical linchpin for our AI Platform team. At the P4 level, you aren't just deploying models; you are designing the fundamental infrastructure that enables dozens of teams to build, deploy, and monitor AI at scale. You will act as a force multiplier, mentoring senior engineers and setting the architectural standards for our MLOps lifecycle—from feature stores and automated retraining to high-performance inference at the edge. A day in the life (Responsibilities) Architectural Leadership: Design and lead the evolution of a unified MLOps platform that supports diverse needs across Toast, ensuring high availability, scalability, and security of ML services. Engineering Excellence: Champion and institutionalize best practices for CI/CD for ML (MLOps), automated testing, and infrastructure-as-code (Terraform). Cross-Functional Synergy: Lead collaborative efforts across Data Engineering, DevOps, and Product teams to bridge the gap between model prototyping and production-grade reliability. Strategic Roadmapping: Partner with leadership and Product Managers to define the 1-2 year technical vision for AI infrastructure, prioritizing long-term stability over short-term fixes. Operational Ownership: Set the standard for observability and incident response for ML systems, driving root-cause analysis for complex system failures. Mentorship: Actively mentor P2 and P3 engineers, fostering a culture of technical rigor and continuous learning. What you'll need to thrive (Requirements) Education: Bachelor’s or Master’s degree in Computer Science, AI, or a related technical field. Experience: A minimum of 10-12+ years of professional software engineering experience, with at least 6-7 years specifically focused on productionizing and scaling ML systems at the enterprise level. Core Tech Stack: Expert-level proficiency in Python, Scala, or Java/Kotlin . Extensive experience with PySpark and high-performance computing. Generative AI & LLMOps: Proven track record of taking LLM applications from research to production, including experience with Vector Databases, LangChain/LangGraph, and A2A protocols . System Design: Superior ability to design distributed systems that handle millions of requests with sub-second latency. Experience with microservice based architecture, preferably with AWS tooling (SageMaker, DynamoDB, Athena, Glue, etc.) Experience in software engineering best practices and tools including object-oriented programming, test-driven development, CI/CD, git, shell scripting, task orchestration, MLflow Profound knowledge of model deployment, orchestration (Apache airflow, Prefect ), scaling, and managing CPU/GPU resources efficiently. Exceptional problem-solving, analytical skills and the ability to tackle complex problems with a critical thinking approach. Outstanding communication and interpersonal skills, coupled with a demonstrated ability to work collaboratively within a team environment. Foundational knowledge in statistical concepts (e.g. classification, regression, etc) and deep learning algorithms (e.g. CNN, RNN) is desirable What will help you stand out Data Strategy: Experience implementing enterprise-grade Fea