Senior Data Scientist, AI

ABOUT POSH We are all social creatures, but the dominant “social” companies today have evolved into digital loneliness machines, driving isolation, anxiety, and mental health challenges around the world. Human connection is lost. Posh is a beacon guiding us back. Posh enables anyone to build an IRL community based on shared interests, while connecting consumers with the communities of people just like them. Founded by event organizers who were frustrated with the growing loneliness epidemic and the tools available to build their own event brand, we’ve built the ultimate platform for launching, monetizing, and finding IRL communities of people just like you. In just 6 years, Posh has grown to a team of 70, expanded to 8M+ users, secured $70m in venture funding, and facilitated over $350M in transactions. ABOUT THE ROLE We are looking for an experienced Senior Data Scientist to own the evaluation framework for our AI agent, the data that feeds it, and the success analysis, testing, and metrics that determine how well it's working. As one of the early data hires at Posh, you'll shape the technical direction of our AI quality strategy and set the standards for how agent performance is defined, measured, and improved over time. You'll work across the full evaluation lifecycle: identifying the right signals, building clean pipelines for evaluation data, designing tests and experiments, and communicating what "good" looks like to both technical and non-technical stakeholders. Your work will directly inform how we iterate on our AI agent and how we know when it's ready to ship. You'll partner closely with Product and Engineering to define success criteria, ensure proper instrumentation, and architect evaluation datasets that reflect real user behavior. You'll establish best practices for data quality, governance, and documentation, ensuring our evaluation framework remains trustworthy and rigorous as we grow. This role offers a high-growth opportunity as we expand our AI and data capabilities. If you're passionate about building from 0 to 1 and making a lasting impact, this is the role for you. This is an in-person position at our New York City office, located in the heart of SoHo. AT A HIGH LEVEL, YOU’LL BE IN CHARGE OF: - Building and Owning the AI Agent Evaluation Framework: Design and maintain the systems and methodologies we use to measure AI agent quality. Define what good looks like, build the rubrics and benchmarks, and own the feedback loops that drive iteration. Build ETL/ELT pipelines that transform raw behavioral, transactional, and interaction data into clean evaluation inputs. - Preparing High-Quality Data for AI Models: Own the data that feeds our evaluation pipeline, from ground truth datasets and labeled examples to behavioral signals and the semantic layer. Ensure every input is reliable, well-structured, and built to last. - Instrumenting Agent Tests, Experiments, and Success Metrics: Build the testing infrastructure to evaluate agent performance across accuracy, relevance, and user satisfaction. Run structured experiments and pre/post analyses to assess the impact of model and product changes, and build dashboards that keep the team aligned on performance trends and regressions before they become problems. - Collaborating with Product and Engineering on Instrumentation: Work closely with Engineering to ensure accurate logging of agent interactions and user signals. Partner with Product to translate business goals into evaluation criteria and measurement requirements. - Ensuring Strong Data Governance and Documentation: Implement best practices for data quality, documentation, observability, and lineage across all evaluation-bound datasets. Be the person who makes sure the foundation doesn't rot. OUR IDEAL CANDIDATE - Possesses 5+ Years of full time Data Experience: Has at least 5 years of hands-on experience in data science or analytics engineering. Demonstrates a strong ability to design, build, and optimize scala