Strategic Projects Lead

ABOUT PARETO Humanity is in a virtuous cycle: human insight improves AI, and better AI expands what people can do. Sustaining it depends on the one input that can't be automated: expert human judgment https://pareto.ai/blog/debating-persuasive-llms-truthful-answers. At Pareto, we build the platform that turns that judgment into the data https://pareto.ai/blog/community-driven-knowledge-resource-ai, evals https://pareto.ai/blog/introducing-attunebench, and RL environments frontier models https://pareto.ai/blog/llm-metacognition-shared-and-shallow learn from. We work with leading frontier labs like Anthropic and GDM, and we give skilled people everywhere a way to shape the future of AI and share in what it creates. This RL environment and human-data infrastructure is already in production. Our job now is to scale it. ABOUT THIS ROLE Pareto builds human training data pipelines for frontier AI labs. As a Strategic Projects Lead, you sit at the center of that work — owning the architecture, execution, and continuous improvement of complex data collection and evaluation workflows from first scoping call to final delivery. This is a technical operations role, not a project management role. You'll be expected to read code, deep dive data, reason about LLM internals, design evaluation frameworks, and — increasingly — deploy and iterate on AI agents to automate the work your pipelines do today. We're actively building toward a model where agentic systems handle quality gates, expert routing, and output review, and SPLs are the people designing and operating those systems. You'll work directly with AI researchers and technical program managers at our client organizations, own delivery against model performance benchmarks, and lead a team of project managers who handle day-to-day execution tracking. What you'll do Pipeline architecture Design end-to-end data collection and evaluation pipelines for RLVR, RLHF, SFT, red-teaming, and model evaluation workflows. This includes expert sampling strategy, annotation schema, rubric structure, inter-rater calibration, and QA system design. You'll be expected to prototype novel workflows quickly, identify architectural risks before launch, and make tradeoff decisions with confidence. You’ll also need to understand how agents interact with tools to solve expert-driven tasks, and you’ll need to communicate with engineering to ensure the environment is built accordingly to enable such tasks. Agentic system deployment Build, test, and iterate on AI agents that automate pipeline tasks — quality gate review, expert matching, output flagging, throughput anomaly detection. You'll work closely with our engineering team to scope agent capabilities, write the prompts and evaluation logic that make them reliable, and monitor their performance in production. This is a growing part of the role; comfort with agentic tooling (LangChain, DSPy, custom tool-use frameworks, or equivalent) is a meaningful differentiator. Quality systems Define data quality standards across annotation, evaluation, and expert output review. Design and run audits using inter-rater reliability metrics, calibration sets, and statistical sampling. You'll be responsible not just for catching quality issues but for building systems that prevent them — automated checks, structured output validation, and model-assisted review layers where appropriate. Aside from programmatic quality testing, you’ll be responsible for spot-checking tasks & understanding what makes a datapoint meaningful & high quality. The ability to do so, and translate your findings into clear feedback and expert guidelines is particularly helpful for this role. Client interface Engage directly with AI researchers, TPMs, and PMs at our client organizations. Translate research-driven requirements — evaluation rubrics, domain coverage targets, latency constraints, benchmark specifications — into operational workflows. Communicate pipeline performance clearly, escalate technical ri