About the job
This job offers an exciting opportunity to engage with cutting-edge AI technology by creating and evaluating tasks for AI coding agents. Contributors will play a vital role in shaping how AI interacts with real-world developer tasks, ensuring that the challenges presented are both realistic and thought-provoking. The team values collaboration and innovation, working together to push the boundaries of AI capabilities.
You'll be responsible for
📝
Creating tasks and evaluation criteria
Designing challenging tasks and criteria within realistic simulated environments to evaluate AI coding agents.🔍
Reviewing code and analyzing performance
Reviewing code written by AI agents, analyzing successes and failures, and designing edge cases.🔄
Iterating based on feedback
Working collaboratively with expert QA reviewers to refine tasks and ensure quality standards are met.Key criteria
💻
Degree in Computer Science or related field
Possession of a relevant degree is essential for understanding complex concepts.View
🛠️
5+ years software development experience
Demonstrated experience in software development, primarily using Python.View
🔍
Experience writing tests
Proven ability to write functional and integration tests, not just executing them.View
View more