Remote | Senior Software Engineer – LLM Evaluation
24-MAG · New York, New York, United States · remote
We are sharing a specialised part-time consulting opportunity for experienced Senior Software Engineers to contribute to advanced large language model evaluation, coding benchmark development, and AI-assisted software-engineering research.
Selected professionals will curate and evaluate code, develop verification mechanisms, assess AI-generated software across multiple programming languages, and help researchers understand where advanced models succeed or fail throughout the software-development lifecycle. The work combines senior-level software engineering judgement with structured evaluation and benchmark design.
Key Responsibilities
Code Curation & Solution Development
- Curate high-quality code examples for model training and benchmarking
- Develop precise reference solutions to software-engineering tasks
- Correct and improve code across multiple programming languages
- Work with Python, JavaScript, ReactJS, C/C++, Java, Rust, and Go
- Maintain strong standards for correctness, clarity, and maintainability
AI-Generated Code Evaluation
- Evaluate AI-generated code for technical correctness
- Assess solutions for efficiency, scalability, and reliability
- Identify implementation weaknesses and recurring error patterns
- Review code quality against professional engineering standards
- Provide structured rationales supporting evaluation decisions
Verification & Automated Assessment
- Design mechanisms that automatically verify software-engineering solutions
- Build agents capable of assessing code quality
- Develop approaches for identifying common model failure patterns
- Create deterministic or structured checks where appropriate
- Support reliable evaluation across repeated coding tasks
Software Engineering Lifecycle Evaluation
- Evaluate model capabilities across the software-development lifecycle
- Assess reasoning around prototyping and architecture design
- Review API design and production implementation decisions
- Evaluate launch, experimentation, monitoring, and operational scenarios
- Identify where models struggle with real-world engineering workflows
Research & Benchmark Collaboration
- Collaborate with researchers and cross-functional technical teams
- Contribute to datasets used for training and benchmarking
- Help define engineering evaluation strategies and quality standards
- Compare model performance against professional engineering expectations
- Support iterative improvements to coding-focused evaluation systems
Ideal Profile
- Several years of professional software-engineering experience
- At least 2 years of continuous full-time experience at a top-tier product company
- Strong full-stack development capabilities
- Experience building scalable, production-grade software
- Deep understanding of software architecture and system design
- Strong knowledge of debugging and code-quality assessment
- Experience reviewing and improving complex production code
- Proficiency in one or more of Python, JavaScript, C/C++, Java, Rust, or Go
- ReactJS experience is valuable for frontend-oriented assignments
- Strong understanding of API design and production implementation
- Familiarity with software monitoring and operational maintenance
- Ability to reason across the complete software-engineering lifecycle
- Strong analytical and problem-solving capabilities
- Excellent written and verbal communication skills
- Ability to produce clear and structured evaluation rationales
- Comfortable collaborating with research and technical teams in a remote environment
Engagement Details
- Part-time independent contractor engagement
- Fully remote
- Minimum commitment: 10 hours per week
- Flexible workload of up to 40 hours per week
- Partial overlap with PST working hours is required
- Initial project duration is approximately 1 month
- Project is expected to begin the following week
- Extension may be available depending on performance and project fit
- No medical or paid-leave benefits are included under the contractor arrangement
- Work will involve coding evaluation, benchmark development, verification systems, and software-engineering research
- Assignments may span Python, JavaScript, ReactJS, C/C++, Java, Rust, Go, and related technologies
- Work must be completed without using confidential, proprietary, unreleased, employer-restricted, client-restricted, or otherwise protected code, datasets, architecture materials, or technical information belonging to any employer, client, institution, or other third party
About the Platform
This opportunity is available through 24-MAG LLC. We connect experienced professionals with remote consulting opportunities across technical, evaluation, and project-based workstreams.
By submitting this application, you acknowledge that your information may be processed by 24-MAG LLC for recruitment and opportunity matching in accordance with our Privacy Policy: