Skip to content

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

4 Commits
 
 

Repository files navigation

Software Engineering with LLM Agents

  • Instructor: Lingming Zhang (lingming)
  • TA: Yuxiang Wei (ywei40)
  • Class Time: Tue/Thu 09:30 AM - 10:45 AM (Central Time)
  • Location: 3025 Campus Instructional Facility
  • Instructor Office Hours: Tue/Thu 10:45 AM - 11:45 AM (Central Time)
  • TA Office Hours: Thu 3:00 PM - 5:00 PM (Central Time) | Zoom Link
  • Communication: Campuswire (Use netid@illinois.edu to join)

Important

Join the Campuswire forum before the first class as all notifications, assignments, and project submissions will be managed there.


📖 Course Overview

Modern Large Language Models (LLMs) and agents have demonstrated remarkable capabilities across diverse fields, with software engineering as one of their most successful applications. This course dives deep into the intersection of LLM agents and software engineering, exploring how recent advances in generative AI can substantially transform the way people build and maintain software systems.

This is a research-driven course targeting students interested in research. Students must possess:

  • Research background in PL/FM/SE or NLP/ML fields.
  • Proficiency in Python programming.
  • Completion of NLP/ML coursework.
  • Solid background in algorithms and strong problem-solving skills.

For course restrictions, see: go.cs.illinois.edu/csregister


Tentative Schedule

Module I: Background and Basics

Date Topic Readings / Resources
01/20 Course Intro How to read a research paper?
Small guide to giving presentations
01/22 Software Engineering basics (I) Compilers: Principles, Techniques, and Tools (book, optional read)
Introduction to Software Testing (book, optional read)
01/27 Software Engineering basics (II) Feedback-directed Random Test Generation
Finding and Understanding Bugs in C Compilers
Fuzzing with Code Fragments
Compiler Validation via Equivalence Modulo Inputs
01/29 LLM basics Attention Is All You Need
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Training language models to follow instructions with human feedback
ReAct: Synergizing Reasoning and Acting in Language Models
02/03 Software Testing with LLMs Large Language Models are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models
Fuzz4All: Universal Fuzzing with Large Language Models
No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation
02/05 Software Debugging with LLMs Less Training, More Repairing Please: Revisiting Automated Program Repair via Zero-shot Learning
Keep the Conversation Going: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT

Module II: Software Engineering Agents

Date Topic Readings / Resources
02/10 Coding agents SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
OpenHands: An Open Platform for AI Software Developers as Generalist Agents
Additional: The 100 line AI agent that solves GitHub issues & more
Additional: Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scaling
02/12 Coding agents with SE insights AutoCodeRover: Autonomous Program Improvement
Agentless: Demystifying LLM-based Software Engineering Agents
Additional: Prometheus: Unified Knowledge Graphs for Issue Resolution in Multilingual Codebases
02/17 Coding agents with memory supports EXPEREPAIR: Dual-Memory Enhanced LLM-based Repository-Level Program Repair
Confucius Code Agent: Scalable Agent Scaffolding for Real-World Codebases
Additional: A-MEM: Agentic Memory for LLM Agents
Additional: Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory
Additional: Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning
Additional: Recursive Language Models
02/19 Proposal Presentation
02/24 Coding agents for scientific discovery AlphaEvolve: A coding agent for scientific and algorithmic discovery
ShinkaEvolve: Towards Open-Ended And Sample-Efficient Program Evolution
02/26 Self-improving coding agents (I) A Self-Improving Coding Agent
Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents
Automated Design of Agentic Systems
03/03 Self-improving coding agents (II) Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?
Huxley-Gödel Machine: Human-Level Coding Agent Development by an Approximation of the Optimal Self-Improving Machine
GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning

Module III: Benchmarks and Datasets

Date Topic Readings / Resources
03/05 Software benchmarks SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks
Additional: Introducing SWE-bench Verified
03/10 Multilingual benchmarks Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving
SWE-PolyBench: A multi-language benchmark for repository-level evaluation of coding agents
03/12 Automated benchmark construction SWE-bench Goes Live!
SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents
03/17 Spring Break
03/19 Spring Break
03/24 Environments for training coding agents Training Software Engineering Agents and Verifiers with SWE-Gym
R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents
SWE-Universe: Scale Real-World Verifiable Environments to Millions
Additional: Skywork-SWE: Unveiling Data Scaling Laws for Software Engineering in LLMs
03/26 Synthetic training data generation (I) SWE-smith: Scaling Data for Software Engineering Agents
SWE-Synth: Synthesizing Verifiable Bug-Fix Data to Enable Large Language Models in Resolving Real-World Bugs
SERA: Soft-Verified Efficient Repository Agents
03/31 Project Midterm Presentation
04/02 Project Midterm Presentation
04/07 Synthetic training data generation (II) SWE-Mirror: Scaling Issue-Resolving Datasets by Mirroring Issues Across Repositories
BugPilot: Complex Bug Generation for Efficient Learning of SWE Skills
Training Versatile Coding Agents in Synthetic Environments

Module IV: Training Software Agents with RL

Date Topic Readings / Resources
04/09 RL on simple code and math data DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
Kimi k1.5: Scaling Reinforcement Learning with LLMs
04/14 Scaling RL to real-world software data (I) SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution
Kimi-Dev: Agentless Training as Skill Prior for SWE-Agents
04/16 Scaling RL to real-world software data (II) DeepSWE: Training a Fully Open-sourced, State-of-the-Art Coding Agent by Scaling RL
SWE-RM: Execution-free Feedback For Software Engineering Agents
MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention
04/21 Training LLMs with agentic intelligence CWM: An Open-Weights LLM for Research on Code Generation with World Models
Kimi K2: Open Agentic Intelligence
Qwen3-Coder-Next Technical Report
04/23 Training superintelligent coding agents Toward Training Superintelligent Software Agents through Self-Play SWE-RL
Self-Adapting Language Models
04/28 Project Final Presentation
04/30 Project Final Presentation
05/05 Yinfang Chen (Invited Speaker) Zoom Link

🏫 Class Organization

There is no required textbook. The course is discussion-based, and students are expected to read assigned papers before each class.

During class, students may be randomly selected to discuss the following aspects of a paper:

  1. Problem: What is the problem and why does it matter?
  2. Solution: What is the proposed solution and how does it differ from prior work?
  3. Evaluation: What benchmarks and metrics were used? Is it convincing?
  4. Results: What were the results and did they meet expectations?
  5. Critique: What are the strengths (pros) and limitations (cons)?
  6. Future Work: What are the potential next steps?

📊 Grading Details

There is no exam. Grades are calculated based on the following:

Component Weight Description
Homework Assignments 20% Released via Campuswire ("Assignments" page). No late submissions without prior approval/documentation.
Paper Presentation 20% Lead discussion for one paper. Select at least five classes you would like to present by Jan. 30th (submission link shown in Campuswire "Assignments"). Upload your initial slides to Campuswire a week before your presentation slot for comments, and upload the final version of the slides 48 hours before the lecture. Make it clear if you reuse any of the original slides from the authors.
Class Participation 10% This is a discussion-based course, so it does matter that you show up in our class meetings and participate in the discussion.
Course Project 50% The best way to learn software engineering is go there and do software engineering! You will undertake your own course project in a group (3-5 students). We will provide a list of directions (available on Campuswire) to get you started thinking, but I highly encourage you to pursue your own ideas. You are encouraged to use GitHub to host your development history and all the code/data. For the teams proposing your own ideas, you are required to meet with Lingming before Feb. 9th to discuss your proposal.

Project Breakdown

  • Proposal submission/presentation: 5%
  • Midterm project report/presentation: 20%.
  • Final project report/presentation: 25%.

Grading Scale

Grade Percent Grade Percent
A 93% C+ 77%
A- 90% C 73%
B+ 87% C- 70%
B 83% D+ 67%
B- 80% D 63%
F <60% D- 60%

About

cs598lmz

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors