Co-Lab Student Developer Reflections: Building a Co-Lab Co-Pilot

This blog post was written by Jonathan Reyes, Trinity '26.  Jonathan is a Co-Lab student employee on our Dev Team and worked on this project throughout the 23-24 Academic Year.

Last summer I did Code+, a Duke program where students work on real software projects. AI was dominating headlines at the time, and our project was right in the middle of it: an AI-powered customer service chatbot for OIT. My first real look at what it takes to build something with language models.

After the program ended, I found myself eager to continue working. I'd missed the deadline to apply for a student developer position at the Co-Lab, so I reached out to my Code+ coordinator, Jen Vizas, and told her I wanted to keep working on AI projects. She connected me with Michael Faber, Senior Manager at the Co-Lab, and we set up a meeting with Danai Adkisson and Daniel Davis, who would become my project lead.

My proposal was to build on what I'd started over the summer, which is a conversational assistant to help students find relevant information. Michael was on board, and we started brainstorming how AI could fit into the Co-Lab's existing ecosystem. We landed on a clear goal: use the wealth of Pathways data to improve the student experience.

Content-Based Filtering 

We started without generative AI to prove the core concept first. The result was a recommendation system using content-based filtering, suggesting modules similar to ones a user has already taken. Under the hood, it used classical machine learning and natural language processing approaches like TF-IDF for text vectorization and a sigmoid kernel function to calculate similarity between module descriptions. A solid starting point before bringing in anything more complex.

Function Calling 

That fall, OpenAI released a new set of tools for building conversational agents: Threads for managing longer conversations, Retrieval for storing text, and improvements to function calling with their latest model, GPT-4. Our plan was to use function calling to retrieve course descriptions via API, which is a function that, given a course name, would pull back the relevant data. But it had a fundamental problem. Without the user explicitly mentioning a course name, the model had no reliable way to know what to look up. OpenAI's newer models like gpt-3.5-turbo-0125 and gpt-4-turbo-preview had improved at deciding when to invoke functions, but inconsistencies remained. And GPT-4, the only model that handled it well enough, was too expensive to deploy at scale. It wasn't going to work.

LangChain and SQL 

So we looked at a different approach entirely. What if we just queried a database directly? That's when we found LangChain, a framework for building context-aware applications with language models. It works through sequences of calls, or "chains," that can include model interactions, tools, and data preprocessing steps. One useful feature was its SQL agent, which converts user questions into SQL queries, executes them, and returns a response based on the results. It also used dynamic few-shot prompting, a technique that uses FAISS (Facebook AI Similarity Search) and OpenAI's embeddings model to pull in relevant example queries and sharpen the model's accuracy.

In practice, it didn't hold up. Despite our efforts to improve accuracy with dynamic few-shot prompting, it consistently struggled to generate correct SQL queries. Fixing its mistakes required too much manual intervention to be sustainable, and we ran into cost issues on the Azure SQL server side. The expenses from database operations piled up faster than any savings we were getting from token optimization, making the whole approach economically unviable. We moved on.

Vector Search 

The one idea from LangChain that stuck was using similarity search to pass only the most relevant data to the model. That got us thinking: why not embed the entire database? Vector embeddings weren't new to us because we had worked with the technology earlier in the year, but this was a much larger scale. Michael connected me with Daniel Medina, a graduate student in Jon Reifschneider's CREATE Lab, whose experience with vector databases helped us map out what we actually needed to build.

We swapped out the Azure SQL server for MongoDB and its Atlas Vector Search tool. Using OpenAI's text-embedding-3-small model, we embedded both user queries and all course descriptions in our database. Atlas Vector Search then uses the HNSW (Hierarchical Navigable Small Worlds) algorithm to find the most relevant courses for a given query, scoring each one. The top result gets pulled into the model's prompt before the response is returned to the user.

Next Steps

The current pipeline routes every user query through a single assistant node, which decides which tools to call based on context. It works well for a general-purpose chatbot, but as the number of tools grows, so does the complexity of keeping it reliable.

Rather than one assistant handling everything, user queries would first be categorized by intent, then routed to a dedicated sub-graph built specifically for that type of request. A question about course schedules goes to one workflow, a question about instructors goes to another. Each workflow can be improved independently without affecting the rest of the system, making the overall assistant more predictable as a result.

What started as a summer project has turned into something I'm genuinely proud of. From content-based filtering to vector search to specialized workflows, each iteration taught us something. The first approach is rarely the right one, and that's okay. There's still a lot to build, but the foundation is there.