In-IDE Generation-based Information Support with a Large Language Model

In the ever-evolving field of software development, understanding code is crucial. However, developers often face challenges due to the scarcity or absence of code comments and documentation. This issue has been further complicated by the rise of large language model (LLM) based code generation tools. To address these challenges, a research study by Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers investigates an LLM-based conversational UI in the IDE.

The researchers developed an IDE Plugin that interacts with back-ends like OpenAI's GPT-3.5 and GPT-4. This plugin allows users to make high-level requests such as explaining a highlighted section of code, explaining key domain-specific terms, or providing usage examples for an API. The goal is to provide developers with a more contextually aware tool that can answer queries based on the developer's programming context.

The effectiveness of this tool was tested through an exploratory user study involving 32 participants. The study aimed to measure the usefulness and effectiveness of the LLM-powered information support tool. Participants were asked to complete a series of tasks and then evaluate their experiences using both the tool and traditional web search.

The results were promising. Participants rated the LLM information support tool higher on perceived usefulness and usability than search engines. They felt less rushed and more successful in accomplishing tasks with the LLM tool than with search engines. However, the benefits varied based on the participants' experience levels. Professional programmers benefited more from the tool than students, who accepted code suggested by the LLM with little scrutiny and made less progress.

These findings suggest that while LLM-powered tools can enhance developer productivity, they must be carefully designed to adapt to the user's experience level. This is a crucial point for future development and usage of LLM-based tools in code understanding. The study provides a new perspective on how AI can be used to aid developers in their tasks, and how it can be improved to cater to the varying experience levels of its users.

Read the whole article here: http://arxiv.org/abs/2307.08177v1