GuardRails: Automated Suggestions for Clarifying Ambiguous Purpose Statements

VSCode extension to help developers write complete and unambiguous docstrings for their Python functions. Install it now from the Visual Studio Code Marketplace.

Important

Our paper was presented at the 6th Annual COMPUTE Conference by ACM India. Check it out here (or here on arXiv).

Warning

Due to recent updates in the Copilot's VSCode extension, functionality critical for GuardRails is no longer supported. Please hang on while we update GuardRails.

Requirements

You need Python 3 installed on your system to use GuardRails.

Note: GuardRails is fully tested only on Windows and currently provides experimental support for Linux.

Installing Python Dependencies

$ python -m pip install hypothesis hypothesis[cli] hypothesis[ghostwriter] black
$ python -m pip install git+https://github.com/mrigankpawagi/mutpy.git#egg=mutpy
$ python -m pip install func-timeout

A Motivating Example

def first_nonzero(nums: list[float]) -> float:
    """ Return the first non-zero value in nums.
    
    >>> first_nonzero([0.0 , 3.7 , 0.0])
    3.7
    """

This is an example of an incomplete function definition in Python. It has a purpose statement (called the docstring) which contains a functional example (called a doctest). The purpose statement is ambiguous because it does not specify the intended behavior on lists with no non-zero elements.

Contributions

Heuristic

We propose a novel heuristic that suggests such inputs using Large Language Models (LLMs). Using these suggestions, the programmer may choose to clarify the purpose statement (e.g., by providing a functional example that specifies the intended behavior on such an input). Check out our paper for more details on our heuristic.

Open Dataset

To assess the quality of inputs suggested by our heuristic, and to facilitate future research, we create an open dataset of purpose statements with known ambiguities. For each function, we provide four variants of prompts.

• S (Signature)
• SP (S + Purpose statement)
• SP1 (SP + 1 functional example)
• SPx (SP + More functional examples)

Our dataset is publicly available here. We use this dataset to compare GuardRails with test cases generated by GitHub Copilot Chat. More details on this are available in our paper.