Remove python docstrings and Single-line comments #81
Conversation
- Updated to enhance comment removal logic. - Improved handling of Python docstrings and hash comments. - Refactored regex patterns for better accuracy. - Updated test cases in to reflect changes. - Ensured consistent use of double quotes in strings.
- Removed and statements from class.
WalkthroughThe changes introduced enhance the Changes
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Files selected for processing (2)
- src/core/file/fileManipulator.ts (4 hunks)
- tests/core/file/fileManipulator.test.ts (23 hunks)
Additional context used
Biome
src/core/file/fileManipulator.ts
[error] 60-60: Use !== instead of !=.
!= is only allowed when comparing againstnull!= is only allowed when comparing against null
Using != may be unsafe if you are relying on type coercion
Unsafe fix: Use !==(lint/suspicious/noDoubleEquals)
[error] 163-163: Template literals are preferred over string concatenation.
Unsafe fix: Use a template literal.
(lint/style/useTemplate)
[error] 169-169: Template literals are preferred over string concatenation.
Unsafe fix: Use a template literal.
(lint/style/useTemplate)
[error] 177-177: Reassigning a function parameter is confusing.
The parameter is declared here:
Use a local variable instead.
(lint/style/noParameterAssign)
[error] 178-178: Reassigning a function parameter is confusing.
The parameter is declared here:
Use a local variable instead.
(lint/style/noParameterAssign)
Additional comments not posted (16)
tests/core/file/fileManipulator.test.ts (16)
Line range hint
153-173: Test case "Python comment, docstring removal" effectively validates comment and docstring removal.This test case ensures that both single-line comments and docstrings are properly removed from Python code, enhancing test coverage.
175-191: Test case "Python docstring removal mixing string declaration" correctly handles mixed string declarations.This test validates that string literals containing triple quotes are preserved while actual docstrings are removed.
193-213: Test case "Python comment f-string is not removed" ensures f-strings are preserved.This test confirms that f-strings and their contents remain intact during comment and docstring removal.
215-233: Test case "Python comment multi-line string literal is not removed" accurately preserves multi-line string literals.The test verifies that multi-line string literals are not mistaken for docstrings and are left unmodified.
235-247: Test case "Python nested quotes" correctly handles nested quotes within docstrings.This ensures that docstrings with nested single or double quotes are removed without affecting the integrity of string literals.
249-267: Test case "Python nested triple quotes with different types" effectively tests nested docstrings.The test checks the removal of docstrings containing nested triple quotes of different types, ensuring robustness.
269-281: Test case "Python inline comments" confirms that inline comments are properly removed.This test ensures that comments following code on the same line are correctly eliminated without affecting the code itself.
283-297: Test case "Python multi-line statement with string" preserves multi-line strings and removes comments.This verifies that multi-line statements using backslashes are not disrupted during comment removal.
299-315: Test case "Python docstring with triple quotes inside string literals" ensures string literals are preserved.The test confirms that string literals containing triple quotes are not mistaken for docstrings.
317-341: Test case "Python mixed comments and docstrings" accurately removes all comments and docstrings.This comprehensive test covers multiple scenarios of comments and docstrings interleaved with code, ensuring thorough removal.
343-361: Test case "Python f-strings with triple quotes" preserves f-strings during comment removal.The test ensures that multi-line f-strings with expressions are not altered.
363-373: Test case "Python escaped hash in string" correctly handles escaped characters.This verifies that hashes escaped within strings are not treated as comment indicators.
375-393: Test case "Python nested function with docstrings" properly removes docstrings from nested functions.The test confirms that docstrings in both outer and inner functions are removed without affecting the function definitions.
395-413: Test case "Python comment-like content in string" preserves strings containing comment-like content.This ensures that strings with hash symbols are not inadvertently altered during comment removal.
415-430: Test case "Python docstring with backslashes" correctly handles backslashes within docstrings.The test verifies that backslashes, even when escaping triple quotes, do not interfere with docstring removal.
432-442: Test case "Python mixed single and double quotes" preserves string literals with mixed quotes.This test ensures that strings containing triple quotes are not mistaken for docstrings unless they are actual docstrings.
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
There was a problem hiding this comment.
Actionable comments posted: 1
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (1)
- src/core/file/fileManipulator.ts (4 hunks)
Additional context used
Biome
src/core/file/fileManipulator.ts
[error] 177-177: expected
,but instead found:Remove :
(parse)
[error] 177-177: Expected a semicolon or an implicit semicolon after a statement, but found none
An explicit or implicit semicolon is expected here...
...Which is required to end this statement
(parse)
[error] 182-182: This code is unreachable
... because this statement will return from the function beforehand
(lint/correctness/noUnreachable)
Additional comments not posted (2)
src/core/file/fileManipulator.ts (2)
43-99: LGTM!The
removeDocStringsfunction looks good. It correctly handles both single and double quote docstrings, and the logic to accumulate lines in a buffer until a complete docstring is found is sound.
101-174: LGTM!The
removeHashCommentsfunction looks good. The binary search approach to determine if a hash index falls within a string literal is efficient, and the logic to process the content in a loop, identifying hash comments and appending the relevant portions to the result string is correct.
| removeComments(content: string): string { | ||
| removeComments(content: string): string { | ||
| let result = this.removeDocStrings(content); | ||
| result = this.removeHashComments(result); | ||
| return rtrimLines(result); | ||
| } | ||
| return rtrimLines(content); | ||
| } |
There was a problem hiding this comment.
Fix the syntax error and remove the unreachable code.
The removeComments function has the following issues:
- There is a syntax error in the function signature. The
:after the parameter list should be removed. - The last line of the function is unreachable due to the
returnstatement on the previous line.
Apply this diff to fix the issues:
- removeComments(content: string): string {
+ removeComments(content: string) {
let result = this.removeDocStrings(content);
result = this.removeHashComments(result);
return rtrimLines(result);
}
- return rtrimLines(content);
}Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| removeComments(content: string): string { | |
| removeComments(content: string): string { | |
| let result = this.removeDocStrings(content); | |
| result = this.removeHashComments(result); | |
| return rtrimLines(result); | |
| } | |
| return rtrimLines(content); | |
| } | |
| removeComments(content: string) { | |
| let result = this.removeDocStrings(content); | |
| result = this.removeHashComments(result); | |
| return rtrimLines(result); | |
| } |
Tools
Biome
[error] 177-177: expected
,but instead found:Remove :
(parse)
[error] 177-177: Expected a semicolon or an implicit semicolon after a statement, but found none
An explicit or implicit semicolon is expected here...
...Which is required to end this statement
(parse)
[error] 182-182: This code is unreachable
... because this statement will return from the function beforehand
(lint/correctness/noUnreachable)
Summary
Altered files
src/core/file/fileManipulator.ts:I added the methods removeDocStrings() and removeHashComments() to be used in removeComments() so that it's easier to implement an option to only delete certain types of comments later on.
tests/core/file/fileManipulator.test.ts:There are now 16 tests for Python comment removal.
Observations
Certain edge cases were not considered to avoid complexity.
For example in the code
only the first docstring is removed. But this is not standard practice in Python
Time Complexity
removeDocStringsFunction: O(m * n) where 'm' is the number of lines and 'n' is the average length of each lineremoveHashCommentsFunction: O((m * n) + m log m)Benchmark
For a better idea of the time complexity, I ran a quick test where the algorithm was applied to a repeated string of Python code containing several docstrings and comments.
For 3 million lines, the algorithm ran in around one second on my computer. This is more likely than not a decent time since LLMs won't take this many tokens anyway.
Next steps
I'll be happy to help with the implementation of any of these!
Summary by CodeRabbit
New Features
Bug Fixes
Tests