add edit and document test plan #1348
Conversation
| 1. Verify that you can see a list of code lenses with a Cody icon above the generated code: `Show diff`, `Accept`, `Retry`, and `Undo`. | ||
| <!-- 1. Verify that you can see a diff view of the edit in a new tab by clicking `Show diff`. --> | ||
| 1. Verify that you can prompt Cody to retry the command by clicking `Retry` and entering new instructions. | ||
| 1. Verify that you can undo the edit by clicking `Undo`. |
There was a problem hiding this comment.
This needs to be have way more detail. For Edit & Retry, Undo and Show Diff we need to test:
- Edit the file by adding multiple lines of text before and after the region Cody is editing, while Cody is working and after it has completed, then use (Edit & Retry/Undo/Show Diff)
- Edit the file by removing multiple lines of text before and after the region Cody is editing, while Cody is working and after it has completed, then use (Edit & Retry/Undo/Show Diff)
- Edit the file by removing multiple lines of text before and adding multiple lines of text after the region Cody is editing, while Cody is working and after it has completed, then use (Edit & Retry/Undo/Show Diff)
- When testing Undo specifically, only the changes Cody made should be undone.
- When testing Show Diff specifically, only the changes in the region Cody edited should be highlighted in the diff.
You need a preamble before ALL of these commands which says "when we say CONCURRENT HUMAN EDIT, we mean: no editing, add lines of text before the region Cody is editing, ..." and then in the test plan, you need to say "For each kind of CONCURRENT HUMAN EDIT: ..." and then have the list of instructions with one level of indent.
| 1. Verify that you can prompt Cody to retry the command by clicking `Retry` and entering new instructions. | ||
| 1. Verify that you can undo the edit by clicking `Undo`. | ||
| 1. Verify that the ghost text disappears by clicking `Accept`. | ||
|
|
There was a problem hiding this comment.
You haven't mentioned the keyboard shortcuts:
- That they work.
- That the text on the labels change if you change the key bindings.
Is there a separate "key bindings" test section for that? Are we not testing the keyboard shortcuts? In general, Cody uses a bunch of shortcuts which coincide with cedillas/umlauts type stuff, we should test them. VSCode and JetBrains may handle shortcuts on those kinds of keystrokes differently.
| 1. Highlight a section of code in a file. | ||
| 1. Run the `Document Code` command. | ||
| 1. Verify that an icon appears above the highlighted code with "Cody is working..." while Cody generates documentation. | ||
| <!-- 1. Verify that you can see a diff view of the generated documentation in a new tab by clicking `Show diff`. --> |
There was a problem hiding this comment.
Ditto above, show diff has landed
|
|
||
| ### Document | ||
|
|
||
| <!-- unhide after full document command is implemented --> |
There was a problem hiding this comment.
The following are not hidden, so remove the comment. It is unclear what you're referring to unhiding here.
| <!-- unhide after full document command is implemented --> | ||
|
|
||
| 1. Verify that the option to run the `Document Code` command is available from the right-click menu. | ||
| 1. Highlight a section of code in a file. |
There was a problem hiding this comment.
Let's specify particular file(s) and function(s). Preferably ones which:
- Cover a language where we do have a folding range implementation in TreeSitter and one where we don't, so TypeScript and Ocaml say.
- Functions which are "below the fold"—not visible in the first screenful of code.
- One time, a scroll position where the top of the function is not visible, one time when it is.
| 1. Verify that the ghost text disappears by clicking `Accept`. | ||
| 1. Move your cursor inside a function in the file without highlighting code, before running the `Document Code` command again. | ||
| 1. Verify that Cody adds documentation above the function that contains your cursor. | ||
|
|
There was a problem hiding this comment.
Question whether we want to do this test for each kind of "concurrent human edit." My gut feeling is that it is OK. The code is mostly shared, and we can just pound on concurrent human edits for Edit and we'll find the relevant bugs in Document.
Let's have the test plan for Document cover editing the function body while it is editing. That is a realistic use case that's a bit more rigorous than no editing, without the full rigor of all the kinds of edits.
Co-authored-by: Dominic Cooney <dominic.cooney@sourcegraph.com>
Co-authored-by: Dominic Cooney <dominic.cooney@sourcegraph.com>
Update QA test plan with Edit and Document command instructions. Missing features at the moment are show diff, and undo. I added the instructions anyways, we will unhide them after they are implemented
Test plan
N/A