Preview (#1750)
* update package.json version * 🪄 model dropdown, control plane (#1692) * Nate/control plane client (#1691) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * fix(gui): ctx rendering w/ renderInlineAs: "" (#1541) * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * fix codeqwen autocomplete leading space * add voyage rerank-1 * feat: `--noEmit` for tsc checks in CI (#1559) * docs: update CustomContextProvider docs (#1557) * add stop tokens to qwen prompt * update docs to reflect 3.5 sonnet being best * docs: comment out unused providers (#1561) * import Handlebars * feat: toast notification for config updates (#1560) * feat: toast notification for config updates * feat: only trigger toast on config.json save * displayRawMarkdown option * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * update to reflect 16 stop words limit for deepseek * feat: only trigger config update toast in vscode (#1571) * docs(prompt-files): fix typos + clarify (#1575) * doc: prompt file typo + clarifications * fix: add back correct docs * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * docs(telemetry): add pageviews to tracking list (#1581) * Add reranker configuration options to codebase embedding docs (#1584) - Introduce reranker concept - List available reranker options - Provide configuration instructions - Update keywords to include "reranker" * chore: update pr template with screenshots (#1590) * Refactor ConfirmationDialog to use SecondaryButton for cancel action (#1586) * Added instructions for running docs server locally (#1578) - Added NPM script method - Added VS Code task method - Update contributing guidelines * Update branch policy (#1577) - Change PR target to `dev` branch - Update `CONTRIBUTING.md` instructions * Consolidate example configurations into the main configuration guide (#1579) - Moved examples to configuration.md - Deleted the separate examples.md file - Updated sidebar order and links - Improved readability and structure in configuration.md * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * Update completionProvider.ts (warning tab-autocomplete models) (#1566) * feat: enhanced IndexingProgressBar with blinking dot feature - Integrated BlinkingDot component - Added STATUS_COLORS for various states - Replaced CircleDiv with BlinkingDot in UI - Updated status messages and layout * small UI tweaks * feat(gui): enhance ModelCard, ModelProviderTag, and Toggle components (#1595) - add styling and adjustments to ModelCard - update ModelProviderTag font size - remove box shadow from Toggle component - tweak icon styles in ModelCard - improve alignment and spacing * media query * feat: add best experience onboarding * fix: file rename * stop movement on button hover by keeping same border thickness * fix mistake in setting cursor: pointer * fix when free trial option is shown * Support Node.js versions below 20 for streaming response handling (#1591) - Add fallback for Node < 20 - Implement toAsyncIterable for streaming - Use TextDecoder for manual decoding - Maintain existing streaming for Node 20+ * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * fix hasPassedFTL * fix /edit cancellation from UI * feat: add `applyCodeBlock` experimental prop (#1601) * feat: add new model styling improvements (#1609) * feat: add new model styling improvements * better gap size * feat: update bug_report.yml (#1610) * chore: update bug_report.yml * typo fix * feat: add labels to "Add docs" dialog (#1612) * feat: add labels to "Add docs" dialog * remove autofocus * don't double load config * small fixes * speed up directory traversal, and use correct native path module * option not to show config update toast * merge air-gapped and recommended setup pages * chore: add telemetry for full screen toggle (#1618) * Fix headings in codebase-embeddings.md (#1617) * mention jetbrains * docs: update changie (#1619) * feat: updated changie config * hide toc and autogenerate * Update changelog.mdx * link to deeper explanation of embeddings models * ensure target="_blank" for all links in sidebar * fix gif links in intellij README.md * don't require rust in dependency installation * chore: fix padding on gh button (#1620) * chore: adjust button padding * Update tasks.json * escape colons in diff path * smoother lancedb indexing reporting * smooth progress updates for indexing * fix tsc err * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * docs: add docs and schema for "OS" provider (#1536) * ignore .env * 🚑 fix constant warnings when onboarding with Ollama * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * team analytics * apply control plane settings * workos auth * ide protocol get session info * UI for auth * profile switching * small fixes * updates * refresh tokens * updates * fix tsc errs * model select in toolbar to make room for profile selector * prod client id * link to prod URL * internal beta option * profiles change listener --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * temporary patch for type errs --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * skip extension tests * update package.json version * handlebars import * cmd+I improvements (#1728) * Nate/control plane client (#1691) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * fix(gui): ctx rendering w/ renderInlineAs: "" (#1541) * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * fix codeqwen autocomplete leading space * add voyage rerank-1 * feat: `--noEmit` for tsc checks in CI (#1559) * docs: update CustomContextProvider docs (#1557) * add stop tokens to qwen prompt * update docs to reflect 3.5 sonnet being best * docs: comment out unused providers (#1561) * import Handlebars * feat: toast notification for config updates (#1560) * feat: toast notification for config updates * feat: only trigger toast on config.json save * displayRawMarkdown option * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * update to reflect 16 stop words limit for deepseek * feat: only trigger config update toast in vscode (#1571) * docs(prompt-files): fix typos + clarify (#1575) * doc: prompt file typo + clarifications * fix: add back correct docs * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * docs(telemetry): add pageviews to tracking list (#1581) * Add reranker configuration options to codebase embedding docs (#1584) - Introduce reranker concept - List available reranker options - Provide configuration instructions - Update keywords to include "reranker" * chore: update pr template with screenshots (#1590) * Refactor ConfirmationDialog to use SecondaryButton for cancel action (#1586) * Added instructions for running docs server locally (#1578) - Added NPM script method - Added VS Code task method - Update contributing guidelines * Update branch policy (#1577) - Change PR target to `dev` branch - Update `CONTRIBUTING.md` instructions * Consolidate example configurations into the main configuration guide (#1579) - Moved examples to configuration.md - Deleted the separate examples.md file - Updated sidebar order and links - Improved readability and structure in configuration.md * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * Update completionProvider.ts (warning tab-autocomplete models) (#1566) * feat: enhanced IndexingProgressBar with blinking dot feature - Integrated BlinkingDot component - Added STATUS_COLORS for various states - Replaced CircleDiv with BlinkingDot in UI - Updated status messages and layout * small UI tweaks * feat(gui): enhance ModelCard, ModelProviderTag, and Toggle components (#1595) - add styling and adjustments to ModelCard - update ModelProviderTag font size - remove box shadow from Toggle component - tweak icon styles in ModelCard - improve alignment and spacing * media query * feat: add best experience onboarding * fix: file rename * stop movement on button hover by keeping same border thickness * fix mistake in setting cursor: pointer * fix when free trial option is shown * Support Node.js versions below 20 for streaming response handling (#1591) - Add fallback for Node < 20 - Implement toAsyncIterable for streaming - Use TextDecoder for manual decoding - Maintain existing streaming for Node 20+ * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * fix hasPassedFTL * fix /edit cancellation from UI * feat: add `applyCodeBlock` experimental prop (#1601) * feat: add new model styling improvements (#1609) * feat: add new model styling improvements * better gap size * feat: update bug_report.yml (#1610) * chore: update bug_report.yml * typo fix * feat: add labels to "Add docs" dialog (#1612) * feat: add labels to "Add docs" dialog * remove autofocus * don't double load config * small fixes * speed up directory traversal, and use correct native path module * option not to show config update toast * merge air-gapped and recommended setup pages * chore: add telemetry for full screen toggle (#1618) * Fix headings in codebase-embeddings.md (#1617) * mention jetbrains * docs: update changie (#1619) * feat: updated changie config * hide toc and autogenerate * Update changelog.mdx * link to deeper explanation of embeddings models * ensure target="_blank" for all links in sidebar * fix gif links in intellij README.md * don't require rust in dependency installation * chore: fix padding on gh button (#1620) * chore: adjust button padding * Update tasks.json * escape colons in diff path * smoother lancedb indexing reporting * smooth progress updates for indexing * fix tsc err * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * docs: add docs and schema for "OS" provider (#1536) * ignore .env * 🚑 fix constant warnings when onboarding with Ollama * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * team analytics * apply control plane settings * workos auth * ide protocol get session info * UI for auth * profile switching * small fixes * updates * refresh tokens * updates * fix tsc errs * model select in toolbar to make room for profile selector * prod client id * link to prod URL * internal beta option * profiles change listener --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * temporary patch for type errs * Dev (#1689) * Fix an issue where CMD+K does not clear the terminal when the terminal has focus (#1671) On MacOS, ⌘+K is bound, by default, to Terminal:Clear. Without this change ⌘+K does not clear the terminal but instead iniates a chord sequence and waits for the next stroke of the chord. Co-authored-by: Rob Leidle <rleidle@tesla.com> * Change treeSitter to cache the Language objects it loads from wasm (#1672) Without this change, for a repository with 600 typescript files, the indexer would fail to finish correctly and there would be many of the following errors in the webview console log: 'Unable to load language for file ${path} RuntimeError: table index is out of bounds' The following bash will create a repo that reproduces the problem: current_path="." for ((i=1; i<=20; i++)); do new_folder="folder-$i" mkdir -p "$current_path/$new_folder" current_path="$current_path/$new_folder" for ((a=1; a<=30; a++)); do head -c 10000 /dev/urandom | base64 > "$current_path/file-$a.ts" done done Co-authored-by: Rob Leidle <rleidle@tesla.com> * acknowledge sourcemap flag in esbuild.js * don't run jetbrains-release.yaml on vscode releases * further testing for walkDir * chore: add telemetry to commands (#1673) * test: Add basic unit test to baseLLM (#1668) * update version * test: Add basic unit test to baseLLM --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: inimaz <imaz@ai-verse.com> * feat: add Quick Actions CodeLens feature (#1674) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * add voyage rerank-1 * import Handlebars * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * small UI tweaks * media query * feat: add best experience onboarding * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * chore: add telemetry for full screen toggle (#1618) * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * manually implement stop tokens for hf inference api * chore: onboarding metrics (#1626) * fix: pageview tracking * feat: add onboarding telemetry * create single `onboardingStatus` type * improved var naming * remove console logs * fix windows performance issue * rename vscodeExtension.ts * migration of onboarding variables * "stash" instead of "delete" in indexing progress * fix preview.yaml * also fix main.yaml * Update troubleshooting.md (#1637) * feat: add quick actions * Update index.d.ts * quick actions mvp * update docs * subscribe to vscode change settings * Update commands.ts * cleanup * Update quick-actions.md * Update VerticalPerLineCodeLensProvider.ts * resolve feedback --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: Nate Sesti <33237525+sestinj@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> * chore: add `isCommandEvent` to command telemetry (#1675) * chore: add `isCommandEvent` to command telemetry * Update commands.ts * Nate/better retrieval (#1677) * deduplicatearray tests * break out separate retrieval pipelines * IConfigHandler * tests for codebase indexer * better .continueignore for continue * indexing fixes * ignore .gitignore and .continueignore when indexing * retrieval pipeline improvements * fix formatting err in out .continueignore * add necessary filter to lance_db_cache * update package.json version * skip unused tests * don't ignore .prompt files * update version * Update pull_request_template.md * don't use multi-media format when there are multiple text items * add free trial experience (#1685) * fix: add code range for quick actions/fixes (#1687) * fix: add code range for quick actions/fixes * Update test.js * add pathSep message type * docs improvements * jetbrains fix * update package.json version --------- Co-authored-by: Rob Leidle <rleidle@gmail.com> Co-authored-by: Rob Leidle <rleidle@tesla.com> Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: inimaz <49730431+inimaz@users.noreply.github.com> Co-authored-by: inimaz <imaz@ai-verse.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> * update package.json version * skip extension tests * update package.json version * handlebars import * don't push package.json version change from CI * fix: scroll issues w/ code blocks (#1688) * fix: scroll issues w/ code blocks * chore: name offset * chore: remove docs start * chore: update headings on troubleshooting.md (#1696) * ignore .svn folder for indexing (#1699) * chore: clearer naming for `useSuffix` (#1702) * chore: clearer naming for `useSuffix` * schema updates * fix: cmd+shft+l closes sidebar if focused (#1638) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * add voyage rerank-1 * import Handlebars * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * small UI tweaks * media query * feat: add best experience onboarding * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * chore: add telemetry for full screen toggle (#1618) * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * manually implement stop tokens for hf inference api * chore: onboarding metrics (#1626) * fix: pageview tracking * feat: add onboarding telemetry * create single `onboardingStatus` type * improved var naming * remove console logs * fix windows performance issue * rename vscodeExtension.ts * migration of onboarding variables * "stash" instead of "delete" in indexing progress * fix preview.yaml * also fix main.yaml * Update troubleshooting.md (#1637) * feat: close panel if main input is focused * add skip param * Update TipTapEditor.tsx * merge dev * Update commands.ts --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: Nate Sesti <33237525+sestinj@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> * bugfix: quick actions undefined array (#1704) * feat: add rich quick pick for quick edit (#1706) * feat: multistep quick pick * fix: multi step for context provider * Update test.js * Create deepseek.md (#1708) * style: add max height to model selector (#1712) * update styling * remove testing code * remove comments * feat: add tutorial card (#1716) * feat: create tutorial card * Update TutorialCard.tsx * feat: add file search to quick edit (#1714) * feat: multistep quick pick * fix: multi step for context provider * Update test.js * add file search * complete file search * remove unused import * add comments * Update utils.ts * update naming * chore: move quick pick instantiation (#1723) * Break out into NPM modules (#1726) * config.json types * export all schemas * break out request logic into packages * update openai adapters to export body types * update packaging of npm modules * update packages * continue proxy * mistral * Fix typo in documentation (#1722) In set-up-codestral.md "Trobleshooting" -> "Troubleshooting" * update package.json version * 🪄 model dropdown, control plane (#1692) * Nate/control plane client (#1691) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * fix(gui): ctx rendering w/ renderInlineAs: "" (#1541) * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * fix codeqwen autocomplete leading space * add voyage rerank-1 * feat: `--noEmit` for tsc checks in CI (#1559) * docs: update CustomContextProvider docs (#1557) * add stop tokens to qwen prompt * update docs to reflect 3.5 sonnet being best * docs: comment out unused providers (#1561) * import Handlebars * feat: toast notification for config updates (#1560) * feat: toast notification for config updates * feat: only trigger toast on config.json save * displayRawMarkdown option * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * update to reflect 16 stop words limit for deepseek * feat: only trigger config update toast in vscode (#1571) * docs(prompt-files): fix typos + clarify (#1575) * doc: prompt file typo + clarifications * fix: add back correct docs * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * docs(telemetry): add pageviews to tracking list (#1581) * Add reranker configuration options to codebase embedding docs (#1584) - Introduce reranker concept - List available reranker options - Provide configuration instructions - Update keywords to include "reranker" * chore: update pr template with screenshots (#1590) * Refactor ConfirmationDialog to use SecondaryButton for cancel action (#1586) * Added instructions for running docs server locally (#1578) - Added NPM script method - Added VS Code task method - Update contributing guidelines * Update branch policy (#1577) - Change PR target to `dev` branch - Update `CONTRIBUTING.md` instructions * Consolidate example configurations into the main configuration guide (#1579) - Moved examples to configuration.md - Deleted the separate examples.md file - Updated sidebar order and links - Improved readability and structure in configuration.md * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * Update completionProvider.ts (warning tab-autocomplete models) (#1566) * feat: enhanced IndexingProgressBar with blinking dot feature - Integrated BlinkingDot component - Added STATUS_COLORS for various states - Replaced CircleDiv with BlinkingDot in UI - Updated status messages and layout * small UI tweaks * feat(gui): enhance ModelCard, ModelProviderTag, and Toggle components (#1595) - add styling and adjustments to ModelCard - update ModelProviderTag font size - remove box shadow from Toggle component - tweak icon styles in ModelCard - improve alignment and spacing * media query * feat: add best experience onboarding * fix: file rename * stop movement on button hover by keeping same border thickness * fix mistake in setting cursor: pointer * fix when free trial option is shown * Support Node.js versions below 20 for streaming response handling (#1591) - Add fallback for Node < 20 - Implement toAsyncIterable for streaming - Use TextDecoder for manual decoding - Maintain existing streaming for Node 20+ * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * fix hasPassedFTL * fix /edit cancellation from UI * feat: add `applyCodeBlock` experimental prop (#1601) * feat: add new model styling improvements (#1609) * feat: add new model styling improvements * better gap size * feat: update bug_report.yml (#1610) * chore: update bug_report.yml * typo fix * feat: add labels to "Add docs" dialog (#1612) * feat: add labels to "Add docs" dialog * remove autofocus * don't double load config * small fixes * speed up directory traversal, and use correct native path module * option not to show config update toast * merge air-gapped and recommended setup pages * chore: add telemetry for full screen toggle (#1618) * Fix headings in codebase-embeddings.md (#1617) * mention jetbrains * docs: update changie (#1619) * feat: updated changie config * hide toc and autogenerate * Update changelog.mdx * link to deeper explanation of embeddings models * ensure target="_blank" for all links in sidebar * fix gif links in intellij README.md * don't require rust in dependency installation * chore: fix padding on gh button (#1620) * chore: adjust button padding * Update tasks.json * escape colons in diff path * smoother lancedb indexing reporting * smooth progress updates for indexing * fix tsc err * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * docs: add docs and schema for "OS" provider (#1536) * ignore .env * 🚑 fix constant warnings when onboarding with Ollama * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * team analytics * apply control plane settings * workos auth * ide protocol get session info * UI for auth * profile switching * small fixes * updates * refresh tokens * updates * fix tsc errs * model select in toolbar to make room for profile selector * prod client id * link to prod URL * internal beta option * profiles change listener --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * temporary patch for type errs --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * skip extension tests --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> Co-authored-by: Rob Leidle <rleidle@gmail.com> Co-authored-by: Rob Leidle <rleidle@tesla.com> Co-authored-by: inimaz <49730431+inimaz@users.noreply.github.com> Co-authored-by: inimaz <imaz@ai-verse.com> Co-authored-by: Anaël Bonnafous <71386173+AnaelBonnafous@users.noreply.github.com> Co-authored-by: Huy Tran <119860259+tranquochuy645@users.noreply.github.com> * Dev (#1740) * Nate/control plane client (#1691) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * fix(gui): ctx rendering w/ renderInlineAs: "" (#1541) * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * fix codeqwen autocomplete leading space * add voyage rerank-1 * feat: `--noEmit` for tsc checks in CI (#1559) * docs: update CustomContextProvider docs (#1557) * add stop tokens to qwen prompt * update docs to reflect 3.5 sonnet being best * docs: comment out unused providers (#1561) * import Handlebars * feat: toast notification for config updates (#1560) * feat: toast notification for config updates * feat: only trigger toast on config.json save * displayRawMarkdown option * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * update to reflect 16 stop words limit for deepseek * feat: only trigger config update toast in vscode (#1571) * docs(prompt-files): fix typos + clarify (#1575) * doc: prompt file typo + clarifications * fix: add back correct docs * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * docs(telemetry): add pageviews to tracking list (#1581) * Add reranker configuration options to codebase embedding docs (#1584) - Introduce reranker concept - List available reranker options - Provide configuration instructions - Update keywords to include "reranker" * chore: update pr template with screenshots (#1590) * Refactor ConfirmationDialog to use SecondaryButton for cancel action (#1586) * Added instructions for running docs server locally (#1578) - Added NPM script method - Added VS Code task method - Update contributing guidelines * Update branch policy (#1577) - Change PR target to `dev` branch - Update `CONTRIBUTING.md` instructions * Consolidate example configurations into the main configuration guide (#1579) - Moved examples to configuration.md - Deleted the separate examples.md file - Updated sidebar order and links - Improved readability and structure in configuration.md * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * Update completionProvider.ts (warning tab-autocomplete models) (#1566) * feat: enhanced IndexingProgressBar with blinking dot feature - Integrated BlinkingDot component - Added STATUS_COLORS for various states - Replaced CircleDiv with BlinkingDot in UI - Updated status messages and layout * small UI tweaks * feat(gui): enhance ModelCard, ModelProviderTag, and Toggle components (#1595) - add styling and adjustments to ModelCard - update ModelProviderTag font size - remove box shadow from Toggle component - tweak icon styles in ModelCard - improve alignment and spacing * media query * feat: add best experience onboarding * fix: file rename * stop movement on button hover by keeping same border thickness * fix mistake in setting cursor: pointer * fix when free trial option is shown * Support Node.js versions below 20 for streaming response handling (#1591) - Add fallback for Node < 20 - Implement toAsyncIterable for streaming - Use TextDecoder for manual decoding - Maintain existing streaming for Node 20+ * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * fix hasPassedFTL * fix /edit cancellation from UI * feat: add `applyCodeBlock` experimental prop (#1601) * feat: add new model styling improvements (#1609) * feat: add new model styling improvements * better gap size * feat: update bug_report.yml (#1610) * chore: update bug_report.yml * typo fix * feat: add labels to "Add docs" dialog (#1612) * feat: add labels to "Add docs" dialog * remove autofocus * don't double load config * small fixes * speed up directory traversal, and use correct native path module * option not to show config update toast * merge air-gapped and recommended setup pages * chore: add telemetry for full screen toggle (#1618) * Fix headings in codebase-embeddings.md (#1617) * mention jetbrains * docs: update changie (#1619) * feat: updated changie config * hide toc and autogenerate * Update changelog.mdx * link to deeper explanation of embeddings models * ensure target="_blank" for all links in sidebar * fix gif links in intellij README.md * don't require rust in dependency installation * chore: fix padding on gh button (#1620) * chore: adjust button padding * Update tasks.json * escape colons in diff path * smoother lancedb indexing reporting * smooth progress updates for indexing * fix tsc err * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * docs: add docs and schema for "OS" provider (#1536) * ignore .env * 🚑 fix constant warnings when onboarding with Ollama * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * team analytics * apply control plane settings * workos auth * ide protocol get session info * UI for auth * profile switching * small fixes * updates * refresh tokens * updates * fix tsc errs * model select in toolbar to make room for profile selector * prod client id * link to prod URL * internal beta option * profiles change listener --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * temporary patch for type errs * Dev (#1689) * Fix an issue where CMD+K does not clear the terminal when the terminal has focus (#1671) On MacOS, ⌘+K is bound, by default, to Terminal:Clear. Without this change ⌘+K does not clear the terminal but instead iniates a chord sequence and waits for the next stroke of the chord. Co-authored-by: Rob Leidle <rleidle@tesla.com> * Change treeSitter to cache the Language objects it loads from wasm (#1672) Without this change, for a repository with 600 typescript files, the indexer would fail to finish correctly and there would be many of the following errors in the webview console log: 'Unable to load language for file ${path} RuntimeError: table index is out of bounds' The following bash will create a repo that reproduces the problem: current_path="." for ((i=1; i<=20; i++)); do new_folder="folder-$i" mkdir -p "$current_path/$new_folder" current_path="$current_path/$new_folder" for ((a=1; a<=30; a++)); do head -c 10000 /dev/urandom | base64 > "$current_path/file-$a.ts" done done Co-authored-by: Rob Leidle <rleidle@tesla.com> * acknowledge sourcemap flag in esbuild.js * don't run jetbrains-release.yaml on vscode releases * further testing for walkDir * chore: add telemetry to commands (#1673) * test: Add basic unit test to baseLLM (#1668) * update version * test: Add basic unit test to baseLLM --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: inimaz <imaz@ai-verse.com> * feat: add Quick Actions CodeLens feature (#1674) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * add voyage rerank-1 * import Handlebars * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * small UI tweaks * media query * feat: add best experience onboarding * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * chore: add telemetry for full screen toggle (#1618) * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * manually implement stop tokens for hf inference api * chore: onboarding metrics (#1626) * fix: pageview tracking * feat: add onboarding telemetry * create single `onboardingStatus` type * improved var naming * remove console logs * fix windows performance issue * rename vscodeExtension.ts * migration of onboarding variables * "stash" instead of "delete" in indexing progress * fix preview.yaml * also fix main.yaml * Update troubleshooting.md (#1637) * feat: add quick actions * Update index.d.ts * quick actions mvp * update docs * subscribe to vscode change settings * Update commands.ts * cleanup * Update quick-actions.md * Update VerticalPerLineCodeLensProvider.ts * resolve feedback --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: Nate Sesti <33237525+sestinj@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> * chore: add `isCommandEvent` to command telemetry (#1675) * chore: add `isCommandEvent` to command telemetry * Update commands.ts * Nate/better retrieval (#1677) * deduplicatearray tests * break out separate retrieval pipelines * IConfigHandler * tests for codebase indexer * better .continueignore for continue * indexing fixes * ignore .gitignore and .continueignore when indexing * retrieval pipeline improvements * fix formatting err in out .continueignore * add necessary filter to lance_db_cache * update package.json version * skip unused tests * don't ignore .prompt files * update version * Update pull_request_template.md * don't use multi-media format when there are multiple text items * add free trial experience (#1685) * fix: add code range for quick actions/fixes (#1687) * fix: add code range for quick actions/fixes * Update test.js * add pathSep message type * docs improvements * jetbrains fix * update package.json version --------- Co-authored-by: Rob Leidle <rleidle@gmail.com> Co-authored-by: Rob Leidle <rleidle@tesla.com> Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: inimaz <49730431+inimaz@users.noreply.github.com> Co-authored-by: inimaz <imaz@ai-verse.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> * update package.json version * skip extension tests * update package.json version * handlebars import * don't push package.json version change from CI * fix: scroll issues w/ code blocks (#1688) * fix: scroll issues w/ code blocks * chore: name offset * chore: remove docs start * chore: update headings on troubleshooting.md (#1696) * ignore .svn folder for indexing (#1699) * chore: clearer naming for `useSuffix` (#1702) * chore: clearer naming for `useSuffix` * schema updates * fix: cmd+shft+l closes sidebar if focused (#1638) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * add voyage rerank-1 * import Handlebars * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * small UI tweaks * media query * feat: add best experience onboarding * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * chore: add telemetry for full screen toggle (#1618) * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * manually implement stop tokens for hf inference api * chore: onboarding metrics (#1626) * fix: pageview tracking * feat: add onboarding telemetry * create single `onboardingStatus` type * improved var naming * remove console logs * fix windows performance issue * rename vscodeExtension.ts * migration of onboarding variables * "stash" instead of "delete" in indexing progress * fix preview.yaml * also fix main.yaml * Update troubleshooting.md (#1637) * feat: close panel if main input is focused * add skip param * Update TipTapEditor.tsx * merge dev * Update commands.ts --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: Nate Sesti <33237525+sestinj@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> * bugfix: quick actions undefined array (#1704) * feat: add rich quick pick for quick edit (#1706) * feat: multistep quick pick * fix: multi step for context provider * Update test.js * Create deepseek.md (#1708) * style: add max height to model selector (#1712) * update styling * remove testing code * remove comments * feat: add tutorial card (#1716) * feat: create tutorial card * Update TutorialCard.tsx * feat: add file search to quick edit (#1714) * feat: multistep quick pick * fix: multi step for context provider * Update test.js * add file search * complete file search * remove unused import * add comments * Update utils.ts * update naming * chore: move quick pick instantiation (#1723) * Break out into NPM modules (#1726) * config.json types * export all schemas * break out request logic into packages * update openai adapters to export body types * update packaging of npm modules * update packages * continue proxy * mistral * Fix typo in documentation (#1722) In set-up-codestral.md "Trobleshooting" -> "Troubleshooting" * update package.json version * 🪄 model dropdown, control plane (#1692) * Nate/control plane client (#1691) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * fix(gui): ctx rendering w/ renderInlineAs: "" (#1541) * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * fix codeqwen autocomplete leading space * add voyage rerank-1 * feat: `--noEmit` for tsc checks in CI (#1559) * docs: update CustomContextProvider docs (#1557) * add stop tokens to qwen prompt * update docs to reflect 3.5 sonnet being best * docs: comment out unused providers (#1561) * import Handlebars * feat: toast notification for config updates (#1560) * feat: toast notification for config updates * feat: only trigger toast on config.json save * displayRawMarkdown option * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * update to reflect 16 stop words limit for deepseek * feat: only trigger config update toast in vscode (#1571) * docs(prompt-files): fix typos + clarify (#1575) * doc: prompt file typo + clarifications * fix: add back correct docs * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * docs(telemetry): add pageviews to tracking list (#1581) * Add reranker configuration options to codebase embedding docs (#1584) - Introduce reranker concept - List available reranker options - Provide configuration instructions - Update keywords to include "reranker" * chore: update pr template with screenshots (#1590) * Refactor ConfirmationDialog to use SecondaryButton for cancel action (#1586) * Added instructions for running docs server locally (#1578) - Added NPM script method - Added VS Code task method - Update contributing guidelines * Update branch policy (#1577) - Change PR target to `dev` branch - Update `CONTRIBUTING.md` instructions * Consolidate example configurations into the main configuration guide (#1579) - Moved examples to configuration.md - Deleted the separate examples.md file - Updated sidebar order and links - Improved readability and structure in configuration.md * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * Update completionProvider.ts (warning tab-autocomplete models) (#1566) * feat: enhanced IndexingProgressBar with blinking dot feature - Integrated BlinkingDot component - Added STATUS_COLORS for various states - Replaced CircleDiv with BlinkingDot in UI - Updated status messages and layout * small UI tweaks * feat(gui): enhance ModelCard, ModelProviderTag, and Toggle components (#1595) - add styling and adjustments to ModelCard - update ModelProviderTag font size - remove box shadow from Toggle component - tweak icon styles in ModelCard - improve alignment and spacing * media query * feat: add best experience onboarding * fix: file rename * stop movement on button hover by keeping same border thickness * fix mistake in setting cursor: pointer * fix when free trial option is shown * Support Node.js versions below 20 for streaming response handling (#1591) - Add fallback for Node < 20 - Implement toAsyncIterable for streaming - Use TextDecoder for manual decoding - Maintain existing streaming for Node 20+ * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * fix hasPassedFTL * fix /edit cancellation from UI * feat: add `applyCodeBlock` experimental prop (#1601) * feat: add new model styling improvements (#1609) * feat: add new model styling improvements * better gap size * feat: update bug_report.yml (#1610) * chore: update bug_report.yml * typo fix * feat: add labels to "Add docs" dialog (#1612) * feat: add labels to "Add docs" dialog * remove autofocus * don't double load config * small fixes * speed up directory traversal, and use correct native path module * option not to show config update toast * merge air-gapped and recommended setup pages * chore: add telemetry for full screen toggle (#1618) * Fix headings in codebase-embeddings.md (#1617) * mention jetbrains * docs: update changie (#1619) * feat: updated changie config * hide toc and autogenerate * Update changelog.mdx * link to deeper explanation of embeddings models * ensure target="_blank" for all links in sidebar * fix gif links in intellij README.md * don't require rust in dependency installation * chore: fix padding on gh button (#1620) * chore: adjust button padding * Update tasks.json * escape colons in diff path * smoother lancedb indexing reporting * smooth progress updates for indexing * fix tsc err * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * docs: add docs and schema for "OS" provider (#1536) * ignore .env * 🚑 fix constant warnings when onboarding with Ollama * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * team analytics * apply control plane settings * workos auth * ide protocol get session info * UI for auth * profile switching * small fixes * updates * refresh tokens * updates * fix tsc errs * model select in toolbar to make room for profile selector * prod client id * link to prod URL * internal beta option * profiles change listener --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * temporary patch for type errs --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * skip extension tests * JetBrains status bar spinner for autocomplete * Remove unused images * partial accept jetbrains * fix double-rendering of jetbrains completions * fix offset of multi-line jetbrains completions * enable/disable actions * upgrade to version 0.0.54 * fix: wrap editor buttons when out of space (#1727) * Add .continuerc file to config directory on activation to prevent it's indexing. (#1733) * Updated the content links that were deprecated in the document (#1709) Co-authored-by: Ihe Fan <ih.fan.1027@gmail.com> * Nate/dev (#1739) * JetBrains status bar spinner for autocomplete * Remove unused images * partial accept jetbrains * fix double-rendering of jetbrains completions * fix offset of multi-line jetbrains completions * enable/disable actions * upgrade to version 0.0.54 * llm-info package * small profile switching improvements * instruct cmd+I not to leave placeholders * @ files in context-providers docs * onboarding fix * update jetbrains ci to build gui * feat: single default quick pick to edit (#1743) * feat: single default quick pick to edit * Update quick-actions.md * Update quick-actions.md * use defaultQuickAction * quick fixes * fix: handle line removal diff (#1744) * fix: handle line removal diff * remove comments * Configurable `maxChunkSize` for embedding providers (#1746) * Add `maxChunkSize` to `EmbeddingsProvider` * Add `maxChunkSize` to schema * add support for docs via config (#1594) * mistral mamba (#1748) * 🐍 mistral mamba * autoscale retrieval based on context length * autoscale retrieval based on context length * fix input box movement * update jetbrains build --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> Co-authored-by: Rob Leidle <rleidle@gmail.com> Co-authored-by: Rob Leidle <rleidle@tesla.com> Co-authored-by: inimaz <49730431+inimaz@users.noreply.github.com> Co-authored-by: inimaz <imaz@ai-verse.com> Co-authored-by: Anaël Bonnafous <71386173+AnaelBonnafous@users.noreply.github.com> Co-authored-by: Huy Tran <119860259+tranquochuy645@users.noreply.github.com> Co-authored-by: Gabriel Gordbegli <ggordbegli@gmail.com> Co-authored-by: Raven-1027 <83693755+Raven-1027@users.noreply.github.com> Co-authored-by: Ihe Fan <ih.fan.1027@gmail.com> Co-authored-by: Lukas Kreussel <65088241+LLukas22@users.noreply.github.com> * update jetbrains build * jetbrains build update * Dev (#1780) * Nate/control plane client (#1691) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * fix(gui): ctx rendering w/ renderInlineAs: "" (#1541) * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * fix codeqwen autocomplete leading space * add voyage rerank-1 * feat: `--noEmit` for tsc checks in CI (#1559) * docs: update CustomContextProvider docs (#1557) * add stop tokens to qwen prompt * update docs to reflect 3.5 sonnet being best * docs: comment out unused providers (#1561) * import Handlebars * feat: toast notification for config updates (#1560) * feat: toast notification for config updates * feat: only trigger toast on config.json save * displayRawMarkdown option * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * update to reflect 16 stop words limit for deepseek * feat: only trigger config update toast in vscode (#1571) * docs(prompt-files): fix typos + clarify (#1575) * doc: prompt file typo + clarifications * fix: add back correct docs * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * docs(telemetry): add pageviews to tracking list (#1581) * Add reranker configuration options to codebase embedding docs (#1584) - Introduce reranker concept - List available reranker options - Provide configuration instructions - Update keywords to include "reranker" * chore: update pr template with screenshots (#1590) * Refactor ConfirmationDialog to use SecondaryButton for cancel action (#1586) * Added instructions for running docs server locally (#1578) - Added NPM script method - Added VS Code task method - Update contributing guidelines * Update branch policy (#1577) - Change PR target to `dev` branch - Update `CONTRIBUTING.md` instructions * Consolidate example configurations into the main configuration guide (#1579) - Moved examples to configuration.md - Deleted the separate examples.md file - Updated sidebar order and links - Improved readability and structure in configuration.md * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * Update completionProvider.ts (warning tab-autocomplete models) (#1566) * feat: enhanced IndexingProgressBar with blinking dot feature - Integrated BlinkingDot component - Added STATUS_COLORS for various states - Replaced CircleDiv with BlinkingDot in UI - Updated status messages and layout * small UI tweaks * feat(gui): enhance ModelCard, ModelProviderTag, and Toggle components (#1595) - add styling and adjustments to ModelCard - update ModelProviderTag font size - remove box shadow from Toggle component - tweak icon styles in ModelCard - improve alignment and spacing * media query * feat: add best experience onboarding * fix: file rename * stop movement on button hover by keeping same border thickness * fix mistake in setting cursor: pointer * fix when free trial option is shown * Support Node.js versions below 20 for streaming response handling (#1591) - Add fallback for Node < 20 - Implement toAsyncIterable for streaming - Use TextDecoder for manual decoding - Maintain existing streaming for Node 20+ * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * fix hasPassedFTL * fix /edit cancellation from UI * feat: add `applyCodeBlock` experimental prop (#1601) * feat: add new model styling improvements (#1609) * feat: add new model styling improvements * better gap size * feat: update bug_report.yml (#1610) * chore: update bug_report.yml * typo fix * feat: add labels to "Add docs" dialog (#1612) * feat: add labels to "Add docs" dialog * remove autofocus * don't double load config * small fixes * speed up directory traversal, and use correct native path module * option not to show config update toast * merge air-gapped and recommended setup pages * chore: add telemetry for full screen toggle (#1618) * Fix headings in codebase-embeddings.md (#1617) * mention jetbrains * docs: update changie (#1619) * feat: updated changie config * hide toc and autogenerate * Update changelog.mdx * link to deeper explanation of embeddings models * ensure target="_blank" for all links in sidebar * fix gif links in intellij README.md * don't require rust in dependency installation * chore: fix padding on gh button (#1620) * chore: adjust button padding * Update tasks.json * escape colons in diff path * smoother lancedb indexing reporting * smooth progress updates for indexing * fix tsc err * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * docs: add docs and schema for "OS" provider (#1536) * ignore .env * 🚑 fix constant warnings when onboarding with Ollama * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * team analytics * apply control plane settings * workos auth * ide protocol get session info * UI for auth * profile switching * small fixes * updates * refresh tokens * updates * fix tsc errs * model select in toolbar to make room for profile selector * prod client id * link to prod URL * internal beta option * profiles change listener --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * temporary patch for type errs * Dev (#1689) * Fix an issue where CMD+K does not clear the terminal when the terminal has focus (#1671) On MacOS, ⌘+K is bound, by default, to Terminal:Clear. Without this change ⌘+K does not clear the terminal but instead iniates a chord sequence and waits for the next stroke of the chord. Co-authored-by: Rob Leidle <rleidle@tesla.com> * Change treeSitter to cache the Language objects it loads from wasm (#1672) Without this change, for a repository with 600 typescript files, the indexer would fail to finish correctly and there would be many of the following errors in the webview console log: 'Unable to load language for file ${path} RuntimeError: table index is out of bounds' The following bash will create a repo that reproduces the problem: current_path="." for ((i=1; i<=20; i++)); do new_folder="folder-$i" mkdir -p "$current_path/$new_folder" current_path="$current_path/$new_folder" for ((a=1; a<=30; a++)); do head -c 10000 /dev/urandom | base64 > "$current_path/file-$a.ts" done done Co-authored-by: Rob Leidle <rleidle@tesla.com> * acknowledge sourcemap flag in esbuild.js * don't run jetbrains-release.yaml on vscode releases * further testing for walkDir * chore: add telemetry to commands (#1673) * test: Add basic unit test to baseLLM (#1668) * update version * test: Add basic unit test to baseLLM --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: inimaz <imaz@ai-verse.com> * feat: add Quick Actions CodeLens feature (#1674) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * add voyage rerank-1 * import Handlebars * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * small UI tweaks * media query * feat: add best experience onboarding * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * chore: add telemetry for full screen toggle (#1618) * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * manually implement stop tokens for hf inference api * chore: onboarding metrics (#1626) * fix: pageview tracking * feat: add onboarding telemetry * create single `onboardingStatus` type * improved var naming * remove console logs * fix windows performance issue * rename vscodeExtension.ts * migration of onboarding variables * "stash" instead of "delete" in indexing progress * fix preview.yaml * also fix main.yaml * Update troubleshooting.md (#1637) * feat: add quick actions * Update index.d.ts * quick actions mvp * update docs * subscribe to vscode change settings * Update commands.ts * cleanup * Update quick-actions.md * Update VerticalPerLineCodeLensProvider.ts * resolve feedback --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: Nate Sesti <33237525+sestinj@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> * chore: add `isCommandEvent` to command telemetry (#1675) * chore: add `isCommandEvent` to command telemetry * Update commands.ts * Nate/better retrieval (#1677) * deduplicatearray tests * break out separate retrieval pipelines * IConfigHandler * tests for codebase indexer * better .continueignore for continue * indexing fixes * ignore .gitignore and .continueignore when indexing * retrieval pipeline improvements * fix formatting err in out .continueignore * add necessary filter to lance_db_cache * update package.json version * skip unused tests * don't ignore .prompt files * update version * Update pull_request_template.md * don't use multi-media format when there are multiple text items * add free trial experience (#1685) * fix: add code range for quick actions/fixes (#1687) * fix: add code range for quick actions/fixes * Update test.js * add pathSep message type * docs improvements * jetbrains fix * update package.json version --------- Co-authored-by: Rob Leidle <rleidle@gmail.com> Co-authored-by: Rob Leidle <rleidle@tesla.com> Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: inimaz <49730431+inimaz@users.noreply.github.com> Co-authored-by: inimaz <imaz@ai-verse.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> * update package.json version * skip extension tests * update package.json version * handlebars import * don't push package.json version change from CI * fix: scroll issues w/ code blocks (#1688) * fix: scroll issues w/ code blocks * chore: name offset * chore: remove docs start * chore: update headings on troubleshooting.md (#1696) * ignore .svn folder for indexing (#1699) * chore: clearer naming for `useSuffix` (#1702) * chore: clearer naming for `useSuffix` * schema updates * fix: cmd+shft+l closes sidebar if focused (#1638) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * add voyage rerank-1 * import Handlebars * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * small UI tweaks * media query * feat: add best experience onboarding * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * chore: add telemetry for full screen toggle (#1618) * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * manually implement stop tokens for hf inference api * chore: onboarding metrics (#1626) * fix: pageview tracking * feat: add onboarding telemetry * create single `onboardingStatus` type * improved var naming * remove console logs * fix windows performance issue * rename vscodeExtension.ts * migration of onboarding variables * "stash" instead of "delete" in indexing progress * fix preview.yaml * also fix main.yaml * Update troubleshooting.md (#1637) * feat: close panel if main input is focused * add skip param * Update TipTapEditor.tsx * merge dev * Update commands.ts --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: Nate Sesti <33237525+sestinj@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> * bugfix: quick actions undefined array (#1704) * feat: add rich quick pick for quick edit (#1706) * feat: multistep quick pick * fix: multi step for context provider * Update test.js * Create deepseek.md (#1708) * style: add max height to model selector (#1712) * update styling * remove testing code * remove comments * feat: add tutorial card (#1716) * feat: create tutorial card * Update TutorialCard.tsx * feat: add file search to quick edit (#1714) * feat: multistep quick pick * fix: multi step for context provider * Update test.js * add file search * complete file search * remove unused import * add comments * Update utils.ts * update naming * chore: move quick pick instantiation (#1723) * Break out into NPM modules (#1726) * config.json types * export all schemas * break out request logic into packages * update openai adapters to export body types * update packaging of npm modules * update packages * continue proxy * mistral * Fix typo in documentation (#1722) In set-up-codestral.md "Trobleshooting" -> "Troubleshooting" * update package.json version * 🪄 model dropdown, control plane (#1692) * Nate/control plane client (#1691) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * fix(gui): ctx rendering w/ renderInlineAs: "" (#1541) * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * fix codeqwen autocomplete leading space * add voyage rerank-1 * feat: `--noEmit` for tsc checks in CI (#1559) * docs: update CustomContextProvider docs (#1557) * add stop tokens to qwen prompt * update docs to reflect 3.5 sonnet being best * docs: comment out unused providers (#1561) * import Handlebars * feat: toast notification for config updates (#1560) * feat: toast notification for config updates * feat: only trigger toast on config.json save * displayRawMarkdown option * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * update to reflect 16 stop words limit for deepseek * feat: only trigger config update toast in vscode (#1571) * docs(prompt-files): fix typos + clarify (#1575) * doc: prompt file typo + clarifications * fix: add back correct docs * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * docs(telemetry): add pageviews to tracking list (#1581) * Add reranker configuration options to codebase embedding docs (#1584) - Introduce reranker concept - List available reranker options - Provide configuration instructions - Update keywords to include "reranker" * chore: update pr template with screenshots (#1590) * Refactor ConfirmationDialog to use SecondaryButton for cancel action (#1586) * Added instructions for running docs server locally (#1578) - Added NPM script method - Added VS Code task method - Update contributing guidelines * Update branch policy (#1577) - Change PR target to `dev` branch - Update `CONTRIBUTING.md` instructions * Consolidate example configurations into the main configuration guide (#1579) - Moved examples to configuration.md - Deleted the separate examples.md file - Updated sidebar order and links - Improved readability and structure in configuration.md * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * Update completionProvider.ts (warning tab-autocomplete models) (#1566) * feat: enhanced IndexingProgressBar with blinking dot feature - Integrated BlinkingDot component - Added STATUS_COLORS for various states - Replaced CircleDiv with BlinkingDot in UI - Updated status messages and layout * small UI tweaks * feat(gui): enhance ModelCard, ModelProviderTag, and Toggle components (#1595) - add styling and adjustments to ModelCard - update ModelProviderTag font size - remove box shadow from Toggle component - tweak icon styles in ModelCard - improve alignment and spacing * media query * feat: add best experience onboarding * fix: file rename * stop movement on button hover by keeping same border thickness * fix mistake in setting cursor: pointer * fix when free trial option is shown * Support Node.js versions below 20 for streaming response handling (#1591) - Add fallback for Node < 20 - Implement toAsyncIterable for streaming - Use TextDecoder for manual decoding - Maintain existing streaming for Node 20+ * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * fix hasPassedFTL * fix /edit cancellation from UI * feat: add `applyCodeBlock` experimental prop (#1601) * feat: add new model styling improvements (#1609) * feat: add new model styling improvements * better gap size * feat: update bug_report.yml (#1610) * chore: update bug_report.yml * typo fix * feat: add labels to "Add docs" dialog (#1612) * feat: add labels to "Add docs" dialog * remove autofocus * don't double load config * small fixes * speed up directory traversal, and use correct native path module * option not to show config update toast * merge air-gapped and recommended setup pages * chore: add telemetry for full screen toggle (#1618) * Fix headings in codebase-embeddings.md (#1617) * mention jetbrains * docs: update changie (#1619) * feat: updated changie config * hide toc and autogenerate * Update changelog.mdx * link to deeper explanation of embeddings models * ensure target="_blank" for all links in sidebar * fix gif links in intellij README.md * don't require rust in dependency installation * chore: fix padding on gh button (#1620) * chore: adjust button padding * Update tasks.json * escape colons in diff path * smoother lancedb indexing reporting * smooth progress updates for indexing * fix tsc err * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * docs: add docs and schema for "OS" provider (#1536) * ignore .env * 🚑 fix constant warnings when onboarding with Ollama * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * team analytics * apply control plane settings * workos auth * ide protocol get session info * UI for auth * profile switching * small fixes * updates * refresh tokens * updates * fix tsc errs * model select in toolbar to make room for profile selector * prod client id * link to prod URL * internal beta option * profiles change listener --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * temporary patch for type errs --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * skip extension tests * JetBrains status bar spinner for autocomplete * Remove unused images * partial accept jetbrains * fix double-rendering of jetbrains completions * fix offset of multi-line jetbrains completions * enable/disable actions * upgrade to version 0.0.54 * fix: wrap editor buttons when out of space (#1727) * Add .continuerc file to config directory on activation to prevent it's indexing. (#1733) * Updated the content links that were deprecated in the document (#1709) Co-authored-by: Ihe Fan <ih.fan.1027@gmail.com> * Nate/dev (#1739) * JetBrains status bar spinner for autocomplete * Remove unused images * partial accept jetbrains * fix double-rendering of jetbrains completions * fix offset of multi-line jetbrains completions * enable/disable actions * upgrade to version 0.0.54 * llm-info package * small profile switching improvements * instruct cmd+I not to leave placeholders * @ files in context-providers docs * onboarding fix * update jetbrains ci to build gui * feat: single default quick pick to edit (#1743) * feat: single default quick pick to edit * Update quick-actions.md * Update quick-actions.md * use defaultQuickAction * quick fixes * fix: handle line removal diff (#1744) * fix: handle line removal diff * remove comments * Configurable `maxChunkSize` for embedding providers (#1746) * Add `maxChunkSize` to `EmbeddingsProvider` * Add `maxChunkSize` to schema * add support for docs via config (#1594) * mistral mamba (#1748) * 🐍 mistral mamba * autoscale retrieval based on context length * autoscale retrieval based on context length * fix input box movement * update jetbrains build * test: add myers diff tests (#1754) * feat: enhance help center (#1755) * feat: enhance help center * Update help.tsx * feat: add azure provider config (#1764) * autocomplete reload fix * logout of control plane * Update QuickEditQuickPick.ts (#1772) * Update QuickEditQuickPick.ts * updates * Update QuickEditQuickPick.ts * pass refreshed access token to continue-proxy model * test: add dir read check for walkDir (#1773) * test: add dir read check for walkDir * Update ignore.ts * Update walkDir.test.ts * Update walkDir.test.ts * gpt-4o-mini * gpt-4o-mini * gpt4omini to ui selector * fix: layout alignment (#1779) * fix: delete old docs index on force re-index (#1778) * add "Codebase Force Re-Index" command (#1757) * Add `HuggingFaceTEIReranker` (#1711) * Add `HuggingFaceTEIReranker` * Add `huggingface-tei` to `config_schema.json` * update package.json version * update config schemas --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> Co-authored-by: Rob Leidle <rleidle@gmail.com> Co-authored-by: Rob Leidle <rleidle@tesla.com> Co-authored-by: inimaz <49730431+inimaz@users.noreply.github.com> Co-authored-by: inimaz <imaz@ai-verse.com> Co-authored-by: Anaël Bonnafous <71386173+AnaelBonnafous@users.noreply.github.com> Co-authored-by: Huy Tran <119860259+tranquochuy645@users.noreply.github.com> Co-authored-by: Gabriel Gordbegli <ggordbegli@gmail.com> Co-authored-by: Raven-1027 <83693755+Raven-1027@users.noreply.github.com> Co-authored-by: Ihe Fan <ih.fan.1027@gmail.com> Co-authored-by: Lukas Kreussel <65088241+LLukas22@users.noreply.github.com> * fix tests (#1781) * Nate/control plane client (#1691) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * fix(gui): ctx rendering w/ renderInlineAs: "" (#1541) * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * fix codeqwen autocomplete leading space * add voyage rerank-1 * feat: `--noEmit` for tsc checks in CI (#1559) * docs: update CustomContextProvider docs (#1557) * add stop tokens to qwen prompt * update docs to reflect 3.5 sonnet being best * docs: comment out unused providers (#1561) * import Handlebars * feat: toast notification for config updates (#1560) * feat: toast notification for config updates * feat: only trigger toast on config.json save * displayRawMarkdown option * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * update to reflect 16 stop words limit for deepseek * feat: only trigger config update toast in vscode (#1571) * docs(prompt-files): fix typos + clarify (#1575) * doc: prompt file typo + clarifications * fix: add back correct docs * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * docs(telemetry): add pageviews to tracking list (#1581) * Add reranker configuration options to codebase embedding docs (#1584) - Introduce reranker concept - List available reranker options - Provide configuration instructions - Update keywords to include "reranker" * chore: update pr template with screenshots (#1590) * Refactor ConfirmationDialog to use SecondaryButton for cancel action (#1586) * Added instructions for running docs server locally (#1578) - Added NPM script method - Added VS Code task method - Update contributing guidelines * Update branch policy (#1577) - Change PR target to `dev` branch - Update `CONTRIBUTING.md` instructions * Consolidate example configurations into the main configuration guide (#1579) - Moved examples to configuration.md - Deleted the separate examples.md file - Updated sidebar order and links - Improved readability and structure in configuration.md * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * Update completionProvider.ts (warning tab-autocomplete models) (#1566) * feat: enhanced IndexingProgressBar with blinking dot feature - Integrated BlinkingDot component - Added STATUS_COLORS for various states - Replaced CircleDiv with BlinkingDot in UI - Updated status messages and layout * small UI tweaks * feat(gui): enhance ModelCard, ModelProviderTag, and Toggle components (#1595) - add styling and adjustments to ModelCard - update ModelProviderTag font size - remove box shadow from Toggle component - tweak icon styles in ModelCard - improve alignment and spacing * media query * feat: add best experience onboarding * fix: file rename * stop movement on button hover by keeping same border thickness * fix mistake in setting cursor: pointer * fix when free trial option is shown * Support Node.js versions below 20 for streaming response handling (#1591) - Add fallback for Node < 20 - Implement toAsyncIterable for streaming - Use TextDecoder for manual decoding - Maintain existing streaming for Node 20+ * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * fix hasPassedFTL * fix /edit cancellation from UI * feat: add `applyCodeBlock` experimental prop (#1601) * feat: add new model styling improvements (#1609) * feat: add new model styling improvements * better gap size * feat: update bug_report.yml (#1610) * chore: update bug_report.yml * typo fix * feat: add labels to "Add docs" dialog (#1612) * feat: add labels to "Add docs" dialog * remove autofocus * don't double load config * small fixes * speed up directory traversal, and use correct native path module * option not to show config update toast * merge air-gapped and recommended setup pages * chore: add telemetry for full screen toggle (#1618) * Fix headings in codebase-embeddings.md (#1617) * mention jetbrains * docs: update changie (#1619) * feat: updated changie config * hide toc and autogenerate * Update changelog.mdx * link to deeper explanation of embeddings models * ensure target="_blank" for all links in sidebar * fix gif links in intellij README.md * don't require rust in dependency installation * chore: fix padding on gh button (#1620) * chore: adjust button padding * Update tasks.json * escape colons in diff path * smoother lancedb indexing reporting * smooth progress updates for indexing * fix tsc err * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * docs: add docs and schema for "OS" provider (#1536) * ignore .env * 🚑 fix constant warnings when onboarding with Ollama * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * team analytics * apply control plane settings * workos auth * ide protocol get session info * UI for auth * profile switching * small fixes * updates * refresh tokens * updates * fix tsc errs * model select in toolbar to make room for profile selector * prod client id * link to prod URL * internal beta option * profiles change listener --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * temporary patch for type errs * Dev (#1689) * Fix an issue where CMD+K does not clear the terminal when the terminal has focus (#1671) On MacOS, ⌘+K is bound, by default, to Terminal:Clear. Without this change ⌘+K does not clear the terminal but instead iniates a chord sequence and waits for the next stroke of the chord. Co-authored-by: Rob Leidle <rleidle@tesla.com> * Change treeSitter to cache the Language objects it loads from wasm (#1672) Without this change, for a repository with 600 typescript files, the indexer would fail to finish correctly and there would be many of the following errors in the webview console log: 'Unable to load language for file ${path} RuntimeError: table index is out of bounds' The following bash will create a repo that reproduces the problem: current_path="." for ((i=1; i<=20; i++)); do new_folder="folder-$i" mkdir -p "$current_path/$new_folder" current_path="$current_path/$new_folder" for ((a=1; a<=30; a++)); do head -c 10000 /dev/urandom | base64 > "$current_path/file-$a.ts" done done Co-authored-by: Rob Leidle <rleidle@tesla.com> * acknowledge sourcemap flag in esbuild.js * don't run jetbrains-release.yaml on vscode releases * further testing for walkDir * chore: add telemetry to commands (#1673) * test: Add basic unit test to baseLLM (#1668) * update version * test: Add basic unit test to baseLLM --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: inimaz <imaz@ai-verse.com> * feat: add Quick Actions CodeLens feature (#1674) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * add voyage rerank-1 * import Handlebars * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * small UI tweaks * media query * feat: add best experience onboarding * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * chore: add telemetry for full screen toggle (#1618) * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * manually implement stop tokens for hf inference api * chore: onboarding metrics (#1626) * fix: pageview tracking * feat: add onboarding telemetry * create single `onboardingStatus` type * improved var naming * remove console logs * fix windows performance issue * rename vscodeExtension.ts * migration of onboarding variables * "stash" instead of "delete" in indexing progress * fix preview.yaml * also fix main.yaml * Update troubleshooting.md (#1637) * feat: add quick actions * Update index.d.ts * quick actions mvp * update docs * subscribe to vscode change settings * Update commands.ts * cleanup * Update quick-actions.md * Update VerticalPerLineCodeLensProvider.ts * resolve feedback --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: Nate Sesti <33237525+sestinj@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> * chore: add `isCommandEvent` to command telemetry (#1675) * chore: add `isCommandEvent` to command telemetry * Update commands.ts * Nate/better retrieval (#1677) * deduplicatearray tests * break out separate retrieval pipelines * IConfigHandler * tests for codebase indexer * better .continueignore for continue * indexing fixes * ignore .gitignore and .continueignore when indexing * retrieval pipeline improvements * fix formatting err in out .continueignore * add necessary filter to lance_db_cache * update package.json version * skip unused tests * don't ignore .prompt files * update version * Update pull_request_template.md * don't use multi-media format when there are multiple text items * add free trial experience (#1685) * fix: add code range for quick actions/fixes (#1687) * fix: add code range for quick actions/fixes * Update test.js * add pathSep message type * docs improvements * jetbrains fix * update package.json version --------- Co-authored-by: Rob Leidle <rleidle@gmail.com> Co-authored-by: Rob Leidle <rleidle@tesla.com> Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: inimaz <49730431+inimaz@users.noreply.github.com> Co-authored-by: inimaz <imaz@ai-verse.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> * update package.json version * skip extension tests * update package.json version * handlebars import * don't push package.json version change from CI * fix: scroll issues w/ code blocks (#1688) * fix: scroll issues w/ code blocks * chore: name offset * chore: remove docs start * chore: update headings on troubleshooting.md (#1696) * ignore .svn folder for indexing (#1699) * chore: clearer naming for `useSuffix` (#1702) * chore: clearer naming for `useSuffix` * schema updates * fix: cmd+shft+l closes sidebar if focused (#1638) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * add voyage rerank-1 * import Handlebars * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * small UI tweaks * media query * feat: add best experience onboarding * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * chore: add telemetry for full screen toggle (#1618) * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * manually implement stop tokens for hf inference api * chore: onboarding metrics (#1626) * fix: pageview tracking * feat: add onboarding telemetry * create single `onboardingStatus` type * improved var naming * remove console logs * fix windows performance issue * rename vscodeExtension.ts * migration of onboarding variables * "stash" instead of "delete" in indexing progress * fix preview.yaml * also fix main.yaml * Update troubleshooting.md (#1637) * feat: close panel if main input is focused * add skip param * Update TipTapEditor.tsx * merge dev * Update commands.ts --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: Nate Sesti <33237525+sestinj@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> * bugfix: quick actions undefined array (#1704) * feat: add rich quick pick for quick edit (#1706) * feat: multistep quick pick * fix: multi step for context provider * Update test.js * Create deepseek.md (#1708) * style: add max height to model selector (#1712) * update styling * remove testing code * remove comments * feat: add tutorial card (#1716) * feat: create tutorial card * Update TutorialCard.tsx * feat: add file search to quick edit (#1714) * feat: multistep quick pick * fix: multi step for context provider * Update test.js * add file search * complete file search * remove unused import * add comments * Update utils.ts * update naming * chore: move quick pick instantiation (#1723) * Break out into NPM modules (#1726) * config.json types * export all schemas * break out request logic into packages * update openai adapters to export body types * update packaging of npm modules * update packages * continue proxy * mistral * Fix typo in documentation (#1722) In set-up-codestral.md "Trobleshooting" -> "Troubleshooting" * update package.json version * 🪄 model dropdown, control plane (#1692) * Nate/control plane client (#1691) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * fix(gui): ctx rendering w/ renderInlineAs: "" (#1541) * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * fix codeqwen autocomplete leading space * add voyage rerank-1 * feat: `--noEmit` for tsc checks in CI (#1559) * docs: update CustomContextProvider docs (#1557) * add stop tokens to qwen prompt * update docs to reflect 3.5 sonnet being best * docs: comment out unused providers (#1561) * import Handlebars * feat: toast notification for config updates (#1560) * feat: toast notification for config updates * feat: only trigger toast on config.json save * displayRawMarkdown option * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * update to reflect 16 stop words limit for deepseek * feat: only trigger config update toast in vscode (#1571) * docs(prompt-files): fix typos + clarify (#1575) * doc: prompt file typo + clarifications * fix: add back correct docs * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * docs(telemetry): add pageviews to tracking list (#1581) * Add reranker configuration options to codebase embedding docs (#1584) - Introduce reranker concept - List available reranker options - Provide configuration instructions - Update keywords to include "reranker" * chore: update pr template with screenshots (#1590) * Refactor ConfirmationDialog to use SecondaryButton for cancel action (#1586) * Added instructions for running docs server locally (#1578) - Added NPM script method - Added VS Code task method - Update contributing guidelines * Update branch policy (#1577) - Change PR target to `dev` branch - Update `CONTRIBUTING.md` instructions * Consolidate example configurations into the main configuration guide (#1579) - Moved examples to configuration.md - Deleted the separate examples.md file - Updated sidebar order and links - Improved readability and structure in configuration.md * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * Update completionProvider.ts (warning tab-autocomplete models) (#1566) * feat: enhanced IndexingProgressBar with blinking dot feature - Integrated BlinkingDot component - Added STATUS_COLORS for various states - Replaced CircleDiv with BlinkingDot in UI - Updated status messages and layout * small UI tweaks * feat(gui): enhance ModelCard, ModelProviderTag, and Toggle components (#1595) - add styling and adjustments to ModelCard - update ModelProviderTag font size - remove box shadow from Toggle component - tweak icon styles in ModelCard - improve alignment and spacing * media query * feat: add best experience onboarding * fix: file rename * stop movement on button hover by keeping same border thickness * fix mistake in setting cursor: pointer * fix when free trial option is shown * Support Node.js versions below 20 for streaming response handling (#1591) - Add fallback for Node < 20 - Implement toAsyncIterable for streaming - Use TextDecoder for manual decoding - Maintain existing streaming for Node 20+ * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * fix hasPassedFTL * fix /edit cancellation from UI * feat: add `applyCodeBlock` experimental prop (#1601) * feat: add new model styling improvements (#1609) * feat: add new model styling improvements * better gap size * feat: update bug_report.yml (#1610) * chore: update bug_report.yml * typo fix * feat: add labels to "Add docs" dialog (#1612) * feat: add labels to "Add docs" dialog * remove autofocus * don't double load config * small fixes * speed up directory traversal, and use correct native path module * option not to show config update toast * merge air-gapped and recommended setup pages * chore: add telemetry for full screen toggle (#1618) * Fix headings in codebase-embeddings.md (#1617) * mention jetbrains * docs: update changie (#1619) * feat: updated changie config * hide toc and autogenerate * Update changelog.mdx * link to deeper explanation of embeddings models * ensure target="_blank" for all links in sidebar * fix gif links in intellij README.md * don't require rust in dependency installation * chore: fix padding on gh button (#1620) * chore: adjust button padding * Update tasks.json * escape colons in diff path * smoother lancedb indexing reporting * smooth progress updates for indexing * fix tsc err * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * docs: add docs and schema for "OS" provider (#1536) * ignore .env * 🚑 fix constant warnings when onboarding with Ollama * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * team analytics * apply control plane settings * workos auth * ide protocol get session info * UI for auth * profile switching * small fixes * updates * refresh tokens * updates * fix tsc errs * model select in toolbar to make room for profile selector * prod client id * link to prod URL * internal beta option * profiles change listener --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * temporary patch for type errs --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * skip extension tests * JetBrains status bar spinner for autocomplete * Remove unused images * partial accept jetbrains * fix double-rendering of jetbrains completions * fix offset of multi-line jetbrains completions * enable/disable actions * upgrade to version 0.0.54 * fix: wrap editor buttons when out of space (#1727) * Add .continuerc file to config directory on activation to prevent it's indexing. (#1733) * Updated the content links that were deprecated in the document (#1709) Co-authored-by: Ihe Fan <ih.fan.1027@gmail.com> * Nate/dev (#1739) * JetBrains status bar spinner for autocomplete * Remove unused images * partial accept jetbrains * fix double-rendering of jetbrains completions * fix offset of multi-line jetbrains completions * enable/disable actions * upgrade to version 0.0.54 * llm-info package * small profile switching improvements * instruct cmd+I not to leave placeholders * @ files in context-providers docs * onboarding fix * update jetbrains ci to build gui * feat: single default quick pick to edit (#1743) * feat: single default quick pick to edit * Update quick-actions.md * Update quick-actions.md * use defaultQuickAction * quick fixes * fix: handle line removal diff (#1744) * fix: handle line removal diff * remove comments * Configurable `maxChunkSize` for embedding providers (#1746) * Add `maxChunkSize` to `EmbeddingsProvider` * Add `maxChunkSize` to schema * add support for docs via config (#1594) * mistral mamba (#1748) * 🐍 mistral mamba * autoscale retrieval based on context length * autoscale retrieval based on context length * fix input box movement * update jetbrains build * test: add myers diff tests (#1754) * feat: enhance help center (#1755) * feat: enhance help center * Update help.tsx * feat: add azure provider config (#1764) * autocomplete reload fix * logout of control plane * Update QuickEditQuickPick.ts (#1772) * Update QuickEditQuickPick.ts * updates * Update QuickEditQuickPick.ts * pass refreshed access token to continue-proxy model * test: add dir read check for walkDir (#1773) * test: add dir read check for walkDir * Update ignore.ts * Update walkDir.test.ts * Update walkDir.test.ts * gpt-4o-mini * gpt-4o-mini * gpt4omini to ui selector * fix: layout alignment (#1779) * fix: delete old docs index on force re-index (#1778) * add "Codebase Force Re-Index" command (#1757) * Add `HuggingFaceTEIReranker` (#1711) * Add `HuggingFaceTEIReranker` * Add `huggingface-tei` to `config_schema.json` * update package.json version * update config schemas * skip test failing only due to type error --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> Co-authored-by: Rob Leidle <rleidle@gmail.com> Co-authored-by: Rob Leidle <rleidle@tesla.com> Co-authored-by: inimaz <49730431+inimaz@users.noreply.github.com> Co-authored-by: inimaz <imaz@ai-verse.com> Co-authored-by: Anaël Bonnafous <71386173+AnaelBonnafous@users.noreply.github.com> Co-authored-by: Huy Tran <119860259+tranquochuy645@users.noreply.github.com> Co-authored-by: Gabriel Gordbegli <ggordbegli@gmail.com> Co-authored-by: Raven-1027 <83693755+Raven-1027@users.noreply.github.com> Co-authored-by: Ihe Fan <ih.fan.1027@gmail.com> Co-authored-by: Lukas Kreussel <65088241+LLukas22@users.noreply.github.com> * Dev (#1784) * Nate/control plane client (#1691) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * fix(gui): ctx rendering w/ renderInlineAs: "" (#1541) * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * fix codeqwen autocomplete leading space * add voyage rerank-1 * feat: `--noEmit` for tsc checks in CI (#1559) * docs: update CustomContextProvider docs (#1557) * add stop tokens to qwen prompt * update docs to reflect 3.5 sonnet being best * docs: comment out unused providers (#1561) * import Handlebars * feat: toast notification for config updates (#1560) * feat: toast notification for config updates * feat: only trigger toast on config.json save * displayRawMarkdown option * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * update to reflect 16 stop words limit for deepseek * feat: only trigger config update toast in vscode (#1571) * docs(prompt-files): fix typos + clarify (#1575) * doc: prompt file typo + clarifications * fix: add back correct docs * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * docs(telemetry): add pageviews to tracking list (#1581) * Add reranker configuration options to codebase embedding docs (#1584) - Introduce reranker concept - List available reranker options - Provide configuration instructions - Update keywords to include "reranker" * chore: update pr template with screenshots (#1590) * Refactor ConfirmationDialog to use SecondaryButton for cancel action (#1586) * Added instructions for running docs server locally (#1578) - Added NPM script method - Added VS Code task method - Update contributing guidelines * Update branch policy (#1577) - Change PR target to `dev` branch - Update `CONTRIBUTING.md` instructions * Consolidate example configurations into the main configuration guide (#1579) - Moved examples to configuration.md - Deleted the separate examples.md file - Updated sidebar order and links - Improved readability and structure in configuration.md * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * Update completionProvider.ts (warning tab-autocomplete models) (#1566) * feat: enhanced IndexingProgressBar with blinking dot feature - Integrated BlinkingDot component - Added STATUS_COLORS for various states - Replaced CircleDiv with BlinkingDot in UI - Updated status messages and layout * small UI tweaks * feat(gui): enhance ModelCard, ModelProviderTag, and Toggle components (#1595) - add styling and adjustments to ModelCard - update ModelProviderTag font size - remove box shadow from Toggle component - tweak icon styles in ModelCard - improve alignment and spacing * media query * feat: add best experience onboarding * fix: file rename * stop movement on button hover by keeping same border thickness * fix mistake in setting cursor: pointer * fix when free trial option is shown * Support Node.js versions below 20 for streaming response handling (#1591) - Add fallback for Node < 20 - Implement toAsyncIterable for streaming - Use TextDecoder for manual decoding - Maintain existing streaming for Node 20+ * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * fix hasPassedFTL * fix /edit cancellation from UI * feat: add `applyCodeBlock` experimental prop (#1601) * feat: add new model styling improvements (#1609) * feat: add new model styling improvements * better gap size * feat: update bug_report.yml (#1610) * chore: update bug_report.yml * typo fix * feat: add labels to "Add docs" dialog (#1612) * feat: add labels to "Add docs" dialog * remove autofocus * don't double load config * small fixes * speed up directory traversal, and use correct native path module * option not to show config update toast * merge air-gapped and recommended setup pages * chore: add telemetry for full screen toggle (#1618) * Fix headings in codebase-embeddings.md (#1617) * mention jetbrains * docs: update changie (#1619) * feat: updated changie config * hide toc and autogenerate * Update changelog.mdx * link to deeper explanation of embeddings models * ensure target="_blank" for all links in sidebar * fix gif links in intellij README.md * don't require rust in dependency installation * chore: fix padding on gh button (#1620) * chore: adjust button padding * Update tasks.json * escape colons in diff path * smoother lancedb indexing reporting * smooth progress updates for indexing * fix tsc err * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * docs: add docs and schema for "OS" provider (#1536) * ignore .env * 🚑 fix constant warnings when onboarding with Ollama * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * team analytics * apply control plane settings * workos auth * ide protocol get session info * UI for auth * profile switching * small fixes * updates * refresh tokens * updates * fix tsc errs * model select in toolbar to make room for profile selector * prod client id * link to prod URL * internal beta option * profiles change listener --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * temporary patch for type errs * Dev (#1689) * Fix an issue where CMD+K does not clear the terminal when the terminal has focus (#1671) On MacOS, ⌘+K is bound, by default, to Terminal:Clear. Without this change ⌘+K does not clear the terminal but instead iniates a chord sequence and waits for the next stroke of the chord. Co-authored-by: Rob Leidle <rleidle@tesla.com> * Change treeSitter to cache the Language objects it loads from wasm (#1672) Without this change, for a repository with 600 typescript files, the indexer would fail to finish correctly and there would be many of the following errors in the webview console log: 'Unable to load language for file ${path} RuntimeError: table index is out of bounds' The following bash will create a repo that reproduces the problem: current_path="." for ((i=1; i<=20; i++)); do new_folder="folder-$i" mkdir -p "$current_path/$new_folder" current_path="$current_path/$new_folder" for ((a=1; a<=30; a++)); do head -c 10000 /dev/urandom | base64 > "$current_path/file-$a.ts" done done Co-authored-by: Rob Leidle <rleidle@tesla.com> * acknowledge sourcemap flag in esbuild.js * don't run jetbrains-release.yaml on vscode releases * further testing for walkDir * chore: add telemetry to commands (#1673) * test: Add basic unit test to baseLLM (#1668) * update version * test: Add basic unit test to baseLLM --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: inimaz <imaz@ai-verse.com> * feat: add Quick Actions CodeLens feature (#1674) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * add voyage rerank-1 * import Handlebars * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * small UI tweaks * media query * feat: add best experience onboarding * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * chore: add telemetry for full screen toggle (#1618) * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * manually implement stop tokens for hf inference api * chore: onboarding metrics (#1626) * fix: pageview tracking * feat: add onboarding telemetry * create single `onboardingStatus` type * improved var naming * remove console logs * fix windows performance issue * rename vscodeExtension.ts * migration of onboarding variables * "stash" instead of "delete" in indexing progress * fix preview.yaml * also fix main.yaml * Update troubleshooting.md (#1637) * feat: add quick actions * Update index.d.ts * quick actions mvp * update docs * subscribe to vscode change settings * Update commands.ts * cleanup * Update quick-actions.md * Update VerticalPerLineCodeLensProvider.ts * resolve feedback --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: Nate Sesti <33237525+sestinj@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> * chore: add `isCommandEvent` to command telemetry (#1675) * chore: add `isCommandEvent` to command telemetry * Update commands.ts * Nate/better retrieval (#1677) * deduplicatearray tests * break out separate retrieval pipelines * IConfigHandler * tests for codebase indexer * better .continueignore for continue * indexing fixes * ignore .gitignore and .continueignore when indexing * retrieval pipeline improvements * fix formatting err in out .continueignore * add necessary filter to lance_db_cache * update package.json version * skip unused tests * don't ignore .prompt files * update version * Update pull_request_template.md * don't use multi-media format when there are multiple text items * add free trial experience (#1685) * fix: add code range for quick actions/fixes (#1687) * fix: add code range for quick actions/fixes * Update test.js * add pathSep message type * docs improvements * jetbrains fix * update package.json version --------- Co-authored-by: Rob Leidle <rleidle@gmail.com> Co-authored-by: Rob Leidle <rleidle@tesla.com> Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: inimaz <49730431+inimaz@users.noreply.github.com> Co-authored-by: inimaz <imaz@ai-verse.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> * update package.json version * skip extension tests * update package.json version * handlebars import * don't push package.json version change from CI * fix: scroll issues w/ code blocks (#1688) * fix: scroll issues w/ code blocks * chore: name offset * chore: remove docs start * chore: update headings on troubleshooting.md (#1696) * ignore .svn folder for indexing (#1699) * chore: clearer naming for `useSuffix` (#1702) * chore: clearer naming for `useSuffix` * schema updates * fix: cmd+shft+l closes sidebar if focused (#1638) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * add voyage rerank-1 * import Handlebars * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * small UI tweaks * media query * feat: add best experience onboarding * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * chore: add telemetry for full screen toggle (#1618) * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * manually implement stop tokens for hf inference api * chore: onboarding metrics (#1626) * fix: pageview tracking * feat: add onboarding telemetry * create single `onboardingStatus` type * improved var naming * remove console logs * fix windows performance issue * rename vscodeExtension.ts * migration of onboarding variables * "stash" instead of "delete" in indexing progress * fix preview.yaml * also fix main.yaml * Update troubleshooting.md (#1637) * feat: close panel if main input is focused * add skip param * Update TipTapEditor.tsx * merge dev * Update commands.ts --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: Nate Sesti <33237525+sestinj@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> * bugfix: quick actions undefined array (#1704) * feat: add rich quick pick for quick edit (#1706) * feat: multistep quick pick * fix: multi step for context provider * Update test.js * Create deepseek.md (#1708) * style: add max height to model selector (#1712) * update styling * remove testing code * remove comments * feat: add tutorial card (#1716) * feat: create tutorial card * Update TutorialCard.tsx * feat: add file search to quick edit (#1714) * feat: multistep quick pick * fix: multi step for context provider * Update test.js * add file search * complete file search * remove unused import * add comments * Update utils.ts * update naming * chore: move quick pick instantiation (#1723) * Break out into NPM modules (#1726) * config.json types * export all schemas * break out request logic into packages * update openai adapters to export body types * update packaging of npm modules * update packages * continue proxy * mistral * Fix typo in documentation (#1722) In set-up-codestral.md "Trobleshooting" -> "Troubleshooting" * update package.json version * 🪄 model dropdown, control plane (#1692) * Nate/control plane client (#1691) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * fix(gui): ctx rendering w/ renderInlineAs: "" (#1541) * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * fix codeqwen autocomplete leading space * add voyage rerank-1 * feat: `--noEmit` for tsc checks in CI (#1559) * docs: update CustomContextProvider docs (#1557) * add stop tokens to qwen prompt * update docs to reflect 3.5 sonnet being best * docs: comment out unused providers (#1561) * import Handlebars * feat: toast notification for config updates (#1560) * feat: toast notification for config updates * feat: only trigger toast on config.json save * displayRawMarkdown option * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * update to reflect 16 stop words limit for deepseek * feat: only trigger config update toast in vscode (#1571) * docs(prompt-files): fix typos + clarify (#1575) * doc: prompt file typo + clarifications * fix: add back correct docs * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * docs(telemetry): add pageviews to tracking list (#1581) * Add reranker configuration options to codebase embedding docs (#1584) - Introduce reranker concept - List available reranker options - Provide configuration instructions - Update keywords to include "reranker" * chore: update pr template with screenshots (#1590) * Refactor ConfirmationDialog to use SecondaryButton for cancel action (#1586) * Added instructions for running docs server locally (#1578) - Added NPM script method - Added VS Code task method - Update contributing guidelines * Update branch policy (#1577) - Change PR target to `dev` branch - Update `CONTRIBUTING.md` instructions * Consolidate example configurations into the main configuration guide (#1579) - Moved examples to configuration.md - Deleted the separate examples.md file - Updated sidebar order and links - Improved readability and structure in configuration.md * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * Update completionProvider.ts (warning tab-autocomplete models) (#1566) * feat: enhanced IndexingProgressBar with blinking dot feature - Integrated BlinkingDot component - Added STATUS_COLORS for various states - Replaced CircleDiv with BlinkingDot in UI - Updated status messages and layout * small UI tweaks * feat(gui): enhance ModelCard, ModelProviderTag, and Toggle components (#1595) - add styling and adjustments to ModelCard - update ModelProviderTag font size - remove box shadow from Toggle component - tweak icon styles in ModelCard - improve alignment and spacing * media query * feat: add best experience onboarding * fix: file rename * stop movement on button hover by keeping same border thickness * fix mistake in setting cursor: pointer * fix when free trial option is shown * Support Node.js versions below 20 for streaming response handling (#1591) - Add fallback for Node < 20 - Implement toAsyncIterable for streaming - Use TextDecoder for manual decoding - Maintain existing streaming for Node 20+ * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * fix hasPassedFTL * fix /edit cancellation from UI * feat: add `applyCodeBlock` experimental prop (#1601) * feat: add new model styling improvements (#1609) * feat: add new model styling improvements * better gap size * feat: update bug_report.yml (#1610) * chore: update bug_report.yml * typo fix * feat: add labels to "Add docs" dialog (#1612) * feat: add labels to "Add docs" dialog * remove autofocus * don't double load config * small fixes * speed up directory traversal, and use correct native path module * option not to show config update toast * merge air-gapped and recommended setup pages * chore: add telemetry for full screen toggle (#1618) * Fix headings in codebase-embeddings.md (#1617) * mention jetbrains * docs: update changie (#1619) * feat: updated changie config * hide toc and autogenerate * Update changelog.mdx * link to deeper explanation of embeddings models * ensure target="_blank" for all links in sidebar * fix gif links in intellij README.md * don't require rust in dependency installation * chore: fix padding on gh button (#1620) * chore: adjust button padding * Update tasks.json * escape colons in diff path * smoother lancedb indexing reporting * smooth progress updates for indexing * fix tsc err * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * docs: add docs and schema for "OS" provider (#1536) * ignore .env * 🚑 fix constant warnings when onboarding with Ollama * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * team analytics * apply control plane settings * workos auth * ide protocol get session info * UI for auth * profile switching * small fixes * updates * refresh tokens * updates * fix tsc errs * model select in toolbar to make room for profile selector * prod client id * link to prod URL * internal beta option * profiles change listener --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * temporary patch for type errs --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * skip extension tests * JetBrains status bar spinner for autocomplete * Remove unused images * partial accept jetbrains * fix double-rendering of jetbrains completions * fix offset of multi-line jetbrains completions * enable/disable actions * upgrade to version 0.0.54 * fix: wrap editor buttons when out of space (#1727) * Add .continuerc file to config directory on activation to prevent it's indexing. (#1733) * Updated the content links that were deprecated in the document (#1709) Co-authored-by: Ihe Fan <ih.fan.1027@gmail.com> * Nate/dev (#1739) * JetBrains status bar spinner for autocomplete * Remove unused images * partial accept jetbrains * fix double-rendering of jetbrains completions * fix offset of multi-line jetbrains completions * enable/disable actions * upgrade to version 0.0.54 * llm-info package * small profile switching improvements * instruct cmd+I not to leave placeholders * @ files in context-providers docs * onboarding fix * update jetbrains ci to build gui * feat: single default quick pick to edit (#1743) * feat: single default quick pick to edit * Update quick-actions.md * Update quick-actions.md * use defaultQuickAction * quick fixes * fix: handle line removal diff (#1744) * fix: handle line removal diff * remove comments * Configurable `maxChunkSize` for embedding providers (#1746) * Add `maxChunkSize` to `EmbeddingsProvider` * Add `maxChunkSize` to schema * add support for docs via config (#1594) * mistral mamba (#1748) * 🐍 mistral mamba * autoscale retrieval based on context length * autoscale retrieval based on context length * fix input box movement * update jetbrains build * test: add myers diff tests (#1754) * feat: enhance help center (#1755) * feat: enhance help center * Update help.tsx * feat: add azure provider config (#1764) * autocomplete reload fix * logout of control plane * Update QuickEditQuickPick.ts (#1772) * Update QuickEditQuickPick.ts * updates * Update QuickEditQuickPick.ts * pass refreshed access token to continue-proxy model * test: add dir read check for walkDir (#1773) * test: add dir read check for walkDir * Update ignore.ts * Update walkDir.test.ts * Update walkDir.test.ts * gpt-4o-mini * gpt-4o-mini * gpt4omini to ui selector * fix: layout alignment (#1779) * fix: delete old docs index on force re-index (#1778) * add "Codebase Force Re-Index" command (#1757) * Add `HuggingFaceTEIReranker` (#1711) * Add `HuggingFaceTEIReranker` * Add `huggingface-tei` to `config_schema.json` * update package.json version * update config schemas * skip test failing only due to type error * fix: convert `walkDir` to an async generator (#1783) * walkdir to async generator * add yields * handle ignore files * Update CodebaseIndexer.ts * Update walkDir.ts * chore: bump VS Code ext version --------- Co-authored-by: Nate Sesti <33237525+sestinj@users.noreply.github.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: Rob Leidle <rleidle@gmail.com> Co-authored-by: Rob Leidle <rleidle@tesla.com> Co-authored-by: inimaz <49730431+inimaz@users.noreply.github.com> Co-authored-by: inimaz <imaz@ai-verse.com> Co-authored-by: Anaël Bonnafous <71386173+AnaelBonnafous@users.noreply.github.com> Co-authored-by: Huy Tran <119860259+tranquochuy645@users.noreply.github.com> Co-authored-by: Gabriel Gordbegli <ggordbegli@gmail.com> Co-authored-by: Raven-1027 <83693755+Raven-1027@users.noreply.github.com> Co-authored-by: Ihe Fan <ih.fan.1027@gmail.com> Co-authored-by: Lukas Kreussel <65088241+LLukas22@users.noreply.github.com> * test: skip failing tests * Dev (#1807) * Nate/control plane client (#1691) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * fix(gui): ctx rendering w/ renderInlineAs: "" (#1541) * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * fix codeqwen autocomplete leading space * add voyage rerank-1 * feat: `--noEmit` for tsc checks in CI (#1559) * docs: update CustomContextProvider docs (#1557) * add stop tokens to qwen prompt * update docs to reflect 3.5 sonnet being best * docs: comment out unused providers (#1561) * import Handlebars * feat: toast notification for config updates (#1560) * feat: toast notification for config updates * feat: only trigger toast on config.json save * displayRawMarkdown option * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * update to reflect 16 stop words limit for deepseek * feat: only trigger config update toast in vscode (#1571) * docs(prompt-files): fix typos + clarify (#1575) * doc: prompt file typo + clarifications * fix: add back correct docs * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * docs(telemetry): add pageviews to tracking list (#1581) * Add reranker configuration options to codebase embedding docs (#1584) - Introduce reranker concept - List available reranker options - Provide configuration instructions - Update keywords to include "reranker" * chore: update pr template with screenshots (#1590) * Refactor ConfirmationDialog to use SecondaryButton for cancel action (#1586) * Added instructions for running docs server locally (#1578) - Added NPM script method - Added VS Code task method - Update contributing guidelines * Update branch policy (#1577) - Change PR target to `dev` branch - Update `CONTRIBUTING.md` instructions * Consolidate example configurations into the main configuration guide (#1579) - Moved examples to configuration.md - Deleted the separate examples.md file - Updated sidebar order and links - Improved readability and structure in configuration.md * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * Update completionProvider.ts (warning tab-autocomplete models) (#1566) * feat: enhanced IndexingProgressBar with blinking dot feature - Integrated BlinkingDot component - Added STATUS_COLORS for various states - Replaced CircleDiv with BlinkingDot in UI - Updated status messages and layout * small UI tweaks * feat(gui): enhance ModelCard, ModelProviderTag, and Toggle components (#1595) - add styling and adjustments to ModelCard - update ModelProviderTag font size - remove box shadow from Toggle component - tweak icon styles in ModelCard - improve alignment and spacing * media query * feat: add best experience onboarding * fix: file rename * stop movement on button hover by keeping same border thickness * fix mistake in setting cursor: pointer * fix when free trial option is shown * Support Node.js versions below 20 for streaming response handling (#1591) - Add fallback for Node < 20 - Implement toAsyncIterable for streaming - Use TextDecoder for manual decoding - Maintain existing streaming for Node 20+ * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * fix hasPassedFTL * fix /edit cancellation from UI * feat: add `applyCodeBlock` experimental prop (#1601) * feat: add new model styling improvements (#1609) * feat: add new model styling improvements * better gap size * feat: update bug_report.yml (#1610) * chore: update bug_report.yml * typo fix * feat: add labels to "Add docs" dialog (#1612) * feat: add labels to "Add docs" dialog * remove autofocus * don't double load config * small fixes * speed up directory traversal, and use correct native path module * option not to show config update toast * merge air-gapped and recommended setup pages * chore: add telemetry for full screen toggle (#1618) * Fix headings in codebase-embeddings.md (#1617) * mention jetbrains * docs: update changie (#1619) * feat: updated changie config * hide toc and autogenerate * Update changelog.mdx * link to deeper explanation of embeddings models * ensure target="_blank" for all links in sidebar * fix gif links in intellij README.md * don't require rust in dependency installation * chore: fix padding on gh button (#1620) * chore: adjust button padding * Update tasks.json * escape colons in diff path * smoother lancedb indexing reporting * smooth progress updates for indexing * fix tsc err * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * docs: add docs and schema for "OS" provider (#1536) * ignore .env * 🚑 fix constant warnings when onboarding with Ollama * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * team analytics * apply control plane settings * workos auth * ide protocol get session info * UI for auth * profile switching * small fixes * updates * refresh tokens * updates * fix tsc errs * model select in toolbar to make room for profile selector * prod client id * link to prod URL * internal beta option * profiles change listener --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * temporary patch for type errs * Dev (#1689) * Fix an issue where CMD+K does not clear the terminal when the terminal has focus (#1671) On MacOS, ⌘+K is bound, by default, to Terminal:Clear. Without this change ⌘+K does not clear the terminal but instead iniates a chord sequence and waits for the next stroke of the chord. Co-authored-by: Rob Leidle <rleidle@tesla.com> * Change treeSitter to cache the Language objects it loads from wasm (#1672) Without this change, for a repository with 600 typescript files, the indexer would fail to finish correctly and there would be many of the following errors in the webview console log: 'Unable to load language for file ${path} RuntimeError: table index is out of bounds' The following bash will create a repo that reproduces the problem: current_path="." for ((i=1; i<=20; i++)); do new_folder="folder-$i" mkdir -p "$current_path/$new_folder" current_path="$current_path/$new_folder" for ((a=1; a<=30; a++)); do head -c 10000 /dev/urandom | base64 > "$current_path/file-$a.ts" done done Co-authored-by: Rob Leidle <rleidle@tesla.com> * acknowledge sourcemap flag in esbuild.js * don't run jetbrains-release.yaml on vscode releases * further testing for walkDir * chore: add telemetry to commands (#1673) * test: Add basic unit test to baseLLM (#1668) * update version * test: Add basic unit test to baseLLM --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: inimaz <imaz@ai-verse.com> * feat: add Quick Actions CodeLens feature (#1674) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * add voyage rerank-1 * import Handlebars * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * small UI tweaks * media query * feat: add best experience onboarding * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * chore: add telemetry for full screen toggle (#1618) * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * manually implement stop tokens for hf inference api * chore: onboarding metrics (#1626) * fix: pageview tracking * feat: add onboarding telemetry * create single `onboardingStatus` type * improved var naming * remove console logs * fix windows performance issue * rename vscodeExtension.ts * migration of onboarding variables * "stash" instead of "delete" in indexing progress * fix preview.yaml * also fix main.yaml * Update troubleshooting.md (#1637) * feat: add quick actions * Update index.d.ts * quick actions mvp * update docs * subscribe to vscode change settings * Update commands.ts * cleanup * Update quick-actions.md * Update VerticalPerLineCodeLensProvider.ts * resolve feedback --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: Nate Sesti <33237525+sestinj@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> * chore: add `isCommandEvent` to command telemetry (#1675) * chore: add `isCommandEvent` to command telemetry * Update commands.ts * Nate/better retrieval (#1677) * deduplicatearray tests * break out separate retrieval pipelines * IConfigHandler * tests for codebase indexer * better .continueignore for continue * indexing fixes * ignore .gitignore and .continueignore when indexing * retrieval pipeline improvements * fix formatting err in out .continueignore * add necessary filter to lance_db_cache * update package.json version * skip unused tests * don't ignore .prompt files * update version * Update pull_request_template.md * don't use multi-media format when there are multiple text items * add free trial experience (#1685) * fix: add code range for quick actions/fixes (#1687) * fix: add code range for quick actions/fixes * Update test.js * add pathSep message type * docs improvements * jetbrains fix * update package.json version --------- Co-authored-by: Rob Leidle <rleidle@gmail.com> Co-authored-by: Rob Leidle <rleidle@tesla.com> Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: inimaz <49730431+inimaz@users.noreply.github.com> Co-authored-by: inimaz <imaz@ai-verse.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> * update package.json version * skip extension tests * update package.json version * handlebars import * don't push package.json version change from CI * fix: scroll issues w/ code blocks (#1688) * fix: scroll issues w/ code blocks * chore: name offset * chore: remove docs start * chore: update headings on troubleshooting.md (#1696) * ignore .svn folder for indexing (#1699) * chore: clearer naming for `useSuffix` (#1702) * chore: clearer naming for `useSuffix` * schema updates * fix: cmd+shft+l closes sidebar if focused (#1638) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * add voyage rerank-1 * import Handlebars * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * small UI tweaks * media query * feat: add best experience onboarding * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * chore: add telemetry for full screen toggle (#1618) * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * manually implement stop tokens for hf inference api * chore: onboarding metrics (#1626) * fix: pageview tracking * feat: add onboarding telemetry * create single `onboardingStatus` type * improved var naming * remove console logs * fix windows performance issue * rename vscodeExtension.ts * migration of onboarding variables * "stash" instead of "delete" in indexing progress * fix preview.yaml * also fix main.yaml * Update troubleshooting.md (#1637) * feat: close panel if main input is focused * add skip param * Update TipTapEditor.tsx * merge dev * Update commands.ts --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: Nate Sesti <33237525+sestinj@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> * bugfix: quick actions undefined array (#1704) * feat: add rich quick pick for quick edit (#1706) * feat: multistep quick pick * fix: multi step for context provider * Update test.js * Create deepseek.md (#1708) * style: add max height to model selector (#1712) * update styling * remove testing code * remove comments * feat: add tutorial card (#1716) * feat: create tutorial card * Update TutorialCard.tsx * feat: add file search to quick edit (#1714) * feat: multistep quick pick * fix: multi step for context provider * Update test.js * add file search * complete file search * remove unused import * add comments * Update utils.ts * update naming * chore: move quick pick instantiation (#1723) * Break out into NPM modules (#1726) * config.json types * export all schemas * break out request logic into packages * update openai adapters to export body types * update packaging of npm modules * update packages * continue proxy * mistral * Fix typo in documentation (#1722) In set-up-codestral.md "Trobleshooting" -> "Troubleshooting" * update package.json version * 🪄 model dropdown, control plane (#1692) * Nate/control plane client (#1691) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * fix(gui): ctx rendering w/ renderInlineAs: "" (#1541) * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * fix codeqwen autocomplete leading space * add voyage rerank-1 * feat: `--noEmit` for tsc checks in CI (#1559) * docs: update CustomContextProvider docs (#1557) * add stop tokens to qwen prompt * update docs to reflect 3.5 sonnet being best * docs: comment out unused providers (#1561) * import Handlebars * feat: toast notification for config updates (#1560) * feat: toast notification for config updates * feat: only trigger toast on config.json save * displayRawMarkdown option * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * update to reflect 16 stop words limit for deepseek * feat: only trigger config update toast in vscode (#1571) * docs(prompt-files): fix typos + clarify (#1575) * doc: prompt file typo + clarifications * fix: add back correct docs * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * docs(telemetry): add pageviews to tracking list (#1581) * Add reranker configuration options to codebase embedding docs (#1584) - Introduce reranker concept - List available reranker options - Provide configuration instructions - Update keywords to include "reranker" * chore: update pr template with screenshots (#1590) * Refactor ConfirmationDialog to use SecondaryButton for cancel action (#1586) * Added instructions for running docs server locally (#1578) - Added NPM script method - Added VS Code task method - Update contributing guidelines * Update branch policy (#1577) - Change PR target to `dev` branch - Update `CONTRIBUTING.md` instructions * Consolidate example configurations into the main configuration guide (#1579) - Moved examples to configuration.md - Deleted the separate examples.md file - Updated sidebar order and links - Improved readability and structure in configuration.md * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * Update completionProvider.ts (warning tab-autocomplete models) (#1566) * feat: enhanced IndexingProgressBar with blinking dot feature - Integrated BlinkingDot component - Added STATUS_COLORS for various states - Replaced CircleDiv with BlinkingDot in UI - Updated status messages and layout * small UI tweaks * feat(gui): enhance ModelCard, ModelProviderTag, and Toggle components (#1595) - add styling and adjustments to ModelCard - update ModelProviderTag font size - remove box shadow from Toggle component - tweak icon styles in ModelCard - improve alignment and spacing * media query * feat: add best experience onboarding * fix: file rename * stop movement on button hover by keeping same border thickness * fix mistake in setting cursor: pointer * fix when free trial option is shown * Support Node.js versions below 20 for streaming response handling (#1591) - Add fallback for Node < 20 - Implement toAsyncIterable for streaming - Use TextDecoder for manual decoding - Maintain existing streaming for Node 20+ * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * fix hasPassedFTL * fix /edit cancellation from UI * feat: add `applyCodeBlock` experimental prop (#1601) * feat: add new model styling improvements (#1609) * feat: add new model styling improvements * better gap size * feat: update bug_report.yml (#1610) * chore: update bug_report.yml * typo fix * feat: add labels to "Add docs" dialog (#1612) * feat: add labels to "Add docs" dialog * remove autofocus * don't double load config * small fixes * speed up directory traversal, and use correct native path module * option not to show config update toast * merge air-gapped and recommended setup pages * chore: add telemetry for full screen toggle (#1618) * Fix headings in codebase-embeddings.md (#1617) * mention jetbrains * docs: update changie (#1619) * feat: updated changie config * hide toc and autogenerate * Update changelog.mdx * link to deeper explanation of embeddings models * ensure target="_blank" for all links in sidebar * fix gif links in intellij README.md * don't require rust in dependency installation * chore: fix padding on gh button (#1620) * chore: adjust button padding * Update tasks.json * escape colons in diff path * smoother lancedb indexing reporting * smooth progress updates for indexing * fix tsc err * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * docs: add docs and schema for "OS" provider (#1536) * ignore .env * 🚑 fix constant warnings when onboarding with Ollama * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * team analytics * apply control plane settings * workos auth * ide protocol get session info * UI for auth * profile switching * small fixes * updates * refresh tokens * updates * fix tsc errs * model select in toolbar to make room for profile selector * prod client id * link to prod URL * internal beta option * profiles change listener --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * temporary patch for type errs --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * skip extension tests * JetBrains status bar spinner for autocomplete * Remove unused images * partial accept jetbrains * fix double-rendering of jetbrains completions * fix offset of multi-line jetbrains completions * enable/disable actions * upgrade to version 0.0.54 * fix: wrap editor buttons when out of space (#1727) * Add .continuerc file to config directory on activation to prevent it's indexing. (#1733) * Updated the content links that were deprecated in the document (#1709) Co-authored-by: Ihe Fan <ih.fan.1027@gmail.com> * Nate/dev (#1739) * JetBrains status bar spinner for autocomplete * Remove unused images * partial accept jetbrains * fix double-rendering of jetbrains completions * fix offset of multi-line jetbrains completions * enable/disable actions * upgrade to version 0.0.54 * llm-info package * small profile switching improvements * instruct cmd+I not to leave placeholders * @ files in context-providers docs * onboarding fix * update jetbrains ci to build gui * feat: single default quick pick to edit (#1743) * feat: single default quick pick to edit * Update quick-actions.md * Update quick-actions.md * use defaultQuickAction * quick fixes * fix: handle line removal diff (#1744) * fix: handle line removal diff * remove comments * Configurable `maxChunkSize` for embedding providers (#1746) * Add `maxChunkSize` to `EmbeddingsProvider` * Add `maxChunkSize` to schema * add support for docs via config (#1594) * mistral mamba (#1748) * 🐍 mistral mamba * autoscale retrieval based on context length * autoscale retrieval based on context length * fix input box movement * update jetbrains build * test: add myers diff tests (#1754) * feat: enhance help center (#1755) * feat: enhance help center * Update help.tsx * feat: add azure provider config (#1764) * autocomplete reload fix * logout of control plane * Update QuickEditQuickPick.ts (#1772) * Update QuickEditQuickPick.ts * updates * Update QuickEditQuickPick.ts * pass refreshed access token to continue-proxy model * test: add dir read check for walkDir (#1773) * test: add dir read check for walkDir * Update ignore.ts * Update walkDir.test.ts * Update walkDir.test.ts * gpt-4o-mini * gpt-4o-mini * gpt4omini to ui selector * fix: layout alignment (#1779) * fix: delete old docs index on force re-index (#1778) * add "Codebase Force Re-Index" command (#1757) * Add `HuggingFaceTEIReranker` (#1711) * Add `HuggingFaceTEIReranker` * Add `huggingface-tei` to `config_schema.json` * update package.json version * update config schemas * skip test failing only due to type error * fix: convert `walkDir` to an async generator (#1783) * walkdir to async generator * add yields * handle ignore files * Update CodebaseIndexer.ts * Update walkDir.ts * chore: bump VS Code ext version * fix completions support in ContinueProxy * add test for walkDir * IAnalyticsProvider * feat: crawl `.mdx` docs * update workspaces on session info change * rename from profile to workspace (user-facing) * update config schema files * feat: improve chat thread ui for better readability (#1786) * add "language" template variable for autocomplete * Enable debug logs (#1800) * Add an enableDebugLogs flag * Send logs to VSCode * Refresh index fixes (#1795) * Use the resultType given to markComplete This fixes an issue where if there were multiple result types in itemToAction such as an update with UpdateNewVersion and UpdateOldVersion, only the UpdateOldVersion would be hit (which is a noop) and the UpdateNewVersion would never be fired, causing us to never update the timestamp of the tag_catalog. * Add a Compute and UpdateLastUpdated to refreshIndex Removes the ability to insert duplicate rows if we're just computing the result. Instead we replace the existing row instead. * Don't insert compute tags They are REPLACEd in markComplete now. * docs: change page heading to model providers (#1684) * feat: allow JetBrains users to index docs (#1797) * feat: allow JetBrains users to index docs * update docs service * update global context * throw error popup on jetbrains + transformers * update toast notifications * update typings * fix tsc errors * Index on file change (#1785) * Reload the submenu items when the context provider data is refreshed * update version * Just call refreshSubmenuItems * remove log statements * cleanup * Remove the Promise declaration * Copy currentFiles to a new dict since we're going to mutate it Don't mutate the caller's currentFiles unexpectedly * Remove reIndexFile and just call index/forceReIndex * We MUST await the refreshCodebaseIndex call * Expose send on core * remove unused import * Reindex the workspaces on save * Call core.send Ideally, we can implement externalRequest on the webview later so we don't have to expose messenger to the VSCode extension * Always refreshSubmenuItems in refreshCodebaseIndex * whitespace * Change messenger back to private * Add Pause Codebase On Start setting (#1788) * Add a skipInitialSync function * Add pauseInitialCodebaseIndex * docs * version bump * Rename to pauseCodebaseIndexOnStart * Un-shadow file context provider (#1801) * allow FileContextProvider to handle files * remove commented-out lines * chore: add telemetry to alt-enter (#1805) * Set the indexProgress indicator to paused on startup if pauseCodebaseIndexOnStart is set (#1804) * Fix Input Toolbar layout (#1792) * set to flex flex-column * Remove fixed height and always show the StyledDivs content to hold its size * Add a Delete Message button to the StepContainer (#1790) * Add a delete button * add a delete message action to the GUI * Fix off by one error * format * This index should be less than length * Update to trash icon * Simplify walkDir(...) and improve performance by roughly 10x in larger repos (#1806) The performance of WalkDir is improved by replacing the recursive DFS with a stack based implementation (less memory in-use during peak exploration). In addition, some unnecessary flags and options were removed which allowed the code to be overall simplified. Much less rule matching is occuring with the same files returned. Some performance results on my linux host: small repo: - 649 files returned by walkDir() - 9 .gitignore files - walkDir() old vs new duration: 918 ms to 143 ms medium sized repo: - 9271 returned by walkDir() - 35 .gitignore files - walkDir() old vs new duration: 10,886 ms to 970ms large sized repo: - 197,900 files returned by walkDir() - 929 .gitignore files - walkDir() old vs new duration: 498,151 ms to 37,353 ms Co-authored-by: Rob Leidle <rleidle@tesla.com> * modified assistant indicator in UI * update package.json version --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> Co-authored-by: Rob Leidle <rleidle@gmail.com> Co-authored-by: Rob Leidle <rleidle@tesla.com> Co-authored-by: inimaz <49730431+inimaz@users.noreply.github.com> Co-authored-by: inimaz <imaz@ai-verse.com> Co-authored-by: Anaël Bonnafous <71386173+AnaelBonnafous@users.noreply.github.com> Co-authored-by: Huy Tran <119860259+tranquochuy645@users.noreply.github.com> Co-authored-by: Gabriel Gordbegli <ggordbegli@gmail.com> Co-authored-by: Raven-1027 <83693755+Raven-1027@users.noreply.github.com> Co-authored-by: Ihe Fan <ih.fan.1027@gmail.com> Co-authored-by: Lukas Kreussel <65088241+LLukas22@users.noreply.github.com> Co-authored-by: Josh Vera <josh@joshvera.com> Co-authored-by: Peter Zaback <pzaback@gmail.com> * Improved walkDir performance (#1814) * Nate/control plane client (#1691) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * fix(gui): ctx rendering w/ renderInlineAs: "" (#1541) * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * fix codeqwen autocomplete leading space * add voyage rerank-1 * feat: `--noEmit` for tsc checks in CI (#1559) * docs: update CustomContextProvider docs (#1557) * add stop tokens to qwen prompt * update docs to reflect 3.5 sonnet being best * docs: comment out unused providers (#1561) * import Handlebars * feat: toast notification for config updates (#1560) * feat: toast notification for config updates * feat: only trigger toast on config.json save * displayRawMarkdown option * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * update to reflect 16 stop words limit for deepseek * feat: only trigger config update toast in vscode (#1571) * docs(prompt-files): fix typos + clarify (#1575) * doc: prompt file typo + clarifications * fix: add back correct docs * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * docs(telemetry): add pageviews to tracking list (#1581) * Add reranker configuration options to codebase embedding docs (#1584) - Introduce reranker concept - List available reranker options - Provide configuration instructions - Update keywords to include "reranker" * chore: update pr template with screenshots (#1590) * Refactor ConfirmationDialog to use SecondaryButton for cancel action (#1586) * Added instructions for running docs server locally (#1578) - Added NPM script method - Added VS Code task method - Update contributing guidelines * Update branch policy (#1577) - Change PR target to `dev` branch - Update `CONTRIBUTING.md` instructions * Consolidate example configurations into the main configuration guide (#1579) - Moved examples to configuration.md - Deleted the separate examples.md file - Updated sidebar order and links - Improved readability and structure in configuration.md * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * Update completionProvider.ts (warning tab-autocomplete models) (#1566) * feat: enhanced IndexingProgressBar with blinking dot feature - Integrated BlinkingDot component - Added STATUS_COLORS for various states - Replaced CircleDiv with BlinkingDot in UI - Updated status messages and layout * small UI tweaks * feat(gui): enhance ModelCard, ModelProviderTag, and Toggle components (#1595) - add styling and adjustments to ModelCard - update ModelProviderTag font size - remove box shadow from Toggle component - tweak icon styles in ModelCard - improve alignment and spacing * media query * feat: add best experience onboarding * fix: file rename * stop movement on button hover by keeping same border thickness * fix mistake in setting cursor: pointer * fix when free trial option is shown * Support Node.js versions below 20 for streaming response handling (#1591) - Add fallback for Node < 20 - Implement toAsyncIterable for streaming - Use TextDecoder for manual decoding - Maintain existing streaming for Node 20+ * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * fix hasPassedFTL * fix /edit cancellation from UI * feat: add `applyCodeBlock` experimental prop (#1601) * feat: add new model styling improvements (#1609) * feat: add new model styling improvements * better gap size * feat: update bug_report.yml (#1610) * chore: update bug_report.yml * typo fix * feat: add labels to "Add docs" dialog (#1612) * feat: add labels to "Add docs" dialog * remove autofocus * don't double load config * small fixes * speed up directory traversal, and use correct native path module * option not to show config update toast * merge air-gapped and recommended setup pages * chore: add telemetry for full screen toggle (#1618) * Fix headings in codebase-embeddings.md (#1617) * mention jetbrains * docs: update changie (#1619) * feat: updated changie config * hide toc and autogenerate * Update changelog.mdx * link to deeper explanation of embeddings models * ensure target="_blank" for all links in sidebar * fix gif links in intellij README.md * don't require rust in dependency installation * chore: fix padding on gh button (#1620) * chore: adjust button padding * Update tasks.json * escape colons in diff path * smoother lancedb indexing reporting * smooth progress updates for indexing * fix tsc err * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * docs: add docs and schema for "OS" provider (#1536) * ignore .env * 🚑 fix constant warnings when onboarding with Ollama * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * team analytics * apply control plane settings * workos auth * ide protocol get session info * UI for auth * profile switching * small fixes * updates * refresh tokens * updates * fix tsc errs * model select in toolbar to make room for profile selector * prod client id * link to prod URL * internal beta option * profiles change listener --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * temporary patch for type errs * Dev (#1689) * Fix an issue where CMD+K does not clear the terminal when the terminal has focus (#1671) On MacOS, ⌘+K is bound, by default, to Terminal:Clear. Without this change ⌘+K does not clear the terminal but instead iniates a chord sequence and waits for the next stroke of the chord. Co-authored-by: Rob Leidle <rleidle@tesla.com> * Change treeSitter to cache the Language objects it loads from wasm (#1672) Without this change, for a repository with 600 typescript files, the indexer would fail to finish correctly and there would be many of the following errors in the webview console log: 'Unable to load language for file ${path} RuntimeError: table index is out of bounds' The following bash will create a repo that reproduces the problem: current_path="." for ((i=1; i<=20; i++)); do new_folder="folder-$i" mkdir -p "$current_path/$new_folder" current_path="$current_path/$new_folder" for ((a=1; a<=30; a++)); do head -c 10000 /dev/urandom | base64 > "$current_path/file-$a.ts" done done Co-authored-by: Rob Leidle <rleidle@tesla.com> * acknowledge sourcemap flag in esbuild.js * don't run jetbrains-release.yaml on vscode releases * further testing for walkDir * chore: add telemetry to commands (#1673) * test: Add basic unit test to baseLLM (#1668) * update version * test: Add basic unit test to baseLLM --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: inimaz <imaz@ai-verse.com> * feat: add Quick Actions CodeLens feature (#1674) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * add voyage rerank-1 * import Handlebars * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * small UI tweaks * media query * feat: add best experience onboarding * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * chore: add telemetry for full screen toggle (#1618) * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * manually implement stop tokens for hf inference api * chore: onboarding metrics (#1626) * fix: pageview tracking * feat: add onboarding telemetry * create single `onboardingStatus` type * improved var naming * remove console logs * fix windows performance issue * rename vscodeExtension.ts * migration of onboarding variables * "stash" instead of "delete" in indexing progress * fix preview.yaml * also fix main.yaml * Update troubleshooting.md (#1637) * feat: add quick actions * Update index.d.ts * quick actions mvp * update docs * subscribe to vscode change settings * Update commands.ts * cleanup * Update quick-actions.md * Update VerticalPerLineCodeLensProvider.ts * resolve feedback --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: Nate Sesti <33237525+sestinj@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> * chore: add `isCommandEvent` to command telemetry (#1675) * chore: add `isCommandEvent` to command telemetry * Update commands.ts * Nate/better retrieval (#1677) * deduplicatearray tests * break out separate retrieval pipelines * IConfigHandler * tests for codebase indexer * better .continueignore for continue * indexing fixes * ignore .gitignore and .continueignore when indexing * retrieval pipeline improvements * fix formatting err in out .continueignore * add necessary filter to lance_db_cache * update package.json version * skip unused tests * don't ignore .prompt files * update version * Update pull_request_template.md * don't use multi-media format when there are multiple text items * add free trial experience (#1685) * fix: add code range for quick actions/fixes (#1687) * fix: add code range for quick actions/fixes * Update test.js * add pathSep message type * docs improvements * jetbrains fix * update package.json version --------- Co-authored-by: Rob Leidle <rleidle@gmail.com> Co-authored-by: Rob Leidle <rleidle@tesla.com> Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: inimaz <49730431+inimaz@users.noreply.github.com> Co-authored-by: inimaz <imaz@ai-verse.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> * update package.json version * skip extension tests * update package.json version * handlebars import * don't push package.json version change from CI * fix: scroll issues w/ code blocks (#1688) * fix: scroll issues w/ code blocks * chore: name offset * chore: remove docs start * chore: update headings on troubleshooting.md (#1696) * ignore .svn folder for indexing (#1699) * chore: clearer naming for `useSuffix` (#1702) * chore: clearer naming for `useSuffix` * schema updates * fix: cmd+shft+l closes sidebar if focused (#1638) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * add voyage rerank-1 * import Handlebars * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * small UI tweaks * media query * feat: add best experience onboarding * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * chore: add telemetry for full screen toggle (#1618) * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * manually implement stop tokens for hf inference api * chore: onboarding metrics (#1626) * fix: pageview tracking * feat: add onboarding telemetry * create single `onboardingStatus` type * improved var naming * remove console logs * fix windows performance issue * rename vscodeExtension.ts * migration of onboarding variables * "stash" instead of "delete" in indexing progress * fix preview.yaml * also fix main.yaml * Update troubleshooting.md (#1637) * feat: close panel if main input is focused * add skip param * Update TipTapEditor.tsx * merge dev * Update commands.ts --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: Nate Sesti <33237525+sestinj@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> * bugfix: quick actions undefined array (#1704) * feat: add rich quick pick for quick edit (#1706) * feat: multistep quick pick * fix: multi step for context provider * Update test.js * Create deepseek.md (#1708) * style: add max height to model selector (#1712) * update styling * remove testing code * remove comments * feat: add tutorial card (#1716) * feat: create tutorial card * Update TutorialCard.tsx * feat: add file search to quick edit (#1714) * feat: multistep quick pick * fix: multi step for context provider * Update test.js * add file search * complete file search * remove unused import * add comments * Update utils.ts * update naming * chore: move quick pick instantiation (#1723) * Break out into NPM modules (#1726) * config.json types * export all schemas * break out request logic into packages * update openai adapters to export body types * update packaging of npm modules * update packages * continue proxy * mistral * Fix typo in documentation (#1722) In set-up-codestral.md "Trobleshooting" -> "Troubleshooting" * update package.json version * 🪄 model dropdown, control plane (#1692) * Nate/control plane client (#1691) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * fix(gui): ctx rendering w/ renderInlineAs: "" (#1541) * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * fix codeqwen autocomplete leading space * add voyage rerank-1 * feat: `--noEmit` for tsc checks in CI (#1559) * docs: update CustomContextProvider docs (#1557) * add stop tokens to qwen prompt * update docs to reflect 3.5 sonnet being best * docs: comment out unused providers (#1561) * import Handlebars * feat: toast notification for config updates (#1560) * feat: toast notification for config updates * feat: only trigger toast on config.json save * displayRawMarkdown option * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * update to reflect 16 stop words limit for deepseek * feat: only trigger config update toast in vscode (#1571) * docs(prompt-files): fix typos + clarify (#1575) * doc: prompt file typo + clarifications * fix: add back correct docs * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * docs(telemetry): add pageviews to tracking list (#1581) * Add reranker configuration options to codebase embedding docs (#1584) - Introduce reranker concept - List available reranker options - Provide configuration instructions - Update keywords to include "reranker" * chore: update pr template with screenshots (#1590) * Refactor ConfirmationDialog to use SecondaryButton for cancel action (#1586) * Added instructions for running docs server locally (#1578) - Added NPM script method - Added VS Code task method - Update contributing guidelines * Update branch policy (#1577) - Change PR target to `dev` branch - Update `CONTRIBUTING.md` instructions * Consolidate example configurations into the main configuration guide (#1579) - Moved examples to configuration.md - Deleted the separate examples.md file - Updated sidebar order and links - Improved readability and structure in configuration.md * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * Update completionProvider.ts (warning tab-autocomplete models) (#1566) * feat: enhanced IndexingProgressBar with blinking dot feature - Integrated BlinkingDot component - Added STATUS_COLORS for various states - Replaced CircleDiv with BlinkingDot in UI - Updated status messages and layout * small UI tweaks * feat(gui): enhance ModelCard, ModelProviderTag, and Toggle components (#1595) - add styling and adjustments to ModelCard - update ModelProviderTag font size - remove box shadow from Toggle component - tweak icon styles in ModelCard - improve alignment and spacing * media query * feat: add best experience onboarding * fix: file rename * stop movement on button hover by keeping same border thickness * fix mistake in setting cursor: pointer * fix when free trial option is shown * Support Node.js versions below 20 for streaming response handling (#1591) - Add fallback for Node < 20 - Implement toAsyncIterable for streaming - Use TextDecoder for manual decoding - Maintain existing streaming for Node 20+ * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * fix hasPassedFTL * fix /edit cancellation from UI * feat: add `applyCodeBlock` experimental prop (#1601) * feat: add new model styling improvements (#1609) * feat: add new model styling improvements * better gap size * feat: update bug_report.yml (#1610) * chore: update bug_report.yml * typo fix * feat: add labels to "Add docs" dialog (#1612) * feat: add labels to "Add docs" dialog * remove autofocus * don't double load config * small fixes * speed up directory traversal, and use correct native path module * option not to show config update toast * merge air-gapped and recommended setup pages * chore: add telemetry for full screen toggle (#1618) * Fix headings in codebase-embeddings.md (#1617) * mention jetbrains * docs: update changie (#1619) * feat: updated changie config * hide toc and autogenerate * Update changelog.mdx * link to deeper explanation of embeddings models * ensure target="_blank" for all links in sidebar * fix gif links in intellij README.md * don't require rust in dependency installation * chore: fix padding on gh button (#1620) * chore: adjust button padding * Update tasks.json * escape colons in diff path * smoother lancedb indexing reporting * smooth progress updates for indexing * fix tsc err * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * docs: add docs and schema for "OS" provider (#1536) * ignore .env * 🚑 fix constant warnings when onboarding with Ollama * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * team analytics * apply control plane settings * workos auth * ide protocol get session info * UI for auth * profile switching * small fixes * updates * refresh tokens * updates * fix tsc errs * model select in toolbar to make room for profile selector * prod client id * link to prod URL * internal beta option * profiles change listener --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * temporary patch for type errs --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * skip extension tests * JetBrains status bar spinner for autocomplete * Remove unused images * partial accept jetbrains * fix double-rendering of jetbrains completions * fix offset of multi-line jetbrains completions * enable/disable actions * upgrade to version 0.0.54 * fix: wrap editor buttons when out of space (#1727) * Add .continuerc file to config directory on activation to prevent it's indexing. (#1733) * Updated the content links that were deprecated in the document (#1709) Co-authored-by: Ihe Fan <ih.fan.1027@gmail.com> * Nate/dev (#1739) * JetBrains status bar spinner for autocomplete * Remove unused images * partial accept jetbrains * fix double-rendering of jetbrains completions * fix offset of multi-line jetbrains completions * enable/disable actions * upgrade to version 0.0.54 * llm-info package * small profile switching improvements * instruct cmd+I not to leave placeholders * @ files in context-providers docs * onboarding fix * update jetbrains ci to build gui * feat: single default quick pick to edit (#1743) * feat: single default quick pick to edit * Update quick-actions.md * Update quick-actions.md * use defaultQuickAction * quick fixes * fix: handle line removal diff (#1744) * fix: handle line removal diff * remove comments * Configurable `maxChunkSize` for embedding providers (#1746) * Add `maxChunkSize` to `EmbeddingsProvider` * Add `maxChunkSize` to schema * add support for docs via config (#1594) * mistral mamba (#1748) * 🐍 mistral mamba * autoscale retrieval based on context length * autoscale retrieval based on context length * fix input box movement * update jetbrains build * test: add myers diff tests (#1754) * feat: enhance help center (#1755) * feat: enhance help center * Update help.tsx * feat: add azure provider config (#1764) * autocomplete reload fix * logout of control plane * Update QuickEditQuickPick.ts (#1772) * Update QuickEditQuickPick.ts * updates * Update QuickEditQuickPick.ts * pass refreshed access token to continue-proxy model * test: add dir read check for walkDir (#1773) * test: add dir read check for walkDir * Update ignore.ts * Update walkDir.test.ts * Update walkDir.test.ts * gpt-4o-mini * gpt-4o-mini * gpt4omini to ui selector * fix: layout alignment (#1779) * fix: delete old docs index on force re-index (#1778) * add "Codebase Force Re-Index" command (#1757) * Add `HuggingFaceTEIReranker` (#1711) * Add `HuggingFaceTEIReranker` * Add `huggingface-tei` to `config_schema.json` * update package.json version * update config schemas * skip test failing only due to type error * fix: convert `walkDir` to an async generator (#1783) * walkdir to async generator * add yields * handle ignore files * Update CodebaseIndexer.ts * Update walkDir.ts * chore: bump VS Code ext version * fix completions support in ContinueProxy * add test for walkDir * IAnalyticsProvider * feat: crawl `.mdx` docs * update workspaces on session info change * rename from profile to workspace (user-facing) * update config schema files * feat: improve chat thread ui for better readability (#1786) * add "language" template variable for autocomplete * Enable debug logs (#1800) * Add an enableDebugLogs flag * Send logs to VSCode * Refresh index fixes (#1795) * Use the resultType given to markComplete This fixes an issue where if there were multiple result types in itemToAction such as an update with UpdateNewVersion and UpdateOldVersion, only the UpdateOldVersion would be hit (which is a noop) and the UpdateNewVersion would never be fired, causing us to never update the timestamp of the tag_catalog. * Add a Compute and UpdateLastUpdated to refreshIndex Removes the ability to insert duplicate rows if we're just computing the result. Instead we replace the existing row instead. * Don't insert compute tags They are REPLACEd in markComplete now. * docs: change page heading to model providers (#1684) * feat: allow JetBrains users to index docs (#1797) * feat: allow JetBrains users to index docs * update docs service * update global context * throw error popup on jetbrains + transformers * update toast notifications * update typings * fix tsc errors * Index on file change (#1785) * Reload the submenu items when the context provider data is refreshed * update version * Just call refreshSubmenuItems * remove log statements * cleanup * Remove the Promise declaration * Copy currentFiles to a new dict since we're going to mutate it Don't mutate the caller's currentFiles unexpectedly * Remove reIndexFile and just call index/forceReIndex * We MUST await the refreshCodebaseIndex call * Expose send on core * remove unused import * Reindex the workspaces on save * Call core.send Ideally, we can implement externalRequest on the webview later so we don't have to expose messenger to the VSCode extension * Always refreshSubmenuItems in refreshCodebaseIndex * whitespace * Change messenger back to private * Add Pause Codebase On Start setting (#1788) * Add a skipInitialSync function * Add pauseInitialCodebaseIndex * docs * version bump * Rename to pauseCodebaseIndexOnStart * Un-shadow file context provider (#1801) * allow FileContextProvider to handle files * remove commented-out lines * chore: add telemetry to alt-enter (#1805) * Set the indexProgress indicator to paused on startup if pauseCodebaseIndexOnStart is set (#1804) * Fix Input Toolbar layout (#1792) * set to flex flex-column * Remove fixed height and always show the StyledDivs content to hold its size * Add a Delete Message button to the StepContainer (#1790) * Add a delete button * add a delete message action to the GUI * Fix off by one error * format * This index should be less than length * Update to trash icon * Simplify walkDir(...) and improve performance by roughly 10x in larger repos (#1806) The performance of WalkDir is improved by replacing the recursive DFS with a stack based implementation (less memory in-use during peak exploration). In addition, some unnecessary flags and options were removed which allowed the code to be overall simplified. Much less rule matching is occuring with the same files returned. Some performance results on my linux host: small repo: - 649 files returned by walkDir() - 9 .gitignore files - walkDir() old vs new duration: 918 ms to 143 ms medium sized repo: - 9271 returned by walkDir() - 35 .gitignore files - walkDir() old vs new duration: 10,886 ms to 970ms large sized repo: - 197,900 files returned by walkDir() - 929 .gitignore files - walkDir() old vs new duration: 498,151 ms to 37,353 ms Co-authored-by: Rob Leidle <rleidle@tesla.com> * modified assistant indicator in UI * update package.json version * feat: add Llama 3.1 8B to cloudflare provider options (#1811) Co-authored-by: jdelorey@cloudflare.com <jdelorey@cloudflare.com> * increase gemini pro contextLength (#1809) * fix indentation (#1808) * Llama 3.1 405b model selection (#1813) * add support for llama3 405b * 3.1 * name mapping * updates to model setup * fix walkDir tests on windows --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> Co-authored-by: Rob Leidle <rleidle@gmail.com> Co-authored-by: Rob Leidle <rleidle@tesla.com> Co-authored-by: inimaz <49730431+inimaz@users.noreply.github.com> Co-authored-by: inimaz <imaz@ai-verse.com> Co-authored-by: Anaël Bonnafous <71386173+AnaelBonnafous@users.noreply.github.com> Co-authored-by: Huy Tran <119860259+tranquochuy645@users.noreply.github.com> Co-authored-by: Gabriel Gordbegli <ggordbegli@gmail.com> Co-authored-by: Raven-1027 <83693755+Raven-1027@users.noreply.github.com> Co-authored-by: Ihe Fan <ih.fan.1027@gmail.com> Co-authored-by: Lukas Kreussel <65088241+LLukas22@users.noreply.github.com> Co-authored-by: Josh Vera <josh@joshvera.com> Co-authored-by: Peter Zaback <pzaback@gmail.com> Co-authored-by: James Delorey <deloreyjmi@gmail.com> Co-authored-by: jdelorey@cloudflare.com <jdelorey@cloudflare.com> * config loading (#1820) * Nate/control plane client (#1691) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * fix(gui): ctx rendering w/ renderInlineAs: "" (#1541) * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * fix codeqwen autocomplete leading space * add voyage rerank-1 * feat: `--noEmit` for tsc checks in CI (#1559) * docs: update CustomContextProvider docs (#1557) * add stop tokens to qwen prompt * update docs to reflect 3.5 sonnet being best * docs: comment out unused providers (#1561) * import Handlebars * feat: toast notification for config updates (#1560) * feat: toast notification for config updates * feat: only trigger toast on config.json save * displayRawMarkdown option * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * update to reflect 16 stop words limit for deepseek * feat: only trigger config update toast in vscode (#1571) * docs(prompt-files): fix typos + clarify (#1575) * doc: prompt file typo + clarifications * fix: add back correct docs * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * docs(telemetry): add pageviews to tracking list (#1581) * Add reranker configuration options to codebase embedding docs (#1584) - Introduce reranker concept - List available reranker options - Provide configuration instructions - Update keywords to include "reranker" * chore: update pr template with screenshots (#1590) * Refactor ConfirmationDialog to use SecondaryButton for cancel action (#1586) * Added instructions for running docs server locally (#1578) - Added NPM script method - Added VS Code task method - Update contributing guidelines * Update branch policy (#1577) - Change PR target to `dev` branch - Update `CONTRIBUTING.md` instructions * Consolidate example configurations into the main configuration guide (#1579) - Moved examples to configuration.md - Deleted the separate examples.md file - Updated sidebar order and links - Improved readability and structure in configuration.md * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * Update completionProvider.ts (warning tab-autocomplete models) (#1566) * feat: enhanced IndexingProgressBar with blinking dot feature - Integrated BlinkingDot component - Added STATUS_COLORS for various states - Replaced CircleDiv with BlinkingDot in UI - Updated status messages and layout * small UI tweaks * feat(gui): enhance ModelCard, ModelProviderTag, and Toggle components (#1595) - add styling and adjustments to ModelCard - update ModelProviderTag font size - remove box shadow from Toggle component - tweak icon styles in ModelCard - improve alignment and spacing * media query * feat: add best experience onboarding * fix: file rename * stop movement on button hover by keeping same border thickness * fix mistake in setting cursor: pointer * fix when free trial option is shown * Support Node.js versions below 20 for streaming response handling (#1591) - Add fallback for Node < 20 - Implement toAsyncIterable for streaming - Use TextDecoder for manual decoding - Maintain existing streaming for Node 20+ * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * fix hasPassedFTL * fix /edit cancellation from UI * feat: add `applyCodeBlock` experimental prop (#1601) * feat: add new model styling improvements (#1609) * feat: add new model styling improvements * better gap size * feat: update bug_report.yml (#1610) * chore: update bug_report.yml * typo fix * feat: add labels to "Add docs" dialog (#1612) * feat: add labels to "Add docs" dialog * remove autofocus * don't double load config * small fixes * speed up directory traversal, and use correct native path module * option not to show config update toast * merge air-gapped and recommended setup pages * chore: add telemetry for full screen toggle (#1618) * Fix headings in codebase-embeddings.md (#1617) * mention jetbrains * docs: update changie (#1619) * feat: updated changie config * hide toc and autogenerate * Update changelog.mdx * link to deeper explanation of embeddings models * ensure target="_blank" for all links in sidebar * fix gif links in intellij README.md * don't require rust in dependency installation * chore: fix padding on gh button (#1620) * chore: adjust button padding * Update tasks.json * escape colons in diff path * smoother lancedb indexing reporting * smooth progress updates for indexing * fix tsc err * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * docs: add docs and schema for "OS" provider (#1536) * ignore .env * 🚑 fix constant warnings when onboarding with Ollama * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * team analytics * apply control plane settings * workos auth * ide protocol get session info * UI for auth * profile switching * small fixes * updates * refresh tokens * updates * fix tsc errs * model select in toolbar to make room for profile selector * prod client id * link to prod URL * internal beta option * profiles change listener --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * temporary patch for type errs * Dev (#1689) * Fix an issue where CMD+K does not clear the terminal when the terminal has focus (#1671) On MacOS, ⌘+K is bound, by default, to Terminal:Clear. Without this change ⌘+K does not clear the terminal but instead iniates a chord sequence and waits for the next stroke of the chord. Co-authored-by: Rob Leidle <rleidle@tesla.com> * Change treeSitter to cache the Language objects it loads from wasm (#1672) Without this change, for a repository with 600 typescript files, the indexer would fail to finish correctly and there would be many of the following errors in the webview console log: 'Unable to load language for file ${path} RuntimeError: table index is out of bounds' The following bash will create a repo that reproduces the problem: current_path="." for ((i=1; i<=20; i++)); do new_folder="folder-$i" mkdir -p "$current_path/$new_folder" current_path="$current_path/$new_folder" for ((a=1; a<=30; a++)); do head -c 10000 /dev/urandom | base64 > "$current_path/file-$a.ts" done done Co-authored-by: Rob Leidle <rleidle@tesla.com> * acknowledge sourcemap flag in esbuild.js * don't run jetbrains-release.yaml on vscode releases * further testing for walkDir * chore: add telemetry to commands (#1673) * test: Add basic unit test to baseLLM (#1668) * update version * test: Add basic unit test to baseLLM --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: inimaz <imaz@ai-verse.com> * feat: add Quick Actions CodeLens feature (#1674) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * add voyage rerank-1 * import Handlebars * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * small UI tweaks * media query * feat: add best experience onboarding * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * chore: add telemetry for full screen toggle (#1618) * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * manually implement stop tokens for hf inference api * chore: onboarding metrics (#1626) * fix: pageview tracking * feat: add onboarding telemetry * create single `onboardingStatus` type * improved var naming * remove console logs * fix windows performance issue * rename vscodeExtension.ts * migration of onboarding variables * "stash" instead of "delete" in indexing progress * fix preview.yaml * also fix main.yaml * Update troubleshooting.md (#1637) * feat: add quick actions * Update index.d.ts * quick actions mvp * update docs * subscribe to vscode change settings * Update commands.ts * cleanup * Update quick-actions.md * Update VerticalPerLineCodeLensProvider.ts * resolve feedback --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: Nate Sesti <33237525+sestinj@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> * chore: add `isCommandEvent` to command telemetry (#1675) * chore: add `isCommandEvent` to command telemetry * Update commands.ts * Nate/better retrieval (#1677) * deduplicatearray tests * break out separate retrieval pipelines * IConfigHandler * tests for codebase indexer * better .continueignore for continue * indexing fixes * ignore .gitignore and .continueignore when indexing * retrieval pipeline improvements * fix formatting err in out .continueignore * add necessary filter to lance_db_cache * update package.json version * skip unused tests * don't ignore .prompt files * update version * Update pull_request_template.md * don't use multi-media format when there are multiple text items * add free trial experience (#1685) * fix: add code range for quick actions/fixes (#1687) * fix: add code range for quick actions/fixes * Update test.js * add pathSep message type * docs improvements * jetbrains fix * update package.json version --------- Co-authored-by: Rob Leidle <rleidle@gmail.com> Co-authored-by: Rob Leidle <rleidle@tesla.com> Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: inimaz <49730431+inimaz@users.noreply.github.com> Co-authored-by: inimaz <imaz@ai-verse.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> * update package.json version * skip extension tests * update package.json version * handlebars import * don't push package.json version change from CI * fix: scroll issues w/ code blocks (#1688) * fix: scroll issues w/ code blocks * chore: name offset * chore: remove docs start * chore: update headings on troubleshooting.md (#1696) * ignore .svn folder for indexing (#1699) * chore: clearer naming for `useSuffix` (#1702) * chore: clearer naming for `useSuffix` * schema updates * fix: cmd+shft+l closes sidebar if focused (#1638) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * add voyage rerank-1 * import Handlebars * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * small UI tweaks * media query * feat: add best experience onboarding * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * chore: add telemetry for full screen toggle (#1618) * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * manually implement stop tokens for hf inference api * chore: onboarding metrics (#1626) * fix: pageview tracking * feat: add onboarding telemetry * create single `onboardingStatus` type * improved var naming * remove console logs * fix windows performance issue * rename vscodeExtension.ts * migration of onboarding variables * "stash" instead of "delete" in indexing progress * fix preview.yaml * also fix main.yaml * Update troubleshooting.md (#1637) * feat: close panel if main input is focused * add skip param * Update TipTapEditor.tsx * merge dev * Update commands.ts --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: Nate Sesti <33237525+sestinj@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> * bugfix: quick actions undefined array (#1704) * feat: add rich quick pick for quick edit (#1706) * feat: multistep quick pick * fix: multi step for context provider * Update test.js * Create deepseek.md (#1708) * style: add max height to model selector (#1712) * update styling * remove testing code * remove comments * feat: add tutorial card (#1716) * feat: create tutorial card * Update TutorialCard.tsx * feat: add file search to quick edit (#1714) * feat: multistep quick pick * fix: multi step for context provider * Update test.js * add file search * complete file search * remove unused import * add comments * Update utils.ts * update naming * chore: move quick pick instantiation (#1723) * Break out into NPM modules (#1726) * config.json types * export all schemas * break out request logic into packages * update openai adapters to export body types * update packaging of npm modules * update packages * continue proxy * mistral * Fix typo in documentation (#1722) In set-up-codestral.md "Trobleshooting" -> "Troubleshooting" * update package.json version * 🪄 model dropdown, control plane (#1692) * Nate/control plane client (#1691) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * fix(gui): ctx rendering w/ renderInlineAs: "" (#1541) * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * fix codeqwen autocomplete leading space * add voyage rerank-1 * feat: `--noEmit` for tsc checks in CI (#1559) * docs: update CustomContextProvider docs (#1557) * add stop tokens to qwen prompt * update docs to reflect 3.5 sonnet being best * docs: comment out unused providers (#1561) * import Handlebars * feat: toast notification for config updates (#1560) * feat: toast notification for config updates * feat: only trigger toast on config.json save * displayRawMarkdown option * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * update to reflect 16 stop words limit for deepseek * feat: only trigger config update toast in vscode (#1571) * docs(prompt-files): fix typos + clarify (#1575) * doc: prompt file typo + clarifications * fix: add back correct docs * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * docs(telemetry): add pageviews to tracking list (#1581) * Add reranker configuration options to codebase embedding docs (#1584) - Introduce reranker concept - List available reranker options - Provide configuration instructions - Update keywords to include "reranker" * chore: update pr template with screenshots (#1590) * Refactor ConfirmationDialog to use SecondaryButton for cancel action (#1586) * Added instructions for running docs server locally (#1578) - Added NPM script method - Added VS Code task method - Update contributing guidelines * Update branch policy (#1577) - Change PR target to `dev` branch - Update `CONTRIBUTING.md` instructions * Consolidate example configurations into the main configuration guide (#1579) - Moved examples to configuration.md - Deleted the separate examples.md file - Updated sidebar order and links - Improved readability and structure in configuration.md * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * Update completionProvider.ts (warning tab-autocomplete models) (#1566) * feat: enhanced IndexingProgressBar with blinking dot feature - Integrated BlinkingDot component - Added STATUS_COLORS for various states - Replaced CircleDiv with BlinkingDot in UI - Updated status messages and layout * small UI tweaks * feat(gui): enhance ModelCard, ModelProviderTag, and Toggle components (#1595) - add styling and adjustments to ModelCard - update ModelProviderTag font size - remove box shadow from Toggle component - tweak icon styles in ModelCard - improve alignment and spacing * media query * feat: add best experience onboarding * fix: file rename * stop movement on button hover by keeping same border thickness * fix mistake in setting cursor: pointer * fix when free trial option is shown * Support Node.js versions below 20 for streaming response handling (#1591) - Add fallback for Node < 20 - Implement toAsyncIterable for streaming - Use TextDecoder for manual decoding - Maintain existing streaming for Node 20+ * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * fix hasPassedFTL * fix /edit cancellation from UI * feat: add `applyCodeBlock` experimental prop (#1601) * feat: add new model styling improvements (#1609) * feat: add new model styling improvements * better gap size * feat: update bug_report.yml (#1610) * chore: update bug_report.yml * typo fix * feat: add labels to "Add docs" dialog (#1612) * feat: add labels to "Add docs" dialog * remove autofocus * don't double load config * small fixes * speed up directory traversal, and use correct native path module * option not to show config update toast * merge air-gapped and recommended setup pages * chore: add telemetry for full screen toggle (#1618) * Fix headings in codebase-embeddings.md (#1617) * mention jetbrains * docs: update changie (#1619) * feat: updated changie config * hide toc and autogenerate * Update changelog.mdx * link to deeper explanation of embeddings models * ensure target="_blank" for all links in sidebar * fix gif links in intellij README.md * don't require rust in dependency installation * chore: fix padding on gh button (#1620) * chore: adjust button padding * Update tasks.json * escape colons in diff path * smoother lancedb indexing reporting * smooth progress updates for indexing * fix tsc err * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * docs: add docs and schema for "OS" provider (#1536) * ignore .env * 🚑 fix constant warnings when onboarding with Ollama * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * team analytics * apply control plane settings * workos auth * ide protocol get session info * UI for auth * profile switching * small fixes * updates * refresh tokens * updates * fix tsc errs * model select in toolbar to make room for profile selector * prod client id * link to prod URL * internal beta option * profiles change listener --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * temporary patch for type errs --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * skip extension tests * JetBrains status bar spinner for autocomplete * Remove unused images * partial accept jetbrains * fix double-rendering of jetbrains completions * fix offset of multi-line jetbrains completions * enable/disable actions * upgrade to version 0.0.54 * fix: wrap editor buttons when out of space (#1727) * Add .continuerc file to config directory on activation to prevent it's indexing. (#1733) * Updated the content links that were deprecated in the document (#1709) Co-authored-by: Ihe Fan <ih.fan.1027@gmail.com> * Nate/dev (#1739) * JetBrains status bar spinner for autocomplete * Remove unused images * partial accept jetbrains * fix double-rendering of jetbrains completions * fix offset of multi-line jetbrains completions * enable/disable actions * upgrade to version 0.0.54 * llm-info package * small profile switching improvements * instruct cmd+I not to leave placeholders * @ files in context-providers docs * onboarding fix * update jetbrains ci to build gui * feat: single default quick pick to edit (#1743) * feat: single default quick pick to edit * Update quick-actions.md * Update quick-actions.md * use defaultQuickAction * quick fixes * fix: handle line removal diff (#1744) * fix: handle line removal diff * remove comments * Configurable `maxChunkSize` for embedding providers (#1746) * Add `maxChunkSize` to `EmbeddingsProvider` * Add `maxChunkSize` to schema * add support for docs via config (#1594) * mistral mamba (#1748) * 🐍 mistral mamba * autoscale retrieval based on context length * autoscale retrieval based on context length * fix input box movement * update jetbrains build * test: add myers diff tests (#1754) * feat: enhance help center (#1755) * feat: enhance help center * Update help.tsx * feat: add azure provider config (#1764) * autocomplete reload fix * logout of control plane * Update QuickEditQuickPick.ts (#1772) * Update QuickEditQuickPick.ts * updates * Update QuickEditQuickPick.ts * pass refreshed access token to continue-proxy model * test: add dir read check for walkDir (#1773) * test: add dir read check for walkDir * Update ignore.ts * Update walkDir.test.ts * Update walkDir.test.ts * gpt-4o-mini * gpt-4o-mini * gpt4omini to ui selector * fix: layout alignment (#1779) * fix: delete old docs index on force re-index (#1778) * add "Codebase Force Re-Index" command (#1757) * Add `HuggingFaceTEIReranker` (#1711) * Add `HuggingFaceTEIReranker` * Add `huggingface-tei` to `config_schema.json` * update package.json version * update config schemas * skip test failing only due to type error * fix: convert `walkDir` to an async generator (#1783) * walkdir to async generator * add yields * handle ignore files * Update CodebaseIndexer.ts * Update walkDir.ts * chore: bump VS Code ext version * fix completions support in ContinueProxy * add test for walkDir * IAnalyticsProvider * feat: crawl `.mdx` docs * update workspaces on session info change * rename from profile to workspace (user-facing) * update config schema files * feat: improve chat thread ui for better readability (#1786) * add "language" template variable for autocomplete * Enable debug logs (#1800) * Add an enableDebugLogs flag * Send logs to VSCode * Refresh index fixes (#1795) * Use the resultType given to markComplete This fixes an issue where if there were multiple result types in itemToAction such as an update with UpdateNewVersion and UpdateOldVersion, only the UpdateOldVersion would be hit (which is a noop) and the UpdateNewVersion would never be fired, causing us to never update the timestamp of the tag_catalog. * Add a Compute and UpdateLastUpdated to refreshIndex Removes the ability to insert duplicate rows if we're just computing the result. Instead we replace the existing row instead. * Don't insert compute tags They are REPLACEd in markComplete now. * docs: change page heading to model providers (#1684) * feat: allow JetBrains users to index docs (#1797) * feat: allow JetBrains users to index docs * update docs service * update global context * throw error popup on jetbrains + transformers * update toast notifications * update typings * fix tsc errors * Index on file change (#1785) * Reload the submenu items when the context provider data is refreshed * update version * Just call refreshSubmenuItems * remove log statements * cleanup * Remove the Promise declaration * Copy currentFiles to a new dict since we're going to mutate it Don't mutate the caller's currentFiles unexpectedly * Remove reIndexFile and just call index/forceReIndex * We MUST await the refreshCodebaseIndex call * Expose send on core * remove unused import * Reindex the workspaces on save * Call core.send Ideally, we can implement externalRequest on the webview later so we don't have to expose messenger to the VSCode extension * Always refreshSubmenuItems in refreshCodebaseIndex * whitespace * Change messenger back to private * Add Pause Codebase On Start setting (#1788) * Add a skipInitialSync function * Add pauseInitialCodebaseIndex * docs * version bump * Rename to pauseCodebaseIndexOnStart * Un-shadow file context provider (#1801) * allow FileContextProvider to handle files * remove commented-out lines * chore: add telemetry to alt-enter (#1805) * Set the indexProgress indicator to paused on startup if pauseCodebaseIndexOnStart is set (#1804) * Fix Input Toolbar layout (#1792) * set to flex flex-column * Remove fixed height and always show the StyledDivs content to hold its size * Add a Delete Message button to the StepContainer (#1790) * Add a delete button * add a delete message action to the GUI * Fix off by one error * format * This index should be less than length * Update to trash icon * Simplify walkDir(...) and improve performance by roughly 10x in larger repos (#1806) The performance of WalkDir is improved by replacing the recursive DFS with a stack based implementation (less memory in-use during peak exploration). In addition, some unnecessary flags and options were removed which allowed the code to be overall simplified. Much less rule matching is occuring with the same files returned. Some performance results on my linux host: small repo: - 649 files returned by walkDir() - 9 .gitignore files - walkDir() old vs new duration: 918 ms to 143 ms medium sized repo: - 9271 returned by walkDir() - 35 .gitignore files - walkDir() old vs new duration: 10,886 ms to 970ms large sized repo: - 197,900 files returned by walkDir() - 929 .gitignore files - walkDir() old vs new duration: 498,151 ms to 37,353 ms Co-authored-by: Rob Leidle <rleidle@tesla.com> * modified assistant indicator in UI * update package.json version * feat: add Llama 3.1 8B to cloudflare provider options (#1811) Co-authored-by: jdelorey@cloudflare.com <jdelorey@cloudflare.com> * increase gemini pro contextLength (#1809) * fix indentation (#1808) * Llama 3.1 405b model selection (#1813) * add support for llama3 405b * 3.1 * name mapping * updates to model setup * fix walkDir tests on windows * chore: add docs as top level config (#1816) * chore: add docs as top level config * Update core.ts * feat: make @codebase a hardcoded ctx provider (#1818) * chore: add docs as top level config * Update core.ts * feat: make @codebase a hardcoded ctx provider * docs schema * Nate/dev (#1819) * llm-info package * small profile switching improvements * instruct cmd+I not to leave placeholders * @ files in context-providers docs * onboarding fix * jetbrains login * fix jetbrains login * change beta wording * proxy context provider * fix proxy context provider * revert to prod id, url * update package.json version --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> Co-authored-by: Rob Leidle <rleidle@gmail.com> Co-authored-by: Rob Leidle <rleidle@tesla.com> Co-authored-by: inimaz <49730431+inimaz@users.noreply.github.com> Co-authored-by: inimaz <imaz@ai-verse.com> Co-authored-by: Anaël Bonnafous <71386173+AnaelBonnafous@users.noreply.github.com> Co-authored-by: Huy Tran <119860259+tranquochuy645@users.noreply.github.com> Co-authored-by: Gabriel Gordbegli <ggordbegli@gmail.com> Co-authored-by: Raven-1027 <83693755+Raven-1027@users.noreply.github.com> Co-authored-by: Ihe Fan <ih.fan.1027@gmail.com> Co-authored-by: Lukas Kreussel <65088241+LLukas22@users.noreply.github.com> Co-authored-by: Josh Vera <josh@joshvera.com> Co-authored-by: Peter Zaback <pzaback@gmail.com> Co-authored-by: James Delorey <deloreyjmi@gmail.com> Co-authored-by: jdelorey@cloudflare.com <jdelorey@cloudflare.com> * Improved indexing performance (#1845) * Nate/control plane client (#1691) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * fix(gui): ctx rendering w/ renderInlineAs: "" (#1541) * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * fix codeqwen autocomplete leading space * add voyage rerank-1 * feat: `--noEmit` for tsc checks in CI (#1559) * docs: update CustomContextProvider docs (#1557) * add stop tokens to qwen prompt * update docs to reflect 3.5 sonnet being best * docs: comment out unused providers (#1561) * import Handlebars * feat: toast notification for config updates (#1560) * feat: toast notification for config updates * feat: only trigger toast on config.json save * displayRawMarkdown option * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * update to reflect 16 stop words limit for deepseek * feat: only trigger config update toast in vscode (#1571) * docs(prompt-files): fix typos + clarify (#1575) * doc: prompt file typo + clarifications * fix: add back correct docs * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * docs(telemetry): add pageviews to tracking list (#1581) * Add reranker configuration options to codebase embedding docs (#1584) - Introduce reranker concept - List available reranker options - Provide configuration instructions - Update keywords to include "reranker" * chore: update pr template with screenshots (#1590) * Refactor ConfirmationDialog to use SecondaryButton for cancel action (#1586) * Added instructions for running docs server locally (#1578) - Added NPM script method - Added VS Code task method - Update contributing guidelines * Update branch policy (#1577) - Change PR target to `dev` branch - Update `CONTRIBUTING.md` instructions * Consolidate example configurations into the main configuration guide (#1579) - Moved examples to configuration.md - Deleted the separate examples.md file - Updated sidebar order and links - Improved readability and structure in configuration.md * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * Update completionProvider.ts (warning tab-autocomplete models) (#1566) * feat: enhanced IndexingProgressBar with blinking dot feature - Integrated BlinkingDot component - Added STATUS_COLORS for various states - Replaced CircleDiv with BlinkingDot in UI - Updated status messages and layout * small UI tweaks * feat(gui): enhance ModelCard, ModelProviderTag, and Toggle components (#1595) - add styling and adjustments to ModelCard - update ModelProviderTag font size - remove box shadow from Toggle component - tweak icon styles in ModelCard - improve alignment and spacing * media query * feat: add best experience onboarding * fix: file rename * stop movement on button hover by keeping same border thickness * fix mistake in setting cursor: pointer * fix when free trial option is shown * Support Node.js versions below 20 for streaming response handling (#1591) - Add fallback for Node < 20 - Implement toAsyncIterable for streaming - Use TextDecoder for manual decoding - Maintain existing streaming for Node 20+ * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * fix hasPassedFTL * fix /edit cancellation from UI * feat: add `applyCodeBlock` experimental prop (#1601) * feat: add new model styling improvements (#1609) * feat: add new model styling improvements * better gap size * feat: update bug_report.yml (#1610) * chore: update bug_report.yml * typo fix * feat: add labels to "Add docs" dialog (#1612) * feat: add labels to "Add docs" dialog * remove autofocus * don't double load config * small fixes * speed up directory traversal, and use correct native path module * option not to show config update toast * merge air-gapped and recommended setup pages * chore: add telemetry for full screen toggle (#1618) * Fix headings in codebase-embeddings.md (#1617) * mention jetbrains * docs: update changie (#1619) * feat: updated changie config * hide toc and autogenerate * Update changelog.mdx * link to deeper explanation of embeddings models * ensure target="_blank" for all links in sidebar * fix gif links in intellij README.md * don't require rust in dependency installation * chore: fix padding on gh button (#1620) * chore: adjust button padding * Update tasks.json * escape colons in diff path * smoother lancedb indexing reporting * smooth progress updates for indexing * fix tsc err * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * docs: add docs and schema for "OS" provider (#1536) * ignore .env * 🚑 fix constant warnings when onboarding with Ollama * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * team analytics * apply control plane settings * workos auth * ide protocol get session info * UI for auth * profile switching * small fixes * updates * refresh tokens * updates * fix tsc errs * model select in toolbar to make room for profile selector * prod client id * link to prod URL * internal beta option * profiles change listener --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * temporary patch for type errs * Dev (#1689) * Fix an issue where CMD+K does not clear the terminal when the terminal has focus (#1671) On MacOS, ⌘+K is bound, by default, to Terminal:Clear. Without this change ⌘+K does not clear the terminal but instead iniates a chord sequence and waits for the next stroke of the chord. Co-authored-by: Rob Leidle <rleidle@tesla.com> * Change treeSitter to cache the Language objects it loads from wasm (#1672) Without this change, for a repository with 600 typescript files, the indexer would fail to finish correctly and there would be many of the following errors in the webview console log: 'Unable to load language for file ${path} RuntimeError: table index is out of bounds' The following bash will create a repo that reproduces the problem: current_path="." for ((i=1; i<=20; i++)); do new_folder="folder-$i" mkdir -p "$current_path/$new_folder" current_path="$current_path/$new_folder" for ((a=1; a<=30; a++)); do head -c 10000 /dev/urandom | base64 > "$current_path/file-$a.ts" done done Co-authored-by: Rob Leidle <rleidle@tesla.com> * acknowledge sourcemap flag in esbuild.js * don't run jetbrains-release.yaml on vscode releases * further testing for walkDir * chore: add telemetry to commands (#1673) * test: Add basic unit test to baseLLM (#1668) * update version * test: Add basic unit test to baseLLM --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: inimaz <imaz@ai-verse.com> * feat: add Quick Actions CodeLens feature (#1674) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * add voyage rerank-1 * import Handlebars * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * small UI tweaks * media query * feat: add best experience onboarding * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * chore: add telemetry for full screen toggle (#1618) * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * manually implement stop tokens for hf inference api * chore: onboarding metrics (#1626) * fix: pageview tracking * feat: add onboarding telemetry * create single `onboardingStatus` type * improved var naming * remove console logs * fix windows performance issue * rename vscodeExtension.ts * migration of onboarding variables * "stash" instead of "delete" in indexing progress * fix preview.yaml * also fix main.yaml * Update troubleshooting.md (#1637) * feat: add quick actions * Update index.d.ts * quick actions mvp * update docs * subscribe to vscode change settings * Update commands.ts * cleanup * Update quick-actions.md * Update VerticalPerLineCodeLensProvider.ts * resolve feedback --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: Nate Sesti <33237525+sestinj@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> * chore: add `isCommandEvent` to command telemetry (#1675) * chore: add `isCommandEvent` to command telemetry * Update commands.ts * Nate/better retrieval (#1677) * deduplicatearray tests * break out separate retrieval pipelines * IConfigHandler * tests for codebase indexer * better .continueignore for continue * indexing fixes * ignore .gitignore and .continueignore when indexing * retrieval pipeline improvements * fix formatting err in out .continueignore * add necessary filter to lance_db_cache * update package.json version * skip unused tests * don't ignore .prompt files * update version * Update pull_request_template.md * don't use multi-media format when there are multiple text items * add free trial experience (#1685) * fix: add code range for quick actions/fixes (#1687) * fix: add code range for quick actions/fixes * Update test.js * add pathSep message type * docs improvements * jetbrains fix * update package.json version --------- Co-authored-by: Rob Leidle <rleidle@gmail.com> Co-authored-by: Rob Leidle <rleidle@tesla.com> Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: inimaz <49730431+inimaz@users.noreply.github.com> Co-authored-by: inimaz <imaz@ai-verse.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> * update package.json version * skip extension tests * update package.json version * handlebars import * don't push package.json version change from CI * fix: scroll issues w/ code blocks (#1688) * fix: scroll issues w/ code blocks * chore: name offset * chore: remove docs start * chore: update headings on troubleshooting.md (#1696) * ignore .svn folder for indexing (#1699) * chore: clearer naming for `useSuffix` (#1702) * chore: clearer naming for `useSuffix` * schema updates * fix: cmd+shft+l closes sidebar if focused (#1638) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * add voyage rerank-1 * import Handlebars * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * small UI tweaks * media query * feat: add best experience onboarding * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * chore: add telemetry for full screen toggle (#1618) * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * manually implement stop tokens for hf inference api * chore: onboarding metrics (#1626) * fix: pageview tracking * feat: add onboarding telemetry * create single `onboardingStatus` type * improved var naming * remove console logs * fix windows performance issue * rename vscodeExtension.ts * migration of onboarding variables * "stash" instead of "delete" in indexing progress * fix preview.yaml * also fix main.yaml * Update troubleshooting.md (#1637) * feat: close panel if main input is focused * add skip param * Update TipTapEditor.tsx * merge dev * Update commands.ts --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: Nate Sesti <33237525+sestinj@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> * bugfix: quick actions undefined array (#1704) * feat: add rich quick pick for quick edit (#1706) * feat: multistep quick pick * fix: multi step for context provider * Update test.js * Create deepseek.md (#1708) * style: add max height to model selector (#1712) * update styling * remove testing code * remove comments * feat: add tutorial card (#1716) * feat: create tutorial card * Update TutorialCard.tsx * feat: add file search to quick edit (#1714) * feat: multistep quick pick * fix: multi step for context provider * Update test.js * add file search * complete file search * remove unused import * add comments * Update utils.ts * update naming * chore: move quick pick instantiation (#1723) * Break out into NPM modules (#1726) * config.json types * export all schemas * break out request logic into packages * update openai adapters to export body types * update packaging of npm modules * update packages * continue proxy * mistral * Fix typo in documentation (#1722) In set-up-codestral.md "Trobleshooting" -> "Troubleshooting" * update package.json version * 🪄 model dropdown, control plane (#1692) * Nate/control plane client (#1691) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * fix(gui): ctx rendering w/ renderInlineAs: "" (#1541) * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * fix codeqwen autocomplete leading space * add voyage rerank-1 * feat: `--noEmit` for tsc checks in CI (#1559) * docs: update CustomContextProvider docs (#1557) * add stop tokens to qwen prompt * update docs to reflect 3.5 sonnet being best * docs: comment out unused providers (#1561) * import Handlebars * feat: toast notification for config updates (#1560) * feat: toast notification for config updates * feat: only trigger toast on config.json save * displayRawMarkdown option * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * update to reflect 16 stop words limit for deepseek * feat: only trigger config update toast in vscode (#1571) * docs(prompt-files): fix typos + clarify (#1575) * doc: prompt file typo + clarifications * fix: add back correct docs * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * docs(telemetry): add pageviews to tracking list (#1581) * Add reranker configuration options to codebase embedding docs (#1584) - Introduce reranker concept - List available reranker options - Provide configuration instructions - Update keywords to include "reranker" * chore: update pr template with screenshots (#1590) * Refactor ConfirmationDialog to use SecondaryButton for cancel action (#1586) * Added instructions for running docs server locally (#1578) - Added NPM script method - Added VS Code task method - Update contributing guidelines * Update branch policy (#1577) - Change PR target to `dev` branch - Update `CONTRIBUTING.md` instructions * Consolidate example configurations into the main configuration guide (#1579) - Moved examples to configuration.md - Deleted the separate examples.md file - Updated sidebar order and links - Improved readability and structure in configuration.md * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * Update completionProvider.ts (warning tab-autocomplete models) (#1566) * feat: enhanced IndexingProgressBar with blinking dot feature - Integrated BlinkingDot component - Added STATUS_COLORS for various states - Replaced CircleDiv with BlinkingDot in UI - Updated status messages and layout * small UI tweaks * feat(gui): enhance ModelCard, ModelProviderTag, and Toggle components (#1595) - add styling and adjustments to ModelCard - update ModelProviderTag font size - remove box shadow from Toggle component - tweak icon styles in ModelCard - improve alignment and spacing * media query * feat: add best experience onboarding * fix: file rename * stop movement on button hover by keeping same border thickness * fix mistake in setting cursor: pointer * fix when free trial option is shown * Support Node.js versions below 20 for streaming response handling (#1591) - Add fallback for Node < 20 - Implement toAsyncIterable for streaming - Use TextDecoder for manual decoding - Maintain existing streaming for Node 20+ * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * fix hasPassedFTL * fix /edit cancellation from UI * feat: add `applyCodeBlock` experimental prop (#1601) * feat: add new model styling improvements (#1609) * feat: add new model styling improvements * better gap size * feat: update bug_report.yml (#1610) * chore: update bug_report.yml * typo fix * feat: add labels to "Add docs" dialog (#1612) * feat: add labels to "Add docs" dialog * remove autofocus * don't double load config * small fixes * speed up directory traversal, and use correct native path module * option not to show config update toast * merge air-gapped and recommended setup pages * chore: add telemetry for full screen toggle (#1618) * Fix headings in codebase-embeddings.md (#1617) * mention jetbrains * docs: update changie (#1619) * feat: updated changie config * hide toc and autogenerate * Update changelog.mdx * link to deeper explanation of embeddings models * ensure target="_blank" for all links in sidebar * fix gif links in intellij README.md * don't require rust in dependency installation * chore: fix padding on gh button (#1620) * chore: adjust button padding * Update tasks.json * escape colons in diff path * smoother lancedb indexing reporting * smooth progress updates for indexing * fix tsc err * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * docs: add docs and schema for "OS" provider (#1536) * ignore .env * 🚑 fix constant warnings when onboarding with Ollama * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * team analytics * apply control plane settings * workos auth * ide protocol get session info * UI for auth * profile switching * small fixes * updates * refresh tokens * updates * fix tsc errs * model select in toolbar to make room for profile selector * prod client id * link to prod URL * internal beta option * profiles change listener --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * temporary patch for type errs --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * skip extension tests * JetBrains status bar spinner for autocomplete * Remove unused images * partial accept jetbrains * fix double-rendering of jetbrains completions * fix offset of multi-line jetbrains completions * enable/disable actions * upgrade to version 0.0.54 * fix: wrap editor buttons when out of space (#1727) * Add .continuerc file to config directory on activation to prevent it's indexing. (#1733) * Updated the content links that were deprecated in the document (#1709) Co-authored-by: Ihe Fan <ih.fan.1027@gmail.com> * Nate/dev (#1739) * JetBrains status bar spinner for autocomplete * Remove unused images * partial accept jetbrains * fix double-rendering of jetbrains completions * fix offset of multi-line jetbrains completions * enable/disable actions * upgrade to version 0.0.54 * llm-info package * small profile switching improvements * instruct cmd+I not to leave placeholders * @ files in context-providers docs * onboarding fix * update jetbrains ci to build gui * feat: single default quick pick to edit (#1743) * feat: single default quick pick to edit * Update quick-actions.md * Update quick-actions.md * use defaultQuickAction * quick fixes * fix: handle line removal diff (#1744) * fix: handle line removal diff * remove comments * Configurable `maxChunkSize` for embedding providers (#1746) * Add `maxChunkSize` to `EmbeddingsProvider` * Add `maxChunkSize` to schema * add support for docs via config (#1594) * mistral mamba (#1748) * 🐍 mistral mamba * autoscale retrieval based on context length * autoscale retrieval based on context length * fix input box movement * update jetbrains build * test: add myers diff tests (#1754) * feat: enhance help center (#1755) * feat: enhance help center * Update help.tsx * feat: add azure provider config (#1764) * autocomplete reload fix * logout of control plane * Update QuickEditQuickPick.ts (#1772) * Update QuickEditQuickPick.ts * updates * Update QuickEditQuickPick.ts * pass refreshed access token to continue-proxy model * test: add dir read check for walkDir (#1773) * test: add dir read check for walkDir * Update ignore.ts * Update walkDir.test.ts * Update walkDir.test.ts * gpt-4o-mini * gpt-4o-mini * gpt4omini to ui selector * fix: layout alignment (#1779) * fix: delete old docs index on force re-index (#1778) * add "Codebase Force Re-Index" command (#1757) * Add `HuggingFaceTEIReranker` (#1711) * Add `HuggingFaceTEIReranker` * Add `huggingface-tei` to `config_schema.json` * update package.json version * update config schemas * skip test failing only due to type error * fix: convert `walkDir` to an async generator (#1783) * walkdir to async generator * add yields * handle ignore files * Update CodebaseIndexer.ts * Update walkDir.ts * chore: bump VS Code ext version * fix completions support in ContinueProxy * add test for walkDir * IAnalyticsProvider * feat: crawl `.mdx` docs * update workspaces on session info change * rename from profile to workspace (user-facing) * update config schema files * feat: improve chat thread ui for better readability (#1786) * add "language" template variable for autocomplete * Enable debug logs (#1800) * Add an enableDebugLogs flag * Send logs to VSCode * Refresh index fixes (#1795) * Use the resultType given to markComplete This fixes an issue where if there were multiple result types in itemToAction such as an update with UpdateNewVersion and UpdateOldVersion, only the UpdateOldVersion would be hit (which is a noop) and the UpdateNewVersion would never be fired, causing us to never update the timestamp of the tag_catalog. * Add a Compute and UpdateLastUpdated to refreshIndex Removes the ability to insert duplicate rows if we're just computing the result. Instead we replace the existing row instead. * Don't insert compute tags They are REPLACEd in markComplete now. * docs: change page heading to model providers (#1684) * feat: allow JetBrains users to index docs (#1797) * feat: allow JetBrains users to index docs * update docs service * update global context * throw error popup on jetbrains + transformers * update toast notifications * update typings * fix tsc errors * Index on file change (#1785) * Reload the submenu items when the context provider data is refreshed * update version * Just call refreshSubmenuItems * remove log statements * cleanup * Remove the Promise declaration * Copy currentFiles to a new dict since we're going to mutate it Don't mutate the caller's currentFiles unexpectedly * Remove reIndexFile and just call index/forceReIndex * We MUST await the refreshCodebaseIndex call * Expose send on core * remove unused import * Reindex the workspaces on save * Call core.send Ideally, we can implement externalRequest on the webview later so we don't have to expose messenger to the VSCode extension * Always refreshSubmenuItems in refreshCodebaseIndex * whitespace * Change messenger back to private * Add Pause Codebase On Start setting (#1788) * Add a skipInitialSync function * Add pauseInitialCodebaseIndex * docs * version bump * Rename to pauseCodebaseIndexOnStart * Un-shadow file context provider (#1801) * allow FileContextProvider to handle files * remove commented-out lines * chore: add telemetry to alt-enter (#1805) * Set the indexProgress indicator to paused on startup if pauseCodebaseIndexOnStart is set (#1804) * Fix Input Toolbar layout (#1792) * set to flex flex-column * Remove fixed height and always show the StyledDivs content to hold its size * Add a Delete Message button to the StepContainer (#1790) * Add a delete button * add a delete message action to the GUI * Fix off by one error * format * This index should be less than length * Update to trash icon * Simplify walkDir(...) and improve performance by roughly 10x in larger repos (#1806) The performance of WalkDir is improved by replacing the recursive DFS with a stack based implementation (less memory in-use during peak exploration). In addition, some unnecessary flags and options were removed which allowed the code to be overall simplified. Much less rule matching is occuring with the same files returned. Some performance results on my linux host: small repo: - 649 files returned by walkDir() - 9 .gitignore files - walkDir() old vs new duration: 918 ms to 143 ms medium sized repo: - 9271 returned by walkDir() - 35 .gitignore files - walkDir() old vs new duration: 10,886 ms to 970ms large sized repo: - 197,900 files returned by walkDir() - 929 .gitignore files - walkDir() old vs new duration: 498,151 ms to 37,353 ms Co-authored-by: Rob Leidle <rleidle@tesla.com> * modified assistant indicator in UI * update package.json version * feat: add Llama 3.1 8B to cloudflare provider options (#1811) Co-authored-by: jdelorey@cloudflare.com <jdelorey@cloudflare.com> * increase gemini pro contextLength (#1809) * fix indentation (#1808) * Llama 3.1 405b model selection (#1813) * add support for llama3 405b * 3.1 * name mapping * updates to model setup * fix walkDir tests on windows * chore: add docs as top level config (#1816) * chore: add docs as top level config * Update core.ts * feat: make @codebase a hardcoded ctx provider (#1818) * chore: add docs as top level config * Update core.ts * feat: make @codebase a hardcoded ctx provider * docs schema * Nate/dev (#1819) * llm-info package * small profile switching improvements * instruct cmd+I not to leave placeholders * @ files in context-providers docs * onboarding fix * jetbrains login * fix jetbrains login * change beta wording * proxy context provider * fix proxy context provider * revert to prod id, url * update package.json version * Add CodeGeeX4 code completion support (#1815) Includes support for context through code snippets and relative filename resolution. * manually stop at stop tokens for all providers * chore: bump tree-sitter (#1828) * Add Missing Tokens for CodeGeeX4 AutoComplete (#1827) Previous attempts at using the normal pipe char with CodeGeeX FIM tokens resulted in empty completion output. However, this was due to <|user|> and <|assistant|> tokens being necessary for completion. The earlier working change ommitted these tokens, and its functionality using deepseek pipes was merely coincidental. This change now properly follows the CodeGeeX infill guides and should match what the model expects. In addition to <|user|> and <|assistant|>, the <|endoftext|> token was added as a stop token. * additional analytics providers * fix import error in jetbrains plugin * explicitly include default context providers * Remove a copy / pasta section that duplicated the work done in ChunkCodebaseIndexer (#1834) This cuts the amount of time spent in this section of indexing in half Co-authored-by: Rob Leidle <rleidle@tesla.com> * respect VS Code isTelemetryEnabled * Update OllamaEmbeddingsProvider.ts (#1839) * feat: include recently + open files in codebase search (#1833) * feat: include recently + open files in codebase search * cleanup * Update BaseRetrievalPipeline.ts * add params object to chunkDocument * Update package-lock.json * Offload all token counting to worker processes as well as some optimizations to do more token counting in parallel Adding the workerpool logic broke gui as vite does not support __dirname or path.join(...). To work around this, moved the gui dependency of stripImages out of countTokens.ts * Create CODE_OF_CONDUCT.md (#1843) * tweaks to new tokenizing * temporarily skip code chunking tests due to worker files not being present * update changelog * update version --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> Co-authored-by: Rob Leidle <rleidle@gmail.com> Co-authored-by: Rob Leidle <rleidle@tesla.com> Co-authored-by: inimaz <49730431+inimaz@users.noreply.github.com> Co-authored-by: inimaz <imaz@ai-verse.com> Co-authored-by: Anaël Bonnafous <71386173+AnaelBonnafous@users.noreply.github.com> Co-authored-by: Huy Tran <119860259+tranquochuy645@users.noreply.github.com> Co-authored-by: Gabriel Gordbegli <ggordbegli@gmail.com> Co-authored-by: Raven-1027 <83693755+Raven-1027@users.noreply.github.com> Co-authored-by: Ihe Fan <ih.fan.1027@gmail.com> Co-authored-by: Lukas Kreussel <65088241+LLukas22@users.noreply.github.com> Co-authored-by: Josh Vera <josh@joshvera.com> Co-authored-by: Peter Zaback <pzaback@gmail.com> Co-authored-by: James Delorey <deloreyjmi@gmail.com> Co-authored-by: jdelorey@cloudflare.com <jdelorey@cloudflare.com> Co-authored-by: ekatiyar <eashaank@gmail.com> * proxy updates (#1861) * Nate/control plane client (#1691) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * fix(gui): ctx rendering w/ renderInlineAs: "" (#1541) * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * fix codeqwen autocomplete leading space * add voyage rerank-1 * feat: `--noEmit` for tsc checks in CI (#1559) * docs: update CustomContextProvider docs (#1557) * add stop tokens to qwen prompt * update docs to reflect 3.5 sonnet being best * docs: comment out unused providers (#1561) * import Handlebars * feat: toast notification for config updates (#1560) * feat: toast notification for config updates * feat: only trigger toast on config.json save * displayRawMarkdown option * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * update to reflect 16 stop words limit for deepseek * feat: only trigger config update toast in vscode (#1571) * docs(prompt-files): fix typos + clarify (#1575) * doc: prompt file typo + clarifications * fix: add back correct docs * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * docs(telemetry): add pageviews to tracking list (#1581) * Add reranker configuration options to codebase embedding docs (#1584) - Introduce reranker concept - List available reranker options - Provide configuration instructions - Update keywords to include "reranker" * chore: update pr template with screenshots (#1590) * Refactor ConfirmationDialog to use SecondaryButton for cancel action (#1586) * Added instructions for running docs server locally (#1578) - Added NPM script method - Added VS Code task method - Update contributing guidelines * Update branch policy (#1577) - Change PR target to `dev` branch - Update `CONTRIBUTING.md` instructions * Consolidate example configurations into the main configuration guide (#1579) - Moved examples to configuration.md - Deleted the separate examples.md file - Updated sidebar order and links - Improved readability and structure in configuration.md * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * Update completionProvider.ts (warning tab-autocomplete models) (#1566) * feat: enhanced IndexingProgressBar with blinking dot feature - Integrated BlinkingDot component - Added STATUS_COLORS for various states - Replaced CircleDiv with BlinkingDot in UI - Updated status messages and layout * small UI tweaks * feat(gui): enhance ModelCard, ModelProviderTag, and Toggle components (#1595) - add styling and adjustments to ModelCard - update ModelProviderTag font size - remove box shadow from Toggle component - tweak icon styles in ModelCard - improve alignment and spacing * media query * feat: add best experience onboarding * fix: file rename * stop movement on button hover by keeping same border thickness * fix mistake in setting cursor: pointer * fix when free trial option is shown * Support Node.js versions below 20 for streaming response handling (#1591) - Add fallback for Node < 20 - Implement toAsyncIterable for streaming - Use TextDecoder for manual decoding - Maintain existing streaming for Node 20+ * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * fix hasPassedFTL * fix /edit cancellation from UI * feat: add `applyCodeBlock` experimental prop (#1601) * feat: add new model styling improvements (#1609) * feat: add new model styling improvements * better gap size * feat: update bug_report.yml (#1610) * chore: update bug_report.yml * typo fix * feat: add labels to "Add docs" dialog (#1612) * feat: add labels to "Add docs" dialog * remove autofocus * don't double load config * small fixes * speed up directory traversal, and use correct native path module * option not to show config update toast * merge air-gapped and recommended setup pages * chore: add telemetry for full screen toggle (#1618) * Fix headings in codebase-embeddings.md (#1617) * mention jetbrains * docs: update changie (#1619) * feat: updated changie config * hide toc and autogenerate * Update changelog.mdx * link to deeper explanation of embeddings models * ensure target="_blank" for all links in sidebar * fix gif links in intellij README.md * don't require rust in dependency installation * chore: fix padding on gh button (#1620) * chore: adjust button padding * Update tasks.json * escape colons in diff path * smoother lancedb indexing reporting * smooth progress updates for indexing * fix tsc err * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * docs: add docs and schema for "OS" provider (#1536) * ignore .env * 🚑 fix constant warnings when onboarding with Ollama * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * team analytics * apply control plane settings * workos auth * ide protocol get session info * UI for auth * profile switching * small fixes * updates * refresh tokens * updates * fix tsc errs * model select in toolbar to make room for profile selector * prod client id * link to prod URL * internal beta option * profiles change listener --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * temporary patch for type errs * Dev (#1689) * Fix an issue where CMD+K does not clear the terminal when the terminal has focus (#1671) On MacOS, ⌘+K is bound, by default, to Terminal:Clear. Without this change ⌘+K does not clear the terminal but instead iniates a chord sequence and waits for the next stroke of the chord. Co-authored-by: Rob Leidle <rleidle@tesla.com> * Change treeSitter to cache the Language objects it loads from wasm (#1672) Without this change, for a repository with 600 typescript files, the indexer would fail to finish correctly and there would be many of the following errors in the webview console log: 'Unable to load language for file ${path} RuntimeError: table index is out of bounds' The following bash will create a repo that reproduces the problem: current_path="." for ((i=1; i<=20; i++)); do new_folder="folder-$i" mkdir -p "$current_path/$new_folder" current_path="$current_path/$new_folder" for ((a=1; a<=30; a++)); do head -c 10000 /dev/urandom | base64 > "$current_path/file-$a.ts" done done Co-authored-by: Rob Leidle <rleidle@tesla.com> * acknowledge sourcemap flag in esbuild.js * don't run jetbrains-release.yaml on vscode releases * further testing for walkDir * chore: add telemetry to commands (#1673) * test: Add basic unit test to baseLLM (#1668) * update version * test: Add basic unit test to baseLLM --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: inimaz <imaz@ai-verse.com> * feat: add Quick Actions CodeLens feature (#1674) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * add voyage rerank-1 * import Handlebars * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * small UI tweaks * media query * feat: add best experience onboarding * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * chore: add telemetry for full screen toggle (#1618) * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * manually implement stop tokens for hf inference api * chore: onboarding metrics (#1626) * fix: pageview tracking * feat: add onboarding telemetry * create single `onboardingStatus` type * improved var naming * remove console logs * fix windows performance issue * rename vscodeExtension.ts * migration of onboarding variables * "stash" instead of "delete" in indexing progress * fix preview.yaml * also fix main.yaml * Update troubleshooting.md (#1637) * feat: add quick actions * Update index.d.ts * quick actions mvp * update docs * subscribe to vscode change settings * Update commands.ts * cleanup * Update quick-actions.md * Update VerticalPerLineCodeLensProvider.ts * resolve feedback --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: Nate Sesti <33237525+sestinj@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> * chore: add `isCommandEvent` to command telemetry (#1675) * chore: add `isCommandEvent` to command telemetry * Update commands.ts * Nate/better retrieval (#1677) * deduplicatearray tests * break out separate retrieval pipelines * IConfigHandler * tests for codebase indexer * better .continueignore for continue * indexing fixes * ignore .gitignore and .continueignore when indexing * retrieval pipeline improvements * fix formatting err in out .continueignore * add necessary filter to lance_db_cache * update package.json version * skip unused tests * don't ignore .prompt files * update version * Update pull_request_template.md * don't use multi-media format when there are multiple text items * add free trial experience (#1685) * fix: add code range for quick actions/fixes (#1687) * fix: add code range for quick actions/fixes * Update test.js * add pathSep message type * docs improvements * jetbrains fix * update package.json version --------- Co-authored-by: Rob Leidle <rleidle@gmail.com> Co-authored-by: Rob Leidle <rleidle@tesla.com> Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: inimaz <49730431+inimaz@users.noreply.github.com> Co-authored-by: inimaz <imaz@ai-verse.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> * update package.json version * skip extension tests * update package.json version * handlebars import * don't push package.json version change from CI * fix: scroll issues w/ code blocks (#1688) * fix: scroll issues w/ code blocks * chore: name offset * chore: remove docs start * chore: update headings on troubleshooting.md (#1696) * ignore .svn folder for indexing (#1699) * chore: clearer naming for `useSuffix` (#1702) * chore: clearer naming for `useSuffix` * schema updates * fix: cmd+shft+l closes sidebar if focused (#1638) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * add voyage rerank-1 * import Handlebars * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * small UI tweaks * media query * feat: add best experience onboarding * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * chore: add telemetry for full screen toggle (#1618) * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * manually implement stop tokens for hf inference api * chore: onboarding metrics (#1626) * fix: pageview tracking * feat: add onboarding telemetry * create single `onboardingStatus` type * improved var naming * remove console logs * fix windows performance issue * rename vscodeExtension.ts * migration of onboarding variables * "stash" instead of "delete" in indexing progress * fix preview.yaml * also fix main.yaml * Update troubleshooting.md (#1637) * feat: close panel if main input is focused * add skip param * Update TipTapEditor.tsx * merge dev * Update commands.ts --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: Nate Sesti <33237525+sestinj@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> * bugfix: quick actions undefined array (#1704) * feat: add rich quick pick for quick edit (#1706) * feat: multistep quick pick * fix: multi step for context provider * Update test.js * Create deepseek.md (#1708) * style: add max height to model selector (#1712) * update styling * remove testing code * remove comments * feat: add tutorial card (#1716) * feat: create tutorial card * Update TutorialCard.tsx * feat: add file search to quick edit (#1714) * feat: multistep quick pick * fix: multi step for context provider * Update test.js * add file search * complete file search * remove unused import * add comments * Update utils.ts * update naming * chore: move quick pick instantiation (#1723) * Break out into NPM modules (#1726) * config.json types * export all schemas * break out request logic into packages * update openai adapters to export body types * update packaging of npm modules * update packages * continue proxy * mistral * Fix typo in documentation (#1722) In set-up-codestral.md "Trobleshooting" -> "Troubleshooting" * update package.json version * 🪄 model dropdown, control plane (#1692) * Nate/control plane client (#1691) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * fix(gui): ctx rendering w/ renderInlineAs: "" (#1541) * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * fix codeqwen autocomplete leading space * add voyage rerank-1 * feat: `--noEmit` for tsc checks in CI (#1559) * docs: update CustomContextProvider docs (#1557) * add stop tokens to qwen prompt * update docs to reflect 3.5 sonnet being best * docs: comment out unused providers (#1561) * import Handlebars * feat: toast notification for config updates (#1560) * feat: toast notification for config updates * feat: only trigger toast on config.json save * displayRawMarkdown option * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * update to reflect 16 stop words limit for deepseek * feat: only trigger config update toast in vscode (#1571) * docs(prompt-files): fix typos + clarify (#1575) * doc: prompt file typo + clarifications * fix: add back correct docs * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * docs(telemetry): add pageviews to tracking list (#1581) * Add reranker configuration options to codebase embedding docs (#1584) - Introduce reranker concept - List available reranker options - Provide configuration instructions - Update keywords to include "reranker" * chore: update pr template with screenshots (#1590) * Refactor ConfirmationDialog to use SecondaryButton for cancel action (#1586) * Added instructions for running docs server locally (#1578) - Added NPM script method - Added VS Code task method - Update contributing guidelines * Update branch policy (#1577) - Change PR target to `dev` branch - Update `CONTRIBUTING.md` instructions * Consolidate example configurations into the main configuration guide (#1579) - Moved examples to configuration.md - Deleted the separate examples.md file - Updated sidebar order and links - Improved readability and structure in configuration.md * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * Update completionProvider.ts (warning tab-autocomplete models) (#1566) * feat: enhanced IndexingProgressBar with blinking dot feature - Integrated BlinkingDot component - Added STATUS_COLORS for various states - Replaced CircleDiv with BlinkingDot in UI - Updated status messages and layout * small UI tweaks * feat(gui): enhance ModelCard, ModelProviderTag, and Toggle components (#1595) - add styling and adjustments to ModelCard - update ModelProviderTag font size - remove box shadow from Toggle component - tweak icon styles in ModelCard - improve alignment and spacing * media query * feat: add best experience onboarding * fix: file rename * stop movement on button hover by keeping same border thickness * fix mistake in setting cursor: pointer * fix when free trial option is shown * Support Node.js versions below 20 for streaming response handling (#1591) - Add fallback for Node < 20 - Implement toAsyncIterable for streaming - Use TextDecoder for manual decoding - Maintain existing streaming for Node 20+ * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * fix hasPassedFTL * fix /edit cancellation from UI * feat: add `applyCodeBlock` experimental prop (#1601) * feat: add new model styling improvements (#1609) * feat: add new model styling improvements * better gap size * feat: update bug_report.yml (#1610) * chore: update bug_report.yml * typo fix * feat: add labels to "Add docs" dialog (#1612) * feat: add labels to "Add docs" dialog * remove autofocus * don't double load config * small fixes * speed up directory traversal, and use correct native path module * option not to show config update toast * merge air-gapped and recommended setup pages * chore: add telemetry for full screen toggle (#1618) * Fix headings in codebase-embeddings.md (#1617) * mention jetbrains * docs: update changie (#1619) * feat: updated changie config * hide toc and autogenerate * Update changelog.mdx * link to deeper explanation of embeddings models * ensure target="_blank" for all links in sidebar * fix gif links in intellij README.md * don't require rust in dependency installation * chore: fix padding on gh button (#1620) * chore: adjust button padding * Update tasks.json * escape colons in diff path * smoother lancedb indexing reporting * smooth progress updates for indexing * fix tsc err * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * docs: add docs and schema for "OS" provider (#1536) * ignore .env * 🚑 fix constant warnings when onboarding with Ollama * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * team analytics * apply control plane settings * workos auth * ide protocol get session info * UI for auth * profile switching * small fixes * updates * refresh tokens * updates * fix tsc errs * model select in toolbar to make room for profile selector * prod client id * link to prod URL * internal beta option * profiles change listener --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * temporary patch for type errs --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * skip extension tests * JetBrains status bar spinner for autocomplete * Remove unused images * partial accept jetbrains * fix double-rendering of jetbrains completions * fix offset of multi-line jetbrains completions * enable/disable actions * upgrade to version 0.0.54 * fix: wrap editor buttons when out of space (#1727) * Add .continuerc file to config directory on activation to prevent it's indexing. (#1733) * Updated the content links that were deprecated in the document (#1709) Co-authored-by: Ihe Fan <ih.fan.1027@gmail.com> * Nate/dev (#1739) * JetBrains status bar spinner for autocomplete * Remove unused images * partial accept jetbrains * fix double-rendering of jetbrains completions * fix offset of multi-line jetbrains completions * enable/disable actions * upgrade to version 0.0.54 * llm-info package * small profile switching improvements * instruct cmd+I not to leave placeholders * @ files in context-providers docs * onboarding fix * update jetbrains ci to build gui * feat: single default quick pick to edit (#1743) * feat: single default quick pick to edit * Update quick-actions.md * Update quick-actions.md * use defaultQuickAction * quick fixes * fix: handle line removal diff (#1744) * fix: handle line removal diff * remove comments * Configurable `maxChunkSize` for embedding providers (#1746) * Add `maxChunkSize` to `EmbeddingsProvider` * Add `maxChunkSize` to schema * add support for docs via config (#1594) * mistral mamba (#1748) * 🐍 mistral mamba * autoscale retrieval based on context length * autoscale retrieval based on context length * fix input box movement * update jetbrains build * test: add myers diff tests (#1754) * feat: enhance help center (#1755) * feat: enhance help center * Update help.tsx * feat: add azure provider config (#1764) * autocomplete reload fix * logout of control plane * Update QuickEditQuickPick.ts (#1772) * Update QuickEditQuickPick.ts * updates * Update QuickEditQuickPick.ts * pass refreshed access token to continue-proxy model * test: add dir read check for walkDir (#1773) * test: add dir read check for walkDir * Update ignore.ts * Update walkDir.test.ts * Update walkDir.test.ts * gpt-4o-mini * gpt-4o-mini * gpt4omini to ui selector * fix: layout alignment (#1779) * fix: delete old docs index on force re-index (#1778) * add "Codebase Force Re-Index" command (#1757) * Add `HuggingFaceTEIReranker` (#1711) * Add `HuggingFaceTEIReranker` * Add `huggingface-tei` to `config_schema.json` * update package.json version * update config schemas * skip test failing only due to type error * fix: convert `walkDir` to an async generator (#1783) * walkdir to async generator * add yields * handle ignore files * Update CodebaseIndexer.ts * Update walkDir.ts * chore: bump VS Code ext version * fix completions support in ContinueProxy * add test for walkDir * IAnalyticsProvider * feat: crawl `.mdx` docs * update workspaces on session info change * rename from profile to workspace (user-facing) * update config schema files * feat: improve chat thread ui for better readability (#1786) * add "language" template variable for autocomplete * Enable debug logs (#1800) * Add an enableDebugLogs flag * Send logs to VSCode * Refresh index fixes (#1795) * Use the resultType given to markComplete This fixes an issue where if there were multiple result types in itemToAction such as an update with UpdateNewVersion and UpdateOldVersion, only the UpdateOldVersion would be hit (which is a noop) and the UpdateNewVersion would never be fired, causing us to never update the timestamp of the tag_catalog. * Add a Compute and UpdateLastUpdated to refreshIndex Removes the ability to insert duplicate rows if we're just computing the result. Instead we replace the existing row instead. * Don't insert compute tags They are REPLACEd in markComplete now. * docs: change page heading to model providers (#1684) * feat: allow JetBrains users to index docs (#1797) * feat: allow JetBrains users to index docs * update docs service * update global context * throw error popup on jetbrains + transformers * update toast notifications * update typings * fix tsc errors * Index on file change (#1785) * Reload the submenu items when the context provider data is refreshed * update version * Just call refreshSubmenuItems * remove log statements * cleanup * Remove the Promise declaration * Copy currentFiles to a new dict since we're going to mutate it Don't mutate the caller's currentFiles unexpectedly * Remove reIndexFile and just call index/forceReIndex * We MUST await the refreshCodebaseIndex call * Expose send on core * remove unused import * Reindex the workspaces on save * Call core.send Ideally, we can implement externalRequest on the webview later so we don't have to expose messenger to the VSCode extension * Always refreshSubmenuItems in refreshCodebaseIndex * whitespace * Change messenger back to private * Add Pause Codebase On Start setting (#1788) * Add a skipInitialSync function * Add pauseInitialCodebaseIndex * docs * version bump * Rename to pauseCodebaseIndexOnStart * Un-shadow file context provider (#1801) * allow FileContextProvider to handle files * remove commented-out lines * chore: add telemetry to alt-enter (#1805) * Set the indexProgress indicator to paused on startup if pauseCodebaseIndexOnStart is set (#1804) * Fix Input Toolbar layout (#1792) * set to flex flex-column * Remove fixed height and always show the StyledDivs content to hold its size * Add a Delete Message button to the StepContainer (#1790) * Add a delete button * add a delete message action to the GUI * Fix off by one error * format * This index should be less than length * Update to trash icon * Simplify walkDir(...) and improve performance by roughly 10x in larger repos (#1806) The performance of WalkDir is improved by replacing the recursive DFS with a stack based implementation (less memory in-use during peak exploration). In addition, some unnecessary flags and options were removed which allowed the code to be overall simplified. Much less rule matching is occuring with the same files returned. Some performance results on my linux host: small repo: - 649 files returned by walkDir() - 9 .gitignore files - walkDir() old vs new duration: 918 ms to 143 ms medium sized repo: - 9271 returned by walkDir() - 35 .gitignore files - walkDir() old vs new duration: 10,886 ms to 970ms large sized repo: - 197,900 files returned by walkDir() - 929 .gitignore files - walkDir() old vs new duration: 498,151 ms to 37,353 ms Co-authored-by: Rob Leidle <rleidle@tesla.com> * modified assistant indicator in UI * update package.json version * feat: add Llama 3.1 8B to cloudflare provider options (#1811) Co-authored-by: jdelorey@cloudflare.com <jdelorey@cloudflare.com> * increase gemini pro contextLength (#1809) * fix indentation (#1808) * Llama 3.1 405b model selection (#1813) * add support for llama3 405b * 3.1 * name mapping * updates to model setup * fix walkDir tests on windows * chore: add docs as top level config (#1816) * chore: add docs as top level config * Update core.ts * feat: make @codebase a hardcoded ctx provider (#1818) * chore: add docs as top level config * Update core.ts * feat: make @codebase a hardcoded ctx provider * docs schema * Nate/dev (#1819) * llm-info package * small profile switching improvements * instruct cmd+I not to leave placeholders * @ files in context-providers docs * onboarding fix * jetbrains login * fix jetbrains login * change beta wording * proxy context provider * fix proxy context provider * revert to prod id, url * update package.json version * Add CodeGeeX4 code completion support (#1815) Includes support for context through code snippets and relative filename resolution. * manually stop at stop tokens for all providers * chore: bump tree-sitter (#1828) * Add Missing Tokens for CodeGeeX4 AutoComplete (#1827) Previous attempts at using the normal pipe char with CodeGeeX FIM tokens resulted in empty completion output. However, this was due to <|user|> and <|assistant|> tokens being necessary for completion. The earlier working change ommitted these tokens, and its functionality using deepseek pipes was merely coincidental. This change now properly follows the CodeGeeX infill guides and should match what the model expects. In addition to <|user|> and <|assistant|>, the <|endoftext|> token was added as a stop token. * additional analytics providers * fix import error in jetbrains plugin * explicitly include default context providers * Remove a copy / pasta section that duplicated the work done in ChunkCodebaseIndexer (#1834) This cuts the amount of time spent in this section of indexing in half Co-authored-by: Rob Leidle <rleidle@tesla.com> * respect VS Code isTelemetryEnabled * Update OllamaEmbeddingsProvider.ts (#1839) * feat: include recently + open files in codebase search (#1833) * feat: include recently + open files in codebase search * cleanup * Update BaseRetrievalPipeline.ts * add params object to chunkDocument * Update package-lock.json * Offload all token counting to worker processes as well as some optimizations to do more token counting in parallel Adding the workerpool logic broke gui as vite does not support __dirname or path.join(...). To work around this, moved the gui dependency of stripImages out of countTokens.ts * Create CODE_OF_CONDUCT.md (#1843) * tweaks to new tokenizing * temporarily skip code chunking tests due to worker files not being present * update changelog * update version * jetbrains refresh token updates * refresh submenu items on workspace switch * OpenAI Adapters (#1859) * embeddings support in openai-adapters * rerank support in openai-adapters * cohere embed/rerank * continue-proxy reranker and embeddings providers * test for openai-adapters * embeddings provider and reranker for continue proxy * fix: missing Content-Type header in OllamaEmbeddingsProvider (#1855) * Add Language Name Param to AutocompleteTemplate (#1853) Language name was previously added as a parameter only for string autocomplete templates. This change provides this same information to template functions that implement the AutocompleteTemplate interface. Language name is explicitly supported for the CodeGeeX4 autocomplete template, and is added here in this change. Explicit language awareness could also be added for other templates in a future change, like the holeFillerTemplate used for models without explicit completion support. * update package.json version --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> Co-authored-by: Rob Leidle <rleidle@gmail.com> Co-authored-by: Rob Leidle <rleidle@tesla.com> Co-authored-by: inimaz <49730431+inimaz@users.noreply.github.com> Co-authored-by: inimaz <imaz@ai-verse.com> Co-authored-by: Anaël Bonnafous <71386173+AnaelBonnafous@users.noreply.github.com> Co-authored-by: Huy Tran <119860259+tranquochuy645@users.noreply.github.com> Co-authored-by: Gabriel Gordbegli <ggordbegli@gmail.com> Co-authored-by: Raven-1027 <83693755+Raven-1027@users.noreply.github.com> Co-authored-by: Ihe Fan <ih.fan.1027@gmail.com> Co-authored-by: Lukas Kreussel <65088241+LLukas22@users.noreply.github.com> Co-authored-by: Josh Vera <josh@joshvera.com> Co-authored-by: Peter Zaback <pzaback@gmail.com> Co-authored-by: James Delorey <deloreyjmi@gmail.com> Co-authored-by: jdelorey@cloudflare.com <jdelorey@cloudflare.com> Co-authored-by: ekatiyar <eashaank@gmail.com> Co-authored-by: Simo <simo.paasisalo@syke.fi> * DeepSeek FIM Support (#1863) * Nate/control plane client (#1691) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * fix(gui): ctx rendering w/ renderInlineAs: "" (#1541) * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * fix codeqwen autocomplete leading space * add voyage rerank-1 * feat: `--noEmit` for tsc checks in CI (#1559) * docs: update CustomContextProvider docs (#1557) * add stop tokens to qwen prompt * update docs to reflect 3.5 sonnet being best * docs: comment out unused providers (#1561) * import Handlebars * feat: toast notification for config updates (#1560) * feat: toast notification for config updates * feat: only trigger toast on config.json save * displayRawMarkdown option * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * update to reflect 16 stop words limit for deepseek * feat: only trigger config update toast in vscode (#1571) * docs(prompt-files): fix typos + clarify (#1575) * doc: prompt file typo + clarifications * fix: add back correct docs * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * docs(telemetry): add pageviews to tracking list (#1581) * Add reranker configuration options to codebase embedding docs (#1584) - Introduce reranker concept - List available reranker options - Provide configuration instructions - Update keywords to include "reranker" * chore: update pr template with screenshots (#1590) * Refactor ConfirmationDialog to use SecondaryButton for cancel action (#1586) * Added instructions for running docs server locally (#1578) - Added NPM script method - Added VS Code task method - Update contributing guidelines * Update branch policy (#1577) - Change PR target to `dev` branch - Update `CONTRIBUTING.md` instructions * Consolidate example configurations into the main configuration guide (#1579) - Moved examples to configuration.md - Deleted the separate examples.md file - Updated sidebar order and links - Improved readability and structure in configuration.md * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * Update completionProvider.ts (warning tab-autocomplete models) (#1566) * feat: enhanced IndexingProgressBar with blinking dot feature - Integrated BlinkingDot component - Added STATUS_COLORS for various states - Replaced CircleDiv with BlinkingDot in UI - Updated status messages and layout * small UI tweaks * feat(gui): enhance ModelCard, ModelProviderTag, and Toggle components (#1595) - add styling and adjustments to ModelCard - update ModelProviderTag font size - remove box shadow from Toggle component - tweak icon styles in ModelCard - improve alignment and spacing * media query * feat: add best experience onboarding * fix: file rename * stop movement on button hover by keeping same border thickness * fix mistake in setting cursor: pointer * fix when free trial option is shown * Support Node.js versions below 20 for streaming response handling (#1591) - Add fallback for Node < 20 - Implement toAsyncIterable for streaming - Use TextDecoder for manual decoding - Maintain existing streaming for Node 20+ * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * fix hasPassedFTL * fix /edit cancellation from UI * feat: add `applyCodeBlock` experimental prop (#1601) * feat: add new model styling improvements (#1609) * feat: add new model styling improvements * better gap size * feat: update bug_report.yml (#1610) * chore: update bug_report.yml * typo fix * feat: add labels to "Add docs" dialog (#1612) * feat: add labels to "Add docs" dialog * remove autofocus * don't double load config * small fixes * speed up directory traversal, and use correct native path module * option not to show config update toast * merge air-gapped and recommended setup pages * chore: add telemetry for full screen toggle (#1618) * Fix headings in codebase-embeddings.md (#1617) * mention jetbrains * docs: update changie (#1619) * feat: updated changie config * hide toc and autogenerate * Update changelog.mdx * link to deeper explanation of embeddings models * ensure target="_blank" for all links in sidebar * fix gif links in intellij README.md * don't require rust in dependency installation * chore: fix padding on gh button (#1620) * chore: adjust button padding * Update tasks.json * escape colons in diff path * smoother lancedb indexing reporting * smooth progress updates for indexing * fix tsc err * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * docs: add docs and schema for "OS" provider (#1536) * ignore .env * 🚑 fix constant warnings when onboarding with Ollama * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * team analytics * apply control plane settings * workos auth * ide protocol get session info * UI for auth * profile switching * small fixes * updates * refresh tokens * updates * fix tsc errs * model select in toolbar to make room for profile selector * prod client id * link to prod URL * internal beta option * profiles change listener --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * temporary patch for type errs * Dev (#1689) * Fix an issue where CMD+K does not clear the terminal when the terminal has focus (#1671) On MacOS, ⌘+K is bound, by default, to Terminal:Clear. Without this change ⌘+K does not clear the terminal but instead iniates a chord sequence and waits for the next stroke of the chord. Co-authored-by: Rob Leidle <rleidle@tesla.com> * Change treeSitter to cache the Language objects it loads from wasm (#1672) Without this change, for a repository with 600 typescript files, the indexer would fail to finish correctly and there would be many of the following errors in the webview console log: 'Unable to load language for file ${path} RuntimeError: table index is out of bounds' The following bash will create a repo that reproduces the problem: current_path="." for ((i=1; i<=20; i++)); do new_folder="folder-$i" mkdir -p "$current_path/$new_folder" current_path="$current_path/$new_folder" for ((a=1; a<=30; a++)); do head -c 10000 /dev/urandom | base64 > "$current_path/file-$a.ts" done done Co-authored-by: Rob Leidle <rleidle@tesla.com> * acknowledge sourcemap flag in esbuild.js * don't run jetbrains-release.yaml on vscode releases * further testing for walkDir * chore: add telemetry to commands (#1673) * test: Add basic unit test to baseLLM (#1668) * update version * test: Add basic unit test to baseLLM --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: inimaz <imaz@ai-verse.com> * feat: add Quick Actions CodeLens feature (#1674) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * add voyage rerank-1 * import Handlebars * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * small UI tweaks * media query * feat: add best experience onboarding * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * chore: add telemetry for full screen toggle (#1618) * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * manually implement stop tokens for hf inference api * chore: onboarding metrics (#1626) * fix: pageview tracking * feat: add onboarding telemetry * create single `onboardingStatus` type * improved var naming * remove console logs * fix windows performance issue * rename vscodeExtension.ts * migration of onboarding variables * "stash" instead of "delete" in indexing progress * fix preview.yaml * also fix main.yaml * Update troubleshooting.md (#1637) * feat: add quick actions * Update index.d.ts * quick actions mvp * update docs * subscribe to vscode change settings * Update commands.ts * cleanup * Update quick-actions.md * Update VerticalPerLineCodeLensProvider.ts * resolve feedback --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: Nate Sesti <33237525+sestinj@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> * chore: add `isCommandEvent` to command telemetry (#1675) * chore: add `isCommandEvent` to command telemetry * Update commands.ts * Nate/better retrieval (#1677) * deduplicatearray tests * break out separate retrieval pipelines * IConfigHandler * tests for codebase indexer * better .continueignore for continue * indexing fixes * ignore .gitignore and .continueignore when indexing * retrieval pipeline improvements * fix formatting err in out .continueignore * add necessary filter to lance_db_cache * update package.json version * skip unused tests * don't ignore .prompt files * update version * Update pull_request_template.md * don't use multi-media format when there are multiple text items * add free trial experience (#1685) * fix: add code range for quick actions/fixes (#1687) * fix: add code range for quick actions/fixes * Update test.js * add pathSep message type * docs improvements * jetbrains fix * update package.json version --------- Co-authored-by: Rob Leidle <rleidle@gmail.com> Co-authored-by: Rob Leidle <rleidle@tesla.com> Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: inimaz <49730431+inimaz@users.noreply.github.com> Co-authored-by: inimaz <imaz@ai-verse.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> * update package.json version * skip extension tests * update package.json version * handlebars import * don't push package.json version change from CI * fix: scroll issues w/ code blocks (#1688) * fix: scroll issues w/ code blocks * chore: name offset * chore: remove docs start * chore: update headings on troubleshooting.md (#1696) * ignore .svn folder for indexing (#1699) * chore: clearer naming for `useSuffix` (#1702) * chore: clearer naming for `useSuffix` * schema updates * fix: cmd+shft+l closes sidebar if focused (#1638) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * add voyage rerank-1 * import Handlebars * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * small UI tweaks * media query * feat: add best experience onboarding * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * chore: add telemetry for full screen toggle (#1618) * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * manually implement stop tokens for hf inference api * chore: onboarding metrics (#1626) * fix: pageview tracking * feat: add onboarding telemetry * create single `onboardingStatus` type * improved var naming * remove console logs * fix windows performance issue * rename vscodeExtension.ts * migration of onboarding variables * "stash" instead of "delete" in indexing progress * fix preview.yaml * also fix main.yaml * Update troubleshooting.md (#1637) * feat: close panel if main input is focused * add skip param * Update TipTapEditor.tsx * merge dev * Update commands.ts --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: Nate Sesti <33237525+sestinj@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> * bugfix: quick actions undefined array (#1704) * feat: add rich quick pick for quick edit (#1706) * feat: multistep quick pick * fix: multi step for context provider * Update test.js * Create deepseek.md (#1708) * style: add max height to model selector (#1712) * update styling * remove testing code * remove comments * feat: add tutorial card (#1716) * feat: create tutorial card * Update TutorialCard.tsx * feat: add file search to quick edit (#1714) * feat: multistep quick pick * fix: multi step for context provider * Update test.js * add file search * complete file search * remove unused import * add comments * Update utils.ts * update naming * chore: move quick pick instantiation (#1723) * Break out into NPM modules (#1726) * config.json types * export all schemas * break out request logic into packages * update openai adapters to export body types * update packaging of npm modules * update packages * continue proxy * mistral * Fix typo in documentation (#1722) In set-up-codestral.md "Trobleshooting" -> "Troubleshooting" * update package.json version * 🪄 model dropdown, control plane (#1692) * Nate/control plane client (#1691) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * fix(gui): ctx rendering w/ renderInlineAs: "" (#1541) * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * fix codeqwen autocomplete leading space * add voyage rerank-1 * feat: `--noEmit` for tsc checks in CI (#1559) * docs: update CustomContextProvider docs (#1557) * add stop tokens to qwen prompt * update docs to reflect 3.5 sonnet being best * docs: comment out unused providers (#1561) * import Handlebars * feat: toast notification for config updates (#1560) * feat: toast notification for config updates * feat: only trigger toast on config.json save * displayRawMarkdown option * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * update to reflect 16 stop words limit for deepseek * feat: only trigger config update toast in vscode (#1571) * docs(prompt-files): fix typos + clarify (#1575) * doc: prompt file typo + clarifications * fix: add back correct docs * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * docs(telemetry): add pageviews to tracking list (#1581) * Add reranker configuration options to codebase embedding docs (#1584) - Introduce reranker concept - List available reranker options - Provide configuration instructions - Update keywords to include "reranker" * chore: update pr template with screenshots (#1590) * Refactor ConfirmationDialog to use SecondaryButton for cancel action (#1586) * Added instructions for running docs server locally (#1578) - Added NPM script method - Added VS Code task method - Update contributing guidelines * Update branch policy (#1577) - Change PR target to `dev` branch - Update `CONTRIBUTING.md` instructions * Consolidate example configurations into the main configuration guide (#1579) - Moved examples to configuration.md - Deleted the separate examples.md file - Updated sidebar order and links - Improved readability and structure in configuration.md * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * Update completionProvider.ts (warning tab-autocomplete models) (#1566) * feat: enhanced IndexingProgressBar with blinking dot feature - Integrated BlinkingDot component - Added STATUS_COLORS for various states - Replaced CircleDiv with BlinkingDot in UI - Updated status messages and layout * small UI tweaks * feat(gui): enhance ModelCard, ModelProviderTag, and Toggle components (#1595) - add styling and adjustments to ModelCard - update ModelProviderTag font size - remove box shadow from Toggle component - tweak icon styles in ModelCard - improve alignment and spacing * media query * feat: add best experience onboarding * fix: file rename * stop movement on button hover by keeping same border thickness * fix mistake in setting cursor: pointer * fix when free trial option is shown * Support Node.js versions below 20 for streaming response handling (#1591) - Add fallback for Node < 20 - Implement toAsyncIterable for streaming - Use TextDecoder for manual decoding - Maintain existing streaming for Node 20+ * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * fix hasPassedFTL * fix /edit cancellation from UI * feat: add `applyCodeBlock` experimental prop (#1601) * feat: add new model styling improvements (#1609) * feat: add new model styling improvements * better gap size * feat: update bug_report.yml (#1610) * chore: update bug_report.yml * typo fix * feat: add labels to "Add docs" dialog (#1612) * feat: add labels to "Add docs" dialog * remove autofocus * don't double load config * small fixes * speed up directory traversal, and use correct native path module * option not to show config update toast * merge air-gapped and recommended setup pages * chore: add telemetry for full screen toggle (#1618) * Fix headings in codebase-embeddings.md (#1617) * mention jetbrains * docs: update changie (#1619) * feat: updated changie config * hide toc and autogenerate * Update changelog.mdx * link to deeper explanation of embeddings models * ensure target="_blank" for all links in sidebar * fix gif links in intellij README.md * don't require rust in dependency installation * chore: fix padding on gh button (#1620) * chore: adjust button padding * Update tasks.json * escape colons in diff path * smoother lancedb indexing reporting * smooth progress updates for indexing * fix tsc err * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * docs: add docs and schema for "OS" provider (#1536) * ignore .env * 🚑 fix constant warnings when onboarding with Ollama * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * team analytics * apply control plane settings * workos auth * ide protocol get session info * UI for auth * profile switching * small fixes * updates * refresh tokens * updates * fix tsc errs * model select in toolbar to make room for profile selector * prod client id * link to prod URL * internal beta option * profiles change listener --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * temporary patch for type errs --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * skip extension tests * JetBrains status bar spinner for autocomplete * Remove unused images * partial accept jetbrains * fix double-rendering of jetbrains completions * fix offset of multi-line jetbrains completions * enable/disable actions * upgrade to version 0.0.54 * fix: wrap editor buttons when out of space (#1727) * Add .continuerc file to config directory on activation to prevent it's indexing. (#1733) * Updated the content links that were deprecated in the document (#1709) Co-authored-by: Ihe Fan <ih.fan.1027@gmail.com> * Nate/dev (#1739) * JetBrains status bar spinner for autocomplete * Remove unused images * partial accept jetbrains * fix double-rendering of jetbrains completions * fix offset of multi-line jetbrains completions * enable/disable actions * upgrade to version 0.0.54 * llm-info package * small profile switching improvements * instruct cmd+I not to leave placeholders * @ files in context-providers docs * onboarding fix * update jetbrains ci to build gui * feat: single default quick pick to edit (#1743) * feat: single default quick pick to edit * Update quick-actions.md * Update quick-actions.md * use defaultQuickAction * quick fixes * fix: handle line removal diff (#1744) * fix: handle line removal diff * remove comments * Configurable `maxChunkSize` for embedding providers (#1746) * Add `maxChunkSize` to `EmbeddingsProvider` * Add `maxChunkSize` to schema * add support for docs via config (#1594) * mistral mamba (#1748) * 🐍 mistral mamba * autoscale retrieval based on context length * autoscale retrieval based on context length * fix input box movement * update jetbrains build * test: add myers diff tests (#1754) * feat: enhance help center (#1755) * feat: enhance help center * Update help.tsx * feat: add azure provider config (#1764) * autocomplete reload fix * logout of control plane * Update QuickEditQuickPick.ts (#1772) * Update QuickEditQuickPick.ts * updates * Update QuickEditQuickPick.ts * pass refreshed access token to continue-proxy model * test: add dir read check for walkDir (#1773) * test: add dir read check for walkDir * Update ignore.ts * Update walkDir.test.ts * Update walkDir.test.ts * gpt-4o-mini * gpt-4o-mini * gpt4omini to ui selector * fix: layout alignment (#1779) * fix: delete old docs index on force re-index (#1778) * add "Codebase Force Re-Index" command (#1757) * Add `HuggingFaceTEIReranker` (#1711) * Add `HuggingFaceTEIReranker` * Add `huggingface-tei` to `config_schema.json` * update package.json version * update config schemas * skip test failing only due to type error * fix: convert `walkDir` to an async generator (#1783) * walkdir to async generator * add yields * handle ignore files * Update CodebaseIndexer.ts * Update walkDir.ts * chore: bump VS Code ext version * fix completions support in ContinueProxy * add test for walkDir * IAnalyticsProvider * feat: crawl `.mdx` docs * update workspaces on session info change * rename from profile to workspace (user-facing) * update config schema files * feat: improve chat thread ui for better readability (#1786) * add "language" template variable for autocomplete * Enable debug logs (#1800) * Add an enableDebugLogs flag * Send logs to VSCode * Refresh index fixes (#1795) * Use the resultType given to markComplete This fixes an issue where if there were multiple result types in itemToAction such as an update with UpdateNewVersion and UpdateOldVersion, only the UpdateOldVersion would be hit (which is a noop) and the UpdateNewVersion would never be fired, causing us to never update the timestamp of the tag_catalog. * Add a Compute and UpdateLastUpdated to refreshIndex Removes the ability to insert duplicate rows if we're just computing the result. Instead we replace the existing row instead. * Don't insert compute tags They are REPLACEd in markComplete now. * docs: change page heading to model providers (#1684) * feat: allow JetBrains users to index docs (#1797) * feat: allow JetBrains users to index docs * update docs service * update global context * throw error popup on jetbrains + transformers * update toast notifications * update typings * fix tsc errors * Index on file change (#1785) * Reload the submenu items when the context provider data is refreshed * update version * Just call refreshSubmenuItems * remove log statements * cleanup * Remove the Promise declaration * Copy currentFiles to a new dict since we're going to mutate it Don't mutate the caller's currentFiles unexpectedly * Remove reIndexFile and just call index/forceReIndex * We MUST await the refreshCodebaseIndex call * Expose send on core * remove unused import * Reindex the workspaces on save * Call core.send Ideally, we can implement externalRequest on the webview later so we don't have to expose messenger to the VSCode extension * Always refreshSubmenuItems in refreshCodebaseIndex * whitespace * Change messenger back to private * Add Pause Codebase On Start setting (#1788) * Add a skipInitialSync function * Add pauseInitialCodebaseIndex * docs * version bump * Rename to pauseCodebaseIndexOnStart * Un-shadow file context provider (#1801) * allow FileContextProvider to handle files * remove commented-out lines * chore: add telemetry to alt-enter (#1805) * Set the indexProgress indicator to paused on startup if pauseCodebaseIndexOnStart is set (#1804) * Fix Input Toolbar layout (#1792) * set to flex flex-column * Remove fixed height and always show the StyledDivs content to hold its size * Add a Delete Message button to the StepContainer (#1790) * Add a delete button * add a delete message action to the GUI * Fix off by one error * format * This index should be less than length * Update to trash icon * Simplify walkDir(...) and improve performance by roughly 10x in larger repos (#1806) The performance of WalkDir is improved by replacing the recursive DFS with a stack based implementation (less memory in-use during peak exploration). In addition, some unnecessary flags and options were removed which allowed the code to be overall simplified. Much less rule matching is occuring with the same files returned. Some performance results on my linux host: small repo: - 649 files returned by walkDir() - 9 .gitignore files - walkDir() old vs new duration: 918 ms to 143 ms medium sized repo: - 9271 returned by walkDir() - 35 .gitignore files - walkDir() old vs new duration: 10,886 ms to 970ms large sized repo: - 197,900 files returned by walkDir() - 929 .gitignore files - walkDir() old vs new duration: 498,151 ms to 37,353 ms Co-authored-by: Rob Leidle <rleidle@tesla.com> * modified assistant indicator in UI * update package.json version * feat: add Llama 3.1 8B to cloudflare provider options (#1811) Co-authored-by: jdelorey@cloudflare.com <jdelorey@cloudflare.com> * increase gemini pro contextLength (#1809) * fix indentation (#1808) * Llama 3.1 405b model selection (#1813) * add support for llama3 405b * 3.1 * name mapping * updates to model setup * fix walkDir tests on windows * chore: add docs as top level config (#1816) * chore: add docs as top level config * Update core.ts * feat: make @codebase a hardcoded ctx provider (#1818) * chore: add docs as top level config * Update core.ts * feat: make @codebase a hardcoded ctx provider * docs schema * Nate/dev (#1819) * llm-info package * small profile switching improvements * instruct cmd+I not to leave placeholders * @ files in context-providers docs * onboarding fix * jetbrains login * fix jetbrains login * change beta wording * proxy context provider * fix proxy context provider * revert to prod id, url * update package.json version * Add CodeGeeX4 code completion support (#1815) Includes support for context through code snippets and relative filename resolution. * manually stop at stop tokens for all providers * chore: bump tree-sitter (#1828) * Add Missing Tokens for CodeGeeX4 AutoComplete (#1827) Previous attempts at using the normal pipe char with CodeGeeX FIM tokens resulted in empty completion output. However, this was due to <|user|> and <|assistant|> tokens being necessary for completion. The earlier working change ommitted these tokens, and its functionality using deepseek pipes was merely coincidental. This change now properly follows the CodeGeeX infill guides and should match what the model expects. In addition to <|user|> and <|assistant|>, the <|endoftext|> token was added as a stop token. * additional analytics providers * fix import error in jetbrains plugin * explicitly include default context providers * Remove a copy / pasta section that duplicated the work done in ChunkCodebaseIndexer (#1834) This cuts the amount of time spent in this section of indexing in half Co-authored-by: Rob Leidle <rleidle@tesla.com> * respect VS Code isTelemetryEnabled * Update OllamaEmbeddingsProvider.ts (#1839) * feat: include recently + open files in codebase search (#1833) * feat: include recently + open files in codebase search * cleanup * Update BaseRetrievalPipeline.ts * add params object to chunkDocument * Update package-lock.json * Offload all token counting to worker processes as well as some optimizations to do more token counting in parallel Adding the workerpool logic broke gui as vite does not support __dirname or path.join(...). To work around this, moved the gui dependency of stripImages out of countTokens.ts * Create CODE_OF_CONDUCT.md (#1843) * tweaks to new tokenizing * temporarily skip code chunking tests due to worker files not being present * update changelog * update version * jetbrains refresh token updates * refresh submenu items on workspace switch * OpenAI Adapters (#1859) * embeddings support in openai-adapters * rerank support in openai-adapters * cohere embed/rerank * continue-proxy reranker and embeddings providers * test for openai-adapters * embeddings provider and reranker for continue proxy * fix: missing Content-Type header in OllamaEmbeddingsProvider (#1855) * Add Language Name Param to AutocompleteTemplate (#1853) Language name was previously added as a parameter only for string autocomplete templates. This change provides this same information to template functions that implement the AutocompleteTemplate interface. Language name is explicitly supported for the CodeGeeX4 autocomplete template, and is added here in this change. Explicit language awareness could also be added for other templates in a future change, like the holeFillerTemplate used for models without explicit completion support. * update package.json version * deepseek FIM endpoint (#1862) * update package.json version --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> Co-authored-by: Rob Leidle <rleidle@gmail.com> Co-authored-by: Rob Leidle <rleidle@tesla.com> Co-authored-by: inimaz <49730431+inimaz@users.noreply.github.com> Co-authored-by: inimaz <imaz@ai-verse.com> Co-authored-by: Anaël Bonnafous <71386173+AnaelBonnafous@users.noreply.github.com> Co-authored-by: Huy Tran <119860259+tranquochuy645@users.noreply.github.com> Co-authored-by: Gabriel Gordbegli <ggordbegli@gmail.com> Co-authored-by: Raven-1027 <83693755+Raven-1027@users.noreply.github.com> Co-authored-by: Ihe Fan <ih.fan.1027@gmail.com> Co-authored-by: Lukas Kreussel <65088241+LLukas22@users.noreply.github.com> Co-authored-by: Josh Vera <josh@joshvera.com> Co-authored-by: Peter Zaback <pzaback@gmail.com> Co-authored-by: James Delorey <deloreyjmi@gmail.com> Co-authored-by: jdelorey@cloudflare.com <jdelorey@cloudflare.com> Co-authored-by: ekatiyar <eashaank@gmail.com> Co-authored-by: Simo <simo.paasisalo@syke.fi> * update gradle version [skip ci] (#1865) * Nate/control plane client (#1691) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * fix(gui): ctx rendering w/ renderInlineAs: "" (#1541) * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * fix codeqwen autocomplete leading space * add voyage rerank-1 * feat: `--noEmit` for tsc checks in CI (#1559) * docs: update CustomContextProvider docs (#1557) * add stop tokens to qwen prompt * update docs to reflect 3.5 sonnet being best * docs: comment out unused providers (#1561) * import Handlebars * feat: toast notification for config updates (#1560) * feat: toast notification for config updates * feat: only trigger toast on config.json save * displayRawMarkdown option * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * update to reflect 16 stop words limit for deepseek * feat: only trigger config update toast in vscode (#1571) * docs(prompt-files): fix typos + clarify (#1575) * doc: prompt file typo + clarifications * fix: add back correct docs * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * docs(telemetry): add pageviews to tracking list (#1581) * Add reranker configuration options to codebase embedding docs (#1584) - Introduce reranker concept - List available reranker options - Provide configuration instructions - Update keywords to include "reranker" * chore: update pr template with screenshots (#1590) * Refactor ConfirmationDialog to use SecondaryButton for cancel action (#1586) * Added instructions for running docs server locally (#1578) - Added NPM script method - Added VS Code task method - Update contributing guidelines * Update branch policy (#1577) - Change PR target to `dev` branch - Update `CONTRIBUTING.md` instructions * Consolidate example configurations into the main configuration guide (#1579) - Moved examples to configuration.md - Deleted the separate examples.md file - Updated sidebar order and links - Improved readability and structure in configuration.md * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * Update completionProvider.ts (warning tab-autocomplete models) (#1566) * feat: enhanced IndexingProgressBar with blinking dot feature - Integrated BlinkingDot component - Added STATUS_COLORS for various states - Replaced CircleDiv with BlinkingDot in UI - Updated status messages and layout * small UI tweaks * feat(gui): enhance ModelCard, ModelProviderTag, and Toggle components (#1595) - add styling and adjustments to ModelCard - update ModelProviderTag font size - remove box shadow from Toggle component - tweak icon styles in ModelCard - improve alignment and spacing * media query * feat: add best experience onboarding * fix: file rename * stop movement on button hover by keeping same border thickness * fix mistake in setting cursor: pointer * fix when free trial option is shown * Support Node.js versions below 20 for streaming response handling (#1591) - Add fallback for Node < 20 - Implement toAsyncIterable for streaming - Use TextDecoder for manual decoding - Maintain existing streaming for Node 20+ * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * fix hasPassedFTL * fix /edit cancellation from UI * feat: add `applyCodeBlock` experimental prop (#1601) * feat: add new model styling improvements (#1609) * feat: add new model styling improvements * better gap size * feat: update bug_report.yml (#1610) * chore: update bug_report.yml * typo fix * feat: add labels to "Add docs" dialog (#1612) * feat: add labels to "Add docs" dialog * remove autofocus * don't double load config * small fixes * speed up directory traversal, and use correct native path module * option not to show config update toast * merge air-gapped and recommended setup pages * chore: add telemetry for full screen toggle (#1618) * Fix headings in codebase-embeddings.md (#1617) * mention jetbrains * docs: update changie (#1619) * feat: updated changie config * hide toc and autogenerate * Update changelog.mdx * link to deeper explanation of embeddings models * ensure target="_blank" for all links in sidebar * fix gif links in intellij README.md * don't require rust in dependency installation * chore: fix padding on gh button (#1620) * chore: adjust button padding * Update tasks.json * escape colons in diff path * smoother lancedb indexing reporting * smooth progress updates for indexing * fix tsc err * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * docs: add docs and schema for "OS" provider (#1536) * ignore .env * 🚑 fix constant warnings when onboarding with Ollama * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * team analytics * apply control plane settings * workos auth * ide protocol get session info * UI for auth * profile switching * small fixes * updates * refresh tokens * updates * fix tsc errs * model select in toolbar to make room for profile selector * prod client id * link to prod URL * internal beta option * profiles change listener --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * temporary patch for type errs * Dev (#1689) * Fix an issue where CMD+K does not clear the terminal when the terminal has focus (#1671) On MacOS, ⌘+K is bound, by default, to Terminal:Clear. Without this change ⌘+K does not clear the terminal but instead iniates a chord sequence and waits for the next stroke of the chord. Co-authored-by: Rob Leidle <rleidle@tesla.com> * Change treeSitter to cache the Language objects it loads from wasm (#1672) Without this change, for a repository with 600 typescript files, the indexer would fail to finish correctly and there would be many of the following errors in the webview console log: 'Unable to load language for file ${path} RuntimeError: table index is out of bounds' The following bash will create a repo that reproduces the problem: current_path="." for ((i=1; i<=20; i++)); do new_folder="folder-$i" mkdir -p "$current_path/$new_folder" current_path="$current_path/$new_folder" for ((a=1; a<=30; a++)); do head -c 10000 /dev/urandom | base64 > "$current_path/file-$a.ts" done done Co-authored-by: Rob Leidle <rleidle@tesla.com> * acknowledge sourcemap flag in esbuild.js * don't run jetbrains-release.yaml on vscode releases * further testing for walkDir * chore: add telemetry to commands (#1673) * test: Add basic unit test to baseLLM (#1668) * update version * test: Add basic unit test to baseLLM --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: inimaz <imaz@ai-verse.com> * feat: add Quick Actions CodeLens feature (#1674) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * add voyage rerank-1 * import Handlebars * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * small UI tweaks * media query * feat: add best experience onboarding * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * chore: add telemetry for full screen toggle (#1618) * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * manually implement stop tokens for hf inference api * chore: onboarding metrics (#1626) * fix: pageview tracking * feat: add onboarding telemetry * create single `onboardingStatus` type * improved var naming * remove console logs * fix windows performance issue * rename vscodeExtension.ts * migration of onboarding variables * "stash" instead of "delete" in indexing progress * fix preview.yaml * also fix main.yaml * Update troubleshooting.md (#1637) * feat: add quick actions * Update index.d.ts * quick actions mvp * update docs * subscribe to vscode change settings * Update commands.ts * cleanup * Update quick-actions.md * Update VerticalPerLineCodeLensProvider.ts * resolve feedback --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: Nate Sesti <33237525+sestinj@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> * chore: add `isCommandEvent` to command telemetry (#1675) * chore: add `isCommandEvent` to command telemetry * Update commands.ts * Nate/better retrieval (#1677) * deduplicatearray tests * break out separate retrieval pipelines * IConfigHandler * tests for codebase indexer * better .continueignore for continue * indexing fixes * ignore .gitignore and .continueignore when indexing * retrieval pipeline improvements * fix formatting err in out .continueignore * add necessary filter to lance_db_cache * update package.json version * skip unused tests * don't ignore .prompt files * update version * Update pull_request_template.md * don't use multi-media format when there are multiple text items * add free trial experience (#1685) * fix: add code range for quick actions/fixes (#1687) * fix: add code range for quick actions/fixes * Update test.js * add pathSep message type * docs improvements * jetbrains fix * update package.json version --------- Co-authored-by: Rob Leidle <rleidle@gmail.com> Co-authored-by: Rob Leidle <rleidle@tesla.com> Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: inimaz <49730431+inimaz@users.noreply.github.com> Co-authored-by: inimaz <imaz@ai-verse.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> * update package.json version * skip extension tests * update package.json version * handlebars import * don't push package.json version change from CI * fix: scroll issues w/ code blocks (#1688) * fix: scroll issues w/ code blocks * chore: name offset * chore: remove docs start * chore: update headings on troubleshooting.md (#1696) * ignore .svn folder for indexing (#1699) * chore: clearer naming for `useSuffix` (#1702) * chore: clearer naming for `useSuffix` * schema updates * fix: cmd+shft+l closes sidebar if focused (#1638) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * add voyage rerank-1 * import Handlebars * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * small UI tweaks * media query * feat: add best experience onboarding * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * chore: add telemetry for full screen toggle (#1618) * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * manually implement stop tokens for hf inference api * chore: onboarding metrics (#1626) * fix: pageview tracking * feat: add onboarding telemetry * create single `onboardingStatus` type * improved var naming * remove console logs * fix windows performance issue * rename vscodeExtension.ts * migration of onboarding variables * "stash" instead of "delete" in indexing progress * fix preview.yaml * also fix main.yaml * Update troubleshooting.md (#1637) * feat: close panel if main input is focused * add skip param * Update TipTapEditor.tsx * merge dev * Update commands.ts --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: Nate Sesti <33237525+sestinj@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> * bugfix: quick actions undefined array (#1704) * feat: add rich quick pick for quick edit (#1706) * feat: multistep quick pick * fix: multi step for context provider * Update test.js * Create deepseek.md (#1708) * style: add max height to model selector (#1712) * update styling * remove testing code * remove comments * feat: add tutorial card (#1716) * feat: create tutorial card * Update TutorialCard.tsx * feat: add file search to quick edit (#1714) * feat: multistep quick pick * fix: multi step for context provider * Update test.js * add file search * complete file search * remove unused import * add comments * Update utils.ts * update naming * chore: move quick pick instantiation (#1723) * Break out into NPM modules (#1726) * config.json types * export all schemas * break out request logic into packages * update openai adapters to export body types * update packaging of npm modules * update packages * continue proxy * mistral * Fix typo in documentation (#1722) In set-up-codestral.md "Trobleshooting" -> "Troubleshooting" * update package.json version * 🪄 model dropdown, control plane (#1692) * Nate/control plane client (#1691) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * fix(gui): ctx rendering w/ renderInlineAs: "" (#1541) * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * fix codeqwen autocomplete leading space * add voyage rerank-1 * feat: `--noEmit` for tsc checks in CI (#1559) * docs: update CustomContextProvider docs (#1557) * add stop tokens to qwen prompt * update docs to reflect 3.5 sonnet being best * docs: comment out unused providers (#1561) * import Handlebars * feat: toast notification for config updates (#1560) * feat: toast notification for config updates * feat: only trigger toast on config.json save * displayRawMarkdown option * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * update to reflect 16 stop words limit for deepseek * feat: only trigger config update toast in vscode (#1571) * docs(prompt-files): fix typos + clarify (#1575) * doc: prompt file typo + clarifications * fix: add back correct docs * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * docs(telemetry): add pageviews to tracking list (#1581) * Add reranker configuration options to codebase embedding docs (#1584) - Introduce reranker concept - List available reranker options - Provide configuration instructions - Update keywords to include "reranker" * chore: update pr template with screenshots (#1590) * Refactor ConfirmationDialog to use SecondaryButton for cancel action (#1586) * Added instructions for running docs server locally (#1578) - Added NPM script method - Added VS Code task method - Update contributing guidelines * Update branch policy (#1577) - Change PR target to `dev` branch - Update `CONTRIBUTING.md` instructions * Consolidate example configurations into the main configuration guide (#1579) - Moved examples to configuration.md - Deleted the separate examples.md file - Updated sidebar order and links - Improved readability and structure in configuration.md * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * Update completionProvider.ts (warning tab-autocomplete models) (#1566) * feat: enhanced IndexingProgressBar with blinking dot feature - Integrated BlinkingDot component - Added STATUS_COLORS for various states - Replaced CircleDiv with BlinkingDot in UI - Updated status messages and layout * small UI tweaks * feat(gui): enhance ModelCard, ModelProviderTag, and Toggle components (#1595) - add styling and adjustments to ModelCard - update ModelProviderTag font size - remove box shadow from Toggle component - tweak icon styles in ModelCard - improve alignment and spacing * media query * feat: add best experience onboarding * fix: file rename * stop movement on button hover by keeping same border thickness * fix mistake in setting cursor: pointer * fix when free trial option is shown * Support Node.js versions below 20 for streaming response handling (#1591) - Add fallback for Node < 20 - Implement toAsyncIterable for streaming - Use TextDecoder for manual decoding - Maintain existing streaming for Node 20+ * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * fix hasPassedFTL * fix /edit cancellation from UI * feat: add `applyCodeBlock` experimental prop (#1601) * feat: add new model styling improvements (#1609) * feat: add new model styling improvements * better gap size * feat: update bug_report.yml (#1610) * chore: update bug_report.yml * typo fix * feat: add labels to "Add docs" dialog (#1612) * feat: add labels to "Add docs" dialog * remove autofocus * don't double load config * small fixes * speed up directory traversal, and use correct native path module * option not to show config update toast * merge air-gapped and recommended setup pages * chore: add telemetry for full screen toggle (#1618) * Fix headings in codebase-embeddings.md (#1617) * mention jetbrains * docs: update changie (#1619) * feat: updated changie config * hide toc and autogenerate * Update changelog.mdx * link to deeper explanation of embeddings models * ensure target="_blank" for all links in sidebar * fix gif links in intellij README.md * don't require rust in dependency installation * chore: fix padding on gh button (#1620) * chore: adjust button padding * Update tasks.json * escape colons in diff path * smoother lancedb indexing reporting * smooth progress updates for indexing * fix tsc err * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * docs: add docs and schema for "OS" provider (#1536) * ignore .env * 🚑 fix constant warnings when onboarding with Ollama * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * team analytics * apply control plane settings * workos auth * ide protocol get session info * UI for auth * profile switching * small fixes * updates * refresh tokens * updates * fix tsc errs * model select in toolbar to make room for profile selector * prod client id * link to prod URL * internal beta option * profiles change listener --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * temporary patch for type errs --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * skip extension tests * JetBrains status bar spinner for autocomplete * Remove unused images * partial accept jetbrains * fix double-rendering of jetbrains completions * fix offset of multi-line jetbrains completions * enable/disable actions * upgrade to version 0.0.54 * fix: wrap editor buttons when out of space (#1727) * Add .continuerc file to config directory on activation to prevent it's indexing. (#1733) * Updated the content links that were deprecated in the document (#1709) Co-authored-by: Ihe Fan <ih.fan.1027@gmail.com> * Nate/dev (#1739) * JetBrains status bar spinner for autocomplete * Remove unused images * partial accept jetbrains * fix double-rendering of jetbrains completions * fix offset of multi-line jetbrains completions * enable/disable actions * upgrade to version 0.0.54 * llm-info package * small profile switching improvements * instruct cmd+I not to leave placeholders * @ files in context-providers docs * onboarding fix * update jetbrains ci to build gui * feat: single default quick pick to edit (#1743) * feat: single default quick pick to edit * Update quick-actions.md * Update quick-actions.md * use defaultQuickAction * quick fixes * fix: handle line removal diff (#1744) * fix: handle line removal diff * remove comments * Configurable `maxChunkSize` for embedding providers (#1746) * Add `maxChunkSize` to `EmbeddingsProvider` * Add `maxChunkSize` to schema * add support for docs via config (#1594) * mistral mamba (#1748) * 🐍 mistral mamba * autoscale retrieval based on context length * autoscale retrieval based on context length * fix input box movement * update jetbrains build * test: add myers diff tests (#1754) * feat: enhance help center (#1755) * feat: enhance help center * Update help.tsx * feat: add azure provider config (#1764) * autocomplete reload fix * logout of control plane * Update QuickEditQuickPick.ts (#1772) * Update QuickEditQuickPick.ts * updates * Update QuickEditQuickPick.ts * pass refreshed access token to continue-proxy model * test: add dir read check for walkDir (#1773) * test: add dir read check for walkDir * Update ignore.ts * Update walkDir.test.ts * Update walkDir.test.ts * gpt-4o-mini * gpt-4o-mini * gpt4omini to ui selector * fix: layout alignment (#1779) * fix: delete old docs index on force re-index (#1778) * add "Codebase Force Re-Index" command (#1757) * Add `HuggingFaceTEIReranker` (#1711) * Add `HuggingFaceTEIReranker` * Add `huggingface-tei` to `config_schema.json` * update package.json version * update config schemas * skip test failing only due to type error * fix: convert `walkDir` to an async generator (#1783) * walkdir to async generator * add yields * handle ignore files * Update CodebaseIndexer.ts * Update walkDir.ts * chore: bump VS Code ext version * fix completions support in ContinueProxy * add test for walkDir * IAnalyticsProvider * feat: crawl `.mdx` docs * update workspaces on session info change * rename from profile to workspace (user-facing) * update config schema files * feat: improve chat thread ui for better readability (#1786) * add "language" template variable for autocomplete * Enable debug logs (#1800) * Add an enableDebugLogs flag * Send logs to VSCode * Refresh index fixes (#1795) * Use the resultType given to markComplete This fixes an issue where if there were multiple result types in itemToAction such as an update with UpdateNewVersion and UpdateOldVersion, only the UpdateOldVersion would be hit (which is a noop) and the UpdateNewVersion would never be fired, causing us to never update the timestamp of the tag_catalog. * Add a Compute and UpdateLastUpdated to refreshIndex Removes the ability to insert duplicate rows if we're just computing the result. Instead we replace the existing row instead. * Don't insert compute tags They are REPLACEd in markComplete now. * docs: change page heading to model providers (#1684) * feat: allow JetBrains users to index docs (#1797) * feat: allow JetBrains users to index docs * update docs service * update global context * throw error popup on jetbrains + transformers * update toast notifications * update typings * fix tsc errors * Index on file change (#1785) * Reload the submenu items when the context provider data is refreshed * update version * Just call refreshSubmenuItems * remove log statements * cleanup * Remove the Promise declaration * Copy currentFiles to a new dict since we're going to mutate it Don't mutate the caller's currentFiles unexpectedly * Remove reIndexFile and just call index/forceReIndex * We MUST await the refreshCodebaseIndex call * Expose send on core * remove unused import * Reindex the workspaces on save * Call core.send Ideally, we can implement externalRequest on the webview later so we don't have to expose messenger to the VSCode extension * Always refreshSubmenuItems in refreshCodebaseIndex * whitespace * Change messenger back to private * Add Pause Codebase On Start setting (#1788) * Add a skipInitialSync function * Add pauseInitialCodebaseIndex * docs * version bump * Rename to pauseCodebaseIndexOnStart * Un-shadow file context provider (#1801) * allow FileContextProvider to handle files * remove commented-out lines * chore: add telemetry to alt-enter (#1805) * Set the indexProgress indicator to paused on startup if pauseCodebaseIndexOnStart is set (#1804) * Fix Input Toolbar layout (#1792) * set to flex flex-column * Remove fixed height and always show the StyledDivs content to hold its size * Add a Delete Message button to the StepContainer (#1790) * Add a delete button * add a delete message action to the GUI * Fix off by one error * format * This index should be less than length * Update to trash icon * Simplify walkDir(...) and improve performance by roughly 10x in larger repos (#1806) The performance of WalkDir is improved by replacing the recursive DFS with a stack based implementation (less memory in-use during peak exploration). In addition, some unnecessary flags and options were removed which allowed the code to be overall simplified. Much less rule matching is occuring with the same files returned. Some performance results on my linux host: small repo: - 649 files returned by walkDir() - 9 .gitignore files - walkDir() old vs new duration: 918 ms to 143 ms medium sized repo: - 9271 returned by walkDir() - 35 .gitignore files - walkDir() old vs new duration: 10,886 ms to 970ms large sized repo: - 197,900 files returned by walkDir() - 929 .gitignore files - walkDir() old vs new duration: 498,151 ms to 37,353 ms Co-authored-by: Rob Leidle <rleidle@tesla.com> * modified assistant indicator in UI * update package.json version * feat: add Llama 3.1 8B to cloudflare provider options (#1811) Co-authored-by: jdelorey@cloudflare.com <jdelorey@cloudflare.com> * increase gemini pro contextLength (#1809) * fix indentation (#1808) * Llama 3.1 405b model selection (#1813) * add support for llama3 405b * 3.1 * name mapping * updates to model setup * fix walkDir tests on windows * chore: add docs as top level config (#1816) * chore: add docs as top level config * Update core.ts * feat: make @codebase a hardcoded ctx provider (#1818) * chore: add docs as top level config * Update core.ts * feat: make @codebase a hardcoded ctx provider * docs schema * Nate/dev (#1819) * llm-info package * small profile switching improvements * instruct cmd+I not to leave placeholders * @ files in context-providers docs * onboarding fix * jetbrains login * fix jetbrains login * change beta wording * proxy context provider * fix proxy context provider * revert to prod id, url * update package.json version * Add CodeGeeX4 code completion support (#1815) Includes support for context through code snippets and relative filename resolution. * manually stop at stop tokens for all providers * chore: bump tree-sitter (#1828) * Add Missing Tokens for CodeGeeX4 AutoComplete (#1827) Previous attempts at using the normal pipe char with CodeGeeX FIM tokens resulted in empty completion output. However, this was due to <|user|> and <|assistant|> tokens being necessary for completion. The earlier working change ommitted these tokens, and its functionality using deepseek pipes was merely coincidental. This change now properly follows the CodeGeeX infill guides and should match what the model expects. In addition to <|user|> and <|assistant|>, the <|endoftext|> token was added as a stop token. * additional analytics providers * fix import error in jetbrains plugin * explicitly include default context providers * Remove a copy / pasta section that duplicated the work done in ChunkCodebaseIndexer (#1834) This cuts the amount of time spent in this section of indexing in half Co-authored-by: Rob Leidle <rleidle@tesla.com> * respect VS Code isTelemetryEnabled * Update OllamaEmbeddingsProvider.ts (#1839) * feat: include recently + open files in codebase search (#1833) * feat: include recently + open files in codebase search * cleanup * Update BaseRetrievalPipeline.ts * add params object to chunkDocument * Update package-lock.json * Offload all token counting to worker processes as well as some optimizations to do more token counting in parallel Adding the workerpool logic broke gui as vite does not support __dirname or path.join(...). To work around this, moved the gui dependency of stripImages out of countTokens.ts * Create CODE_OF_CONDUCT.md (#1843) * tweaks to new tokenizing * temporarily skip code chunking tests due to worker files not being present * update changelog * update version * jetbrains refresh token updates * refresh submenu items on workspace switch * OpenAI Adapters (#1859) * embeddings support in openai-adapters * rerank support in openai-adapters * cohere embed/rerank * continue-proxy reranker and embeddings providers * test for openai-adapters * embeddings provider and reranker for continue proxy * fix: missing Content-Type header in OllamaEmbeddingsProvider (#1855) * Add Language Name Param to AutocompleteTemplate (#1853) Language name was previously added as a parameter only for string autocomplete templates. This change provides this same information to template functions that implement the AutocompleteTemplate interface. Language name is explicitly supported for the CodeGeeX4 autocomplete template, and is added here in this change. Explicit language awareness could also be added for other templates in a future change, like the holeFillerTemplate used for models without explicit completion support. * update package.json version * deepseek FIM endpoint (#1862) * update package.json version * continue teams beta setting * update jetbrains version --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> Co-authored-by: Rob Leidle <rleidle@gmail.com> Co-authored-by: Rob Leidle <rleidle@tesla.com> Co-authored-by: inimaz <49730431+inimaz@users.noreply.github.com> Co-authored-by: inimaz <imaz@ai-verse.com> Co-authored-by: Anaël Bonnafous <71386173+AnaelBonnafous@users.noreply.github.com> Co-authored-by: Huy Tran <119860259+tranquochuy645@users.noreply.github.com> Co-authored-by: Gabriel Gordbegli <ggordbegli@gmail.com> Co-authored-by: Raven-1027 <83693755+Raven-1027@users.noreply.github.com> Co-authored-by: Ihe Fan <ih.fan.1027@gmail.com> Co-authored-by: Lukas Kreussel <65088241+LLukas22@users.noreply.github.com> Co-authored-by: Josh Vera <josh@joshvera.com> Co-authored-by: Peter Zaback <pzaback@gmail.com> Co-authored-by: James Delorey <deloreyjmi@gmail.com> Co-authored-by: jdelorey@cloudflare.com <jdelorey@cloudflare.com> Co-authored-by: ekatiyar <eashaank@gmail.com> Co-authored-by: Simo <simo.paasisalo@syke.fi> --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> Co-authored-by: Rob Leidle <rleidle@gmail.com> Co-authored-by: Rob Leidle <rleidle@tesla.com> Co-authored-by: inimaz <49730431+inimaz@users.noreply.github.com> Co-authored-by: inimaz <imaz@ai-verse.com> Co-authored-by: Anaël Bonnafous <71386173+AnaelBonnafous@users.noreply.github.com> Co-authored-by: Huy Tran <119860259+tranquochuy645@users.noreply.github.com> Co-authored-by: Gabriel Gordbegli <ggordbegli@gmail.com> Co-authored-by: Raven-1027 <83693755+Raven-1027@users.noreply.github.com> Co-authored-by: Ihe Fan <ih.fan.1027@gmail.com> Co-authored-by: Lukas Kreussel <65088241+LLukas22@users.noreply.github.com> Co-authored-by: Josh Vera <josh@joshvera.com> Co-authored-by: Peter Zaback <pzaback@gmail.com> Co-authored-by: James Delorey <deloreyjmi@gmail.com> Co-authored-by: jdelorey@cloudflare.com <jdelorey@cloudflare.com> Co-authored-by: ekatiyar <eashaank@gmail.com> Co-authored-by: Simo <simo.paasisalo@syke.fi>
This commit is contained in:
Nate Sesti
GitHub
parent
9961f85326
commit
d0f3209dff
|
|
@ -0,0 +1,4 @@
|
|||
project: extensions/vscode
|
||||
kind: Added
|
||||
body: Support for Llama 3.1 and gpt-4o-mini
|
||||
time: 2024-07-26T15:35:14.411895-07:00
|
||||
|
|
@ -0,0 +1,4 @@
|
|||
project: extensions/vscode
|
||||
kind: Changed
|
||||
body: Significant improvements to indexing performance
|
||||
time: 2024-07-26T15:34:58.457908-07:00
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
project: extensions/vscode
|
||||
kind: Changed
|
||||
body: Improved @codebase quality by more accurately searching over file names and
|
||||
paths
|
||||
time: 2024-07-26T15:35:42.390496-07:00
|
||||
|
|
@ -16,7 +16,7 @@ name: Build
|
|||
on:
|
||||
# Trigger the workflow on pushes to only the 'main' branch (this avoids duplicate checks being run e.g., for dependabot pull requests)
|
||||
push:
|
||||
branches: [main]
|
||||
branches: [main, preview]
|
||||
# Trigger the workflow on any pull request
|
||||
# pull_request:
|
||||
|
||||
|
|
@ -97,6 +97,12 @@ jobs:
|
|||
path: binary/node_modules
|
||||
key: ${{ runner.os }}-node-${{ hashFiles('binary/package-lock.json') }}
|
||||
|
||||
- name: Cache gui node_modules
|
||||
uses: actions/cache@v3
|
||||
with:
|
||||
path: gui/node_modules
|
||||
key: ${{ runner.os }}-node-${{ hashFiles('gui/package-lock.json') }}
|
||||
|
||||
# npm install core
|
||||
- name: Install core node_modules
|
||||
run: |
|
||||
|
|
@ -109,6 +115,18 @@ jobs:
|
|||
cd ../../binary
|
||||
npm ci
|
||||
|
||||
# npm install gui
|
||||
- name: Install gui node_modules
|
||||
run: |
|
||||
cd ../../gui
|
||||
npm ci
|
||||
|
||||
# build gui
|
||||
- name: Build gui
|
||||
run: |
|
||||
cd ../../gui
|
||||
npm run build
|
||||
|
||||
# Run prepackage.js script
|
||||
- name: Run prepackage script
|
||||
run: |
|
||||
|
|
@ -134,11 +152,11 @@ jobs:
|
|||
id: artifact
|
||||
shell: bash
|
||||
run: |
|
||||
cd ${{ github.workspace }}/extensions/intellij/build/distributions
|
||||
cd ../../extensions/intellij/build/distributions
|
||||
FILENAME=`ls *.zip`
|
||||
unzip "$FILENAME" -d content
|
||||
|
||||
echo "filename=${FILENAME:0:-4}" >> $GITHUB_OUTPUT
|
||||
echo "filename=${FILENAME%.????}" >> $GITHUB_OUTPUT
|
||||
|
||||
# Store already-built plugin as an artifact for downloading
|
||||
- name: Upload artifact
|
||||
|
|
|
|||
|
|
@ -87,6 +87,12 @@ jobs:
|
|||
path: binary/node_modules
|
||||
key: ${{ runner.os }}-node-${{ hashFiles('binary/package-lock.json') }}
|
||||
|
||||
- name: Cache gui node_modules
|
||||
uses: actions/cache@v3
|
||||
with:
|
||||
path: gui/node_modules
|
||||
key: ${{ runner.os }}-node-${{ hashFiles('gui/package-lock.json') }}
|
||||
|
||||
# npm install core
|
||||
- name: Install core node_modules
|
||||
run: |
|
||||
|
|
@ -99,6 +105,18 @@ jobs:
|
|||
cd ../../binary
|
||||
npm ci
|
||||
|
||||
# npm install gui
|
||||
- name: Install gui node_modules
|
||||
run: |
|
||||
cd ../../gui
|
||||
npm ci
|
||||
|
||||
# build gui
|
||||
- name: Build gui
|
||||
run: |
|
||||
cd ../../gui
|
||||
npm run build
|
||||
|
||||
# Run prepackage.js script
|
||||
- name: Run prepackage script
|
||||
run: |
|
||||
|
|
@ -120,6 +138,12 @@ jobs:
|
|||
run: |
|
||||
./gradlew patchChangelog --release-note="$CHANGELOG"
|
||||
|
||||
# Upload artifact as a release asset
|
||||
- name: Upload Release Asset
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: gh release upload ${{ github.event.release.tag_name }} ./build/distributions/*
|
||||
|
||||
# Publish the plugin to JetBrains Marketplace
|
||||
- name: Publish Plugin
|
||||
env:
|
||||
|
|
@ -129,12 +153,6 @@ jobs:
|
|||
PRIVATE_KEY_PASSWORD: ${{ secrets.JETBRAINS_PRIVATE_KEY_PASSWORD }}
|
||||
run: ./gradlew publishPlugin
|
||||
|
||||
# Upload artifact as a release asset
|
||||
- name: Upload Release Asset
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: gh release upload ${{ github.event.release.tag_name }} ./build/distributions/*
|
||||
|
||||
# Create a pull request
|
||||
- name: Create Pull Request
|
||||
if: ${{ steps.properties.outputs.changelog != '' }}
|
||||
|
|
|
|||
|
|
@ -125,19 +125,19 @@ jobs:
|
|||
|
||||
# 4. Run tests for the extension
|
||||
# - name: Install Xvfb for Linux and run tests
|
||||
# run: |
|
||||
# sudo apt-get install -y xvfb # Install Xvfb
|
||||
# Xvfb :99 & # Start Xvfb
|
||||
# export DISPLAY=:99 # Export the display number to the environment
|
||||
# cd extensions/vscode
|
||||
# npm run test
|
||||
# if: matrix.os == 'ubuntu-latest'
|
||||
run: |
|
||||
sudo apt-get install -y xvfb # Install Xvfb
|
||||
Xvfb :99 & # Start Xvfb
|
||||
export DISPLAY=:99 # Export the display number to the environment
|
||||
cd extensions/vscode
|
||||
npm run test
|
||||
if: matrix.os == 'ubuntu-latest'
|
||||
|
||||
# - name: Run extension tests
|
||||
# run: |
|
||||
# cd extensions/vscode
|
||||
# npm run test
|
||||
# if: matrix.os != 'ubuntu-latest'
|
||||
- name: Run extension tests
|
||||
run: |
|
||||
cd extensions/vscode
|
||||
npm run test
|
||||
if: matrix.os != 'ubuntu-latest'
|
||||
|
||||
# 5. Package the extension
|
||||
- name: Package the extension
|
||||
|
|
@ -231,22 +231,22 @@ jobs:
|
|||
npx ovsx publish -p ${{ secrets.VSX_REGISTRY_TOKEN }} --packagePath ../../alpine-x64/*.vsix ../../darwin-arm64/*.vsix ../../darwin-x64/*.vsix ../../linux-arm64/*.vsix ../../linux-armhf/*.vsix ../../linux-x64/*.vsix ../../win32-x64/*.vsix ../../win32-arm64/*.vsix
|
||||
|
||||
# 4. Update the package.json version and push changes
|
||||
- name: Update version in package.json
|
||||
run: |
|
||||
cd extensions/vscode
|
||||
npm version patch
|
||||
# - name: Update version in package.json
|
||||
# run: |
|
||||
# cd extensions/vscode
|
||||
# npm version patch
|
||||
|
||||
- name: Commit changes
|
||||
run: |
|
||||
git config --local user.email "action@github.com"
|
||||
git config --local user.name "GitHub Action"
|
||||
git commit -am "💚 Update package.json version [skip ci]"
|
||||
# - name: Commit changes
|
||||
# run: |
|
||||
# git config --local user.email "action@github.com"
|
||||
# git config --local user.name "GitHub Action"
|
||||
# git commit -am "💚 Update package.json version [skip ci]"
|
||||
|
||||
- name: Push changes
|
||||
uses: ad-m/github-push-action@master
|
||||
with:
|
||||
github_token: ${{ secrets.GITHUB_TOKEN }}
|
||||
branch: ${{ github.ref }}
|
||||
# - name: Push changes
|
||||
# uses: ad-m/github-push-action@master
|
||||
# with:
|
||||
# github_token: ${{ secrets.GITHUB_TOKEN }}
|
||||
# branch: ${{ github.ref }}
|
||||
|
||||
# 5 Send to Discord Webhook
|
||||
- name: Discord Commits
|
||||
|
|
|
|||
|
|
@ -223,19 +223,19 @@ jobs:
|
|||
npx ovsx publish --pre-release -p ${{ secrets.VSX_REGISTRY_TOKEN }} --packagePath ../../alpine-x64/*.vsix ../../darwin-arm64/*.vsix ../../darwin-x64/*.vsix ../../linux-arm64/*.vsix ../../linux-armhf/*.vsix ../../linux-x64/*.vsix ../../win32-x64/*.vsix ../../win32-arm64/*.vsix
|
||||
|
||||
# 4. Update the package.json version and push changes
|
||||
- name: Update version in package.json
|
||||
run: |
|
||||
cd extensions/vscode
|
||||
npm version patch
|
||||
# - name: Update version in package.json
|
||||
# run: |
|
||||
# cd extensions/vscode
|
||||
# npm version patch
|
||||
|
||||
- name: Commit changes
|
||||
run: |
|
||||
git config --local user.email "action@github.com"
|
||||
git config --local user.name "GitHub Action"
|
||||
git commit -am "💚 Update package.json version [skip ci]"
|
||||
# - name: Commit changes
|
||||
# run: |
|
||||
# git config --local user.email "action@github.com"
|
||||
# git config --local user.name "GitHub Action"
|
||||
# git commit -am "💚 Update package.json version [skip ci]"
|
||||
|
||||
- name: Push changes
|
||||
uses: ad-m/github-push-action@master
|
||||
with:
|
||||
github_token: ${{ secrets.GITHUB_TOKEN }}
|
||||
branch: ${{ github.ref }}
|
||||
# - name: Push changes
|
||||
# uses: ad-m/github-push-action@master
|
||||
# with:
|
||||
# github_token: ${{ secrets.GITHUB_TOKEN }}
|
||||
# branch: ${{ github.ref }}
|
||||
|
|
|
|||
|
|
@ -68,7 +68,10 @@
|
|||
"${workspaceFolder}/extensions/vscode/out/extension.js",
|
||||
"/Users/natesesti/.continue/config.ts"
|
||||
],
|
||||
"preLaunchTask": "vscode-extension:build"
|
||||
"preLaunchTask": "vscode-extension:build",
|
||||
"env": {
|
||||
// "CONTROL_PLANE_ENV": "local"
|
||||
}
|
||||
},
|
||||
// Has to be run after starting the server (separately or using the compound configuration)
|
||||
{
|
||||
|
|
@ -95,6 +98,41 @@
|
|||
// Avoid timing out when stopping on breakpoints during debugging in VSCode
|
||||
"MOCHA_TIMEOUT": "0"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "[Core] Jest Test Debugger, Current Open File",
|
||||
"type": "node",
|
||||
"request": "launch",
|
||||
"runtimeArgs": [
|
||||
"--inspect-brk",
|
||||
"${workspaceRoot}/core/node_modules/jest/bin/jest.js",
|
||||
"--runInBand",
|
||||
"--config",
|
||||
"${workspaceRoot}/core/jest.config.js",
|
||||
"${relativeFile}"
|
||||
],
|
||||
"cwd": "${workspaceRoot}/core",
|
||||
"console": "integratedTerminal",
|
||||
"internalConsoleOptions": "neverOpen"
|
||||
}
|
||||
// {
|
||||
// "name": "[openai-adapters] Jest Test Debugger, Current Open File",
|
||||
// "type": "node",
|
||||
// "request": "launch",
|
||||
// "runtimeArgs": [
|
||||
// "--inspect-brk",
|
||||
// "${workspaceRoot}/packages/openai-adapters/node_modules/jest/bin/jest.js",
|
||||
// "--runInBand",
|
||||
// "--config",
|
||||
// "${workspaceRoot}/packages/openai-adapters/jest.config.mjs",
|
||||
// "${relativeFile}"
|
||||
// ],
|
||||
// "cwd": "${workspaceRoot}/packages/openai-adapters",
|
||||
// "console": "integratedTerminal",
|
||||
// "internalConsoleOptions": "neverOpen",
|
||||
// "env": {
|
||||
// "NODE_OPTIONS": "--experimental-vm-modules"
|
||||
// }
|
||||
// }
|
||||
]
|
||||
}
|
||||
|
|
|
|||
|
|
@ -7,10 +7,14 @@
|
|||
"python.testing.unittestEnabled": false,
|
||||
"python.testing.pytestEnabled": true,
|
||||
"terminal.integrated.defaultProfile.linux": "bash",
|
||||
"terminal.integrated.profiles.linux": {
|
||||
"bash": {
|
||||
"path": "bash",
|
||||
"args": ["-l"]
|
||||
}
|
||||
"terminal.integrated.profiles.linux": {
|
||||
"bash": {
|
||||
"path": "bash",
|
||||
"args": ["-l"]
|
||||
}
|
||||
},
|
||||
"search.exclude": {
|
||||
"./binary": true,
|
||||
"**/core/vendor/**": true
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,73 @@
|
|||
# Contributor Covenant Code of Conduct
|
||||
|
||||
## Our Pledge
|
||||
|
||||
In the interest of fostering an open and welcoming environment, we as
|
||||
contributors and maintainers pledge to making participation in our project and
|
||||
our community a harassment-free experience for everyone, regardless of age, body
|
||||
size, disability, ethnicity, gender identity and expression, level of experience,
|
||||
education, socio-economic status, nationality, personal appearance, race,
|
||||
religion, or sexual identity and orientation.
|
||||
|
||||
## Our Standards
|
||||
|
||||
Examples of behavior that contributes to creating a positive environment
|
||||
include:
|
||||
|
||||
* Using welcoming and inclusive language
|
||||
* Being respectful of differing viewpoints and experiences
|
||||
* Gracefully accepting constructive criticism
|
||||
* Focusing on what is best for the community
|
||||
* Showing empathy towards other community members
|
||||
|
||||
Examples of unacceptable behavior by participants include:
|
||||
|
||||
* The use of sexualized language or imagery and unwelcome sexual attention or
|
||||
advances
|
||||
* Trolling, insulting/derogatory comments, and personal or political attacks
|
||||
* Public or private harassment
|
||||
* Publishing others' private information, such as a physical or electronic
|
||||
address, without explicit permission
|
||||
* Other conduct which could reasonably be considered inappropriate in a
|
||||
professional setting
|
||||
|
||||
## Our Responsibilities
|
||||
|
||||
Project maintainers are responsible for clarifying the standards of acceptable
|
||||
behavior and are expected to take appropriate and fair corrective action in
|
||||
response to any instances of unacceptable behavior.
|
||||
|
||||
Project maintainers have the right and responsibility to remove, edit, or
|
||||
reject comments, commits, code, wiki edits, issues, and other contributions
|
||||
that are not aligned to this Code of Conduct, or to ban temporarily or
|
||||
permanently any contributor for other behaviors that they deem inappropriate,
|
||||
threatening, offensive, or harmful.
|
||||
|
||||
## Scope
|
||||
|
||||
This Code of Conduct applies both within project spaces and in public spaces
|
||||
when an individual is representing the project or its community. Examples of
|
||||
representing a project or community include using an official project e-mail
|
||||
address, posting via an official social media account, or acting as an appointed
|
||||
representative at an online or offline event. Representation of a project may be
|
||||
further defined and clarified by project maintainers.
|
||||
|
||||
## Enforcement
|
||||
|
||||
Instances of abusive, harassing, or otherwise unacceptable behavior may be
|
||||
reported by contacting the project team at hi@continue.dev. All
|
||||
complaints will be reviewed and investigated and will result in a response that
|
||||
is deemed necessary and appropriate to the circumstances. The project team is
|
||||
obligated to maintain confidentiality with regard to the reporter of an incident.
|
||||
Further details of specific enforcement policies may be posted separately.
|
||||
|
||||
Project maintainers who do not follow or enforce the Code of Conduct in good
|
||||
faith may face temporary or permanent repercussions as determined by other
|
||||
members of the project's leadership.
|
||||
|
||||
## Attribution
|
||||
|
||||
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
|
||||
available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
|
||||
|
||||
[homepage]: https://www.contributor-covenant.org
|
||||
|
|
@ -158,11 +158,18 @@ async function installNodeModuleInTempDirAndCopyToCurrent(packageName, toCopy) {
|
|||
);
|
||||
});
|
||||
|
||||
fs.copyFileSync(
|
||||
path.join(__dirname, "../core/vendor/tree-sitter.wasm"),
|
||||
path.join(__dirname, "out/tree-sitter.wasm"),
|
||||
);
|
||||
console.log("[info] Copied tree-sitter wasms");
|
||||
const filesToCopy = [
|
||||
"../core/vendor/tree-sitter.wasm",
|
||||
"../core/llm/llamaTokenizerWorkerPool.mjs",
|
||||
"../core/llm/llamaTokenizer.mjs",
|
||||
];
|
||||
for (const f of filesToCopy) {
|
||||
fs.copyFileSync(
|
||||
path.join(__dirname, f),
|
||||
path.join(__dirname, "out", path.basename(f)),
|
||||
);
|
||||
console.log(`[info] Copied ${path.basename(f)}`);
|
||||
}
|
||||
|
||||
console.log("[info] Cleaning up artifacts from previous builds...");
|
||||
|
||||
|
|
|
|||
|
|
@ -46,6 +46,8 @@
|
|||
"dependencies": {
|
||||
"@aws-sdk/client-bedrock-runtime": "^3.574.0",
|
||||
"@aws-sdk/credential-providers": "^3.596.0",
|
||||
"@continuedev/config-types": "^1.0.10",
|
||||
"@continuedev/llm-info": "^1.0.1",
|
||||
"@mozilla/readability": "^0.5.0",
|
||||
"@octokit/rest": "^20.0.2",
|
||||
"@types/jsdom": "^21.1.6",
|
||||
|
|
@ -61,7 +63,7 @@
|
|||
"commander": "^12.0.0",
|
||||
"comment-json": "^4.2.3",
|
||||
"dbinfoz": "^0.1.4",
|
||||
"dotenv": "^16.3.1",
|
||||
"dotenv": "^16.4.5",
|
||||
"fastest-levenshtein": "^1.0.16",
|
||||
"follow-redirects": "^1.15.5",
|
||||
"handlebars": "^4.7.8",
|
||||
|
|
@ -80,6 +82,7 @@
|
|||
"openai": "^4.20.1",
|
||||
"pg": "^8.11.3",
|
||||
"posthog-node": "^3.6.3",
|
||||
"quick-lru": "^7.0.0",
|
||||
"replicate": "^0.26.0",
|
||||
"request": "^2.88.2",
|
||||
"socket.io-client": "^4.7.3",
|
||||
|
|
@ -91,7 +94,9 @@
|
|||
"vectordb": "^0.4.20",
|
||||
"web-tree-sitter": "^0.21.0",
|
||||
"win-ca": "^3.5.1",
|
||||
"yaml": "^2.4.2"
|
||||
"workerpool": "^9.1.3",
|
||||
"yaml": "^2.4.2",
|
||||
"zod": "^3.23.8"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@babel/preset-env": "^7.24.7",
|
||||
|
|
@ -106,6 +111,7 @@
|
|||
"eslint": "^8",
|
||||
"eslint-plugin-import": "^2.29.1",
|
||||
"jest": "^29.7.0",
|
||||
"myers-diff": "^2.1.0",
|
||||
"onnxruntime-common": "1.14.0",
|
||||
"onnxruntime-web": "1.14.0",
|
||||
"ts-jest": "^29.1.1"
|
||||
|
|
|
|||
|
|
@ -107,7 +107,7 @@ describe("Test Suite", () => {
|
|||
|
||||
// Many of the files are only created when trying to load the config
|
||||
const config = await messenger.request(
|
||||
"config/getBrowserSerialized",
|
||||
"config/getSerializedProfileInfo",
|
||||
undefined,
|
||||
);
|
||||
|
||||
|
|
@ -128,8 +128,8 @@ describe("Test Suite", () => {
|
|||
});
|
||||
|
||||
it("should properly edit config", async () => {
|
||||
const config = await messenger.request(
|
||||
"config/getBrowserSerialized",
|
||||
const { config } = await messenger.request(
|
||||
"config/getSerializedProfileInfo",
|
||||
undefined,
|
||||
);
|
||||
expect(config).toHaveProperty("models");
|
||||
|
|
|
|||
|
|
@ -1,11 +1,11 @@
|
|||
import { IDE } from "..";
|
||||
import { RangeInFileWithContents } from "../commands/util";
|
||||
import { PrecalculatedLruCache } from "../util/LruCache";
|
||||
import { IDE } from "../index.js";
|
||||
import { RangeInFileWithContents } from "../commands/util.js";
|
||||
import { PrecalculatedLruCache } from "../util/LruCache.js";
|
||||
import {
|
||||
TSQueryType,
|
||||
getParserForFile,
|
||||
getQueryForFile,
|
||||
} from "../util/treeSitter";
|
||||
} from "../util/treeSitter.js";
|
||||
|
||||
interface FileInfo {
|
||||
imports: { [key: string]: RangeInFileWithContents[] };
|
||||
|
|
|
|||
|
|
@ -90,7 +90,7 @@ This is just another object like the ones in the `"models"` array of `config.jso
|
|||
This object allows you to customize the behavior of tab-autocomplete. The available options are:
|
||||
|
||||
- `useCopyBuffer`: Determines whether the copy buffer will be considered when constructing the prompt. (Boolean)
|
||||
- `useSuffix`: Determines whether to use the file suffix in the prompt. (Boolean)
|
||||
- `useFileSuffix`: Determines whether to use the file suffix in the prompt. (Boolean)
|
||||
- `maxPromptTokens`: The maximum number of prompt tokens to use. A smaller number will yield faster completions, but less context. (Number)
|
||||
- `debounceDelay`: The delay in milliseconds before triggering autocomplete after a keystroke. (Number)
|
||||
- `maxSuffixPercentage`: The maximum percentage of the prompt that can be dedicated to the suffix. (Number)
|
||||
|
|
|
|||
|
|
@ -1,10 +1,9 @@
|
|||
import Handlebars from "handlebars";
|
||||
import ignore from "ignore";
|
||||
import OpenAI from "openai";
|
||||
import path from "path";
|
||||
import { v4 as uuidv4 } from "uuid";
|
||||
import { RangeInFileWithContents } from "../commands/util.js";
|
||||
import { IConfigHandler } from "../config/IConfigHandler.js";
|
||||
import { ConfigHandler } from "../config/ConfigHandler.js";
|
||||
import { TRIAL_FIM_MODEL } from "../config/onboarding.js";
|
||||
import { streamLines } from "../diff/util.js";
|
||||
import {
|
||||
|
|
@ -29,6 +28,7 @@ import AutocompleteLruCache from "./cache.js";
|
|||
import {
|
||||
noFirstCharNewline,
|
||||
onlyWhitespaceAfterEndOfLine,
|
||||
stopAtStopTokens,
|
||||
} from "./charStream.js";
|
||||
import {
|
||||
constructAutocompletePrompt,
|
||||
|
|
@ -50,6 +50,8 @@ import { AutocompleteSnippet } from "./ranking.js";
|
|||
import { RecentlyEditedRange } from "./recentlyEdited.js";
|
||||
import { getTemplateForModel } from "./templates.js";
|
||||
import { GeneratorReuseManager } from "./util.js";
|
||||
// @prettier-ignore
|
||||
import Handlebars from "handlebars";
|
||||
|
||||
export interface AutocompleteInput {
|
||||
completionId: string;
|
||||
|
|
@ -145,7 +147,7 @@ export class CompletionProvider {
|
|||
private static lastUUID: string | undefined = undefined;
|
||||
|
||||
constructor(
|
||||
private readonly configHandler: IConfigHandler,
|
||||
private readonly configHandler: ConfigHandler,
|
||||
private readonly ide: IDE,
|
||||
private readonly getLlm: () => Promise<ILLM | undefined>,
|
||||
private readonly _onError: (e: any) => void,
|
||||
|
|
@ -201,13 +203,17 @@ export class CompletionProvider {
|
|||
const outcome = this._outcomes.get(completionId)!;
|
||||
outcome.accepted = true;
|
||||
logDevData("autocomplete", outcome);
|
||||
Telemetry.capture("autocomplete", {
|
||||
accepted: outcome.accepted,
|
||||
modelName: outcome.modelName,
|
||||
modelProvider: outcome.modelProvider,
|
||||
time: outcome.time,
|
||||
cacheHit: outcome.cacheHit,
|
||||
});
|
||||
Telemetry.capture(
|
||||
"autocomplete",
|
||||
{
|
||||
accepted: outcome.accepted,
|
||||
modelName: outcome.modelName,
|
||||
modelProvider: outcome.modelProvider,
|
||||
time: outcome.time,
|
||||
cacheHit: outcome.cacheHit,
|
||||
},
|
||||
true,
|
||||
);
|
||||
this._outcomes.delete(completionId);
|
||||
|
||||
this.bracketMatchingService.handleAcceptedCompletion(
|
||||
|
|
@ -358,9 +364,13 @@ export class CompletionProvider {
|
|||
outcome.accepted = false;
|
||||
logDevData("autocomplete", outcome);
|
||||
const { prompt, completion, ...restOfOutcome } = outcome;
|
||||
Telemetry.capture("autocomplete", {
|
||||
...restOfOutcome,
|
||||
});
|
||||
Telemetry.capture(
|
||||
"autocomplete",
|
||||
{
|
||||
...restOfOutcome,
|
||||
},
|
||||
true,
|
||||
);
|
||||
this._logRejectionTimeouts.delete(completionId);
|
||||
}, COUNT_COMPLETION_REJECTED_AFTER);
|
||||
this._outcomes.set(completionId, outcome);
|
||||
|
|
@ -572,10 +582,11 @@ export class CompletionProvider {
|
|||
suffix,
|
||||
filename,
|
||||
reponame,
|
||||
language: lang.name,
|
||||
});
|
||||
} else {
|
||||
// Let the template function format snippets
|
||||
prompt = template(prefix, suffix, filepath, reponame, snippets);
|
||||
prompt = template(prefix, suffix, filepath, reponame, lang.name, snippets);
|
||||
}
|
||||
|
||||
// Completion
|
||||
|
|
@ -652,6 +663,7 @@ export class CompletionProvider {
|
|||
lang.endOfLine,
|
||||
fullStop,
|
||||
);
|
||||
charGenerator = stopAtStopTokens(charGenerator, stop);
|
||||
charGenerator = this.bracketMatchingService.stopOnUnmatchedClosingBracket(
|
||||
charGenerator,
|
||||
prefix,
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
import type { LineFilter } from "./lineStream";
|
||||
|
||||
export interface AutocompleteLanguageInfo {
|
||||
name: string;
|
||||
topLevelKeywords: string[];
|
||||
singleLineComment: string;
|
||||
endOfLine: string[];
|
||||
|
|
@ -14,6 +15,7 @@ export interface AutocompleteLanguageInfo {
|
|||
|
||||
// TypeScript
|
||||
export const Typescript = {
|
||||
name: "TypeScript",
|
||||
topLevelKeywords: ["function", "class", "module", "export", "import"],
|
||||
singleLineComment: "//",
|
||||
endOfLine: [";"],
|
||||
|
|
@ -21,6 +23,7 @@ export const Typescript = {
|
|||
|
||||
// Python
|
||||
export const Python = {
|
||||
name: "Python",
|
||||
// """"#" is for .ipynb files, where we add '"""' surrounding markdown blocks.
|
||||
// This stops the model from trying to complete the start of a new markdown block
|
||||
topLevelKeywords: ["def", "class", '"""#'],
|
||||
|
|
@ -30,6 +33,7 @@ export const Python = {
|
|||
|
||||
// Java
|
||||
export const Java = {
|
||||
name: "Java",
|
||||
topLevelKeywords: ["class", "function"],
|
||||
singleLineComment: "//",
|
||||
endOfLine: [";"],
|
||||
|
|
@ -37,6 +41,7 @@ export const Java = {
|
|||
|
||||
// C++
|
||||
export const Cpp = {
|
||||
name: "C++",
|
||||
topLevelKeywords: ["class", "namespace", "template"],
|
||||
singleLineComment: "//",
|
||||
endOfLine: [";"],
|
||||
|
|
@ -44,6 +49,7 @@ export const Cpp = {
|
|||
|
||||
// C#
|
||||
export const CSharp = {
|
||||
name: "C#",
|
||||
topLevelKeywords: ["class", "namespace", "void"],
|
||||
singleLineComment: "//",
|
||||
endOfLine: [";"],
|
||||
|
|
@ -51,6 +57,7 @@ export const CSharp = {
|
|||
|
||||
// C
|
||||
export const C = {
|
||||
name: "C",
|
||||
topLevelKeywords: ["if", "else", "while", "for", "switch", "case"],
|
||||
singleLineComment: "//",
|
||||
endOfLine: [";"],
|
||||
|
|
@ -58,6 +65,7 @@ export const C = {
|
|||
|
||||
// Scala
|
||||
export const Scala = {
|
||||
name: "Scala",
|
||||
topLevelKeywords: ["def", "val", "var", "class", "object", "trait"],
|
||||
singleLineComment: "//",
|
||||
endOfLine: [";"],
|
||||
|
|
@ -65,6 +73,7 @@ export const Scala = {
|
|||
|
||||
// Go
|
||||
export const Go = {
|
||||
name: "Go",
|
||||
topLevelKeywords: ["func", "package", "import", "type"],
|
||||
singleLineComment: "//",
|
||||
endOfLine: [],
|
||||
|
|
@ -72,6 +81,7 @@ export const Go = {
|
|||
|
||||
// Rust
|
||||
export const Rust = {
|
||||
name: "Rust",
|
||||
topLevelKeywords: ["fn", "mod", "pub", "struct", "enum", "trait"],
|
||||
singleLineComment: "//",
|
||||
endOfLine: [";"],
|
||||
|
|
@ -79,6 +89,7 @@ export const Rust = {
|
|||
|
||||
// Haskell
|
||||
export const Haskell = {
|
||||
name: "Haskell",
|
||||
topLevelKeywords: [
|
||||
"data",
|
||||
"type",
|
||||
|
|
@ -95,6 +106,7 @@ export const Haskell = {
|
|||
|
||||
// PHP
|
||||
export const PHP = {
|
||||
name: "PHP",
|
||||
topLevelKeywords: ["function", "class", "namespace", "use"],
|
||||
singleLineComment: "//",
|
||||
endOfLine: [";"],
|
||||
|
|
@ -102,6 +114,7 @@ export const PHP = {
|
|||
|
||||
// Ruby on Rails
|
||||
export const RubyOnRails = {
|
||||
name: "Ruby on Rails",
|
||||
topLevelKeywords: ["def", "class", "module"],
|
||||
singleLineComment: "#",
|
||||
endOfLine: [],
|
||||
|
|
@ -109,6 +122,7 @@ export const RubyOnRails = {
|
|||
|
||||
// Swift
|
||||
export const Swift = {
|
||||
name: "Swift",
|
||||
topLevelKeywords: ["func", "class", "struct", "import"],
|
||||
singleLineComment: "//",
|
||||
endOfLine: [";"],
|
||||
|
|
@ -116,6 +130,7 @@ export const Swift = {
|
|||
|
||||
// Kotlin
|
||||
export const Kotlin = {
|
||||
name: "Kotlin",
|
||||
topLevelKeywords: ["fun", "class", "package", "import"],
|
||||
singleLineComment: "//",
|
||||
endOfLine: [";"],
|
||||
|
|
@ -123,6 +138,7 @@ export const Kotlin = {
|
|||
|
||||
// Ruby
|
||||
export const Ruby = {
|
||||
name: "Ruby",
|
||||
topLevelKeywords: ["class", "module", "def"],
|
||||
singleLineComment: "#",
|
||||
endOfLine: [],
|
||||
|
|
@ -130,6 +146,7 @@ export const Ruby = {
|
|||
|
||||
// Clojure
|
||||
export const Clojure = {
|
||||
name: "Clojure",
|
||||
topLevelKeywords: ["def", "fn", "let", "do", "if", "defn", "ns", "defmacro"],
|
||||
singleLineComment: ";",
|
||||
endOfLine: [],
|
||||
|
|
@ -137,6 +154,7 @@ export const Clojure = {
|
|||
|
||||
// Julia
|
||||
export const Julia = {
|
||||
name: "Julia",
|
||||
topLevelKeywords: [
|
||||
"function",
|
||||
"macro",
|
||||
|
|
@ -155,6 +173,7 @@ export const Julia = {
|
|||
|
||||
// F#
|
||||
export const FSharp = {
|
||||
name: "F#",
|
||||
topLevelKeywords: [
|
||||
"let",
|
||||
"type",
|
||||
|
|
@ -173,6 +192,7 @@ export const FSharp = {
|
|||
|
||||
// R
|
||||
export const R = {
|
||||
name: "R",
|
||||
topLevelKeywords: [
|
||||
"function",
|
||||
"if",
|
||||
|
|
@ -189,6 +209,7 @@ export const R = {
|
|||
|
||||
// Dart
|
||||
export const Dart = {
|
||||
name: "Dart",
|
||||
topLevelKeywords: ["class", "import", "void", "enum"],
|
||||
singleLineComment: "//",
|
||||
endOfLine: [";"],
|
||||
|
|
@ -196,6 +217,7 @@ export const Dart = {
|
|||
|
||||
// Solidity
|
||||
export const Solidity = {
|
||||
name: "Solidity",
|
||||
topLevelKeywords: [
|
||||
"contract",
|
||||
"event",
|
||||
|
|
@ -218,6 +240,7 @@ export const Solidity = {
|
|||
|
||||
// YAML
|
||||
export const YAML: AutocompleteLanguageInfo = {
|
||||
name: "YAML",
|
||||
topLevelKeywords: [],
|
||||
singleLineComment: "#",
|
||||
endOfLine: [],
|
||||
|
|
@ -258,6 +281,7 @@ export const YAML: AutocompleteLanguageInfo = {
|
|||
};
|
||||
|
||||
export const Markdown: AutocompleteLanguageInfo = {
|
||||
name: "Markdown",
|
||||
topLevelKeywords: [],
|
||||
singleLineComment: "",
|
||||
endOfLine: [],
|
||||
|
|
|
|||
|
|
@ -19,6 +19,7 @@ interface AutocompleteTemplate {
|
|||
suffix: string,
|
||||
filepath: string,
|
||||
reponame: string,
|
||||
language: string,
|
||||
snippets: AutocompleteSnippet[],
|
||||
) => string);
|
||||
completionOptions?: Partial<CompletionOptions>;
|
||||
|
|
@ -78,6 +79,7 @@ const codestralMultifileFimTemplate: AutocompleteTemplate = {
|
|||
suffix: string,
|
||||
filepath: string,
|
||||
reponame: string,
|
||||
language: string,
|
||||
snippets: AutocompleteSnippet[],
|
||||
): string => {
|
||||
return `[SUFFIX]${suffix}[PREFIX]${prefix}`;
|
||||
|
|
@ -109,6 +111,7 @@ const starcoder2FimTemplate: AutocompleteTemplate = {
|
|||
suffix: string,
|
||||
filename: string,
|
||||
reponame: string,
|
||||
language: string,
|
||||
snippets: AutocompleteSnippet[],
|
||||
): string => {
|
||||
const otherFiles =
|
||||
|
|
@ -155,9 +158,40 @@ const deepseekFimTemplate: AutocompleteTemplate = {
|
|||
},
|
||||
};
|
||||
|
||||
const deepseekFimTemplateWrongPipeChar: AutocompleteTemplate = {
|
||||
template: "<|fim▁begin|>{{{prefix}}}<|fim▁hole|>{{{suffix}}}<|fim▁end|>",
|
||||
completionOptions: { stop: ["<|fim▁begin|>", "<|fim▁hole|>", "<|fim▁end|>"] },
|
||||
// https://github.com/THUDM/CodeGeeX4/blob/main/guides/Infilling_guideline.md
|
||||
const codegeexFimTemplate: AutocompleteTemplate = {
|
||||
template: (
|
||||
prefix: string,
|
||||
suffix: string,
|
||||
filepath: string,
|
||||
reponame: string,
|
||||
language: string,
|
||||
snippets: AutocompleteSnippet[],
|
||||
): string => {
|
||||
const relativePaths = shortestRelativePaths([
|
||||
...snippets.map((snippet) => snippet.filepath),
|
||||
filepath,
|
||||
]);
|
||||
const baseTemplate = `###PATH:${relativePaths[relativePaths.length - 1]}\n###LANGUAGE:${language}\n###MODE:BLOCK\n<|code_suffix|>${suffix}<|code_prefix|>${prefix}<|code_middle|>`;
|
||||
if (snippets.length == 0) {
|
||||
return `<|user|>\n${baseTemplate}<|assistant|>\n`;
|
||||
}
|
||||
const references = `###REFERENCE:\n${snippets
|
||||
.map((snippet, i) => `###PATH:${relativePaths[i]}\n${snippet.contents}\n`)
|
||||
.join("###REFERENCE:\n")}`;
|
||||
const prompt = `<|user|>\n${references}\n${baseTemplate}<|assistant|>\n`;
|
||||
return prompt;
|
||||
},
|
||||
completionOptions: {
|
||||
stop: [
|
||||
"<|user|>",
|
||||
"<|code_suffix|>",
|
||||
"<|code_prefix|>",
|
||||
"<|code_middle|>",
|
||||
"<|assistant|>",
|
||||
"<|endoftext|>",
|
||||
],
|
||||
},
|
||||
};
|
||||
|
||||
const gptAutocompleteTemplate: AutocompleteTemplate = {
|
||||
|
|
@ -175,6 +209,7 @@ const holeFillerTemplate: AutocompleteTemplate = {
|
|||
suffix: string,
|
||||
filename: string,
|
||||
reponame: string,
|
||||
language: string,
|
||||
snippets: AutocompleteSnippet[],
|
||||
) => {
|
||||
// From https://github.com/VictorTaelin/AI-scripts
|
||||
|
|
@ -307,6 +342,10 @@ export function getTemplateForModel(model: string): AutocompleteTemplate {
|
|||
return deepseekFimTemplate;
|
||||
}
|
||||
|
||||
if (lowerCaseModel.includes("codegeex")) {
|
||||
return codegeexFimTemplate;
|
||||
}
|
||||
|
||||
if (
|
||||
lowerCaseModel.includes("gpt") ||
|
||||
lowerCaseModel.includes("davinci-002") ||
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ import {
|
|||
SlashCommand,
|
||||
SlashCommandDescription,
|
||||
} from "../index.js";
|
||||
import { stripImages } from "../llm/countTokens.js";
|
||||
import { stripImages } from "../llm/images.js";
|
||||
import { renderTemplatedString } from "../llm/llms/index.js";
|
||||
import SlashCommands from "./slash/index.js";
|
||||
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
import { SlashCommand } from "../../index.js";
|
||||
import { stripImages } from "../../llm/countTokens.js";
|
||||
import { stripImages } from "../../llm/images.js";
|
||||
|
||||
const CommitMessageCommand: SlashCommand = {
|
||||
name: "commit",
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
import { ChatMessage, SlashCommand } from "../../index.js";
|
||||
import { stripImages } from "../../llm/countTokens.js";
|
||||
import { stripImages } from "../../llm/images.js";
|
||||
import { removeQuotesAndEscapes } from "../../util/index.js";
|
||||
|
||||
const PROMPT = (
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ import {
|
|||
} from "../../autocomplete/lineStream.js";
|
||||
import { streamLines } from "../../diff/util.js";
|
||||
import { ContextItemWithId, ILLM, SlashCommand } from "../../index.js";
|
||||
import { stripImages } from "../../llm/countTokens.js";
|
||||
import { stripImages } from "../../llm/images.js";
|
||||
import {
|
||||
dedentAndGetCommonWhitespace,
|
||||
getMarkdownLanguageTagForFile,
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
import { ChatMessage, SlashCommand } from "../../index.js";
|
||||
import { stripImages } from "../../llm/countTokens.js";
|
||||
import { stripImages } from "../../llm/images.js";
|
||||
|
||||
const prompt = `
|
||||
Review the following code, focusing on Readability, Maintainability, Code Smells, Speed, and Memory Performance. Provide feedback with these guidelines:
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ import { homedir } from "node:os";
|
|||
import path from "path";
|
||||
import { languageForFilepath } from "../../autocomplete/constructPrompt.js";
|
||||
import { SlashCommand } from "../../index.js";
|
||||
import { stripImages } from "../../llm/countTokens.js";
|
||||
import { stripImages } from "../../llm/images.js";
|
||||
|
||||
// If useful elsewhere, helper funcs should move to core/util/index.ts or similar
|
||||
function getOffsetDatetime(date: Date): Date {
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
import { constants } from "../../deploy/constants.js";
|
||||
import { ChatMessageRole, FetchFunction, SlashCommand } from "../../index.js";
|
||||
import { pruneStringFromBottom, stripImages } from "../../llm/countTokens.js";
|
||||
import { pruneStringFromBottom } from "../../llm/countTokens.js";
|
||||
import { stripImages } from "../../llm/images.js";
|
||||
|
||||
const PROMPT = (
|
||||
input: string,
|
||||
|
|
|
|||
|
|
@ -1,116 +1,277 @@
|
|||
import {
|
||||
ControlPlaneClient,
|
||||
ControlPlaneSessionInfo,
|
||||
} from "../control-plane/client.js";
|
||||
import {
|
||||
BrowserSerializedContinueConfig,
|
||||
ContinueConfig,
|
||||
ContinueRcJson,
|
||||
IContextProvider,
|
||||
IDE,
|
||||
IdeSettings,
|
||||
ILLM,
|
||||
} from "../index.js";
|
||||
import { Telemetry } from "../util/posthog.js";
|
||||
import { IConfigHandler } from "./IConfigHandler.js";
|
||||
import { finalToBrowserConfig, loadFullConfigNode } from "./load.js";
|
||||
import { GlobalContext } from "../util/GlobalContext.js";
|
||||
import { finalToBrowserConfig } from "./load.js";
|
||||
import ControlPlaneProfileLoader from "./profile/ControlPlaneProfileLoader.js";
|
||||
import { IProfileLoader } from "./profile/IProfileLoader.js";
|
||||
import LocalProfileLoader from "./profile/LocalProfileLoader.js";
|
||||
|
||||
export class ConfigHandler implements IConfigHandler {
|
||||
export interface ProfileDescription {
|
||||
title: string;
|
||||
id: string;
|
||||
}
|
||||
|
||||
// Separately manages saving/reloading each profile
|
||||
class ProfileLifecycleManager {
|
||||
private savedConfig: ContinueConfig | undefined;
|
||||
private savedBrowserConfig?: BrowserSerializedContinueConfig;
|
||||
private pendingConfigPromise?: Promise<ContinueConfig>;
|
||||
|
||||
constructor(private readonly profileLoader: IProfileLoader) {}
|
||||
|
||||
get profileId() {
|
||||
return this.profileLoader.profileId;
|
||||
}
|
||||
|
||||
get profileTitle() {
|
||||
return this.profileLoader.profileTitle;
|
||||
}
|
||||
|
||||
get profileDescription(): ProfileDescription {
|
||||
return {
|
||||
title: this.profileTitle,
|
||||
id: this.profileId,
|
||||
};
|
||||
}
|
||||
|
||||
clearConfig() {
|
||||
this.savedConfig = undefined;
|
||||
this.savedBrowserConfig = undefined;
|
||||
this.pendingConfigPromise = undefined;
|
||||
}
|
||||
|
||||
// Clear saved config and reload
|
||||
reloadConfig(): Promise<ContinueConfig> {
|
||||
this.savedConfig = undefined;
|
||||
this.savedBrowserConfig = undefined;
|
||||
this.pendingConfigPromise = undefined;
|
||||
|
||||
return this.profileLoader.doLoadConfig();
|
||||
}
|
||||
|
||||
async loadConfig(
|
||||
additionalContextProviders: IContextProvider[],
|
||||
): Promise<ContinueConfig> {
|
||||
// If we already have a config, return it
|
||||
if (this.savedConfig) {
|
||||
return this.savedConfig;
|
||||
} else if (this.pendingConfigPromise) {
|
||||
return this.pendingConfigPromise;
|
||||
}
|
||||
|
||||
// Set pending config promise
|
||||
this.pendingConfigPromise = new Promise(async (resolve, reject) => {
|
||||
const newConfig = await this.profileLoader.doLoadConfig();
|
||||
|
||||
// Add registered context providers
|
||||
newConfig.contextProviders = (newConfig.contextProviders ?? []).concat(
|
||||
additionalContextProviders,
|
||||
);
|
||||
|
||||
this.savedConfig = newConfig;
|
||||
resolve(newConfig);
|
||||
});
|
||||
|
||||
// Wait for the config promise to resolve
|
||||
this.savedConfig = await this.pendingConfigPromise;
|
||||
this.pendingConfigPromise = undefined;
|
||||
return this.savedConfig;
|
||||
}
|
||||
|
||||
async getSerializedConfig(
|
||||
additionalContextProviders: IContextProvider[],
|
||||
): Promise<BrowserSerializedContinueConfig> {
|
||||
if (!this.savedBrowserConfig) {
|
||||
const continueConfig = await this.loadConfig(additionalContextProviders);
|
||||
this.savedBrowserConfig = finalToBrowserConfig(continueConfig);
|
||||
}
|
||||
return this.savedBrowserConfig;
|
||||
}
|
||||
}
|
||||
|
||||
export class ConfigHandler {
|
||||
private readonly globalContext = new GlobalContext();
|
||||
private additionalContextProviders: IContextProvider[] = [];
|
||||
private profiles: ProfileLifecycleManager[];
|
||||
private selectedProfileId: string;
|
||||
|
||||
// This will be the local profile
|
||||
private get fallbackProfile() {
|
||||
return this.profiles[0];
|
||||
}
|
||||
|
||||
get currentProfile() {
|
||||
return (
|
||||
this.profiles.find((p) => p.profileId === this.selectedProfileId) ??
|
||||
this.fallbackProfile
|
||||
);
|
||||
}
|
||||
|
||||
get inactiveProfiles() {
|
||||
return this.profiles.filter((p) => p.profileId !== this.selectedProfileId);
|
||||
}
|
||||
|
||||
private async fetchControlPlaneProfiles() {
|
||||
// Get the profiles and create their lifecycle managers
|
||||
this.controlPlaneClient.listWorkspaces().then(async (workspaces) => {
|
||||
this.profiles = this.profiles.filter(
|
||||
(profile) => profile.profileId === "local",
|
||||
);
|
||||
workspaces.forEach((workspace) => {
|
||||
const profileLoader = new ControlPlaneProfileLoader(
|
||||
workspace.id,
|
||||
workspace.name,
|
||||
this.controlPlaneClient,
|
||||
this.ide,
|
||||
this.ideSettingsPromise,
|
||||
this.writeLog,
|
||||
this.reloadConfig.bind(this),
|
||||
);
|
||||
this.profiles.push(new ProfileLifecycleManager(profileLoader));
|
||||
});
|
||||
|
||||
this.notifyProfileListeners(
|
||||
this.profiles.map((profile) => profile.profileDescription),
|
||||
);
|
||||
|
||||
// Check the last selected workspace, and reload if it isn't local
|
||||
const workspaceId = await this.getWorkspaceId();
|
||||
const lastSelectedWorkspaceIds =
|
||||
this.globalContext.get("lastSelectedProfileForWorkspace") ?? {};
|
||||
const selectedWorkspaceId = lastSelectedWorkspaceIds[workspaceId];
|
||||
if (selectedWorkspaceId) {
|
||||
this.selectedProfileId = selectedWorkspaceId;
|
||||
this.loadConfig();
|
||||
} else {
|
||||
// Otherwise we stick with local profile, and record choice
|
||||
lastSelectedWorkspaceIds[workspaceId] = this.selectedProfileId;
|
||||
this.globalContext.update(
|
||||
"lastSelectedProfileForWorkspace",
|
||||
lastSelectedWorkspaceIds,
|
||||
);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
constructor(
|
||||
private readonly ide: IDE,
|
||||
private ideSettingsPromise: Promise<IdeSettings>,
|
||||
private readonly writeLog: (text: string) => Promise<void>,
|
||||
private controlPlaneClient: ControlPlaneClient,
|
||||
) {
|
||||
this.ide = ide;
|
||||
this.ideSettingsPromise = ideSettingsPromise;
|
||||
this.writeLog = writeLog;
|
||||
|
||||
// Set local profile as default
|
||||
const localProfileLoader = new LocalProfileLoader(
|
||||
ide,
|
||||
ideSettingsPromise,
|
||||
controlPlaneClient,
|
||||
writeLog,
|
||||
);
|
||||
this.profiles = [new ProfileLifecycleManager(localProfileLoader)];
|
||||
this.selectedProfileId = localProfileLoader.profileId;
|
||||
|
||||
// Always load local profile immediately in case control plane doesn't load
|
||||
try {
|
||||
this.loadConfig();
|
||||
} catch (e) {
|
||||
console.error("Failed to load config: ", e);
|
||||
}
|
||||
|
||||
// Load control plane profiles
|
||||
this.fetchControlPlaneProfiles();
|
||||
}
|
||||
|
||||
async setSelectedProfile(profileId: string) {
|
||||
this.selectedProfileId = profileId;
|
||||
const newConfig = await this.loadConfig();
|
||||
this.notifyConfigListeners(newConfig);
|
||||
const selectedProfiles =
|
||||
this.globalContext.get("lastSelectedProfileForWorkspace") ?? {};
|
||||
selectedProfiles[await this.getWorkspaceId()] = profileId;
|
||||
this.globalContext.update(
|
||||
"lastSelectedProfileForWorkspace",
|
||||
selectedProfiles,
|
||||
);
|
||||
}
|
||||
|
||||
// A unique ID for the current workspace, built from folder names
|
||||
private async getWorkspaceId(): Promise<string> {
|
||||
const dirs = await this.ide.getWorkspaceDirs();
|
||||
return dirs.join("&");
|
||||
}
|
||||
|
||||
// Automatically refresh config when Continue-related IDE (e.g. VS Code) settings are changed
|
||||
updateIdeSettings(ideSettings: IdeSettings) {
|
||||
this.ideSettingsPromise = Promise.resolve(ideSettings);
|
||||
this.reloadConfig();
|
||||
}
|
||||
|
||||
updateControlPlaneSessionInfo(
|
||||
sessionInfo: ControlPlaneSessionInfo | undefined,
|
||||
) {
|
||||
this.controlPlaneClient = new ControlPlaneClient(
|
||||
Promise.resolve(sessionInfo),
|
||||
);
|
||||
this.fetchControlPlaneProfiles();
|
||||
}
|
||||
|
||||
private profilesListeners: ((profiles: ProfileDescription[]) => void)[] = [];
|
||||
onDidChangeAvailableProfiles(
|
||||
listener: (profiles: ProfileDescription[]) => void,
|
||||
) {
|
||||
this.profilesListeners.push(listener);
|
||||
}
|
||||
|
||||
private notifyProfileListeners(profiles: ProfileDescription[]) {
|
||||
for (const listener of this.profilesListeners) {
|
||||
listener(profiles);
|
||||
}
|
||||
}
|
||||
|
||||
private notifyConfigListeners(newConfig: ContinueConfig) {
|
||||
// Notify listeners that config changed
|
||||
for (const listener of this.updateListeners) {
|
||||
listener(newConfig);
|
||||
}
|
||||
}
|
||||
|
||||
private updateListeners: ((newConfig: ContinueConfig) => void)[] = [];
|
||||
onConfigUpdate(listener: (newConfig: ContinueConfig) => void) {
|
||||
this.updateListeners.push(listener);
|
||||
}
|
||||
|
||||
async reloadConfig() {
|
||||
this.savedConfig = undefined;
|
||||
this.savedBrowserConfig = undefined;
|
||||
this._pendingConfigPromise = undefined;
|
||||
|
||||
const newConfig = await this.loadConfig();
|
||||
|
||||
for (const listener of this.updateListeners) {
|
||||
listener(newConfig);
|
||||
}
|
||||
// TODO: this isn't right, there are two different senses in which you want to "reload"
|
||||
const newConfig = await this.currentProfile.reloadConfig();
|
||||
this.inactiveProfiles.forEach((profile) => profile.clearConfig());
|
||||
this.notifyConfigListeners(newConfig);
|
||||
}
|
||||
|
||||
async getSerializedConfig(): Promise<BrowserSerializedContinueConfig> {
|
||||
if (!this.savedBrowserConfig) {
|
||||
this.savedConfig = await this.loadConfig();
|
||||
this.savedBrowserConfig = finalToBrowserConfig(this.savedConfig);
|
||||
}
|
||||
return this.savedBrowserConfig;
|
||||
getSerializedConfig(): Promise<BrowserSerializedContinueConfig> {
|
||||
return this.currentProfile.getSerializedConfig(
|
||||
this.additionalContextProviders,
|
||||
);
|
||||
}
|
||||
|
||||
listProfiles(): ProfileDescription[] {
|
||||
return this.profiles.map((p) => p.profileDescription);
|
||||
}
|
||||
|
||||
private _pendingConfigPromise?: Promise<ContinueConfig>;
|
||||
async loadConfig(): Promise<ContinueConfig> {
|
||||
if (this.savedConfig) {
|
||||
return this.savedConfig;
|
||||
} else if (this._pendingConfigPromise) {
|
||||
return this._pendingConfigPromise;
|
||||
}
|
||||
|
||||
this._pendingConfigPromise = new Promise(async (resolve, reject) => {
|
||||
let workspaceConfigs: ContinueRcJson[] = [];
|
||||
try {
|
||||
workspaceConfigs = await this.ide.getWorkspaceConfigs();
|
||||
} catch (e) {
|
||||
console.warn("Failed to load workspace configs");
|
||||
}
|
||||
|
||||
const ideInfo = await this.ide.getIdeInfo();
|
||||
const uniqueId = await this.ide.getUniqueId();
|
||||
const ideSettings = await this.ideSettingsPromise;
|
||||
|
||||
const newConfig = await loadFullConfigNode(
|
||||
this.ide,
|
||||
workspaceConfigs,
|
||||
ideSettings,
|
||||
ideInfo.ideType,
|
||||
uniqueId,
|
||||
this.writeLog,
|
||||
);
|
||||
newConfig.allowAnonymousTelemetry =
|
||||
newConfig.allowAnonymousTelemetry &&
|
||||
(await this.ide.isTelemetryEnabled());
|
||||
|
||||
// Setup telemetry only after (and if) we know it is enabled
|
||||
await Telemetry.setup(
|
||||
newConfig.allowAnonymousTelemetry ?? true,
|
||||
await this.ide.getUniqueId(),
|
||||
ideInfo.extensionVersion,
|
||||
);
|
||||
|
||||
(newConfig.contextProviders ?? []).push(
|
||||
...this.additionalContextProviders,
|
||||
);
|
||||
|
||||
this.savedConfig = newConfig;
|
||||
resolve(newConfig);
|
||||
});
|
||||
|
||||
this.savedConfig = await this._pendingConfigPromise;
|
||||
this._pendingConfigPromise = undefined;
|
||||
return this.savedConfig;
|
||||
return this.currentProfile.loadConfig(this.additionalContextProviders);
|
||||
}
|
||||
|
||||
async llmFromTitle(title?: string): Promise<ILLM> {
|
||||
|
|
|
|||
|
|
@ -1,17 +0,0 @@
|
|||
import {
|
||||
BrowserSerializedContinueConfig,
|
||||
ContinueConfig,
|
||||
IContextProvider,
|
||||
IdeSettings,
|
||||
ILLM,
|
||||
} from "../index.js";
|
||||
|
||||
export interface IConfigHandler {
|
||||
updateIdeSettings(ideSettings: IdeSettings): void;
|
||||
onConfigUpdate(listener: (newConfig: ContinueConfig) => void): void;
|
||||
reloadConfig(): Promise<void>;
|
||||
getSerializedConfig(): Promise<BrowserSerializedContinueConfig>;
|
||||
loadConfig(): Promise<ContinueConfig>;
|
||||
llmFromTitle(title?: string): Promise<ILLM>;
|
||||
registerCustomContextProvider(contextProvider: IContextProvider): void;
|
||||
}
|
||||
|
|
@ -32,39 +32,21 @@ export const FREE_TRIAL_MODELS: ModelDescription[] = [
|
|||
},
|
||||
];
|
||||
|
||||
export const defaultConfig: SerializedContinueConfig = {
|
||||
models: [],
|
||||
customCommands: [
|
||||
{
|
||||
name: "test",
|
||||
prompt:
|
||||
"{{{ input }}}\n\nWrite a comprehensive set of unit tests for the selected code. It should setup, run tests that check for correctness including important edge cases, and teardown. Ensure that the tests are complete and sophisticated. Give the tests just as chat output, don't edit any file.",
|
||||
description: "Write unit tests for highlighted code",
|
||||
},
|
||||
],
|
||||
tabAutocompleteModel: {
|
||||
title: "Starcoder2 3b",
|
||||
provider: "ollama",
|
||||
model: "starcoder2:3b",
|
||||
},
|
||||
};
|
||||
export const defaultContextProvidersVsCode: ContextProviderWithParams[] = [
|
||||
{ name: "code", params: {} },
|
||||
{ name: "docs", params: {} },
|
||||
{ name: "diff", params: {} },
|
||||
{ name: "terminal", params: {} },
|
||||
{ name: "problems", params: {} },
|
||||
{ name: "folder", params: {} },
|
||||
{ name: "codebase", params: {} },
|
||||
];
|
||||
|
||||
export const defaultConfigJetBrains: SerializedContinueConfig = {
|
||||
models: FREE_TRIAL_MODELS,
|
||||
customCommands: [
|
||||
{
|
||||
name: "test",
|
||||
prompt:
|
||||
"{{{ input }}}\n\nWrite a comprehensive set of unit tests for the selected code. It should setup, run tests that check for correctness including important edge cases, and teardown. Ensure that the tests are complete and sophisticated. Give the tests just as chat output, don't edit any file.",
|
||||
description: "Write unit tests for highlighted code",
|
||||
},
|
||||
],
|
||||
tabAutocompleteModel: {
|
||||
title: "Starcoder2 3b",
|
||||
provider: "ollama",
|
||||
model: "starcoder2:3b",
|
||||
},
|
||||
};
|
||||
export const defaultContextProvidersJetBrains: ContextProviderWithParams[] = [
|
||||
{ name: "diff", params: {} },
|
||||
{ name: "folder", params: {} },
|
||||
{ name: "codebase", params: {} },
|
||||
];
|
||||
|
||||
export const defaultSlashCommandsVscode: SlashCommandDescription[] = [
|
||||
{
|
||||
|
|
@ -108,18 +90,40 @@ export const defaultSlashCommandsJetBrains = [
|
|||
},
|
||||
];
|
||||
|
||||
export const defaultContextProvidersVsCode: ContextProviderWithParams[] = [
|
||||
{ name: "code", params: {} },
|
||||
{ name: "docs", params: {} },
|
||||
{ name: "diff", params: {} },
|
||||
{ name: "terminal", params: {} },
|
||||
{ name: "problems", params: {} },
|
||||
{ name: "folder", params: {} },
|
||||
{ name: "codebase", params: {} },
|
||||
];
|
||||
export const defaultConfig: SerializedContinueConfig = {
|
||||
models: [],
|
||||
customCommands: [
|
||||
{
|
||||
name: "test",
|
||||
prompt:
|
||||
"{{{ input }}}\n\nWrite a comprehensive set of unit tests for the selected code. It should setup, run tests that check for correctness including important edge cases, and teardown. Ensure that the tests are complete and sophisticated. Give the tests just as chat output, don't edit any file.",
|
||||
description: "Write unit tests for highlighted code",
|
||||
},
|
||||
],
|
||||
tabAutocompleteModel: {
|
||||
title: "Starcoder2 3b",
|
||||
provider: "ollama",
|
||||
model: "starcoder2:3b",
|
||||
},
|
||||
contextProviders: defaultContextProvidersVsCode,
|
||||
slashCommands: defaultSlashCommandsVscode,
|
||||
};
|
||||
|
||||
export const defaultContextProvidersJetBrains: ContextProviderWithParams[] = [
|
||||
{ name: "diff", params: {} },
|
||||
{ name: "folder", params: {} },
|
||||
{ name: "codebase", params: {} },
|
||||
];
|
||||
export const defaultConfigJetBrains: SerializedContinueConfig = {
|
||||
models: FREE_TRIAL_MODELS,
|
||||
customCommands: [
|
||||
{
|
||||
name: "test",
|
||||
prompt:
|
||||
"{{{ input }}}\n\nWrite a comprehensive set of unit tests for the selected code. It should setup, run tests that check for correctness including important edge cases, and teardown. Ensure that the tests are complete and sophisticated. Give the tests just as chat output, don't edit any file.",
|
||||
description: "Write unit tests for highlighted code",
|
||||
},
|
||||
],
|
||||
tabAutocompleteModel: {
|
||||
title: "Starcoder2 3b",
|
||||
provider: "ollama",
|
||||
model: "starcoder2:3b",
|
||||
},
|
||||
contextProviders: defaultContextProvidersJetBrains,
|
||||
slashCommands: defaultSlashCommandsJetBrains,
|
||||
};
|
||||
|
|
|
|||
|
|
@ -36,6 +36,8 @@ import CustomLLMClass from "../llm/llms/CustomLLM.js";
|
|||
import FreeTrial from "../llm/llms/FreeTrial.js";
|
||||
import { llmFromDescription } from "../llm/llms/index.js";
|
||||
|
||||
import CodebaseContextProvider from "../context/providers/CodebaseContextProvider.js";
|
||||
import ContinueProxyContextProvider from "../context/providers/ContinueProxyContextProvider.js";
|
||||
import { fetchwithRequestOptions } from "../util/fetchWithOptions.js";
|
||||
import { copyOf } from "../util/index.js";
|
||||
import mergeJson from "../util/merge.js";
|
||||
|
|
@ -46,7 +48,6 @@ import {
|
|||
getConfigJsonPathForRemote,
|
||||
getConfigTsPath,
|
||||
getContinueDotEnv,
|
||||
migrate,
|
||||
readAllGlobalPromptFiles,
|
||||
} from "../util/paths.js";
|
||||
import {
|
||||
|
|
@ -60,6 +61,7 @@ import {
|
|||
getPromptFiles,
|
||||
slashCommandFromPromptFile,
|
||||
} from "./promptFile.js";
|
||||
|
||||
const { execSync } = require("child_process");
|
||||
|
||||
function resolveSerializedConfig(filepath: string): SerializedContinueConfig {
|
||||
|
|
@ -95,32 +97,22 @@ function loadSerializedConfig(
|
|||
workspaceConfigs: ContinueRcJson[],
|
||||
ideSettings: IdeSettings,
|
||||
ideType: IdeType,
|
||||
overrideConfigJson: SerializedContinueConfig | undefined,
|
||||
): SerializedContinueConfig {
|
||||
const configPath = getConfigJsonPath(ideType);
|
||||
let config: SerializedContinueConfig;
|
||||
try {
|
||||
config = resolveSerializedConfig(configPath);
|
||||
} catch (e) {
|
||||
throw new Error(`Failed to parse config.json: ${e}`);
|
||||
let config: SerializedContinueConfig = overrideConfigJson!;
|
||||
if (!config) {
|
||||
try {
|
||||
config = resolveSerializedConfig(configPath);
|
||||
} catch (e) {
|
||||
throw new Error(`Failed to parse config.json: ${e}`);
|
||||
}
|
||||
}
|
||||
|
||||
if (config.allowAnonymousTelemetry === undefined) {
|
||||
config.allowAnonymousTelemetry = true;
|
||||
}
|
||||
|
||||
migrate("codeContextProvider", () => {
|
||||
const gpt = config.models.find(
|
||||
(model) =>
|
||||
model.model.startsWith("gpt-4") && model.provider === "free-trial",
|
||||
);
|
||||
if (gpt) {
|
||||
gpt.systemMessage =
|
||||
"You are an expert software developer. You give helpful and concise responses.";
|
||||
}
|
||||
|
||||
fs.writeFileSync(configPath, JSON.stringify(config, undefined, 2), "utf8");
|
||||
});
|
||||
|
||||
if (ideSettings.remoteConfigServerUrl) {
|
||||
try {
|
||||
const remoteConfigJson = resolveSerializedConfig(
|
||||
|
|
@ -157,6 +149,7 @@ function loadSerializedConfig(
|
|||
async function serializedToIntermediateConfig(
|
||||
initial: SerializedContinueConfig,
|
||||
ide: IDE,
|
||||
loadPromptFiles: boolean = true,
|
||||
): Promise<Config> {
|
||||
const slashCommands: SlashCommand[] = [];
|
||||
for (const command of initial.slashCommands || []) {
|
||||
|
|
@ -172,25 +165,27 @@ async function serializedToIntermediateConfig(
|
|||
const workspaceDirs = await ide.getWorkspaceDirs();
|
||||
const promptFolder = initial.experimental?.promptPath;
|
||||
|
||||
let promptFiles: { path: string; content: string }[] = [];
|
||||
promptFiles = (
|
||||
await Promise.all(
|
||||
workspaceDirs.map((dir) =>
|
||||
getPromptFiles(
|
||||
ide,
|
||||
path.join(dir, promptFolder ?? DEFAULT_PROMPTS_FOLDER),
|
||||
if (loadPromptFiles) {
|
||||
let promptFiles: { path: string; content: string }[] = [];
|
||||
promptFiles = (
|
||||
await Promise.all(
|
||||
workspaceDirs.map((dir) =>
|
||||
getPromptFiles(
|
||||
ide,
|
||||
path.join(dir, promptFolder ?? DEFAULT_PROMPTS_FOLDER),
|
||||
),
|
||||
),
|
||||
),
|
||||
)
|
||||
)
|
||||
)
|
||||
.flat()
|
||||
.filter(({ path }) => path.endsWith(".prompt"));
|
||||
.flat()
|
||||
.filter(({ path }) => path.endsWith(".prompt"));
|
||||
|
||||
// Also read from ~/.continue/.prompts
|
||||
promptFiles.push(...readAllGlobalPromptFiles());
|
||||
// Also read from ~/.continue/.prompts
|
||||
promptFiles.push(...readAllGlobalPromptFiles());
|
||||
|
||||
for (const file of promptFiles) {
|
||||
slashCommands.push(slashCommandFromPromptFile(file.path, file.content));
|
||||
for (const file of promptFiles) {
|
||||
slashCommands.push(slashCommandFromPromptFile(file.path, file.content));
|
||||
}
|
||||
}
|
||||
|
||||
const config: Config = {
|
||||
|
|
@ -221,9 +216,11 @@ async function intermediateToFinalConfig(
|
|||
ideSettings: IdeSettings,
|
||||
uniqueId: string,
|
||||
writeLog: (log: string) => Promise<void>,
|
||||
workOsAccessToken: string | undefined,
|
||||
allowFreeTrial: boolean = true,
|
||||
): Promise<ContinueConfig> {
|
||||
// Auto-detect models
|
||||
const models: BaseLLM[] = [];
|
||||
let models: BaseLLM[] = [];
|
||||
for (const desc of config.models) {
|
||||
if (isModelDescription(desc)) {
|
||||
const llm = await llmFromDescription(
|
||||
|
|
@ -304,15 +301,20 @@ async function intermediateToFinalConfig(
|
|||
};
|
||||
}
|
||||
|
||||
// Obtain auth token (only if free trial being used)
|
||||
const freeTrialModels = models.filter(
|
||||
(model) => model.providerName === "free-trial",
|
||||
);
|
||||
if (freeTrialModels.length > 0) {
|
||||
const ghAuthToken = await ide.getGitHubAuthToken();
|
||||
for (const model of freeTrialModels) {
|
||||
(model as FreeTrial).setupGhAuthToken(ghAuthToken);
|
||||
if (allowFreeTrial) {
|
||||
// Obtain auth token (iff free trial being used)
|
||||
const freeTrialModels = models.filter(
|
||||
(model) => model.providerName === "free-trial",
|
||||
);
|
||||
if (freeTrialModels.length > 0) {
|
||||
const ghAuthToken = await ide.getGitHubAuthToken();
|
||||
for (const model of freeTrialModels) {
|
||||
(model as FreeTrial).setupGhAuthToken(ghAuthToken);
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// Remove free trial models
|
||||
models = models.filter((model) => model.providerName !== "free-trial");
|
||||
}
|
||||
|
||||
// Tab autocomplete model
|
||||
|
|
@ -336,6 +338,10 @@ async function intermediateToFinalConfig(
|
|||
);
|
||||
|
||||
if (llm?.providerName === "free-trial") {
|
||||
if (!allowFreeTrial) {
|
||||
// This shouldn't happen
|
||||
throw new Error("Free trial cannot be used with control plane");
|
||||
}
|
||||
const ghAuthToken = await ide.getGitHubAuthToken();
|
||||
(llm as FreeTrial).setupGhAuthToken(ghAuthToken);
|
||||
}
|
||||
|
|
@ -348,16 +354,39 @@ async function intermediateToFinalConfig(
|
|||
).filter((x) => x !== undefined) as BaseLLM[];
|
||||
}
|
||||
|
||||
// These context providers are always included, regardless of what, if anything,
|
||||
// the user has configured in config.json
|
||||
const DEFAULT_CONTEXT_PROVIDERS = [
|
||||
new FileContextProvider({}),
|
||||
new CodebaseContextProvider({}),
|
||||
];
|
||||
|
||||
const DEFAULT_CONTEXT_PROVIDERS_TITLES = DEFAULT_CONTEXT_PROVIDERS.map(
|
||||
({ description: { title } }) => title,
|
||||
);
|
||||
|
||||
// Context providers
|
||||
const contextProviders: IContextProvider[] = [new FileContextProvider({})];
|
||||
const contextProviders: IContextProvider[] = DEFAULT_CONTEXT_PROVIDERS;
|
||||
|
||||
for (const provider of config.contextProviders || []) {
|
||||
if (isContextProviderWithParams(provider)) {
|
||||
const cls = contextProviderClassFromName(provider.name) as any;
|
||||
if (!cls) {
|
||||
console.warn(`Unknown context provider ${provider.name}`);
|
||||
if (!DEFAULT_CONTEXT_PROVIDERS_TITLES.includes(provider.name)) {
|
||||
console.warn(`Unknown context provider ${provider.name}`);
|
||||
}
|
||||
|
||||
continue;
|
||||
}
|
||||
contextProviders.push(new cls(provider.params));
|
||||
const instance: IContextProvider = new cls(provider.params);
|
||||
|
||||
// Handle continue-proxy
|
||||
if (instance.description.title === "continue-proxy") {
|
||||
(instance as ContinueProxyContextProvider).workOsAccessToken =
|
||||
workOsAccessToken;
|
||||
}
|
||||
|
||||
contextProviders.push(instance);
|
||||
} else {
|
||||
contextProviders.push(new CustomContextProviderClass(provider));
|
||||
}
|
||||
|
|
@ -537,10 +566,21 @@ async function loadFullConfigNode(
|
|||
ideType: IdeType,
|
||||
uniqueId: string,
|
||||
writeLog: (log: string) => Promise<void>,
|
||||
workOsAccessToken: string | undefined,
|
||||
overrideConfigJson: SerializedContinueConfig | undefined,
|
||||
): Promise<ContinueConfig> {
|
||||
let serialized = loadSerializedConfig(workspaceConfigs, ideSettings, ideType);
|
||||
// Serialized config
|
||||
let serialized = loadSerializedConfig(
|
||||
workspaceConfigs,
|
||||
ideSettings,
|
||||
ideType,
|
||||
overrideConfigJson,
|
||||
);
|
||||
|
||||
// Convert serialized to intermediate config
|
||||
let intermediate = await serializedToIntermediateConfig(serialized, ide);
|
||||
|
||||
// Apply config.ts to modify intermediate config
|
||||
const configJsContents = await buildConfigTs();
|
||||
if (configJsContents) {
|
||||
try {
|
||||
|
|
@ -557,7 +597,7 @@ async function loadFullConfigNode(
|
|||
}
|
||||
}
|
||||
|
||||
// Remote config.js
|
||||
// Apply remote config.js to modify intermediate config
|
||||
if (ideSettings.remoteConfigServerUrl) {
|
||||
try {
|
||||
const configJsPathForRemote = getConfigJsPathForRemote(
|
||||
|
|
@ -574,12 +614,14 @@ async function loadFullConfigNode(
|
|||
}
|
||||
}
|
||||
|
||||
// Convert to final config format
|
||||
const finalConfig = await intermediateToFinalConfig(
|
||||
intermediate,
|
||||
ide,
|
||||
ideSettings,
|
||||
uniqueId,
|
||||
writeLog,
|
||||
workOsAccessToken,
|
||||
);
|
||||
return finalConfig;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -10,11 +10,6 @@ export function setupApiKeysMode(
|
|||
return {
|
||||
...config,
|
||||
models: config.models.filter((model) => model.provider !== "free-trial"),
|
||||
tabAutocompleteModel: {
|
||||
title: "Tab Autocomplete",
|
||||
provider: "free-trial",
|
||||
model: TRIAL_FIM_MODEL,
|
||||
},
|
||||
embeddingsProvider: {
|
||||
provider: "free-trial",
|
||||
},
|
||||
|
|
|
|||
|
|
@ -0,0 +1,57 @@
|
|||
import { ConfigJson } from "@continuedev/config-types";
|
||||
import {
|
||||
ContinueConfig,
|
||||
IDE,
|
||||
IdeSettings,
|
||||
SerializedContinueConfig,
|
||||
} from "../..";
|
||||
import { ControlPlaneClient } from "../../control-plane/client";
|
||||
import { IProfileLoader } from "./IProfileLoader";
|
||||
import doLoadConfig from "./doLoadConfig";
|
||||
|
||||
export default class ControlPlaneProfileLoader implements IProfileLoader {
|
||||
private static RELOAD_INTERVAL = 1000 * 60 * 15; // every 15 minutes
|
||||
|
||||
readonly profileId: string;
|
||||
profileTitle: string;
|
||||
|
||||
workspaceSettings: ConfigJson | undefined;
|
||||
|
||||
constructor(
|
||||
private readonly workspaceId: string,
|
||||
private workspaceTitle: string,
|
||||
private readonly controlPlaneClient: ControlPlaneClient,
|
||||
private readonly ide: IDE,
|
||||
private ideSettingsPromise: Promise<IdeSettings>,
|
||||
private writeLog: (message: string) => Promise<void>,
|
||||
private readonly onReload: () => void,
|
||||
) {
|
||||
this.profileId = workspaceId;
|
||||
this.profileTitle = workspaceTitle;
|
||||
|
||||
setInterval(async () => {
|
||||
this.workspaceSettings =
|
||||
await this.controlPlaneClient.getSettingsForWorkspace(this.profileId);
|
||||
this.onReload();
|
||||
}, ControlPlaneProfileLoader.RELOAD_INTERVAL);
|
||||
}
|
||||
|
||||
async doLoadConfig(): Promise<ContinueConfig> {
|
||||
const settings =
|
||||
this.workspaceSettings ??
|
||||
((await this.controlPlaneClient.getSettingsForWorkspace(
|
||||
this.profileId,
|
||||
)) as any);
|
||||
const serializedConfig: SerializedContinueConfig = settings;
|
||||
|
||||
return doLoadConfig(
|
||||
this.ide,
|
||||
this.ideSettingsPromise,
|
||||
this.controlPlaneClient,
|
||||
this.writeLog,
|
||||
serializedConfig,
|
||||
);
|
||||
}
|
||||
|
||||
setIsActive(isActive: boolean): void {}
|
||||
}
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
// ProfileHandlers manage the loading of a config, allowing us to abstract over different ways of getting to a ContinueConfig
|
||||
|
||||
import { ContinueConfig } from "../..";
|
||||
|
||||
// After we have the ContinueConfig, the ConfigHandler takes care of everything else (loading models, lifecycle, etc.)
|
||||
export interface IProfileLoader {
|
||||
profileTitle: string;
|
||||
profileId: string;
|
||||
doLoadConfig(): Promise<ContinueConfig>;
|
||||
setIsActive(isActive: boolean): void;
|
||||
}
|
||||
|
|
@ -0,0 +1,29 @@
|
|||
import { ContinueConfig, IDE, IdeSettings } from "../..";
|
||||
import { ControlPlaneClient } from "../../control-plane/client";
|
||||
import doLoadConfig from "./doLoadConfig";
|
||||
import { IProfileLoader } from "./IProfileLoader";
|
||||
|
||||
export default class LocalProfileLoader implements IProfileLoader {
|
||||
static ID = "local";
|
||||
profileId = LocalProfileLoader.ID;
|
||||
profileTitle = "Local Config";
|
||||
|
||||
constructor(
|
||||
private ide: IDE,
|
||||
private ideSettingsPromise: Promise<IdeSettings>,
|
||||
private controlPlaneClient: ControlPlaneClient,
|
||||
private writeLog: (message: string) => Promise<void>,
|
||||
) {}
|
||||
|
||||
async doLoadConfig(): Promise<ContinueConfig> {
|
||||
return doLoadConfig(
|
||||
this.ide,
|
||||
this.ideSettingsPromise,
|
||||
this.controlPlaneClient,
|
||||
this.writeLog,
|
||||
undefined,
|
||||
);
|
||||
}
|
||||
|
||||
setIsActive(isActive: boolean): void {}
|
||||
}
|
||||
|
|
@ -0,0 +1,82 @@
|
|||
import {
|
||||
ContinueRcJson,
|
||||
IDE,
|
||||
IdeSettings,
|
||||
SerializedContinueConfig,
|
||||
} from "../..";
|
||||
import { ContinueProxyReranker } from "../../context/rerankers/ContinueProxyReranker";
|
||||
import { ControlPlaneClient } from "../../control-plane/client";
|
||||
import { TeamAnalytics } from "../../control-plane/TeamAnalytics";
|
||||
import ContinueProxyEmbeddingsProvider from "../../indexing/embeddings/ContinueProxyEmbeddingsProvider";
|
||||
import ContinueProxy from "../../llm/llms/stubs/ContinueProxy";
|
||||
import { Telemetry } from "../../util/posthog";
|
||||
import { loadFullConfigNode } from "../load";
|
||||
|
||||
export default async function doLoadConfig(
|
||||
ide: IDE,
|
||||
ideSettingsPromise: Promise<IdeSettings>,
|
||||
controlPlaneClient: ControlPlaneClient,
|
||||
writeLog: (message: string) => Promise<void>,
|
||||
overrideConfigJson: SerializedContinueConfig | undefined,
|
||||
) {
|
||||
let workspaceConfigs: ContinueRcJson[] = [];
|
||||
try {
|
||||
workspaceConfigs = await ide.getWorkspaceConfigs();
|
||||
} catch (e) {
|
||||
console.warn("Failed to load workspace configs");
|
||||
}
|
||||
|
||||
const ideInfo = await ide.getIdeInfo();
|
||||
const uniqueId = await ide.getUniqueId();
|
||||
const ideSettings = await ideSettingsPromise;
|
||||
const workOsAccessToken = await controlPlaneClient.getAccessToken();
|
||||
|
||||
const newConfig = await loadFullConfigNode(
|
||||
ide,
|
||||
workspaceConfigs,
|
||||
ideSettings,
|
||||
ideInfo.ideType,
|
||||
uniqueId,
|
||||
writeLog,
|
||||
workOsAccessToken,
|
||||
overrideConfigJson,
|
||||
);
|
||||
newConfig.allowAnonymousTelemetry =
|
||||
newConfig.allowAnonymousTelemetry && (await ide.isTelemetryEnabled());
|
||||
|
||||
// Setup telemetry only after (and if) we know it is enabled
|
||||
await Telemetry.setup(
|
||||
newConfig.allowAnonymousTelemetry ?? true,
|
||||
await ide.getUniqueId(),
|
||||
ideInfo.extensionVersion,
|
||||
);
|
||||
|
||||
if (newConfig.analytics) {
|
||||
await TeamAnalytics.setup(
|
||||
newConfig.analytics as any, // TODO: Need to get rid of index.d.ts once and for all
|
||||
uniqueId,
|
||||
ideInfo.extensionVersion,
|
||||
);
|
||||
}
|
||||
|
||||
[...newConfig.models, ...(newConfig.tabAutocompleteModels ?? [])].forEach(
|
||||
async (model) => {
|
||||
if (model.providerName === "continue-proxy") {
|
||||
(model as ContinueProxy).workOsAccessToken = workOsAccessToken;
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
if (newConfig.embeddingsProvider?.providerName === "continue-proxy") {
|
||||
(
|
||||
newConfig.embeddingsProvider as ContinueProxyEmbeddingsProvider
|
||||
).workOsAccessToken = workOsAccessToken;
|
||||
}
|
||||
|
||||
if (newConfig.reranker?.name === "continue-proxy") {
|
||||
(newConfig.reranker as ContinueProxyReranker).workOsAccessToken =
|
||||
workOsAccessToken;
|
||||
}
|
||||
|
||||
return newConfig;
|
||||
}
|
||||
|
|
@ -3,7 +3,7 @@ import path from "path";
|
|||
import * as YAML from "yaml";
|
||||
import type { IDE, SlashCommand } from "..";
|
||||
import { walkDir } from "../indexing/walkDir";
|
||||
import { stripImages } from "../llm/countTokens.js";
|
||||
import { stripImages } from "../llm/images";
|
||||
import { renderTemplatedString } from "../llm/llms/index.js";
|
||||
import { getBasename } from "../util/index.js";
|
||||
|
||||
|
|
|
|||
|
|
@ -523,6 +523,7 @@ declare global {
|
|||
| "gpt-4-32k"
|
||||
| "gpt-4-turbo"
|
||||
| "gpt-4o"
|
||||
| "gpt-4o-mini"
|
||||
| "gpt-4-turbo-preview"
|
||||
| "gpt-4-vision-preview"
|
||||
// Mistral
|
||||
|
|
@ -676,7 +677,7 @@ declare global {
|
|||
export interface TabAutocompleteOptions {
|
||||
disable: boolean;
|
||||
useCopyBuffer: boolean;
|
||||
useSuffix: boolean;
|
||||
useFileSuffix: boolean;
|
||||
maxPromptTokens: number;
|
||||
debounceDelay: number;
|
||||
maxSuffixPercentage: number;
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ import type {
|
|||
IContextProvider,
|
||||
LoadSubmenuItemsArgs,
|
||||
} from "../index.js";
|
||||
|
||||
export abstract class BaseContextProvider implements IContextProvider {
|
||||
options: { [key: string]: any };
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,76 @@
|
|||
import { CONTROL_PLANE_URL } from "../../control-plane/client.js";
|
||||
import {
|
||||
ContextItem,
|
||||
ContextProviderDescription,
|
||||
ContextProviderExtras,
|
||||
ContextSubmenuItem,
|
||||
LoadSubmenuItemsArgs,
|
||||
} from "../../index.js";
|
||||
import { BaseContextProvider } from "../index.js";
|
||||
|
||||
class ContinueProxyContextProvider extends BaseContextProvider {
|
||||
static description: ContextProviderDescription = {
|
||||
title: "continue-proxy",
|
||||
displayTitle: "Continue Proxy",
|
||||
description: "Retrieve a context item from a Continue for Teams add-on",
|
||||
type: "submenu",
|
||||
};
|
||||
|
||||
workOsAccessToken: string | undefined = undefined;
|
||||
|
||||
override get description(): ContextProviderDescription {
|
||||
return {
|
||||
title:
|
||||
this.options.title || ContinueProxyContextProvider.description.title,
|
||||
displayTitle:
|
||||
this.options.displayTitle ||
|
||||
ContinueProxyContextProvider.description.displayTitle,
|
||||
description:
|
||||
this.options.description ||
|
||||
ContinueProxyContextProvider.description.description,
|
||||
type: this.options.type || ContinueProxyContextProvider.description.type,
|
||||
};
|
||||
}
|
||||
|
||||
async loadSubmenuItems(
|
||||
args: LoadSubmenuItemsArgs,
|
||||
): Promise<ContextSubmenuItem[]> {
|
||||
const response = await args.fetch(
|
||||
new URL(`/proxy/context/${this.options.id}/list`, CONTROL_PLANE_URL),
|
||||
{
|
||||
method: "GET",
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
Authorization: `Bearer ${this.workOsAccessToken}`,
|
||||
},
|
||||
},
|
||||
);
|
||||
const data = await response.json();
|
||||
return data.items;
|
||||
}
|
||||
|
||||
async getContextItems(
|
||||
query: string,
|
||||
extras: ContextProviderExtras,
|
||||
): Promise<ContextItem[]> {
|
||||
const response = await extras.fetch(
|
||||
new URL(`/proxy/context/${this.options.id}/retrieve`, CONTROL_PLANE_URL),
|
||||
{
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
Authorization: `Bearer ${this.workOsAccessToken}`,
|
||||
},
|
||||
body: JSON.stringify({
|
||||
query: query || "",
|
||||
fullInput: extras.fullInput,
|
||||
}),
|
||||
},
|
||||
);
|
||||
|
||||
const items: any = await response.json();
|
||||
return items;
|
||||
}
|
||||
}
|
||||
|
||||
export default ContinueProxyContextProvider;
|
||||
|
|
@ -1,14 +1,16 @@
|
|||
import fetch from "node-fetch";
|
||||
import {
|
||||
Chunk,
|
||||
ContextItem,
|
||||
ContextProviderDescription,
|
||||
ContextProviderExtras,
|
||||
ContextSubmenuItem,
|
||||
EmbeddingsProvider,
|
||||
LoadSubmenuItemsArgs,
|
||||
Reranker,
|
||||
} from "../../index.js";
|
||||
import { DocsService } from "../../indexing/docs/DocsService.js";
|
||||
import configs from "../../indexing/docs/preIndexedDocs.js";
|
||||
import TransformersJsEmbeddingsProvider from "../../indexing/embeddings/TransformersJsEmbeddingsProvider.js";
|
||||
import preIndexedDocs from "../../indexing/docs/preIndexedDocs.js";
|
||||
import { Telemetry } from "../../util/posthog.js";
|
||||
import { BaseContextProvider } from "../index.js";
|
||||
|
||||
class DocsContextProvider extends BaseContextProvider {
|
||||
|
|
@ -20,6 +22,7 @@ class DocsContextProvider extends BaseContextProvider {
|
|||
description: "Type to search docs",
|
||||
type: "submenu",
|
||||
};
|
||||
|
||||
private docsService: DocsService;
|
||||
|
||||
constructor(options: any) {
|
||||
|
|
@ -27,34 +30,96 @@ class DocsContextProvider extends BaseContextProvider {
|
|||
this.docsService = DocsService.getInstance();
|
||||
}
|
||||
|
||||
private async _getIconDataUrl(url: string): Promise<string | undefined> {
|
||||
private async _rerankChunks(
|
||||
chunks: Chunk[],
|
||||
reranker: NonNullable<ContextProviderExtras["reranker"]>,
|
||||
fullInput: ContextProviderExtras["fullInput"],
|
||||
) {
|
||||
let chunksCopy = [...chunks];
|
||||
|
||||
try {
|
||||
const response = await fetch(url);
|
||||
if (!response.headers.get("content-type")?.startsWith("image/")) {
|
||||
console.log("Not an image: ", await response.text());
|
||||
return undefined;
|
||||
}
|
||||
const buffer = await response.buffer();
|
||||
const base64data = buffer.toString("base64");
|
||||
return `data:${response.headers.get("content-type")};base64,${base64data}`;
|
||||
const scores = await reranker.rerank(fullInput, chunksCopy);
|
||||
|
||||
chunksCopy.sort(
|
||||
(a, b) => scores[chunksCopy.indexOf(b)] - scores[chunksCopy.indexOf(a)],
|
||||
);
|
||||
|
||||
chunksCopy = chunksCopy.splice(
|
||||
0,
|
||||
this.options?.nFinal ?? DocsContextProvider.DEFAULT_N_FINAL,
|
||||
);
|
||||
} catch (e) {
|
||||
console.log("E: ", e);
|
||||
return undefined;
|
||||
console.warn(`Failed to rerank docs results: ${e}`);
|
||||
|
||||
chunksCopy = chunksCopy.splice(
|
||||
0,
|
||||
this.options?.nFinal ?? DocsContextProvider.DEFAULT_N_FINAL,
|
||||
);
|
||||
}
|
||||
|
||||
return chunksCopy;
|
||||
}
|
||||
|
||||
private _sortByPreIndexedDocs(
|
||||
submenuItems: ContextSubmenuItem[],
|
||||
): ContextSubmenuItem[] {
|
||||
// Sort submenuItems such that the objects with titles which don't occur in configs occur first, and alphabetized
|
||||
return submenuItems.sort((a, b) => {
|
||||
const aTitleInConfigs = a.metadata?.preIndexed ?? false;
|
||||
const bTitleInConfigs = b.metadata?.preIndexed ?? false;
|
||||
|
||||
// Primary criterion: Items not in configs come first
|
||||
if (!aTitleInConfigs && bTitleInConfigs) {
|
||||
return -1;
|
||||
} else if (aTitleInConfigs && !bTitleInConfigs) {
|
||||
return 1;
|
||||
} else {
|
||||
// Secondary criterion: Alphabetical order when both items are in the same category
|
||||
return a.title.toString().localeCompare(b.title.toString());
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
async getContextItems(
|
||||
query: string,
|
||||
extras: ContextProviderExtras,
|
||||
): Promise<ContextItem[]> {
|
||||
// Not supported in JetBrains IDEs right now
|
||||
if ((await extras.ide.getIdeInfo()).ideType === "jetbrains") {
|
||||
throw new Error(
|
||||
"The @docs context provider is not currently supported in JetBrains IDEs. We'll have an update soon!",
|
||||
const ideInfo = await extras.ide.getIdeInfo();
|
||||
const isJetBrains = ideInfo.ideType === "jetbrains";
|
||||
|
||||
const isJetBrainsAndPreIndexedDocsProvider =
|
||||
this.docsService.isJetBrainsAndPreIndexedDocsProvider(
|
||||
ideInfo,
|
||||
extras.embeddingsProvider.id,
|
||||
);
|
||||
|
||||
if (isJetBrainsAndPreIndexedDocsProvider) {
|
||||
extras.ide.errorPopup(
|
||||
`${DocsService.preIndexedDocsEmbeddingsProvider.id} is configured as ` +
|
||||
"the embeddings provider, but it cannot be used with JetBrains. " +
|
||||
"Please select a different embeddings provider to use the '@docs' " +
|
||||
"context provider.",
|
||||
);
|
||||
|
||||
return [];
|
||||
}
|
||||
|
||||
const preIndexedDoc = preIndexedDocs[query];
|
||||
|
||||
let embeddingsProvider: EmbeddingsProvider;
|
||||
|
||||
if (!!preIndexedDoc && !isJetBrains) {
|
||||
// Pre-indexed docs should be filtered out in `loadSubmenuItems`,
|
||||
// for JetBrains users, but we sanity check that here
|
||||
Telemetry.capture("docs_pre_indexed_doc_used", {
|
||||
doc: preIndexedDoc["title"],
|
||||
});
|
||||
|
||||
embeddingsProvider = DocsService.preIndexedDocsEmbeddingsProvider;
|
||||
} else {
|
||||
embeddingsProvider = extras.embeddingsProvider;
|
||||
}
|
||||
|
||||
const embeddingsProvider = new TransformersJsEmbeddingsProvider();
|
||||
const [vector] = await embeddingsProvider.embed([extras.fullInput]);
|
||||
|
||||
let chunks = await this.docsService.retrieve(
|
||||
|
|
@ -65,22 +130,11 @@ class DocsContextProvider extends BaseContextProvider {
|
|||
);
|
||||
|
||||
if (extras.reranker) {
|
||||
try {
|
||||
const scores = await extras.reranker.rerank(extras.fullInput, chunks);
|
||||
chunks.sort(
|
||||
(a, b) => scores[chunks.indexOf(b)] - scores[chunks.indexOf(a)],
|
||||
);
|
||||
chunks = chunks.splice(
|
||||
0,
|
||||
this.options?.nFinal ?? DocsContextProvider.DEFAULT_N_FINAL,
|
||||
);
|
||||
} catch (e) {
|
||||
console.warn(`Failed to rerank docs results: ${e}`);
|
||||
chunks = chunks.splice(
|
||||
0,
|
||||
this.options?.nFinal ?? DocsContextProvider.DEFAULT_N_FINAL,
|
||||
);
|
||||
}
|
||||
chunks = await this._rerankChunks(
|
||||
chunks,
|
||||
extras.reranker,
|
||||
extras.fullInput,
|
||||
);
|
||||
}
|
||||
|
||||
return [
|
||||
|
|
@ -95,7 +149,7 @@ class DocsContextProvider extends BaseContextProvider {
|
|||
.slice(1)
|
||||
.join("/")
|
||||
: chunk.otherMetadata?.title || chunk.filepath,
|
||||
description: chunk.filepath, // new URL(chunk.filepath, query).toString(),
|
||||
description: chunk.filepath,
|
||||
content: chunk.content,
|
||||
}))
|
||||
.reverse(),
|
||||
|
|
@ -103,7 +157,10 @@ class DocsContextProvider extends BaseContextProvider {
|
|||
name: "Instructions",
|
||||
description: "Instructions",
|
||||
content:
|
||||
"Use the above documentation to answer the following question. You should not reference anything outside of what is shown, unless it is a commonly known concept. Reference URLs whenever possible using markdown formatting. If there isn't enough information to answer the question, suggest where the user might look to learn more.",
|
||||
"Use the above documentation to answer the following question. You should not reference " +
|
||||
"anything outside of what is shown, unless it is a commonly known concept. Reference URLs " +
|
||||
"whenever possible using markdown formatting. If there isn't enough information to answer " +
|
||||
"the question, suggest where the user might look to learn more.",
|
||||
},
|
||||
];
|
||||
}
|
||||
|
|
@ -111,57 +168,50 @@ class DocsContextProvider extends BaseContextProvider {
|
|||
async loadSubmenuItems(
|
||||
args: LoadSubmenuItemsArgs,
|
||||
): Promise<ContextSubmenuItem[]> {
|
||||
const docs = await this.docsService.list();
|
||||
const submenuItems: ContextSubmenuItem[] = docs.map((doc) => ({
|
||||
title: doc.title,
|
||||
description: new URL(doc.baseUrl).hostname,
|
||||
id: doc.baseUrl,
|
||||
metadata: {
|
||||
preIndexed: !!configs.find((config) => config.title === doc.title),
|
||||
},
|
||||
}));
|
||||
const ideInfo = await args.ide.getIdeInfo();
|
||||
const isJetBrains = ideInfo.ideType === "jetbrains";
|
||||
const configSites = [
|
||||
...new Set([...(this.options?.sites || []), ...(args.config.docs || [])]),
|
||||
];
|
||||
const submenuItemsMap = new Map<string, ContextSubmenuItem>();
|
||||
|
||||
submenuItems.push(
|
||||
...configs
|
||||
// After it's actually downloaded, we don't want to show twice
|
||||
.filter(
|
||||
(config) => !submenuItems.some((item) => item.id === config.startUrl),
|
||||
)
|
||||
.map((config) => ({
|
||||
title: config.title,
|
||||
description: new URL(config.startUrl).hostname,
|
||||
id: config.startUrl,
|
||||
if (!isJetBrains) {
|
||||
// Currently, we generate and host embeddings for pre-indexed docs using transformers.js.
|
||||
// However, we don't ship transformers.js with the JetBrains extension.
|
||||
// So, we only include pre-indexed docs in the submenu for non-JetBrains IDEs.
|
||||
for (const { startUrl, title } of Object.values(preIndexedDocs)) {
|
||||
submenuItemsMap.set(startUrl, {
|
||||
title,
|
||||
id: startUrl,
|
||||
description: new URL(startUrl).hostname,
|
||||
metadata: {
|
||||
preIndexed: true,
|
||||
},
|
||||
// iconUrl: config.faviconUrl,
|
||||
})),
|
||||
);
|
||||
|
||||
// Sort submenuItems such that the objects with titles which don't occur in configs occur first, and alphabetized
|
||||
submenuItems.sort((a, b) => {
|
||||
const aTitleInConfigs = a.metadata?.preIndexed;
|
||||
const bTitleInConfigs = b.metadata?.preIndexed;
|
||||
|
||||
// Primary criterion: Items not in configs come first
|
||||
if (!aTitleInConfigs && bTitleInConfigs) {
|
||||
return -1;
|
||||
} else if (aTitleInConfigs && !bTitleInConfigs) {
|
||||
return 1;
|
||||
} else {
|
||||
// Secondary criterion: Alphabetical order when both items are in the same category
|
||||
return a.title.toString().localeCompare(b.title.toString());
|
||||
});
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
// const icons = await Promise.all(
|
||||
// submenuItems.map(async (item) =>
|
||||
// item.iconUrl ? this._getIconDataUrl(item.iconUrl) : undefined,
|
||||
// ),
|
||||
// );
|
||||
// icons.forEach((icon, i) => {
|
||||
// submenuItems[i].iconUrl = icon;
|
||||
// });
|
||||
for (const { title, baseUrl } of await this.docsService.list()) {
|
||||
submenuItemsMap.set(baseUrl, {
|
||||
title,
|
||||
id: baseUrl,
|
||||
description: new URL(baseUrl).hostname,
|
||||
});
|
||||
}
|
||||
|
||||
for (const { startUrl, title } of configSites) {
|
||||
submenuItemsMap.set(startUrl, {
|
||||
title,
|
||||
id: startUrl,
|
||||
description: new URL(startUrl).hostname,
|
||||
});
|
||||
}
|
||||
|
||||
const submenuItems = Array.from(submenuItemsMap.values());
|
||||
|
||||
if (!isJetBrains) {
|
||||
return this._sortByPreIndexedDocs(submenuItems);
|
||||
}
|
||||
|
||||
return submenuItems;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -2,6 +2,7 @@ import { ContextProviderName } from "../../index.js";
|
|||
import { BaseContextProvider } from "../index.js";
|
||||
import CodeContextProvider from "./CodeContextProvider.js";
|
||||
import CodebaseContextProvider from "./CodebaseContextProvider.js";
|
||||
import ContinueProxyContextProvider from "./ContinueProxyContextProvider.js";
|
||||
import CurrentFileContextProvider from "./CurrentFileContextProvider.js";
|
||||
import DatabaseContextProvider from "./DatabaseContextProvider.js";
|
||||
import DiffContextProvider from "./DiffContextProvider.js";
|
||||
|
|
@ -40,7 +41,6 @@ const Providers: (typeof BaseContextProvider)[] = [
|
|||
HttpContextProvider,
|
||||
SearchContextProvider,
|
||||
OSContextProvider,
|
||||
CodebaseContextProvider,
|
||||
ProblemsContextProvider,
|
||||
FolderContextProvider,
|
||||
DocsContextProvider,
|
||||
|
|
@ -51,16 +51,11 @@ const Providers: (typeof BaseContextProvider)[] = [
|
|||
CodeContextProvider,
|
||||
CurrentFileContextProvider,
|
||||
URLContextProvider,
|
||||
ContinueProxyContextProvider,
|
||||
];
|
||||
|
||||
export function contextProviderClassFromName(
|
||||
name: ContextProviderName,
|
||||
): typeof BaseContextProvider | undefined {
|
||||
const cls = Providers.find((cls) => cls.description.title === name);
|
||||
|
||||
if (!cls) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
return cls;
|
||||
return Providers.find((cls) => cls.description.title === name);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,46 @@
|
|||
import fetch from "node-fetch";
|
||||
import { CONTROL_PLANE_URL } from "../../control-plane/client.js";
|
||||
import { Chunk, Reranker } from "../../index.js";
|
||||
|
||||
export class ContinueProxyReranker implements Reranker {
|
||||
name = "continue-proxy";
|
||||
|
||||
private _workOsAccessToken: string | undefined = undefined;
|
||||
|
||||
get workOsAccessToken(): string | undefined {
|
||||
return this._workOsAccessToken;
|
||||
}
|
||||
|
||||
set workOsAccessToken(value: string | undefined) {
|
||||
if (this._workOsAccessToken !== value) {
|
||||
this._workOsAccessToken = value;
|
||||
this.params.apiKey = value!;
|
||||
}
|
||||
}
|
||||
|
||||
constructor(
|
||||
private readonly params: {
|
||||
apiKey: string;
|
||||
model?: string;
|
||||
},
|
||||
) {}
|
||||
|
||||
async rerank(query: string, chunks: Chunk[]): Promise<number[]> {
|
||||
const url = new URL("/model-proxy/v1/rerank", CONTROL_PLANE_URL);
|
||||
const resp = await fetch(url, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
Authorization: `Bearer ${this.params.apiKey}`,
|
||||
},
|
||||
body: JSON.stringify({
|
||||
query,
|
||||
documents: chunks.map((chunk) => chunk.content),
|
||||
model: this.params.model,
|
||||
}),
|
||||
});
|
||||
const data: any = await resp.json();
|
||||
const results = data.data.sort((a: any, b: any) => a.index - b.index);
|
||||
return results.map((result: any) => result.relevance_score);
|
||||
}
|
||||
}
|
||||
|
|
@ -1,7 +1,9 @@
|
|||
import { RerankerName } from "../../index.js";
|
||||
import { CohereReranker } from "./cohere.js";
|
||||
import { ContinueProxyReranker } from "./ContinueProxyReranker.js";
|
||||
import { FreeTrialReranker } from "./freeTrial.js";
|
||||
import { LLMReranker } from "./llm.js";
|
||||
import { HuggingFaceTEIReranker } from "./tei.js";
|
||||
import { VoyageReranker } from "./voyage.js";
|
||||
|
||||
export const AllRerankers: { [key in RerankerName]: any } = {
|
||||
|
|
@ -9,4 +11,6 @@ export const AllRerankers: { [key in RerankerName]: any } = {
|
|||
llm: LLMReranker,
|
||||
voyage: VoyageReranker,
|
||||
"free-trial": FreeTrialReranker,
|
||||
"huggingface-tei": HuggingFaceTEIReranker,
|
||||
"continue-proxy": ContinueProxyReranker,
|
||||
};
|
||||
|
|
|
|||
|
|
@ -0,0 +1,49 @@
|
|||
import fetch from "node-fetch";
|
||||
import { Chunk, Reranker } from "../../index.js";
|
||||
|
||||
export class HuggingFaceTEIReranker implements Reranker {
|
||||
name = "huggingface-tei";
|
||||
|
||||
static defaultOptions = {
|
||||
apiBase: "http://localhost:8080",
|
||||
truncate: true,
|
||||
truncation_direction: "Right"
|
||||
};
|
||||
|
||||
constructor(
|
||||
private readonly params: {
|
||||
apiBase?: string;
|
||||
truncate?: boolean;
|
||||
truncation_direction?: string;
|
||||
},
|
||||
) {}
|
||||
|
||||
async rerank(query: string, chunks: Chunk[]): Promise<number[]> {
|
||||
let apiBase = this.params.apiBase ?? HuggingFaceTEIReranker.defaultOptions.apiBase;
|
||||
if (!apiBase.endsWith("/")) {
|
||||
apiBase += "/";
|
||||
}
|
||||
|
||||
const resp = await fetch(new URL("rerank", apiBase), {
|
||||
method: "POST",
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
query: query,
|
||||
return_text: false,
|
||||
raw_scores: false,
|
||||
texts: chunks.map((chunk) => chunk.content),
|
||||
truncation_direction: this.params.truncation_direction ?? HuggingFaceTEIReranker.defaultOptions.truncation_direction,
|
||||
truncate: this.params.truncate ?? HuggingFaceTEIReranker.defaultOptions.truncate
|
||||
}),
|
||||
});
|
||||
|
||||
if (!resp.ok) {
|
||||
throw new Error(await resp.text());
|
||||
}
|
||||
|
||||
const data = (await resp.json()) as any;
|
||||
// Resort into original order and extract scores
|
||||
const results = data.sort((a: any, b: any) => a.index - b.index);
|
||||
return results.map((result: any) => result.score);
|
||||
}
|
||||
}
|
||||
|
|
@ -4,9 +4,12 @@ import {
|
|||
EmbeddingsProvider,
|
||||
IDE,
|
||||
Reranker,
|
||||
} from "../../..";
|
||||
import { LanceDbIndex } from "../../../indexing/LanceDbIndex";
|
||||
import { retrieveFts } from "../fullTextSearch";
|
||||
} from "../../../index.js";
|
||||
import { chunkDocument } from "../../../indexing/chunk/chunk.js";
|
||||
import { LanceDbIndex } from "../../../indexing/LanceDbIndex.js";
|
||||
import { MAX_CHUNK_SIZE } from "../../../llm/constants.js";
|
||||
import { retrieveFts } from "../fullTextSearch.js";
|
||||
import { recentlyEditedFilesCache } from "../recentlyEditedFilesCache.js";
|
||||
|
||||
export interface RetrievalPipelineOptions {
|
||||
ide: IDE;
|
||||
|
|
@ -32,6 +35,43 @@ export default class BaseRetrievalPipeline implements IRetrievalPipeline {
|
|||
);
|
||||
}
|
||||
|
||||
protected async retrieveAndChunkRecentlyEditedFiles(
|
||||
n: number,
|
||||
): Promise<Chunk[]> {
|
||||
const recentlyEditedFilesSlice = Array.from(
|
||||
recentlyEditedFilesCache.keys(),
|
||||
).slice(0, n);
|
||||
|
||||
// If the number of recently edited files is less than the retrieval limit,
|
||||
// include additional open files. This is useful in the case where a user
|
||||
// has many tabs open and reloads their IDE. They now have 0 recently edited files,
|
||||
// but many open tabs that represent what they were working on prior to reload.
|
||||
if (recentlyEditedFilesSlice.length < n) {
|
||||
const openFiles = await this.options.ide.getOpenFiles();
|
||||
recentlyEditedFilesSlice.push(
|
||||
...openFiles.slice(0, n - recentlyEditedFilesSlice.length),
|
||||
);
|
||||
}
|
||||
|
||||
const chunks: Chunk[] = [];
|
||||
|
||||
for (const filepath of recentlyEditedFilesSlice) {
|
||||
const contents = await this.options.ide.readFile(filepath);
|
||||
const fileChunks = chunkDocument({
|
||||
filepath,
|
||||
contents,
|
||||
maxChunkSize: MAX_CHUNK_SIZE,
|
||||
digest: filepath,
|
||||
});
|
||||
|
||||
for await (const chunk of fileChunks) {
|
||||
chunks.push(chunk);
|
||||
}
|
||||
}
|
||||
|
||||
return chunks;
|
||||
}
|
||||
|
||||
protected async retrieveFts(input: string, n: number): Promise<Chunk[]> {
|
||||
return retrieveFts(
|
||||
input,
|
||||
|
|
|
|||
|
|
@ -1,26 +1,38 @@
|
|||
import { Chunk } from "../../..";
|
||||
import { deduplicateChunks } from "../util";
|
||||
import BaseRetrievalPipeline from "./BaseRetrievalPipeline";
|
||||
import { Chunk } from "../../../index.js";
|
||||
import { deduplicateChunks } from "../util.js";
|
||||
import BaseRetrievalPipeline from "./BaseRetrievalPipeline.js";
|
||||
|
||||
export default class NoRerankerRetrievalPipeline extends BaseRetrievalPipeline {
|
||||
async run(): Promise<Chunk[]> {
|
||||
const { input } = this.options;
|
||||
const { input, nFinal } = this.options;
|
||||
|
||||
// We give 1/4 weight to recently edited files, 1/4 to full text search,
|
||||
// and the remaining 1/2 to embeddings
|
||||
const recentlyEditedNFinal = nFinal * 0.25;
|
||||
const ftsNFinal = nFinal * 0.25;
|
||||
const embeddingsNFinal = nFinal - recentlyEditedNFinal - ftsNFinal;
|
||||
|
||||
// Get all retrieval results
|
||||
const retrievalResults: Chunk[] = [];
|
||||
|
||||
// Full-text search
|
||||
const ftsResults = await this.retrieveFts(input, this.options.nFinal / 2);
|
||||
retrievalResults.push(...ftsResults);
|
||||
const ftsChunks = await this.retrieveFts(input, ftsNFinal);
|
||||
|
||||
// Embeddings
|
||||
const embeddingResults = await this.retrieveEmbeddings(
|
||||
const embeddingsChunks = await this.retrieveEmbeddings(
|
||||
input,
|
||||
this.options.nFinal / 2,
|
||||
embeddingsNFinal,
|
||||
);
|
||||
retrievalResults.push(...embeddingResults);
|
||||
|
||||
const finalResults: Chunk[] = deduplicateChunks(retrievalResults);
|
||||
return finalResults;
|
||||
const recentlyEditedFilesChunks =
|
||||
await this.retrieveAndChunkRecentlyEditedFiles(recentlyEditedNFinal);
|
||||
|
||||
retrievalResults.push(
|
||||
...recentlyEditedFilesChunks,
|
||||
...ftsChunks,
|
||||
...embeddingsChunks,
|
||||
);
|
||||
|
||||
const deduplicatedRetrievalResults: Chunk[] =
|
||||
deduplicateChunks(retrievalResults);
|
||||
|
||||
return deduplicatedRetrievalResults;
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,27 +1,30 @@
|
|||
import { Chunk } from "../../..";
|
||||
import { RETRIEVAL_PARAMS } from "../../../util/parameters";
|
||||
import { deduplicateChunks } from "../util";
|
||||
import BaseRetrievalPipeline from "./BaseRetrievalPipeline";
|
||||
import { Chunk } from "../../../index.js";
|
||||
import { RETRIEVAL_PARAMS } from "../../../util/parameters.js";
|
||||
import { recentlyEditedFilesCache } from "../recentlyEditedFilesCache.js";
|
||||
import { deduplicateChunks } from "../util.js";
|
||||
import BaseRetrievalPipeline from "./BaseRetrievalPipeline.js";
|
||||
|
||||
export default class RerankerRetrievalPipeline extends BaseRetrievalPipeline {
|
||||
private async _retrieveInitial(): Promise<Chunk[]> {
|
||||
const { input, nRetrieve } = this.options;
|
||||
|
||||
// Get all retrieval results
|
||||
const retrievalResults: Chunk[] = [];
|
||||
|
||||
// Full-text search
|
||||
const ftsResults = await this.retrieveFts(input, nRetrieve / 2);
|
||||
retrievalResults.push(...ftsResults);
|
||||
const ftsChunks = await this.retrieveFts(input, nRetrieve);
|
||||
const embeddingsChunks = await this.retrieveEmbeddings(input, nRetrieve);
|
||||
const recentlyEditedFilesChunks =
|
||||
await this.retrieveAndChunkRecentlyEditedFiles(nRetrieve);
|
||||
|
||||
// Embeddings
|
||||
const embeddingResults = await this.retrieveEmbeddings(input, nRetrieve);
|
||||
retrievalResults.push(
|
||||
...embeddingResults.slice(0, nRetrieve - ftsResults.length),
|
||||
...recentlyEditedFilesChunks,
|
||||
...ftsChunks,
|
||||
...embeddingsChunks,
|
||||
);
|
||||
|
||||
const results: Chunk[] = deduplicateChunks(retrievalResults);
|
||||
return results;
|
||||
const deduplicatedRetrievalResults: Chunk[] =
|
||||
deduplicateChunks(retrievalResults);
|
||||
|
||||
return deduplicatedRetrievalResults;
|
||||
}
|
||||
|
||||
private async _rerank(input: string, chunks: Chunk[]): Promise<Chunk[]> {
|
||||
|
|
@ -74,12 +77,8 @@ export default class RerankerRetrievalPipeline extends BaseRetrievalPipeline {
|
|||
}
|
||||
|
||||
async run(): Promise<Chunk[]> {
|
||||
// Retrieve initial results
|
||||
let results = await this._retrieveInitial();
|
||||
|
||||
// Rerank
|
||||
const { input } = this.options;
|
||||
results = await this._rerank(input, results);
|
||||
const intialResults = await this._retrieveInitial();
|
||||
const rankedResults = await this._rerank(this.options.input, intialResults);
|
||||
|
||||
// // // Expand top reranked results
|
||||
// const expanded = await this._expandRankedResults(results);
|
||||
|
|
@ -93,7 +92,7 @@ export default class RerankerRetrievalPipeline extends BaseRetrievalPipeline {
|
|||
|
||||
// TODO: stitch together results
|
||||
|
||||
return results;
|
||||
return rankedResults;
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,15 @@
|
|||
import QuickLRU from "quick-lru";
|
||||
import { ToWebviewOrCoreFromIdeProtocol } from "../../protocol/ide.js";
|
||||
|
||||
// The cache key and value are both a filepath string
|
||||
export type RecentlyEditedFilesCacheKeyAndValue =
|
||||
ToWebviewOrCoreFromIdeProtocol["didChangeActiveTextEditor"][0]["filepath"];
|
||||
|
||||
const MAX_NUM_RECENTLY_EDITED_FILES = 100;
|
||||
|
||||
export const recentlyEditedFilesCache = new QuickLRU<
|
||||
RecentlyEditedFilesCacheKeyAndValue,
|
||||
RecentlyEditedFilesCacheKeyAndValue
|
||||
>({
|
||||
maxSize: MAX_NUM_RECENTLY_EDITED_FILES,
|
||||
});
|
||||
|
|
@ -3,9 +3,8 @@ import {
|
|||
ContextItem,
|
||||
ContextProviderExtras,
|
||||
} from "../../index.js";
|
||||
|
||||
import TransformersJsEmbeddingsProvider from "../../indexing/embeddings/TransformersJsEmbeddingsProvider.js";
|
||||
import { getRelativePath } from "../../util/index.js";
|
||||
import { RETRIEVAL_PARAMS } from "../../util/parameters.js";
|
||||
import { RetrievalPipelineOptions } from "./pipelines/BaseRetrievalPipeline.js";
|
||||
import NoRerankerRetrievalPipeline from "./pipelines/NoRerankerRetrievalPipeline.js";
|
||||
import RerankerRetrievalPipeline from "./pipelines/RerankerRetrievalPipeline.js";
|
||||
|
|
@ -20,22 +19,20 @@ export async function retrieveContextItemsFromEmbeddings(
|
|||
}
|
||||
|
||||
// transformers.js not supported in JetBrains IDEs right now
|
||||
if (
|
||||
extras.embeddingsProvider.id === "all-MiniLM-L6-v2" &&
|
||||
(await extras.ide.getIdeInfo()).ideType === "jetbrains"
|
||||
) {
|
||||
|
||||
const isJetBrainsAndTransformersJs =
|
||||
extras.embeddingsProvider.id === TransformersJsEmbeddingsProvider.model &&
|
||||
(await extras.ide.getIdeInfo()).ideType === "jetbrains";
|
||||
|
||||
if (isJetBrainsAndTransformersJs) {
|
||||
throw new Error(
|
||||
"The transformers.js context provider is not currently supported in JetBrains. For now, you can use Ollama to set up local embeddings, or use our 'free-trial' embeddings provider. See here to learn more: https://docs.continue.dev/features/codebase-embeddings#embeddings-providers",
|
||||
"The 'transformers.js' context provider is not currently supported in JetBrains. " +
|
||||
"For now, you can use Ollama to set up local embeddings, or use our 'free-trial' " +
|
||||
"embeddings provider. See here to learn more: " +
|
||||
"https://docs.continue.dev/walkthroughs/codebase-embeddings#embeddings-providers",
|
||||
);
|
||||
}
|
||||
|
||||
const nFinal = options?.nFinal || RETRIEVAL_PARAMS.nFinal;
|
||||
const useReranking = extras.reranker !== undefined;
|
||||
const nRetrieve =
|
||||
useReranking === false
|
||||
? nFinal
|
||||
: options?.nRetrieve || RETRIEVAL_PARAMS.nRetrieve;
|
||||
|
||||
// Get tags to retrieve for
|
||||
const workspaceDirs = await extras.ide.getWorkspaceDirs();
|
||||
|
||||
|
|
@ -43,6 +40,14 @@ export async function retrieveContextItemsFromEmbeddings(
|
|||
throw new Error("No workspace directories found");
|
||||
}
|
||||
|
||||
// Fill half of the context length, up to a max of 100 snippets
|
||||
const contextLength = extras.llm.contextLength;
|
||||
const tokensPerSnippet = 512;
|
||||
const nFinal =
|
||||
options?.nFinal ?? Math.min(50, contextLength / tokensPerSnippet / 2);
|
||||
const useReranking = !!extras.reranker;
|
||||
const nRetrieve = useReranking ? options?.nRetrieve || 2 * nFinal : nFinal;
|
||||
|
||||
const branches = (await Promise.race([
|
||||
Promise.all(workspaceDirs.map((dir) => extras.ide.getBranch(dir))),
|
||||
new Promise((resolve) => {
|
||||
|
|
@ -51,6 +56,7 @@ export async function retrieveContextItemsFromEmbeddings(
|
|||
}, 500);
|
||||
}),
|
||||
])) as string[];
|
||||
|
||||
const tags: BranchAndDir[] = workspaceDirs.map((directory, i) => ({
|
||||
directory,
|
||||
branch: branches[i],
|
||||
|
|
@ -59,6 +65,7 @@ export async function retrieveContextItemsFromEmbeddings(
|
|||
const pipelineType = useReranking
|
||||
? RerankerRetrievalPipeline
|
||||
: NoRerankerRetrievalPipeline;
|
||||
|
||||
const pipelineOptions: RetrievalPipelineOptions = {
|
||||
nFinal,
|
||||
nRetrieve,
|
||||
|
|
@ -69,6 +76,7 @@ export async function retrieveContextItemsFromEmbeddings(
|
|||
ide: extras.ide,
|
||||
input: extras.fullInput,
|
||||
};
|
||||
|
||||
const pipeline = new pipelineType(pipelineOptions);
|
||||
const results = await pipeline.run();
|
||||
|
||||
|
|
@ -80,7 +88,9 @@ export async function retrieveContextItemsFromEmbeddings(
|
|||
|
||||
return [
|
||||
...results.map((r) => {
|
||||
const name = `${getRelativePath(r.filepath, workspaceDirs)} (${r.startLine}-${r.endLine})`;
|
||||
const name = `${getRelativePath(r.filepath, workspaceDirs)} (${
|
||||
r.startLine
|
||||
}-${r.endLine})`;
|
||||
const description = `${r.filepath} (${r.startLine}-${r.endLine})`;
|
||||
return {
|
||||
name,
|
||||
|
|
|
|||
|
|
@ -0,0 +1,55 @@
|
|||
import { Analytics } from "@continuedev/config-types";
|
||||
import os from "node:os";
|
||||
import ContinueProxyAnalyticsProvider from "./analytics/ContinueProxyAnalyticsProvider";
|
||||
import { IAnalyticsProvider } from "./analytics/IAnalyticsProvider";
|
||||
import LogStashAnalyticsProvider from "./analytics/LogStashAnalyticsProvider";
|
||||
import PostHogAnalyticsProvider from "./analytics/PostHogAnalyticsProvider";
|
||||
|
||||
function createAnalyticsProvider(
|
||||
config: Analytics,
|
||||
): IAnalyticsProvider | undefined {
|
||||
// @ts-ignore
|
||||
switch (config.provider) {
|
||||
case "posthog":
|
||||
return new PostHogAnalyticsProvider();
|
||||
case "logstash":
|
||||
return new LogStashAnalyticsProvider();
|
||||
case "continue-proxy":
|
||||
return new ContinueProxyAnalyticsProvider();
|
||||
default:
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
export class TeamAnalytics {
|
||||
static provider: IAnalyticsProvider | undefined = undefined;
|
||||
static uniqueId = "NOT_UNIQUE";
|
||||
static os: string | undefined = undefined;
|
||||
static extensionVersion: string | undefined = undefined;
|
||||
|
||||
static async capture(event: string, properties: { [key: string]: any }) {
|
||||
TeamAnalytics.provider?.capture(event, {
|
||||
...properties,
|
||||
os: TeamAnalytics.os,
|
||||
extensionVersion: TeamAnalytics.extensionVersion,
|
||||
});
|
||||
}
|
||||
|
||||
static async setup(
|
||||
config: Analytics,
|
||||
uniqueId: string,
|
||||
extensionVersion: string,
|
||||
) {
|
||||
TeamAnalytics.uniqueId = uniqueId;
|
||||
TeamAnalytics.os = os.platform();
|
||||
TeamAnalytics.extensionVersion = extensionVersion;
|
||||
|
||||
if (!config) {
|
||||
await TeamAnalytics.provider?.shutdown();
|
||||
TeamAnalytics.provider = undefined;
|
||||
} else {
|
||||
TeamAnalytics.provider = createAnalyticsProvider(config);
|
||||
await TeamAnalytics.provider?.setup(config, uniqueId);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,32 @@
|
|||
import { Analytics } from "@continuedev/config-types";
|
||||
import fetch from "node-fetch";
|
||||
import { CONTROL_PLANE_URL } from "../client";
|
||||
import { IAnalyticsProvider } from "./IAnalyticsProvider";
|
||||
|
||||
export default class ContinueProxyAnalyticsProvider
|
||||
implements IAnalyticsProvider
|
||||
{
|
||||
uniqueId?: string;
|
||||
addOnId?: string;
|
||||
|
||||
async capture(
|
||||
event: string,
|
||||
properties: { [key: string]: any },
|
||||
): Promise<void> {
|
||||
fetch(new URL(`/proxy/analytics/${this.addOnId}`, CONTROL_PLANE_URL), {
|
||||
method: "POST",
|
||||
body: JSON.stringify({
|
||||
event,
|
||||
properties,
|
||||
uniqueId: this.uniqueId,
|
||||
}),
|
||||
});
|
||||
}
|
||||
|
||||
async setup(config: Analytics, uniqueId: string): Promise<void> {
|
||||
this.uniqueId = uniqueId;
|
||||
this.addOnId = config.url?.split("/").slice(-1)[0];
|
||||
}
|
||||
|
||||
async shutdown(): Promise<void> {}
|
||||
}
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
import { Analytics } from "@continuedev/config-types";
|
||||
|
||||
export interface AnalyticsMetadata {
|
||||
extensionVersion: string;
|
||||
}
|
||||
|
||||
export interface IAnalyticsProvider {
|
||||
capture(event: string, properties: { [key: string]: any }): Promise<void>;
|
||||
setup(config: Analytics, uniqueId: string): Promise<void>;
|
||||
shutdown(): Promise<void>;
|
||||
}
|
||||
|
|
@ -0,0 +1,42 @@
|
|||
import { Analytics } from "@continuedev/config-types";
|
||||
import net from "node:net";
|
||||
import { IAnalyticsProvider } from "./IAnalyticsProvider";
|
||||
|
||||
export default class LogStashAnalyticsProvider implements IAnalyticsProvider {
|
||||
private host?: string;
|
||||
private port?: number;
|
||||
private uniqueId?: string;
|
||||
|
||||
async capture(
|
||||
event: string,
|
||||
properties: { [key: string]: any },
|
||||
): Promise<void> {
|
||||
if (this.host === undefined || this.port === undefined) {
|
||||
console.warn("LogStashAnalyticsProvider not set up yet.");
|
||||
}
|
||||
|
||||
const payload = {
|
||||
event,
|
||||
properties,
|
||||
uniqueId: this.uniqueId,
|
||||
};
|
||||
const client = new net.Socket();
|
||||
|
||||
client.connect(this.port!, this.host!, () => {
|
||||
client.write(JSON.stringify(payload));
|
||||
client.end();
|
||||
});
|
||||
}
|
||||
|
||||
async setup(config: Analytics, uniqueId: string): Promise<void> {
|
||||
if (!config.url) {
|
||||
throw new Error("Missing url in analytics config");
|
||||
}
|
||||
const url = new URL(config.url);
|
||||
this.host = url.hostname;
|
||||
this.port = parseInt(url.port);
|
||||
this.uniqueId = uniqueId;
|
||||
}
|
||||
|
||||
async shutdown(): Promise<void> {}
|
||||
}
|
||||
|
|
@ -0,0 +1,37 @@
|
|||
import { Analytics } from "@continuedev/config-types";
|
||||
import { IAnalyticsProvider } from "./IAnalyticsProvider";
|
||||
|
||||
export default class PostHogAnalyticsProvider implements IAnalyticsProvider {
|
||||
client?: any;
|
||||
uniqueId?: string;
|
||||
|
||||
async capture(
|
||||
event: string,
|
||||
properties: { [key: string]: any },
|
||||
): Promise<void> {
|
||||
this.client?.capture({
|
||||
distinctId: this.uniqueId,
|
||||
event,
|
||||
properties,
|
||||
});
|
||||
}
|
||||
|
||||
async setup(config: Analytics, uniqueId: string): Promise<void> {
|
||||
if (!config || !config.clientKey || !config.url) {
|
||||
this.client = undefined;
|
||||
} else {
|
||||
try {
|
||||
this.uniqueId = uniqueId;
|
||||
|
||||
const { PostHog } = await import("posthog-node");
|
||||
this.client = new PostHog(config.clientKey, {
|
||||
host: config.url,
|
||||
});
|
||||
} catch (e) {
|
||||
console.error(`Failed to setup telemetry: ${e}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async shutdown(): Promise<void> {}
|
||||
}
|
||||
|
|
@ -0,0 +1,20 @@
|
|||
import { v4 as uuidv4 } from "uuid";
|
||||
|
||||
const CLIENT_ID = "client_01J0FW6XN8N2XJAECF7NE0Y65J";
|
||||
// const CLIENT_ID = "client_01J0FW6XCPMJMQ3CG51RB4HBZQ"; // Staging
|
||||
|
||||
export async function getAuthUrlForTokenPage(): Promise<string> {
|
||||
const url = new URL("https://api.workos.com/user_management/authorize");
|
||||
const params = {
|
||||
response_type: "code",
|
||||
client_id: CLIENT_ID,
|
||||
redirect_uri: "https://app.continue.dev/tokens/callback",
|
||||
// redirect_uri: "http://localhost:3000/tokens/callback",
|
||||
state: uuidv4(),
|
||||
provider: "authkit",
|
||||
};
|
||||
Object.keys(params).forEach((key) =>
|
||||
url.searchParams.append(key, params[key as keyof typeof params]),
|
||||
);
|
||||
return url.toString();
|
||||
}
|
||||
|
|
@ -0,0 +1,97 @@
|
|||
import { ConfigJson } from "@continuedev/config-types";
|
||||
import fetch, { RequestInit, Response } from "node-fetch";
|
||||
import { ModelDescription } from "..";
|
||||
|
||||
export interface ControlPlaneSessionInfo {
|
||||
accessToken: string;
|
||||
account: {
|
||||
label: string;
|
||||
id: string;
|
||||
};
|
||||
}
|
||||
|
||||
export interface ControlPlaneWorkspace {
|
||||
id: string;
|
||||
name: string;
|
||||
settings: ConfigJson;
|
||||
}
|
||||
|
||||
export interface ControlPlaneModelDescription extends ModelDescription {}
|
||||
|
||||
export const CONTROL_PLANE_URL =
|
||||
process.env.CONTROL_PLANE_ENV === "local"
|
||||
? "http://localhost:3001"
|
||||
: "https://control-plane-api-service-i3dqylpbqa-uc.a.run.app";
|
||||
|
||||
export class ControlPlaneClient {
|
||||
private static URL = CONTROL_PLANE_URL;
|
||||
private static ACCESS_TOKEN_VALID_FOR_MS = 1000 * 60 * 5; // 5 minutes
|
||||
|
||||
private lastAccessTokenRefresh = 0;
|
||||
|
||||
constructor(
|
||||
private readonly sessionInfoPromise: Promise<
|
||||
ControlPlaneSessionInfo | undefined
|
||||
>,
|
||||
) {}
|
||||
|
||||
get userId(): Promise<string | undefined> {
|
||||
return this.sessionInfoPromise.then(
|
||||
(sessionInfo) => sessionInfo?.account.id,
|
||||
);
|
||||
}
|
||||
|
||||
async getAccessToken(): Promise<string | undefined> {
|
||||
return (await this.sessionInfoPromise)?.accessToken;
|
||||
}
|
||||
|
||||
private async request(path: string, init: RequestInit): Promise<Response> {
|
||||
const accessToken = await this.getAccessToken();
|
||||
if (!accessToken) {
|
||||
throw new Error("No access token");
|
||||
}
|
||||
const resp = await fetch(new URL(path, ControlPlaneClient.URL).toString(), {
|
||||
...init,
|
||||
headers: {
|
||||
...init.headers,
|
||||
Authorization: `Bearer ${accessToken}`,
|
||||
},
|
||||
});
|
||||
|
||||
if (!resp.ok) {
|
||||
throw new Error(
|
||||
`Control plane request failed: ${resp.status} ${await resp.text()}`,
|
||||
);
|
||||
}
|
||||
|
||||
return resp;
|
||||
}
|
||||
|
||||
public async listWorkspaces(): Promise<ControlPlaneWorkspace[]> {
|
||||
const userId = await this.userId;
|
||||
if (!userId) {
|
||||
return [];
|
||||
}
|
||||
|
||||
try {
|
||||
const resp = await this.request(`/workspaces`, {
|
||||
method: "GET",
|
||||
});
|
||||
return (await resp.json()) as any;
|
||||
} catch (e) {
|
||||
return [];
|
||||
}
|
||||
}
|
||||
|
||||
async getSettingsForWorkspace(workspaceId: string): Promise<ConfigJson> {
|
||||
const userId = await this.userId;
|
||||
if (!userId) {
|
||||
throw new Error("No user id");
|
||||
}
|
||||
|
||||
const resp = await this.request(`/workspaces/${workspaceId}`, {
|
||||
method: "GET",
|
||||
});
|
||||
return ((await resp.json()) as any).settings;
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,128 @@
|
|||
import { z } from "zod";
|
||||
|
||||
const modelDescriptionSchema = z.object({
|
||||
title: z.string(),
|
||||
provider: z.enum([
|
||||
"openai",
|
||||
"anthropic",
|
||||
"cohere",
|
||||
"ollama",
|
||||
"huggingface-tgi",
|
||||
"huggingface-inference-api",
|
||||
"replicate",
|
||||
"gemini",
|
||||
"mistral",
|
||||
"bedrock",
|
||||
"cloudflare",
|
||||
"azure",
|
||||
]),
|
||||
model: z.string(),
|
||||
apiKey: z.string().optional(),
|
||||
apiBase: z.string().optional(),
|
||||
contextLength: z.number().optional(),
|
||||
template: z
|
||||
.enum([
|
||||
"llama2",
|
||||
"alpaca",
|
||||
"zephyr",
|
||||
"phi2",
|
||||
"phind",
|
||||
"anthropic",
|
||||
"chatml",
|
||||
"none",
|
||||
"openchat",
|
||||
"deepseek",
|
||||
"xwin-coder",
|
||||
"neural-chat",
|
||||
"codellama-70b",
|
||||
"llava",
|
||||
"gemma",
|
||||
"llama3",
|
||||
])
|
||||
.optional(),
|
||||
completionOptions: z
|
||||
.object({
|
||||
temperature: z.number().optional(),
|
||||
topP: z.number().optional(),
|
||||
topK: z.number().optional(),
|
||||
minP: z.number().optional(),
|
||||
presencePenalty: z.number().optional(),
|
||||
frequencyPenalty: z.number().optional(),
|
||||
mirostat: z.number().optional(),
|
||||
stop: z.array(z.string()).optional(),
|
||||
maxTokens: z.number().optional(),
|
||||
numThreads: z.number().optional(),
|
||||
keepAlive: z.number().optional(),
|
||||
raw: z.boolean().optional(),
|
||||
stream: z.boolean().optional(),
|
||||
})
|
||||
.optional(),
|
||||
systemMessage: z.string().optional(),
|
||||
requestOptions: z
|
||||
.object({
|
||||
timeout: z.number().optional(),
|
||||
verifySsl: z.boolean().optional(),
|
||||
caBundlePath: z.union([z.string(), z.array(z.string())]).optional(),
|
||||
proxy: z.string().optional(),
|
||||
headers: z.record(z.string()).optional(),
|
||||
extraBodyProperties: z.record(z.any()).optional(),
|
||||
noProxy: z.array(z.string()).optional(),
|
||||
})
|
||||
.optional(),
|
||||
promptTemplates: z.record(z.string()).optional(),
|
||||
});
|
||||
|
||||
const embeddingsProviderSchema = z.object({
|
||||
provider: z.enum([
|
||||
"transformers.js",
|
||||
"ollama",
|
||||
"openai",
|
||||
"cohere",
|
||||
"free-trial",
|
||||
"gemini",
|
||||
]),
|
||||
apiBase: z.string().optional(),
|
||||
apiKey: z.string().optional(),
|
||||
model: z.string().optional(),
|
||||
engine: z.string().optional(),
|
||||
apiType: z.string().optional(),
|
||||
apiVersion: z.string().optional(),
|
||||
requestOptions: z
|
||||
.object({
|
||||
timeout: z.number().optional(),
|
||||
verifySsl: z.boolean().optional(),
|
||||
caBundlePath: z.union([z.string(), z.array(z.string())]).optional(),
|
||||
proxy: z.string().optional(),
|
||||
headers: z.record(z.string()).optional(),
|
||||
extraBodyProperties: z.record(z.any()).optional(),
|
||||
noProxy: z.array(z.string()).optional(),
|
||||
})
|
||||
.optional(),
|
||||
});
|
||||
|
||||
const rerankerSchema = z.object({
|
||||
name: z.enum(["cohere", "voyage", "llm"]),
|
||||
params: z.record(z.any()).optional(),
|
||||
});
|
||||
|
||||
const analyticsSchema = z.object({
|
||||
url: z.string().optional(),
|
||||
clientKey: z.string().optional(),
|
||||
});
|
||||
|
||||
export type ControlPlaneAnalytics = z.infer<typeof analyticsSchema>;
|
||||
|
||||
const devDataSchema = z.object({
|
||||
url: z.string().optional(),
|
||||
});
|
||||
|
||||
export const controlPlaneSettingsSchema = z.object({
|
||||
models: z.array(modelDescriptionSchema),
|
||||
tabAutocompleteModel: modelDescriptionSchema,
|
||||
embeddingsModel: embeddingsProviderSchema,
|
||||
reranker: rerankerSchema,
|
||||
analytics: analyticsSchema,
|
||||
devData: devDataSchema,
|
||||
});
|
||||
|
||||
export type ControlPlaneSettings = z.infer<typeof controlPlaneSettingsSchema>;
|
||||
254
core/core.ts
254
core/core.ts
|
|
@ -1,13 +1,13 @@
|
|||
import { v4 as uuidv4 } from "uuid";
|
||||
import type {
|
||||
ContextItemId,
|
||||
EmbeddingsProvider,
|
||||
IDE,
|
||||
IndexingProgressUpdate,
|
||||
SiteIndexingConfig,
|
||||
} from ".";
|
||||
import { CompletionProvider } from "./autocomplete/completionProvider.js";
|
||||
import { ConfigHandler } from "./config/ConfigHandler.js";
|
||||
import { IConfigHandler } from "./config/IConfigHandler";
|
||||
import {
|
||||
setupApiKeysMode,
|
||||
setupFreeTrialMode,
|
||||
|
|
@ -16,10 +16,12 @@ import {
|
|||
} from "./config/onboarding.js";
|
||||
import { createNewPromptFile } from "./config/promptFile.js";
|
||||
import { addModel, addOpenAIKey, deleteModel } from "./config/util.js";
|
||||
import { recentlyEditedFilesCache } from "./context/retrieval/recentlyEditedFilesCache.js";
|
||||
import { ContinueServerClient } from "./continueServer/stubs/client.js";
|
||||
import { getAuthUrlForTokenPage } from "./control-plane/auth/index.js";
|
||||
import { ControlPlaneClient } from "./control-plane/client";
|
||||
import { CodebaseIndexer, PauseToken } from "./indexing/CodebaseIndexer.js";
|
||||
import { DocsService } from "./indexing/docs/DocsService";
|
||||
import TransformersJsEmbeddingsProvider from "./indexing/embeddings/TransformersJsEmbeddingsProvider.js";
|
||||
import { DocsService } from "./indexing/docs/DocsService.js";
|
||||
import Ollama from "./llm/llms/Ollama.js";
|
||||
import type { FromCoreProtocol, ToCoreProtocol } from "./protocol";
|
||||
import { GlobalContext } from "./util/GlobalContext.js";
|
||||
|
|
@ -34,11 +36,12 @@ import { streamDiffLines } from "./util/verticalEdit.js";
|
|||
|
||||
export class Core {
|
||||
// implements IMessenger<ToCoreProtocol, FromCoreProtocol>
|
||||
configHandler: IConfigHandler;
|
||||
configHandler: ConfigHandler;
|
||||
codebaseIndexerPromise: Promise<CodebaseIndexer>;
|
||||
completionProvider: CompletionProvider;
|
||||
continueServerClientPromise: Promise<ContinueServerClient>;
|
||||
indexingState: IndexingProgressUpdate;
|
||||
controlPlaneClient: ControlPlaneClient;
|
||||
private globalContext = new GlobalContext();
|
||||
private docsService = DocsService.getInstance();
|
||||
private readonly indexingPauseToken = new PauseToken(
|
||||
|
|
@ -64,6 +67,14 @@ export class Core {
|
|||
return this.messenger.invoke(messageType, data);
|
||||
}
|
||||
|
||||
send<T extends keyof FromCoreProtocol>(
|
||||
messageType: T,
|
||||
data: FromCoreProtocol[T][0],
|
||||
messageId?: string,
|
||||
): string {
|
||||
return this.messenger.send(messageType, data);
|
||||
}
|
||||
|
||||
// TODO: It shouldn't actually need an IDE type, because this can happen
|
||||
// through the messenger (it does in the case of any non-VS Code IDEs already)
|
||||
constructor(
|
||||
|
|
@ -72,16 +83,39 @@ export class Core {
|
|||
private readonly onWrite: (text: string) => Promise<void> = async () => {},
|
||||
) {
|
||||
this.indexingState = { status: "loading", desc: "loading", progress: 0 };
|
||||
|
||||
const ideSettingsPromise = messenger.request("getIdeSettings", undefined);
|
||||
const sessionInfoPromise = messenger.request("getControlPlaneSessionInfo", {
|
||||
silent: true,
|
||||
});
|
||||
|
||||
this.controlPlaneClient = new ControlPlaneClient(sessionInfoPromise);
|
||||
|
||||
this.configHandler = new ConfigHandler(
|
||||
this.ide,
|
||||
ideSettingsPromise,
|
||||
this.onWrite,
|
||||
this.controlPlaneClient,
|
||||
);
|
||||
|
||||
this.configHandler.onConfigUpdate(
|
||||
(() => this.messenger.send("configUpdate", undefined)).bind(this),
|
||||
);
|
||||
|
||||
this.configHandler.onConfigUpdate(async ({ embeddingsProvider }) => {
|
||||
if (
|
||||
await this.shouldReindexDocsOnNewEmbeddingsProvider(
|
||||
embeddingsProvider.id,
|
||||
)
|
||||
) {
|
||||
await this.reindexDocsOnNewEmbeddingsProvider(embeddingsProvider);
|
||||
}
|
||||
});
|
||||
|
||||
this.configHandler.onDidChangeAvailableProfiles((profiles) =>
|
||||
this.messenger.send("didChangeAvailableProfiles", { profiles }),
|
||||
);
|
||||
|
||||
// Codebase Indexer and ContinueServerClient depend on IdeSettings
|
||||
let codebaseIndexerResolve: (_: any) => void | undefined;
|
||||
this.codebaseIndexerPromise = new Promise(
|
||||
|
|
@ -108,9 +142,21 @@ export class Core {
|
|||
continueServerClient,
|
||||
),
|
||||
);
|
||||
this.ide
|
||||
.getWorkspaceDirs()
|
||||
.then((dirs) => this.refreshCodebaseIndex(dirs));
|
||||
|
||||
// Index on initialization
|
||||
this.ide.getWorkspaceDirs().then(async (dirs) => {
|
||||
// Respect pauseCodebaseIndexOnStart user settings
|
||||
if (ideSettings.pauseCodebaseIndexOnStart) {
|
||||
await this.messenger.request("indexProgress", {
|
||||
progress: 100,
|
||||
desc: "Initial Indexing Skipped",
|
||||
status: "paused",
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
this.refreshCodebaseIndex(dirs);
|
||||
});
|
||||
});
|
||||
|
||||
const getLlm = async () => {
|
||||
|
|
@ -206,22 +252,14 @@ export class Core {
|
|||
on("config/ideSettingsUpdate", (msg) => {
|
||||
this.configHandler.updateIdeSettings(msg.data);
|
||||
});
|
||||
on("config/listProfiles", (msg) => {
|
||||
return this.configHandler.listProfiles();
|
||||
});
|
||||
|
||||
// Context providers
|
||||
on("context/addDocs", async (msg) => {
|
||||
const siteIndexingConfig: SiteIndexingConfig = {
|
||||
startUrl: msg.data.startUrl,
|
||||
rootUrl: msg.data.rootUrl,
|
||||
title: msg.data.title,
|
||||
maxDepth: msg.data.maxDepth,
|
||||
faviconUrl: new URL("/favicon.ico", msg.data.rootUrl).toString(),
|
||||
};
|
||||
await this.getEmbeddingsProviderAndIndexDoc(msg.data);
|
||||
|
||||
for await (const _ of this.docsService.indexAndAdd(
|
||||
siteIndexingConfig,
|
||||
new TransformersJsEmbeddingsProvider(),
|
||||
)) {
|
||||
}
|
||||
this.ide.infoPopup(`Successfully indexed ${msg.data.title}`);
|
||||
this.messenger.send("refreshSubmenuItems", undefined);
|
||||
});
|
||||
|
|
@ -230,11 +268,36 @@ export class Core {
|
|||
await this.docsService.delete(baseUrl);
|
||||
this.messenger.send("refreshSubmenuItems", undefined);
|
||||
});
|
||||
on("context/indexDocs", async (msg) => {
|
||||
const config = await this.config();
|
||||
const provider: any = config.contextProviders?.find(
|
||||
(provider) => provider.description.title === "docs",
|
||||
);
|
||||
|
||||
if (!provider) {
|
||||
this.ide.infoPopup("No docs in configuration");
|
||||
return;
|
||||
}
|
||||
|
||||
const siteIndexingOptions: SiteIndexingConfig[] = ((mProvider) => [
|
||||
...new Set([
|
||||
...(mProvider?.options?.sites || []),
|
||||
...(config.docs || []),
|
||||
]),
|
||||
])({ ...provider });
|
||||
|
||||
for (const site of siteIndexingOptions) {
|
||||
await this.getEmbeddingsProviderAndIndexDoc(site, msg.data.reIndex);
|
||||
}
|
||||
|
||||
this.ide.infoPopup("Docs indexing completed");
|
||||
});
|
||||
on("context/loadSubmenuItems", async (msg) => {
|
||||
const config = await this.config();
|
||||
const items = config.contextProviders
|
||||
const items = await config.contextProviders
|
||||
?.find((provider) => provider.description.title === msg.data.title)
|
||||
?.loadSubmenuItems({
|
||||
config,
|
||||
ide: this.ide,
|
||||
fetch: (url, init) =>
|
||||
fetchwithRequestOptions(url, init, config.requestOptions),
|
||||
|
|
@ -268,9 +331,13 @@ export class Core {
|
|||
fetchwithRequestOptions(url, init, config.requestOptions),
|
||||
});
|
||||
|
||||
Telemetry.capture("useContextProvider", {
|
||||
name: provider.description.title,
|
||||
});
|
||||
Telemetry.capture(
|
||||
"useContextProvider",
|
||||
{
|
||||
name: provider.description.title,
|
||||
},
|
||||
true,
|
||||
);
|
||||
|
||||
return items.map((item) => ({
|
||||
...item,
|
||||
|
|
@ -282,12 +349,15 @@ export class Core {
|
|||
}
|
||||
});
|
||||
|
||||
on("config/getBrowserSerialized", (msg) => {
|
||||
return this.configHandler.getSerializedConfig();
|
||||
on("config/getSerializedProfileInfo", async (msg) => {
|
||||
return {
|
||||
config: await this.configHandler.getSerializedConfig(),
|
||||
profileId: this.configHandler.currentProfile.profileId,
|
||||
};
|
||||
});
|
||||
|
||||
async function* llmStreamChat(
|
||||
configHandler: IConfigHandler,
|
||||
configHandler: ConfigHandler,
|
||||
abortedMessageIds: Set<string>,
|
||||
msg: Message<ToCoreProtocol["llm/streamChat"][0]>,
|
||||
) {
|
||||
|
|
@ -322,7 +392,7 @@ export class Core {
|
|||
);
|
||||
|
||||
async function* llmStreamComplete(
|
||||
configHandler: IConfigHandler,
|
||||
configHandler: ConfigHandler,
|
||||
abortedMessageIds: Set<string>,
|
||||
|
||||
msg: Message<ToCoreProtocol["llm/streamComplete"][0]>,
|
||||
|
|
@ -388,7 +458,7 @@ export class Core {
|
|||
});
|
||||
|
||||
async function* runNodeJsSlashCommand(
|
||||
configHandler: IConfigHandler,
|
||||
configHandler: ConfigHandler,
|
||||
abortedMessageIds: Set<string>,
|
||||
msg: Message<ToCoreProtocol["command/run"][0]>,
|
||||
messenger: IMessenger<ToCoreProtocol, FromCoreProtocol>,
|
||||
|
|
@ -413,9 +483,13 @@ export class Core {
|
|||
throw new Error(`Unknown slash command ${slashCommandName}`);
|
||||
}
|
||||
|
||||
Telemetry.capture("useSlashCommand", {
|
||||
name: slashCommandName,
|
||||
});
|
||||
Telemetry.capture(
|
||||
"useSlashCommand",
|
||||
{
|
||||
name: slashCommandName,
|
||||
},
|
||||
true,
|
||||
);
|
||||
|
||||
const checkActiveInterval = setInterval(() => {
|
||||
if (abortedMessageIds.has(msg.messageId)) {
|
||||
|
|
@ -477,7 +551,7 @@ export class Core {
|
|||
});
|
||||
|
||||
async function* streamDiffLinesGenerator(
|
||||
configHandler: IConfigHandler,
|
||||
configHandler: ConfigHandler,
|
||||
abortedMessageIds: Set<string>,
|
||||
msg: Message<ToCoreProtocol["streamDiffLines"][0]>,
|
||||
) {
|
||||
|
|
@ -566,7 +640,7 @@ export class Core {
|
|||
});
|
||||
on("index/forceReIndex", async (msg) => {
|
||||
const dirs = msg.data ? [msg.data] : await this.ide.getWorkspaceDirs();
|
||||
this.refreshCodebaseIndex(dirs);
|
||||
await this.refreshCodebaseIndex(dirs);
|
||||
});
|
||||
on("index/setPaused", (msg) => {
|
||||
new GlobalContext().update("indexingPaused", msg.data);
|
||||
|
|
@ -579,6 +653,22 @@ export class Core {
|
|||
this.messenger.request("indexProgress", this.indexingState);
|
||||
}
|
||||
});
|
||||
|
||||
on("didChangeSelectedProfile", (msg) => {
|
||||
this.configHandler.setSelectedProfile(msg.data.id);
|
||||
this.configHandler.reloadConfig();
|
||||
});
|
||||
on("didChangeControlPlaneSessionInfo", async (msg) => {
|
||||
this.configHandler.updateControlPlaneSessionInfo(msg.data.sessionInfo);
|
||||
});
|
||||
on("auth/getAuthUrl", async (msg) => {
|
||||
const url = await getAuthUrlForTokenPage();
|
||||
return { url };
|
||||
});
|
||||
|
||||
on("didChangeActiveTextEditor", ({ data: { filepath } }) => {
|
||||
recentlyEditedFilesCache.set(filepath, filepath);
|
||||
});
|
||||
}
|
||||
|
||||
private indexingCancellationController: AbortController | undefined;
|
||||
|
|
@ -595,5 +685,103 @@ export class Core {
|
|||
this.messenger.request("indexProgress", update);
|
||||
this.indexingState = update;
|
||||
}
|
||||
|
||||
this.messenger.send("refreshSubmenuItems", undefined);
|
||||
}
|
||||
|
||||
private async shouldReindexDocsOnNewEmbeddingsProvider(
|
||||
curEmbeddingsProviderId: EmbeddingsProvider["id"],
|
||||
): Promise<boolean> {
|
||||
const ideInfo = await this.ide.getIdeInfo();
|
||||
const isJetBrainsAndPreIndexedDocsProvider =
|
||||
this.docsService.isJetBrainsAndPreIndexedDocsProvider(
|
||||
ideInfo,
|
||||
curEmbeddingsProviderId,
|
||||
);
|
||||
|
||||
if (isJetBrainsAndPreIndexedDocsProvider) {
|
||||
try {
|
||||
this.ide.errorPopup(
|
||||
"The 'transformers.js' embeddings provider currently cannot be used to index " +
|
||||
"documentation in JetBrains. To enable documentation indexing, you can use " +
|
||||
"any of the other providers described in the docs: " +
|
||||
"https://docs.continue.dev/walkthroughs/codebase-embeddings#embeddings-providers",
|
||||
);
|
||||
} catch (error) {
|
||||
console.error("Failed to show error popup:", error);
|
||||
}
|
||||
this.globalContext.update(
|
||||
"curEmbeddingsProviderId",
|
||||
curEmbeddingsProviderId,
|
||||
);
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
const lastEmbeddingsProviderId = this.globalContext.get(
|
||||
"curEmbeddingsProviderId",
|
||||
);
|
||||
|
||||
if (!lastEmbeddingsProviderId) {
|
||||
// If it's the first time we're setting the `curEmbeddingsProviderId`
|
||||
// global state, we don't need to reindex docs
|
||||
this.globalContext.update(
|
||||
"curEmbeddingsProviderId",
|
||||
curEmbeddingsProviderId,
|
||||
);
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
return lastEmbeddingsProviderId !== curEmbeddingsProviderId;
|
||||
}
|
||||
|
||||
private async getEmbeddingsProviderAndIndexDoc(
|
||||
site: SiteIndexingConfig,
|
||||
reIndex: boolean = false,
|
||||
): Promise<void> {
|
||||
const config = await this.config();
|
||||
const { embeddingsProvider } = config;
|
||||
|
||||
for await (const update of this.docsService.indexAndAdd(
|
||||
site,
|
||||
embeddingsProvider,
|
||||
reIndex,
|
||||
)) {
|
||||
// Temporary disabled posting progress updates to the UI due to
|
||||
// possible collision with code indexing progress updates.
|
||||
// this.messenger.request("indexProgress", update);
|
||||
// this.indexingState = update;
|
||||
}
|
||||
}
|
||||
|
||||
private async reindexDocsOnNewEmbeddingsProvider(
|
||||
embeddingsProvider: EmbeddingsProvider,
|
||||
) {
|
||||
const docs = await this.docsService.list();
|
||||
|
||||
if (docs.length === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
this.ide.infoPopup("Reindexing docs with new embeddings provider");
|
||||
|
||||
for (const { title, baseUrl } of docs) {
|
||||
await this.docsService.delete(baseUrl);
|
||||
|
||||
const generator = this.docsService.indexAndAdd(
|
||||
{ title, startUrl: baseUrl, rootUrl: baseUrl },
|
||||
embeddingsProvider,
|
||||
);
|
||||
|
||||
while (!(await generator.next()).done) {}
|
||||
}
|
||||
|
||||
// Important that this only is invoked after we have successfully
|
||||
// cleared and reindex the docs so that the table cannot end up in an
|
||||
// invalid state.
|
||||
this.globalContext.update("curEmbeddingsProviderId", embeddingsProvider.id);
|
||||
|
||||
this.ide.infoPopup("Completed reindexing of all docs");
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,11 +1,11 @@
|
|||
import { DiffLine } from "../index.js";
|
||||
import { DiffLine, DiffLineType } from "../index.js";
|
||||
import { LineStream, matchLine } from "./util.js";
|
||||
|
||||
/**
|
||||
* https://blog.jcoglan.com/2017/02/12/the-myers-diff-algorithm-part-1/
|
||||
* Invariants:
|
||||
* - new + same = newLines.length
|
||||
* - old + same = oldLines.length
|
||||
* - old + same = oldLinesCopy.length
|
||||
* ^ (above two guarantee that all lines get represented)
|
||||
* - Lines are always output in order, at least among old and new separately
|
||||
*/
|
||||
|
|
@ -13,52 +13,77 @@ export async function* streamDiff(
|
|||
oldLines: string[],
|
||||
newLines: LineStream,
|
||||
): AsyncGenerator<DiffLine> {
|
||||
const mutatedOldLines = [...oldLines]; // be careful
|
||||
const oldLinesCopy = [...oldLines];
|
||||
|
||||
// If one indentation mistake is made, others are likely. So we are more permissive about matching
|
||||
let seenIndentationMistake = false;
|
||||
|
||||
let newLineResult = await newLines.next();
|
||||
while (oldLines.length > 0 && !newLineResult.done) {
|
||||
const [matchIndex, isPerfectMatch, newLine] = matchLine(
|
||||
|
||||
while (oldLinesCopy.length > 0 && !newLineResult.done) {
|
||||
const { matchIndex, isPerfectMatch, newLine } = matchLine(
|
||||
newLineResult.value,
|
||||
oldLines,
|
||||
oldLinesCopy,
|
||||
seenIndentationMistake,
|
||||
);
|
||||
|
||||
if (!seenIndentationMistake && newLineResult.value !== newLine) {
|
||||
seenIndentationMistake = true;
|
||||
}
|
||||
|
||||
if (matchIndex < 0) {
|
||||
// Insert new line
|
||||
yield { type: "new", line: newLine };
|
||||
let type: DiffLineType;
|
||||
|
||||
let isLineRemoval = false;
|
||||
const isNewLine = matchIndex === -1;
|
||||
|
||||
if (isNewLine) {
|
||||
type = "new";
|
||||
} else {
|
||||
// Insert all deleted lines before match
|
||||
for (let i = 0; i < matchIndex; i++) {
|
||||
yield { type: "old", line: oldLines.shift()! };
|
||||
yield { type: "old", line: oldLinesCopy.shift()! };
|
||||
}
|
||||
|
||||
if (isPerfectMatch) {
|
||||
// Same
|
||||
yield { type: "same", line: oldLines.shift()! };
|
||||
} else {
|
||||
// Delete old line and insert the new
|
||||
yield { type: "old", line: oldLines.shift()! };
|
||||
yield { type: "new", line: newLine };
|
||||
}
|
||||
type = isPerfectMatch ? "same" : "old";
|
||||
}
|
||||
|
||||
newLineResult = await newLines.next();
|
||||
switch (type) {
|
||||
case "new":
|
||||
yield { type, line: newLine };
|
||||
break;
|
||||
|
||||
case "same":
|
||||
yield { type, line: oldLinesCopy.shift()! };
|
||||
break;
|
||||
|
||||
case "old":
|
||||
yield { type, line: oldLinesCopy.shift()! };
|
||||
|
||||
if (oldLinesCopy[0] !== newLine) {
|
||||
yield { type: "new", line: newLine };
|
||||
} else {
|
||||
isLineRemoval = true;
|
||||
}
|
||||
|
||||
break;
|
||||
|
||||
default:
|
||||
console.error(`Error streaming diff, unrecognized diff type: ${type}`);
|
||||
}
|
||||
|
||||
if (!isLineRemoval) {
|
||||
newLineResult = await newLines.next();
|
||||
}
|
||||
}
|
||||
|
||||
// Once at the edge, only one choice
|
||||
if (newLineResult.done === true && oldLines.length > 0) {
|
||||
for (const oldLine of oldLines) {
|
||||
if (newLineResult.done && oldLinesCopy.length > 0) {
|
||||
for (const oldLine of oldLinesCopy) {
|
||||
yield { type: "old", line: oldLine };
|
||||
}
|
||||
}
|
||||
|
||||
if (!newLineResult.done && oldLines.length === 0) {
|
||||
if (!newLineResult.done && oldLinesCopy.length === 0) {
|
||||
yield { type: "new", line: newLineResult.value };
|
||||
for await (const newLine of newLines) {
|
||||
yield { type: "new", line: newLine };
|
||||
|
|
|
|||
|
|
@ -1,9 +1,19 @@
|
|||
import { distance } from "fastest-levenshtein";
|
||||
import { ChatMessage } from "../index.js";
|
||||
import { stripImages } from "../llm/countTokens.js";
|
||||
import { stripImages } from "../llm/images.js";
|
||||
|
||||
export type LineStream = AsyncGenerator<string>;
|
||||
|
||||
export type MatchLineResult = {
|
||||
/**
|
||||
* -1 if it's a new line, otherwise the index of the first match
|
||||
* in the old lines.
|
||||
*/
|
||||
matchIndex: number;
|
||||
isPerfectMatch: boolean;
|
||||
newLine: string;
|
||||
};
|
||||
|
||||
function linesMatchPerfectly(lineA: string, lineB: string): boolean {
|
||||
return lineA === lineB && lineA !== "";
|
||||
}
|
||||
|
|
@ -18,6 +28,7 @@ function linesMatch(lineA: string, lineB: string, linesBetween = 0): boolean {
|
|||
}
|
||||
|
||||
const d = distance(lineA, lineB);
|
||||
|
||||
return (
|
||||
// Should be more unlikely for lines to fuzzy match if they are further away
|
||||
(d / Math.max(lineA.length, lineB.length) < 0.5 - linesBetween * 0.05 ||
|
||||
|
|
@ -27,6 +38,8 @@ function linesMatch(lineA: string, lineB: string, linesBetween = 0): boolean {
|
|||
}
|
||||
|
||||
/**
|
||||
* Used to find a match for a new line in an array of old lines.
|
||||
*
|
||||
* Return the index of the first match and whether it is a perfect match
|
||||
* Also return a version of the line with correct indentation if needs fixing
|
||||
*/
|
||||
|
|
@ -34,21 +47,26 @@ export function matchLine(
|
|||
newLine: string,
|
||||
oldLines: string[],
|
||||
permissiveAboutIndentation = false,
|
||||
): [number, boolean, string] {
|
||||
): MatchLineResult {
|
||||
// Only match empty lines if it's the next one:
|
||||
if (newLine.trim() === "" && oldLines[0]?.trim() === "") {
|
||||
return [0, true, newLine.trim()];
|
||||
return {
|
||||
matchIndex: 0,
|
||||
isPerfectMatch: true,
|
||||
newLine: newLine.trim(),
|
||||
};
|
||||
}
|
||||
|
||||
const isEndBracket = END_BRACKETS.includes(newLine.trim());
|
||||
|
||||
for (let i = 0; i < oldLines.length; i++) {
|
||||
// Don't match end bracket lines if too far away
|
||||
if (i > 4 && isEndBracket) {
|
||||
return [-1, false, newLine];
|
||||
return { matchIndex: -1, isPerfectMatch: false, newLine };
|
||||
}
|
||||
|
||||
if (linesMatchPerfectly(newLine, oldLines[i])) {
|
||||
return [i, true, newLine];
|
||||
return { matchIndex: i, isPerfectMatch: true, newLine };
|
||||
}
|
||||
if (linesMatch(newLine, oldLines[i], i)) {
|
||||
// This is a way to fix indentation, but only for sufficiently long lines to avoid matching whitespace or short lines
|
||||
|
|
@ -56,13 +74,17 @@ export function matchLine(
|
|||
newLine.trimStart() === oldLines[i].trimStart() &&
|
||||
(permissiveAboutIndentation || newLine.trim().length > 8)
|
||||
) {
|
||||
return [i, true, oldLines[i]];
|
||||
return {
|
||||
matchIndex: i,
|
||||
isPerfectMatch: true,
|
||||
newLine: oldLines[i],
|
||||
};
|
||||
}
|
||||
return [i, false, newLine];
|
||||
return { matchIndex: i, isPerfectMatch: false, newLine };
|
||||
}
|
||||
}
|
||||
|
||||
return [-1, false, newLine];
|
||||
return { matchIndex: -1, isPerfectMatch: false, newLine };
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
|
|||
|
|
@ -137,6 +137,7 @@ export interface ContextProviderExtras {
|
|||
}
|
||||
|
||||
export interface LoadSubmenuItemsArgs {
|
||||
config: ContinueConfig;
|
||||
ide: IDE;
|
||||
fetch: FetchFunction;
|
||||
}
|
||||
|
|
@ -374,8 +375,10 @@ export type CustomLLM = RequireAtLeastOne<
|
|||
|
||||
// IDE
|
||||
|
||||
export type DiffLineType = "new" | "old" | "same";
|
||||
|
||||
export interface DiffLine {
|
||||
type: "new" | "old" | "same";
|
||||
type: DiffLineType;
|
||||
line: string;
|
||||
}
|
||||
|
||||
|
|
@ -419,6 +422,9 @@ export interface IdeSettings {
|
|||
remoteConfigServerUrl: string | undefined;
|
||||
remoteConfigSyncPeriod: number;
|
||||
userToken: string;
|
||||
enableControlServerBeta: boolean;
|
||||
pauseCodebaseIndexOnStart: boolean;
|
||||
enableDebugLogs: boolean;
|
||||
}
|
||||
|
||||
export interface IDE {
|
||||
|
|
@ -595,6 +601,7 @@ export type ModelName =
|
|||
| "gpt-3.5-turbo-0613"
|
||||
| "gpt-4-32k"
|
||||
| "gpt-4o"
|
||||
| "gpt-4o-mini"
|
||||
| "gpt-4-turbo"
|
||||
| "gpt-4-turbo-preview"
|
||||
| "gpt-4-vision-preview"
|
||||
|
|
@ -732,7 +739,9 @@ export type EmbeddingsProviderName =
|
|||
| "openai"
|
||||
| "cohere"
|
||||
| "free-trial"
|
||||
| "gemini";
|
||||
| "gemini"
|
||||
| "continue-proxy"
|
||||
| "deepinfra";
|
||||
|
||||
export interface EmbedOptions {
|
||||
apiBase?: string;
|
||||
|
|
@ -742,6 +751,7 @@ export interface EmbedOptions {
|
|||
apiType?: string;
|
||||
apiVersion?: string;
|
||||
requestOptions?: RequestOptions;
|
||||
maxChunkSize?: number;
|
||||
}
|
||||
|
||||
export interface EmbeddingsProviderDescription extends EmbedOptions {
|
||||
|
|
@ -750,10 +760,18 @@ export interface EmbeddingsProviderDescription extends EmbedOptions {
|
|||
|
||||
export interface EmbeddingsProvider {
|
||||
id: string;
|
||||
providerName: EmbeddingsProviderName;
|
||||
maxChunkSize: number;
|
||||
embed(chunks: string[]): Promise<number[][]>;
|
||||
}
|
||||
|
||||
export type RerankerName = "cohere" | "voyage" | "llm" | "free-trial";
|
||||
export type RerankerName =
|
||||
| "cohere"
|
||||
| "voyage"
|
||||
| "llm"
|
||||
| "free-trial"
|
||||
| "huggingface-tei"
|
||||
| "continue-proxy";
|
||||
|
||||
export interface RerankerDescription {
|
||||
name: RerankerName;
|
||||
|
|
@ -768,7 +786,7 @@ export interface Reranker {
|
|||
export interface TabAutocompleteOptions {
|
||||
disable: boolean;
|
||||
useCopyBuffer: boolean;
|
||||
useSuffix: boolean;
|
||||
useFileSuffix: boolean;
|
||||
maxPromptTokens: number;
|
||||
debounceDelay: number;
|
||||
maxSuffixPercentage: number;
|
||||
|
|
@ -845,6 +863,12 @@ interface ExperimentalConfig {
|
|||
quickActions?: QuickActionConfig[];
|
||||
}
|
||||
|
||||
interface AnalyticsConfig {
|
||||
type: string;
|
||||
url?: string;
|
||||
clientKey?: string;
|
||||
}
|
||||
|
||||
// config.json
|
||||
export interface SerializedContinueConfig {
|
||||
env?: string[];
|
||||
|
|
@ -865,6 +889,7 @@ export interface SerializedContinueConfig {
|
|||
ui?: ContinueUIConfig;
|
||||
reranker?: RerankerDescription;
|
||||
experimental?: ExperimentalConfig;
|
||||
analytics?: AnalyticsConfig;
|
||||
}
|
||||
|
||||
export type ConfigMergeType = "merge" | "overwrite";
|
||||
|
|
@ -915,6 +940,8 @@ export interface Config {
|
|||
reranker?: RerankerDescription | Reranker;
|
||||
/** Experimental configuration */
|
||||
experimental?: ExperimentalConfig;
|
||||
/** Analytics configuration */
|
||||
analytics?: AnalyticsConfig;
|
||||
}
|
||||
|
||||
// in the actual Continue source code
|
||||
|
|
@ -935,6 +962,8 @@ export interface ContinueConfig {
|
|||
ui?: ContinueUIConfig;
|
||||
reranker?: Reranker;
|
||||
experimental?: ExperimentalConfig;
|
||||
analytics?: AnalyticsConfig;
|
||||
docs?: SiteIndexingConfig[];
|
||||
}
|
||||
|
||||
export interface BrowserSerializedContinueConfig {
|
||||
|
|
@ -952,4 +981,5 @@ export interface BrowserSerializedContinueConfig {
|
|||
ui?: ContinueUIConfig;
|
||||
reranker?: RerankerDescription;
|
||||
experimental?: ExperimentalConfig;
|
||||
analytics?: AnalyticsConfig;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
import { IConfigHandler } from "../config/IConfigHandler.js";
|
||||
import { ConfigHandler } from "../config/ConfigHandler.js";
|
||||
import { IContinueServerClient } from "../continueServer/interface.js";
|
||||
import { IDE, IndexTag, IndexingProgressUpdate } from "../index.js";
|
||||
import { CodeSnippetsCodebaseIndex } from "./CodeSnippetsIndex.js";
|
||||
|
|
@ -6,7 +6,7 @@ import { FullTextSearchCodebaseIndex } from "./FullTextSearch.js";
|
|||
import { LanceDbIndex } from "./LanceDbIndex.js";
|
||||
import { ChunkCodebaseIndex } from "./chunk/ChunkCodebaseIndex.js";
|
||||
import { getComputeDeleteAddRemove } from "./refreshIndex.js";
|
||||
import { CodebaseIndex } from "./types.js";
|
||||
import { CodebaseIndex, IndexResultType } from "./types.js";
|
||||
import { walkDir } from "./walkDir.js";
|
||||
|
||||
export class PauseToken {
|
||||
|
|
@ -23,7 +23,7 @@ export class PauseToken {
|
|||
|
||||
export class CodebaseIndexer {
|
||||
constructor(
|
||||
private readonly configHandler: IConfigHandler,
|
||||
private readonly configHandler: ConfigHandler,
|
||||
private readonly ide: IDE,
|
||||
private readonly pauseToken: PauseToken,
|
||||
private readonly continueServerClient: IContinueServerClient,
|
||||
|
|
@ -36,6 +36,7 @@ export class CodebaseIndexer {
|
|||
new ChunkCodebaseIndex(
|
||||
this.ide.readFile.bind(this.ide),
|
||||
this.continueServerClient,
|
||||
config.embeddingsProvider.maxChunkSize,
|
||||
), // Chunking must come first
|
||||
new LanceDbIndex(
|
||||
config.embeddingsProvider,
|
||||
|
|
@ -111,7 +112,7 @@ export class CodebaseIndexer {
|
|||
branch,
|
||||
artifactId: codebaseIndex.artifactId,
|
||||
};
|
||||
const [results, markComplete] = await getComputeDeleteAddRemove(
|
||||
const [results, lastUpdated, markComplete] = await getComputeDeleteAddRemove(
|
||||
tag,
|
||||
{ ...stats },
|
||||
(filepath) => this.ide.readFile(filepath),
|
||||
|
|
@ -158,6 +159,10 @@ export class CodebaseIndexer {
|
|||
};
|
||||
}
|
||||
|
||||
lastUpdated.forEach((lastUpdated, path) => {
|
||||
markComplete([lastUpdated], IndexResultType.UpdateLastUpdated);
|
||||
});
|
||||
|
||||
completedRelativeExpectedTime += codebaseIndex.relativeExpectedTime;
|
||||
yield {
|
||||
progress:
|
||||
|
|
|
|||
|
|
@ -9,7 +9,6 @@ import {
|
|||
IndexTag,
|
||||
IndexingProgressUpdate,
|
||||
} from "../index.js";
|
||||
import { MAX_CHUNK_SIZE } from "../llm/constants.js";
|
||||
import { getBasename } from "../util/index.js";
|
||||
import { getLanceDbPath, migrate } from "../util/paths.js";
|
||||
import { chunkDocument } from "./chunk/chunk.js";
|
||||
|
|
@ -36,8 +35,6 @@ export class LanceDbIndex implements CodebaseIndex {
|
|||
return `vectordb::${this.embeddingsProvider.id}`;
|
||||
}
|
||||
|
||||
static MAX_CHUNK_SIZE = MAX_CHUNK_SIZE;
|
||||
|
||||
constructor(
|
||||
private readonly embeddingsProvider: EmbeddingsProvider,
|
||||
private readonly readFile: (filepath: string) => Promise<string>,
|
||||
|
|
@ -64,10 +61,13 @@ export class LanceDbIndex implements CodebaseIndex {
|
|||
migrate(
|
||||
"lancedb_sqlite_artifact_id_column",
|
||||
async () => {
|
||||
await db.exec(
|
||||
`ALTER TABLE lance_db_cache ADD COLUMN artifact_id TEXT NOT NULL DEFAULT 'UNDEFINED'`,
|
||||
);
|
||||
resolve(undefined);
|
||||
try {
|
||||
await db.exec(
|
||||
"ALTER TABLE lance_db_cache ADD COLUMN artifact_id TEXT NOT NULL DEFAULT 'UNDEFINED'",
|
||||
);
|
||||
} finally {
|
||||
resolve(undefined);
|
||||
}
|
||||
},
|
||||
() => resolve(undefined),
|
||||
),
|
||||
|
|
@ -96,12 +96,12 @@ export class LanceDbIndex implements CodebaseIndex {
|
|||
|
||||
let hasEmptyChunks = false;
|
||||
|
||||
for await (const chunk of chunkDocument(
|
||||
items[i].path,
|
||||
content,
|
||||
LanceDbIndex.MAX_CHUNK_SIZE,
|
||||
items[i].cacheKey,
|
||||
)) {
|
||||
for await (const chunk of chunkDocument({
|
||||
filepath: items[i].path,
|
||||
contents: content,
|
||||
maxChunkSize: this.embeddingsProvider.maxChunkSize,
|
||||
digest: items[i].cacheKey,
|
||||
})) {
|
||||
if (chunk.content.length == 0) {
|
||||
hasEmptyChunks = true;
|
||||
break;
|
||||
|
|
|
|||
|
|
@ -1,6 +1,5 @@
|
|||
import { IContinueServerClient } from "../../continueServer/interface.js";
|
||||
import { Chunk, IndexTag, IndexingProgressUpdate } from "../../index.js";
|
||||
import { MAX_CHUNK_SIZE } from "../../llm/constants.js";
|
||||
import { getBasename } from "../../util/index.js";
|
||||
import { DatabaseConnection, SqliteDb, tagToString } from "../refreshIndex.js";
|
||||
import {
|
||||
|
|
@ -19,6 +18,7 @@ export class ChunkCodebaseIndex implements CodebaseIndex {
|
|||
constructor(
|
||||
private readonly readFile: (filepath: string) => Promise<string>,
|
||||
private readonly continueServerClient: IContinueServerClient,
|
||||
private readonly maxChunkSize: number,
|
||||
) {
|
||||
this.readFile = readFile;
|
||||
}
|
||||
|
|
@ -105,13 +105,15 @@ export class ChunkCodebaseIndex implements CodebaseIndex {
|
|||
const item = results.compute[i];
|
||||
|
||||
// Insert chunks
|
||||
for await (const chunk of chunkDocument(
|
||||
item.path,
|
||||
contents[i],
|
||||
MAX_CHUNK_SIZE,
|
||||
item.cacheKey,
|
||||
)) {
|
||||
handleChunk(chunk);
|
||||
if (contents.length) {
|
||||
for await (const chunk of chunkDocument({
|
||||
filepath: item.path,
|
||||
contents: contents[i],
|
||||
maxChunkSize: this.maxChunkSize,
|
||||
digest: item.cacheKey,
|
||||
})) {
|
||||
await handleChunk(chunk);
|
||||
}
|
||||
}
|
||||
|
||||
accumulatedProgress =
|
||||
|
|
@ -125,17 +127,22 @@ export class ChunkCodebaseIndex implements CodebaseIndex {
|
|||
}
|
||||
|
||||
// Add tag
|
||||
for (const item of results.addTag) {
|
||||
const chunksWithPath = await db.all(
|
||||
"SELECT * FROM chunks WHERE cacheKey = ?",
|
||||
[item.cacheKey],
|
||||
);
|
||||
const addContents = await Promise.all(
|
||||
results.addTag.map(({ path }) => this.readFile(path)),
|
||||
);
|
||||
for (let i = 0; i < results.addTag.length; i++) {
|
||||
const item = results.addTag[i];
|
||||
|
||||
for (const chunk of chunksWithPath) {
|
||||
await db.run("INSERT INTO chunk_tags (chunkId, tag) VALUES (?, ?)", [
|
||||
chunk.id,
|
||||
tagString,
|
||||
]);
|
||||
// Insert chunks
|
||||
if (contents.length) {
|
||||
for await (const chunk of chunkDocument({
|
||||
filepath: item.path,
|
||||
contents: contents[i],
|
||||
maxChunkSize: this.maxChunkSize,
|
||||
digest: item.cacheKey,
|
||||
})) {
|
||||
handleChunk(chunk);
|
||||
}
|
||||
}
|
||||
|
||||
markComplete([item], IndexResultType.AddTag);
|
||||
|
|
|
|||
|
|
@ -1,10 +1,10 @@
|
|||
import { ChunkWithoutID } from "../../index.js";
|
||||
import { countTokens } from "../../llm/countTokens.js";
|
||||
import { countTokensAsync } from "../../llm/countTokens.js";
|
||||
|
||||
export function* basicChunker(
|
||||
export async function* basicChunker(
|
||||
contents: string,
|
||||
maxChunkSize: number,
|
||||
): Generator<ChunkWithoutID> {
|
||||
): AsyncGenerator<ChunkWithoutID> {
|
||||
if (contents.trim().length === 0) {
|
||||
return;
|
||||
}
|
||||
|
|
@ -14,18 +14,24 @@ export function* basicChunker(
|
|||
let startLine = 0;
|
||||
let currLine = 0;
|
||||
|
||||
for (const line of contents.split("\n")) {
|
||||
const lineTokens = countTokens(line);
|
||||
if (chunkTokens + lineTokens > maxChunkSize - 5) {
|
||||
const lineTokens = await Promise.all(contents.split("\n").map(async l => {
|
||||
return {
|
||||
line: l,
|
||||
tokenCount: await countTokensAsync(l),
|
||||
};
|
||||
}));
|
||||
|
||||
for (const lt of lineTokens) {
|
||||
if (chunkTokens + lt.tokenCount > maxChunkSize - 5) {
|
||||
yield { content: chunkContent, startLine, endLine: currLine - 1 };
|
||||
chunkContent = "";
|
||||
chunkTokens = 0;
|
||||
startLine = currLine;
|
||||
}
|
||||
|
||||
if (lineTokens < maxChunkSize) {
|
||||
chunkContent += `${line}\n`;
|
||||
chunkTokens += lineTokens + 1;
|
||||
if (lt.tokenCount < maxChunkSize) {
|
||||
chunkContent += `${lt.line}\n`;
|
||||
chunkTokens += lt.tokenCount + 1;
|
||||
}
|
||||
|
||||
currLine++;
|
||||
|
|
|
|||
|
|
@ -1,10 +1,16 @@
|
|||
import { Chunk, ChunkWithoutID } from "../../index.js";
|
||||
import { MAX_CHUNK_SIZE } from "../../llm/constants.js";
|
||||
import { countTokens } from "../../llm/countTokens.js";
|
||||
import { countTokens, countTokensAsync } from "../../llm/countTokens.js";
|
||||
import { supportedLanguages } from "../../util/treeSitter.js";
|
||||
import { basicChunker } from "./basic.js";
|
||||
import { codeChunker } from "./code.js";
|
||||
|
||||
export type ChunkDocumentParam = {
|
||||
filepath: string;
|
||||
contents: string;
|
||||
maxChunkSize: number;
|
||||
digest: string;
|
||||
};
|
||||
|
||||
async function* chunkDocumentWithoutId(
|
||||
filepath: string,
|
||||
contents: string,
|
||||
|
|
@ -31,32 +37,41 @@ async function* chunkDocumentWithoutId(
|
|||
yield* basicChunker(contents, maxChunkSize);
|
||||
}
|
||||
|
||||
export async function* chunkDocument(
|
||||
filepath: string,
|
||||
contents: string,
|
||||
maxChunkSize: number,
|
||||
digest: string,
|
||||
): AsyncGenerator<Chunk> {
|
||||
export async function* chunkDocument({
|
||||
filepath,
|
||||
contents,
|
||||
maxChunkSize,
|
||||
digest,
|
||||
}: ChunkDocumentParam): AsyncGenerator<Chunk> {
|
||||
let index = 0;
|
||||
const chunkPromises: Promise<Chunk | undefined>[] = [];
|
||||
for await (const chunkWithoutId of chunkDocumentWithoutId(
|
||||
filepath,
|
||||
contents,
|
||||
maxChunkSize,
|
||||
)) {
|
||||
if (countTokens(chunkWithoutId.content) > MAX_CHUNK_SIZE) {
|
||||
console.warn(
|
||||
`Chunk with more than ${maxChunkSize} tokens constructed: `,
|
||||
chunkPromises.push(new Promise(async (resolve) => {
|
||||
if (await countTokensAsync(chunkWithoutId.content) > maxChunkSize) {
|
||||
console.warn(
|
||||
`Chunk with more than ${maxChunkSize} tokens constructed: `,
|
||||
filepath,
|
||||
countTokens(chunkWithoutId.content),
|
||||
);
|
||||
return resolve(undefined);
|
||||
}
|
||||
resolve({
|
||||
...chunkWithoutId,
|
||||
digest,
|
||||
index,
|
||||
filepath,
|
||||
countTokens(chunkWithoutId.content),
|
||||
);
|
||||
continue;
|
||||
}
|
||||
yield {
|
||||
...chunkWithoutId,
|
||||
digest,
|
||||
index,
|
||||
filepath,
|
||||
};
|
||||
});
|
||||
}));
|
||||
index++;
|
||||
}
|
||||
for await (const chunk of chunkPromises) {
|
||||
if (!chunk) {
|
||||
continue;
|
||||
}
|
||||
yield chunk;
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
import { SyntaxNode } from "web-tree-sitter";
|
||||
import { ChunkWithoutID } from "../../index.js";
|
||||
import { countTokens } from "../../llm/countTokens.js";
|
||||
import { countTokens, countTokensAsync } from "../../llm/countTokens.js";
|
||||
import { getParserForFile } from "../../util/treeSitter.js";
|
||||
|
||||
function collapsedReplacement(node: SyntaxNode): string {
|
||||
|
|
@ -22,14 +22,14 @@ function firstChild(
|
|||
return node.children.find((child) => child.type === grammarName) || null;
|
||||
}
|
||||
|
||||
function collapseChildren(
|
||||
async function collapseChildren(
|
||||
node: SyntaxNode,
|
||||
code: string,
|
||||
blockTypes: string[],
|
||||
collapseTypes: string[],
|
||||
collapseBlockTypes: string[],
|
||||
maxChunkSize: number,
|
||||
): string {
|
||||
): Promise<string> {
|
||||
code = code.slice(0, node.endIndex);
|
||||
const block = firstChild(node, blockTypes);
|
||||
const collapsedChildren = [];
|
||||
|
|
@ -58,7 +58,7 @@ function collapseChildren(
|
|||
code = code.slice(node.startIndex);
|
||||
let removedChild = false;
|
||||
while (
|
||||
countTokens(code.trim()) > maxChunkSize &&
|
||||
(await countTokensAsync(code.trim())) > maxChunkSize &&
|
||||
collapsedChildren.length > 0
|
||||
) {
|
||||
removedChild = true;
|
||||
|
|
@ -106,11 +106,11 @@ export const FUNCTION_DECLARATION_NODE_TYPEs = [
|
|||
"method_declaration",
|
||||
];
|
||||
|
||||
function constructClassDefinitionChunk(
|
||||
async function constructClassDefinitionChunk(
|
||||
node: SyntaxNode,
|
||||
code: string,
|
||||
maxChunkSize: number,
|
||||
): string {
|
||||
): Promise<string> {
|
||||
return collapseChildren(
|
||||
node,
|
||||
code,
|
||||
|
|
@ -121,11 +121,11 @@ function constructClassDefinitionChunk(
|
|||
);
|
||||
}
|
||||
|
||||
function constructFunctionDefinitionChunk(
|
||||
async function constructFunctionDefinitionChunk(
|
||||
node: SyntaxNode,
|
||||
code: string,
|
||||
maxChunkSize: number,
|
||||
): string {
|
||||
): Promise<string> {
|
||||
const bodyNode = node.children[node.children.length - 1];
|
||||
const funcText =
|
||||
code.slice(node.startIndex, bodyNode.startIndex) +
|
||||
|
|
@ -153,7 +153,7 @@ const collapsedNodeConstructors: {
|
|||
node: SyntaxNode,
|
||||
code: string,
|
||||
maxChunkSize: number,
|
||||
) => string;
|
||||
) => Promise<string>;
|
||||
} = {
|
||||
// Classes, structs, etc
|
||||
class_definition: constructClassDefinitionChunk,
|
||||
|
|
@ -165,37 +165,50 @@ const collapsedNodeConstructors: {
|
|||
function_item: constructFunctionDefinitionChunk,
|
||||
};
|
||||
|
||||
function* getSmartCollapsedChunks(
|
||||
async function maybeYieldChunk(
|
||||
node: SyntaxNode,
|
||||
code: string,
|
||||
maxChunkSize: number,
|
||||
root = true,
|
||||
): Generator<ChunkWithoutID> {
|
||||
): Promise<ChunkWithoutID | undefined> {
|
||||
// Keep entire text if not over size
|
||||
if (
|
||||
(root || node.type in collapsedNodeConstructors) &&
|
||||
countTokens(node.text) < maxChunkSize
|
||||
) {
|
||||
yield {
|
||||
content: node.text,
|
||||
startLine: node.startPosition.row,
|
||||
endLine: node.endPosition.row,
|
||||
};
|
||||
if (root || node.type in collapsedNodeConstructors) {
|
||||
const tokenCount = await countTokensAsync(node.text);
|
||||
if (tokenCount < maxChunkSize) {
|
||||
return {
|
||||
content: node.text,
|
||||
startLine: node.startPosition.row,
|
||||
endLine: node.endPosition.row,
|
||||
};
|
||||
}
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
async function* getSmartCollapsedChunks(
|
||||
node: SyntaxNode,
|
||||
code: string,
|
||||
maxChunkSize: number,
|
||||
root = true,
|
||||
): AsyncGenerator<ChunkWithoutID> {
|
||||
const chunk = await maybeYieldChunk(node, code, maxChunkSize, root);
|
||||
if (chunk) {
|
||||
yield chunk;
|
||||
return;
|
||||
}
|
||||
|
||||
// If a collapsed form is defined, use that
|
||||
if (node.type in collapsedNodeConstructors) {
|
||||
yield {
|
||||
content: collapsedNodeConstructors[node.type](node, code, maxChunkSize),
|
||||
content: await collapsedNodeConstructors[node.type](node, code, maxChunkSize),
|
||||
startLine: node.startPosition.row,
|
||||
endLine: node.endPosition.row,
|
||||
};
|
||||
}
|
||||
|
||||
// Recurse (because even if collapsed version was shown, want to show the children in full somewhere)
|
||||
for (const child of node.children) {
|
||||
yield* getSmartCollapsedChunks(child, code, maxChunkSize, false);
|
||||
const generators = node.children.map((child) => getSmartCollapsedChunks(child, code, maxChunkSize, false));
|
||||
for (const generator of generators) {
|
||||
yield* generator;
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -78,7 +78,7 @@ export async function* markdownChunker(
|
|||
if (hLevel > 4) {
|
||||
const header = findHeader(content.split("\n"));
|
||||
|
||||
for (const chunk of basicChunker(content, maxChunkSize)) {
|
||||
for await (const chunk of basicChunker(content, maxChunkSize)) {
|
||||
yield {
|
||||
...chunk,
|
||||
otherMetadata: {
|
||||
|
|
|
|||
|
|
@ -3,15 +3,16 @@ import sqlite3 from "sqlite3";
|
|||
import {
|
||||
Chunk,
|
||||
EmbeddingsProvider,
|
||||
IdeInfo,
|
||||
IndexingProgressUpdate,
|
||||
SiteIndexingConfig,
|
||||
} from "../../index.js";
|
||||
import { getDocsSqlitePath, getLanceDbPath } from "../../util/paths.js";
|
||||
|
||||
import { Article, chunkArticle, pageToArticle } from "./article.js";
|
||||
import { crawlPage } from "./crawl.js";
|
||||
import { downloadFromS3, SiteIndexingResults } from "./preIndexed.js";
|
||||
import { default as configs } from "./preIndexedDocs.js";
|
||||
import preIndexedDocs from "./preIndexedDocs.js";
|
||||
import TransformersJsEmbeddingsProvider from "../embeddings/TransformersJsEmbeddingsProvider.js";
|
||||
|
||||
// Purposefully lowercase because lancedb converts
|
||||
interface LanceDbDocsRow {
|
||||
|
|
@ -29,7 +30,10 @@ interface LanceDbDocsRow {
|
|||
export class DocsService {
|
||||
private static instance: DocsService;
|
||||
private static DOCS_TABLE_NAME = "docs";
|
||||
public static preIndexedDocsEmbeddingsProvider =
|
||||
new TransformersJsEmbeddingsProvider();
|
||||
private _sqliteTable: Database | undefined;
|
||||
private docsIndexingQueue: Set<string> = new Set();
|
||||
|
||||
public static getInstance(): DocsService {
|
||||
if (!DocsService.instance) {
|
||||
|
|
@ -69,32 +73,35 @@ export class DocsService {
|
|||
nested = false,
|
||||
): Promise<Chunk[]> {
|
||||
const lance = await this.getLanceDb();
|
||||
const db = await this.getSqliteTable();
|
||||
const tableNames = await lance.tableNames();
|
||||
const preIndexedDoc = preIndexedDocs[baseUrl];
|
||||
const isPreIndexedDoc = !!preIndexedDoc;
|
||||
let shouldDownloadPreIndexedDoc =
|
||||
!tableNames.includes(DocsService.DOCS_TABLE_NAME) && isPreIndexedDoc;
|
||||
|
||||
const downloadDocs = async () => {
|
||||
const config = configs.find((config) => config.startUrl === baseUrl);
|
||||
if (config) {
|
||||
await this.downloadPreIndexedDocs(embeddingsProviderId, config.title);
|
||||
return await this.retrieve(
|
||||
baseUrl,
|
||||
vector,
|
||||
nRetrieve,
|
||||
embeddingsProviderId,
|
||||
true,
|
||||
);
|
||||
}
|
||||
return undefined;
|
||||
const downloadAndRetrievePreIndexedDoc = async (
|
||||
preIndexedDoc: SiteIndexingConfig,
|
||||
) => {
|
||||
await this.downloadAndAddPreIndexedDocs(
|
||||
embeddingsProviderId,
|
||||
preIndexedDoc.title,
|
||||
);
|
||||
|
||||
return await this.retrieve(
|
||||
baseUrl,
|
||||
vector,
|
||||
nRetrieve,
|
||||
embeddingsProviderId,
|
||||
true,
|
||||
);
|
||||
};
|
||||
|
||||
const tableNames = await lance.tableNames();
|
||||
if (!tableNames.includes(DocsService.DOCS_TABLE_NAME)) {
|
||||
const downloaded = await downloadDocs();
|
||||
if (downloaded) {
|
||||
return downloaded;
|
||||
}
|
||||
if (shouldDownloadPreIndexedDoc) {
|
||||
return await downloadAndRetrievePreIndexedDoc(preIndexedDoc!);
|
||||
}
|
||||
|
||||
const table = await lance.openTable(DocsService.DOCS_TABLE_NAME);
|
||||
|
||||
let docs: LanceDbDocsRow[] = await table
|
||||
.search(vector)
|
||||
.limit(nRetrieve)
|
||||
|
|
@ -103,11 +110,11 @@ export class DocsService {
|
|||
|
||||
docs = docs.filter((doc) => doc.baseurl === baseUrl);
|
||||
|
||||
if ((!docs || docs.length === 0) && !nested) {
|
||||
const downloaded = await downloadDocs();
|
||||
if (downloaded) {
|
||||
return downloaded;
|
||||
}
|
||||
shouldDownloadPreIndexedDoc =
|
||||
(!docs || docs.length === 0) && !nested && isPreIndexedDoc;
|
||||
|
||||
if (shouldDownloadPreIndexedDoc) {
|
||||
return await downloadAndRetrievePreIndexedDoc(preIndexedDoc!);
|
||||
}
|
||||
|
||||
return docs.map((doc) => ({
|
||||
|
|
@ -183,7 +190,7 @@ export class DocsService {
|
|||
return !!doc;
|
||||
}
|
||||
|
||||
private async downloadPreIndexedDocs(
|
||||
private async downloadAndAddPreIndexedDocs(
|
||||
embeddingsProviderId: string,
|
||||
title: string,
|
||||
) {
|
||||
|
|
@ -204,10 +211,16 @@ export class DocsService {
|
|||
async *indexAndAdd(
|
||||
siteIndexingConfig: SiteIndexingConfig,
|
||||
embeddingsProvider: EmbeddingsProvider,
|
||||
reIndex: boolean = false,
|
||||
): AsyncGenerator<IndexingProgressUpdate> {
|
||||
const startUrl = new URL(siteIndexingConfig.startUrl);
|
||||
const startUrl = new URL(siteIndexingConfig.startUrl.toString());
|
||||
|
||||
if (await this.has(siteIndexingConfig.startUrl.toString())) {
|
||||
if (this.docsIndexingQueue.has(startUrl.toString())) {
|
||||
console.log("Already in queue");
|
||||
return;
|
||||
}
|
||||
|
||||
if (!reIndex && (await this.has(startUrl.toString()))) {
|
||||
yield {
|
||||
progress: 1,
|
||||
desc: "Already indexed",
|
||||
|
|
@ -216,6 +229,9 @@ export class DocsService {
|
|||
return;
|
||||
}
|
||||
|
||||
// Mark the site as currently being indexed
|
||||
this.docsIndexingQueue.add(startUrl.toString());
|
||||
|
||||
yield {
|
||||
progress: 0,
|
||||
desc: "Finding subpages",
|
||||
|
|
@ -223,41 +239,56 @@ export class DocsService {
|
|||
};
|
||||
|
||||
const articles: Article[] = [];
|
||||
let processedPages = 0;
|
||||
let maxKnownPages = 1;
|
||||
|
||||
// Crawl pages and retrieve info as articles
|
||||
for await (const page of crawlPage(startUrl, siteIndexingConfig.maxDepth)) {
|
||||
processedPages++;
|
||||
const article = pageToArticle(page);
|
||||
if (!article) {
|
||||
continue;
|
||||
}
|
||||
articles.push(article);
|
||||
|
||||
// Use a heuristic approach for progress calculation
|
||||
const progress = Math.min(processedPages / maxKnownPages, 1);
|
||||
|
||||
yield {
|
||||
progress: 0,
|
||||
progress, // Yield the heuristic progress
|
||||
desc: `Finding subpages (${page.path})`,
|
||||
status: "indexing",
|
||||
};
|
||||
|
||||
// Increase maxKnownPages to delay progress reaching 100% too soon
|
||||
if (processedPages === maxKnownPages) {
|
||||
maxKnownPages *= 2;
|
||||
}
|
||||
}
|
||||
|
||||
const chunks: Chunk[] = [];
|
||||
const embeddings: number[][] = [];
|
||||
|
||||
// Create embeddings of retrieved articles
|
||||
console.log("Creating Embeddings for ", articles.length, " articles");
|
||||
for (const article of articles) {
|
||||
console.log(`Creating embeddings for ${articles.length} articles`);
|
||||
|
||||
for (let i = 0; i < articles.length; i++) {
|
||||
const article = articles[i];
|
||||
yield {
|
||||
progress: Math.max(1, Math.floor(100 / (articles.length + 1))),
|
||||
desc: `${article.subpath}`,
|
||||
progress: i / articles.length,
|
||||
desc: `Creating Embeddings: ${article.subpath}`,
|
||||
status: "indexing",
|
||||
};
|
||||
|
||||
try {
|
||||
const subpathEmbeddings = await embeddingsProvider.embed(
|
||||
chunkArticle(article).map((chunk) => {
|
||||
chunks.push(chunk);
|
||||
chunkArticle(article, embeddingsProvider.maxChunkSize).map(
|
||||
(chunk) => {
|
||||
chunks.push(chunk);
|
||||
|
||||
return chunk.content;
|
||||
}),
|
||||
return chunk.content;
|
||||
},
|
||||
),
|
||||
);
|
||||
|
||||
embeddings.push(...subpathEmbeddings);
|
||||
|
|
@ -268,7 +299,20 @@ export class DocsService {
|
|||
|
||||
// Add docs to databases
|
||||
console.log("Adding ", embeddings.length, " embeddings to db");
|
||||
yield {
|
||||
progress: 0.5,
|
||||
desc: `Adding ${embeddings.length} embeddings to db`,
|
||||
status: "indexing",
|
||||
};
|
||||
|
||||
// Clear old index if re-indexing.
|
||||
if (reIndex) {
|
||||
console.log("Deleting old embeddings");
|
||||
await this.delete(startUrl.toString());
|
||||
}
|
||||
|
||||
await this.add(siteIndexingConfig.title, startUrl, chunks, embeddings);
|
||||
this.docsIndexingQueue.delete(startUrl.toString());
|
||||
|
||||
yield {
|
||||
progress: 1,
|
||||
|
|
@ -276,4 +320,15 @@ export class DocsService {
|
|||
status: "done",
|
||||
};
|
||||
}
|
||||
|
||||
public isJetBrainsAndPreIndexedDocsProvider(
|
||||
ideInfo: IdeInfo,
|
||||
embeddingsProviderId: EmbeddingsProvider["id"],
|
||||
): boolean {
|
||||
const isJetBrains = ideInfo.ideType === "jetbrains";
|
||||
const isPreIndexedDocsProvider =
|
||||
embeddingsProviderId === DocsService.preIndexedDocsEmbeddingsProvider.id;
|
||||
|
||||
return isJetBrains && isPreIndexedDocsProvider;
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,7 +1,6 @@
|
|||
import { Readability } from "@mozilla/readability";
|
||||
import { JSDOM } from "jsdom";
|
||||
import { Chunk } from "../../index.js";
|
||||
import { MAX_CHUNK_SIZE } from "../../llm/constants.js";
|
||||
import { cleanFragment, cleanHeader } from "../chunk/markdown.js";
|
||||
import { PageData } from "./crawl.js";
|
||||
|
||||
|
|
@ -21,6 +20,7 @@ function breakdownArticleComponent(
|
|||
url: string,
|
||||
article: ArticleComponent,
|
||||
subpath: string,
|
||||
max_chunk_size: number,
|
||||
): Chunk[] {
|
||||
const chunks: Chunk[] = [];
|
||||
|
||||
|
|
@ -32,7 +32,7 @@ function breakdownArticleComponent(
|
|||
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
const line = lines[i];
|
||||
if (content.length + line.length <= MAX_CHUNK_SIZE) {
|
||||
if (content.length + line.length <= max_chunk_size) {
|
||||
content += `${line}\n`;
|
||||
endLine = i;
|
||||
} else {
|
||||
|
|
@ -79,7 +79,10 @@ function breakdownArticleComponent(
|
|||
return chunks.filter((c) => c.content.trim().length > 20);
|
||||
}
|
||||
|
||||
export function chunkArticle(articleResult: Article): Chunk[] {
|
||||
export function chunkArticle(
|
||||
articleResult: Article,
|
||||
maxChunkSize: number,
|
||||
): Chunk[] {
|
||||
let chunks: Chunk[] = [];
|
||||
|
||||
for (const article of articleResult.article_components) {
|
||||
|
|
@ -87,6 +90,7 @@ export function chunkArticle(articleResult: Article): Chunk[] {
|
|||
articleResult.url,
|
||||
article,
|
||||
articleResult.subpath,
|
||||
maxChunkSize,
|
||||
);
|
||||
chunks = [...chunks, ...articleChunks];
|
||||
}
|
||||
|
|
|
|||
|
|
@ -16,7 +16,7 @@ const IGNORE_PATHS_ENDING_IN = [
|
|||
"changelog.html",
|
||||
];
|
||||
|
||||
const GITHUB_PATHS_TO_TRAVERSE = ["/blob/", "/tree/"];
|
||||
const markdownRegex = new RegExp(/\.(md|mdx)$/);
|
||||
|
||||
async function getDefaultBranch(owner: string, repo: string): Promise<string> {
|
||||
const octokit = new Octokit({ auth: undefined });
|
||||
|
|
@ -53,7 +53,10 @@ async function crawlGithubRepo(baseUrl: URL) {
|
|||
);
|
||||
|
||||
const paths = tree.data.tree
|
||||
.filter((file: any) => file.type === "blob" && file.path?.endsWith(".md"))
|
||||
.filter(
|
||||
(file: any) =>
|
||||
file.type === "blob" && markdownRegex.test(file.path ?? ""),
|
||||
)
|
||||
.map((file: any) => baseUrl.pathname + "/tree/main/" + file.path);
|
||||
|
||||
return paths;
|
||||
|
|
@ -142,7 +145,10 @@ export async function* crawlPage(
|
|||
url: URL,
|
||||
maxDepth: number = 3,
|
||||
): AsyncGenerator<PageData> {
|
||||
console.log("Starting crawl from: ", url, " - Max Depth: ", maxDepth);
|
||||
console.log(
|
||||
`Starting crawl from: ${url.toString()} - Max Depth: ${maxDepth}`,
|
||||
);
|
||||
|
||||
const { baseUrl, basePath } = splitUrl(url);
|
||||
let paths: { path: string; depth: number }[] = [{ path: basePath, depth: 0 }];
|
||||
|
||||
|
|
|
|||
|
|
@ -1,301 +1,305 @@
|
|||
/* eslint-disable @typescript-eslint/naming-convention */
|
||||
import { SiteIndexingConfig } from "../../index.js";
|
||||
|
||||
const configs: SiteIndexingConfig[] = [
|
||||
{
|
||||
const preIndexedDocs: Record<
|
||||
SiteIndexingConfig["startUrl"],
|
||||
SiteIndexingConfig
|
||||
> = {
|
||||
"https://jinja.palletsprojects.com/en/3.1.x/": {
|
||||
title: "Jinja",
|
||||
startUrl: "https://jinja.palletsprojects.com/en/3.1.x/",
|
||||
rootUrl: "https://jinja.palletsprojects.com/en/3.1.x/",
|
||||
faviconUrl: "https://jinja.palletsprojects.com/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://react.dev/reference/": {
|
||||
title: "React",
|
||||
startUrl: "https://react.dev/reference/",
|
||||
rootUrl: "https://react.dev/reference/",
|
||||
faviconUrl: "https://react.dev/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://posthog.com/docs": {
|
||||
title: "PostHog",
|
||||
startUrl: "https://posthog.com/docs",
|
||||
rootUrl: "https://posthog.com/docs",
|
||||
faviconUrl: "https://posthog.com/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://expressjs.com/en/5x/api.html": {
|
||||
title: "Express",
|
||||
startUrl: "https://expressjs.com/en/5x/api.html",
|
||||
rootUrl: "https://expressjs.com/en/5x/",
|
||||
faviconUrl: "https://expressjs.com/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://platform.openai.com/docs/": {
|
||||
title: "OpenAI",
|
||||
startUrl: "https://platform.openai.com/docs/",
|
||||
rootUrl: "https://platform.openai.com/docs/",
|
||||
faviconUrl: "https://platform.openai.com/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://www.prisma.io/docs": {
|
||||
title: "Prisma",
|
||||
startUrl: "https://www.prisma.io/docs",
|
||||
rootUrl: "https://www.prisma.io/docs",
|
||||
faviconUrl: "https://www.prisma.io/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://boto3.amazonaws.com/v1/documentation/api/latest/index.html": {
|
||||
title: "Boto3",
|
||||
startUrl:
|
||||
"https://boto3.amazonaws.com/v1/documentation/api/latest/index.html",
|
||||
rootUrl: "https://boto3.amazonaws.com/v1/documentation/api/latest/",
|
||||
faviconUrl: "https://boto3.amazonaws.com/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://pytorch.org/docs/stable/": {
|
||||
title: "Pytorch",
|
||||
startUrl: "https://pytorch.org/docs/stable/",
|
||||
rootUrl: "https://pytorch.org/docs/stable/",
|
||||
faviconUrl: "https://pytorch.org/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://redis.io/docs/": {
|
||||
title: "Redis",
|
||||
startUrl: "https://redis.io/docs/",
|
||||
rootUrl: "https://redis.io/docs/",
|
||||
faviconUrl: "https://redis.io/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://axios-http.com/docs/intro": {
|
||||
title: "Axios",
|
||||
startUrl: "https://axios-http.com/docs/intro",
|
||||
rootUrl: "https://axios-http.com/docs",
|
||||
faviconUrl: "https://axios-http.com/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://redwoodjs.com/docs/introduction": {
|
||||
title: "Redwood JS",
|
||||
startUrl: "https://redwoodjs.com/docs/introduction",
|
||||
rootUrl: "https://redwoodjs.com/docs",
|
||||
faviconUrl: "https://redwoodjs.com/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://graphql.org/learn/": {
|
||||
title: "GraphQL",
|
||||
startUrl: "https://graphql.org/learn/",
|
||||
rootUrl: "https://graphql.org/learn/",
|
||||
faviconUrl: "https://graphql.org/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://www.typescriptlang.org/docs/": {
|
||||
title: "Typescript",
|
||||
startUrl: "https://www.typescriptlang.org/docs/",
|
||||
rootUrl: "https://www.typescriptlang.org/docs/",
|
||||
faviconUrl: "https://www.typescriptlang.org/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://jestjs.io/docs/getting-started": {
|
||||
title: "Jest",
|
||||
startUrl: "https://jestjs.io/docs/getting-started",
|
||||
rootUrl: "https://jestjs.io/docs",
|
||||
faviconUrl: "https://jestjs.io/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://tailwindcss.com/docs/installation": {
|
||||
title: "Tailwind CSS",
|
||||
startUrl: "https://tailwindcss.com/docs/installation",
|
||||
rootUrl: "https://tailwindcss.com/docs",
|
||||
faviconUrl: "https://tailwindcss.com/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://vuejs.org/guide/introduction.html": {
|
||||
title: "Vue.js",
|
||||
startUrl: "https://vuejs.org/guide/introduction.html",
|
||||
rootUrl: "https://vuejs.org",
|
||||
faviconUrl: "https://vuejs.org/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://svelte.dev/docs/introduction": {
|
||||
title: "Svelte",
|
||||
startUrl: "https://svelte.dev/docs/introduction",
|
||||
rootUrl: "https://svelte.dev/docs",
|
||||
faviconUrl: "https://svelte.dev/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://docs.github.com/en/actions": {
|
||||
title: "GitHub Actions",
|
||||
startUrl: "https://docs.github.com/en/actions",
|
||||
rootUrl: "https://docs.github.com/en/actions",
|
||||
faviconUrl: "https://docs.github.com/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://nodejs.org/docs/latest/api/": {
|
||||
title: "NodeJS",
|
||||
startUrl: "https://nodejs.org/docs/latest/api/",
|
||||
rootUrl: "https://nodejs.org/docs/latest/api/",
|
||||
faviconUrl: "https://nodejs.org/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://socket.io/docs/v4/": {
|
||||
title: "Socket.io",
|
||||
startUrl: "https://socket.io/docs/v4/",
|
||||
rootUrl: "https://socket.io/docs/v4/",
|
||||
faviconUrl: "https://socket.io/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://docs.gradle.org/current/userguide/userguide.html": {
|
||||
title: "Gradle",
|
||||
startUrl: "https://docs.gradle.org/current/userguide/userguide.html",
|
||||
rootUrl: "https://docs.gradle.org/current",
|
||||
faviconUrl: "https://docs.gradle.org/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://redux-toolkit.js.org/introduction/getting-started": {
|
||||
title: "Redux Toolkit",
|
||||
startUrl: "https://redux-toolkit.js.org/introduction/getting-started",
|
||||
rootUrl: "https://redux-toolkit.js.org",
|
||||
faviconUrl: "https://redux-toolkit.js.org/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://docs.trychroma.com/": {
|
||||
title: "Chroma",
|
||||
startUrl: "https://docs.trychroma.com/",
|
||||
rootUrl: "https://docs.trychroma.com/",
|
||||
faviconUrl: "https://docs.trychroma.com/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://www.sqlite.org/docs.html": {
|
||||
title: "SQLite",
|
||||
startUrl: "https://www.sqlite.org/docs.html",
|
||||
rootUrl: "https://www.sqlite.org",
|
||||
faviconUrl: "https://www.sqlite.org/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://redux.js.org/introduction/getting-started": {
|
||||
title: "Redux",
|
||||
startUrl: "https://redux.js.org/introduction/getting-started",
|
||||
rootUrl: "https://redux.js.org",
|
||||
faviconUrl: "https://redux.js.org/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://prettier.io/docs/en/": {
|
||||
title: "Prettier",
|
||||
startUrl: "https://prettier.io/docs/en/",
|
||||
rootUrl: "https://prettier.io/docs/en/",
|
||||
faviconUrl: "https://prettier.io/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://code.visualstudio.com/api": {
|
||||
title: "VS Code Extension API",
|
||||
startUrl: "https://code.visualstudio.com/api",
|
||||
rootUrl: "https://code.visualstudio.com/api",
|
||||
faviconUrl: "https://code.visualstudio.com/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://docs.continue.dev/intro": {
|
||||
title: "Continue",
|
||||
startUrl: "https://docs.continue.dev/intro",
|
||||
rootUrl: "https://docs.continue.dev",
|
||||
faviconUrl: "https://docs.continue.dev/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://api.jquery.com/": {
|
||||
title: "jQuery",
|
||||
startUrl: "https://api.jquery.com/",
|
||||
rootUrl: "https://api.jquery.com/",
|
||||
faviconUrl: "https://api.jquery.com/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://docs.python.org/3/": {
|
||||
title: "Python",
|
||||
startUrl: "https://docs.python.org/3/",
|
||||
rootUrl: "https://docs.python.org/3/",
|
||||
faviconUrl: "https://docs.python.org/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://doc.rust-lang.org/book/": {
|
||||
title: "Rust",
|
||||
startUrl: "https://doc.rust-lang.org/book/",
|
||||
rootUrl: "https://doc.rust-lang.org/book/",
|
||||
faviconUrl: "https://doc.rust-lang.org/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://plugins.jetbrains.com/docs/intellij/welcome.html": {
|
||||
title: "IntelliJ Platform SDK",
|
||||
startUrl: "https://plugins.jetbrains.com/docs/intellij/welcome.html",
|
||||
rootUrl: "https://plugins.jetbrains.com/docs/intellij",
|
||||
faviconUrl: "https://plugins.jetbrains.com/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://docs.docker.com/": {
|
||||
title: "Docker",
|
||||
startUrl: "https://docs.docker.com/",
|
||||
rootUrl: "https://docs.docker.com/",
|
||||
faviconUrl: "https://docs.docker.com/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://docs.npmjs.com/": {
|
||||
title: "NPM",
|
||||
startUrl: "https://docs.npmjs.com/",
|
||||
rootUrl: "https://docs.npmjs.com/",
|
||||
faviconUrl: "https://docs.npmjs.com/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://tiptap.dev/docs/editor/introduction": {
|
||||
title: "TipTap",
|
||||
startUrl: "https://tiptap.dev/docs/editor/introduction",
|
||||
rootUrl: "https://tiptap.dev/docs",
|
||||
faviconUrl: "https://tiptap.dev/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://esbuild.github.io/": {
|
||||
title: "esbuild",
|
||||
startUrl: "https://esbuild.github.io/",
|
||||
rootUrl: "https://esbuild.github.io/",
|
||||
faviconUrl: "https://esbuild.github.io/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://tree-sitter.github.io/tree-sitter/": {
|
||||
title: "Tree Sitter",
|
||||
startUrl: "https://tree-sitter.github.io/tree-sitter/",
|
||||
rootUrl: "https://tree-sitter.github.io/tree-sitter/",
|
||||
faviconUrl: "https://tree-sitter.github.io/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://docs.netlify.com/": {
|
||||
title: "Netlify",
|
||||
startUrl: "https://docs.netlify.com/",
|
||||
rootUrl: "https://docs.netlify.com/",
|
||||
faviconUrl: "https://docs.netlify.com/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://replicate.com/docs": {
|
||||
title: "Replicate",
|
||||
startUrl: "https://replicate.com/docs",
|
||||
rootUrl: "https://replicate.com/docs",
|
||||
faviconUrl: "https://replicate.com/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://www.w3schools.com/html/default.asp": {
|
||||
title: "HTML",
|
||||
startUrl: "https://www.w3schools.com/html/default.asp",
|
||||
rootUrl: "https://www.w3schools.com/html",
|
||||
faviconUrl: "https://www.w3schools.com/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://www.w3schools.com/css/default.asp": {
|
||||
title: "CSS",
|
||||
startUrl: "https://www.w3schools.com/css/default.asp",
|
||||
rootUrl: "https://www.w3schools.com/css",
|
||||
faviconUrl: "https://www.w3schools.com/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://python.langchain.com/docs/get_started/introduction": {
|
||||
title: "Langchain",
|
||||
startUrl: "https://python.langchain.com/docs/get_started/introduction",
|
||||
rootUrl: "https://python.langchain.com/docs",
|
||||
faviconUrl: "https://python.langchain.com/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://developer.woocommerce.com/docs/": {
|
||||
title: "WooCommerce",
|
||||
startUrl: "https://developer.woocommerce.com/docs/",
|
||||
rootUrl: "https://developer.woocommerce.com/docs/",
|
||||
faviconUrl: "https://developer.woocommerce.com/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://developer.wordpress.org/reference/": {
|
||||
title: "WordPress",
|
||||
startUrl: "https://developer.wordpress.org/reference/",
|
||||
rootUrl: "https://developer.wordpress.org/reference/",
|
||||
faviconUrl: "https://developer.wordpress.org/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://doc.qt.io/qtforpython-6/quickstart.html": {
|
||||
title: "PySide6",
|
||||
startUrl: "https://doc.qt.io/qtforpython-6/quickstart.html",
|
||||
rootUrl: "https://doc.qt.io/qtforpython-6/api.html",
|
||||
faviconUrl: "https://doc.qt.io/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://getbootstrap.com/docs/5.3/getting-started/introduction/": {
|
||||
title: "Bootstrap",
|
||||
startUrl: "https://getbootstrap.com/docs/5.3/getting-started/introduction/",
|
||||
rootUrl: "https://getbootstrap.com/docs/5.3/",
|
||||
faviconUrl: "https://getbootstrap.com/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://alpinejs.dev/start-here": {
|
||||
title: "Alpine.js",
|
||||
startUrl: "https://alpinejs.dev/start-here",
|
||||
rootUrl: "https://alpinejs.dev/",
|
||||
faviconUrl: "https://alpinejs.dev/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://learn.microsoft.com/en-us/dotnet/csharp/": {
|
||||
title: "C# Language Reference",
|
||||
startUrl: "https://learn.microsoft.com/en-us/dotnet/csharp/",
|
||||
rootUrl: "https://learn.microsoft.com/en-us/dotnet/csharp/",
|
||||
faviconUrl: "https://learn.microsoft.com/favicon.ico",
|
||||
},
|
||||
{
|
||||
"https://docs.godotengine.org/en/latest/": {
|
||||
title: "Godot",
|
||||
startUrl: "https://docs.godotengine.org/en/latest/",
|
||||
rootUrl: "https://docs.godotengine.org/en/latest/",
|
||||
faviconUrl: "https://godotengine.org/favicon.ico",
|
||||
},
|
||||
];
|
||||
};
|
||||
|
||||
export default configs;
|
||||
export default preIndexedDocs;
|
||||
|
|
|
|||
|
|
@ -1,9 +1,12 @@
|
|||
import {
|
||||
EmbedOptions,
|
||||
EmbeddingsProvider,
|
||||
EmbeddingsProviderName,
|
||||
FetchFunction,
|
||||
} from "../../index.js";
|
||||
|
||||
import { MAX_CHUNK_SIZE } from "../../llm/constants.js";
|
||||
|
||||
export interface IBaseEmbeddingsProvider extends EmbeddingsProvider {
|
||||
options: EmbedOptions;
|
||||
fetch: FetchFunction;
|
||||
|
|
@ -15,6 +18,11 @@ abstract class BaseEmbeddingsProvider implements IBaseEmbeddingsProvider {
|
|||
static maxBatchSize: IBaseEmbeddingsProvider["maxBatchSize"];
|
||||
static defaultOptions: IBaseEmbeddingsProvider["defaultOptions"];
|
||||
|
||||
static providerName: EmbeddingsProviderName;
|
||||
get providerName(): EmbeddingsProviderName {
|
||||
return (this.constructor as typeof BaseEmbeddingsProvider).providerName;
|
||||
}
|
||||
|
||||
options: IBaseEmbeddingsProvider["options"];
|
||||
fetch: IBaseEmbeddingsProvider["fetch"];
|
||||
id: IBaseEmbeddingsProvider["id"];
|
||||
|
|
@ -29,11 +37,22 @@ abstract class BaseEmbeddingsProvider implements IBaseEmbeddingsProvider {
|
|||
...options,
|
||||
};
|
||||
this.fetch = fetch;
|
||||
this.id = `${this.constructor.name}::${this.options.model}`;
|
||||
// Include the `max_chunk_size` if it is not the default, since we need to create other indices for different chunk_sizes
|
||||
if (this.maxChunkSize !== MAX_CHUNK_SIZE) {
|
||||
this.id = `${this.constructor.name}::${this.options.model}::${this.maxChunkSize}`;
|
||||
} else {
|
||||
this.id = `${this.constructor.name}::${this.options.model}`;
|
||||
}
|
||||
}
|
||||
defaultOptions?: EmbedOptions | undefined;
|
||||
maxBatchSize?: number | undefined;
|
||||
|
||||
abstract embed(chunks: string[]): Promise<number[][]>;
|
||||
|
||||
get maxChunkSize(): number {
|
||||
return this.options.maxChunkSize ?? MAX_CHUNK_SIZE;
|
||||
}
|
||||
|
||||
static getBatchedChunks(chunks: string[]): string[][] {
|
||||
if (!this.maxBatchSize) {
|
||||
console.warn(
|
||||
|
|
|
|||
|
|
@ -1,11 +1,13 @@
|
|||
import { Response } from "node-fetch";
|
||||
import { EmbedOptions } from "../../index.js";
|
||||
import { EmbeddingsProviderName, EmbedOptions } from "../../index.js";
|
||||
import { withExponentialBackoff } from "../../util/withExponentialBackoff.js";
|
||||
import BaseEmbeddingsProvider from "./BaseEmbeddingsProvider.js";
|
||||
|
||||
class CohereEmbeddingsProvider extends BaseEmbeddingsProvider {
|
||||
static maxBatchSize = 96;
|
||||
|
||||
static providerName: EmbeddingsProviderName = "cohere";
|
||||
|
||||
static defaultOptions: Partial<EmbedOptions> | undefined = {
|
||||
apiBase: "https://api.cohere.ai/v1/",
|
||||
model: "embed-english-v3.0",
|
||||
|
|
|
|||
|
|
@ -0,0 +1,25 @@
|
|||
import { EmbeddingsProviderName, EmbedOptions } from "../..";
|
||||
import { CONTROL_PLANE_URL } from "../../control-plane/client";
|
||||
import OpenAIEmbeddingsProvider from "./OpenAIEmbeddingsProvider";
|
||||
|
||||
class ContinueProxyEmbeddingsProvider extends OpenAIEmbeddingsProvider {
|
||||
static providerName: EmbeddingsProviderName = "continue-proxy";
|
||||
static defaultOptions: Partial<EmbedOptions> | undefined = {
|
||||
apiBase: new URL("/model-proxy/v1", CONTROL_PLANE_URL).toString(),
|
||||
};
|
||||
|
||||
private _workOsAccessToken: string | undefined = undefined;
|
||||
|
||||
get workOsAccessToken(): string | undefined {
|
||||
return this._workOsAccessToken;
|
||||
}
|
||||
|
||||
set workOsAccessToken(value: string | undefined) {
|
||||
if (this._workOsAccessToken !== value) {
|
||||
this._workOsAccessToken = value;
|
||||
this.options.apiKey = value;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export default ContinueProxyEmbeddingsProvider;
|
||||
|
|
@ -1,8 +1,9 @@
|
|||
import { EmbedOptions } from "../../index.js";
|
||||
import { EmbeddingsProviderName, EmbedOptions } from "../../index.js";
|
||||
import { withExponentialBackoff } from "../../util/withExponentialBackoff.js";
|
||||
import BaseEmbeddingsProvider from "./BaseEmbeddingsProvider.js";
|
||||
|
||||
class DeepInfraEmbeddingsProvider extends BaseEmbeddingsProvider {
|
||||
static providerName: EmbeddingsProviderName = "deepinfra";
|
||||
static defaultOptions: Partial<EmbedOptions> | undefined = {
|
||||
model: "sentence-transformers/all-MiniLM-L6-v2",
|
||||
};
|
||||
|
|
|
|||
|
|
@ -1,11 +1,16 @@
|
|||
import { Response } from "node-fetch";
|
||||
import { getHeaders } from "../../continueServer/stubs/headers.js";
|
||||
import { constants } from "../../deploy/constants.js";
|
||||
import { EmbedOptions, FetchFunction } from "../../index.js";
|
||||
import {
|
||||
EmbeddingsProviderName,
|
||||
EmbedOptions,
|
||||
FetchFunction,
|
||||
} from "../../index.js";
|
||||
import { withExponentialBackoff } from "../../util/withExponentialBackoff.js";
|
||||
import BaseEmbeddingsProvider from "./BaseEmbeddingsProvider.js";
|
||||
|
||||
class FreeTrialEmbeddingsProvider extends BaseEmbeddingsProvider {
|
||||
static providerName: EmbeddingsProviderName = "free-trial";
|
||||
static maxBatchSize = 128;
|
||||
|
||||
static defaultOptions: Partial<EmbedOptions> | undefined = {
|
||||
|
|
|
|||
|
|
@ -1,17 +1,17 @@
|
|||
import { Response } from "node-fetch";
|
||||
import { withExponentialBackoff } from "../../util/withExponentialBackoff.js";
|
||||
import BaseEmbeddingsProvider, {
|
||||
IBaseEmbeddingsProvider,
|
||||
} from "./BaseEmbeddingsProvider.js";
|
||||
import {
|
||||
EmbedContentRequest,
|
||||
EmbedContentResponse,
|
||||
} from "@google/generative-ai";
|
||||
import { Response } from "node-fetch";
|
||||
import { EmbeddingsProviderName } from "../../index.js";
|
||||
import { withExponentialBackoff } from "../../util/withExponentialBackoff.js";
|
||||
import BaseEmbeddingsProvider from "./BaseEmbeddingsProvider.js";
|
||||
|
||||
/**
|
||||
* [View the Gemini Text Embedding docs.](https://ai.google.dev/gemini-api/docs/models/gemini#text-embedding-and-embedding)
|
||||
*/
|
||||
class GeminiEmbeddingsProvider extends BaseEmbeddingsProvider {
|
||||
static providerName: EmbeddingsProviderName = "gemini";
|
||||
static maxBatchSize = 2048;
|
||||
|
||||
static defaultOptions = {
|
||||
|
|
|
|||
|
|
@ -1,10 +1,11 @@
|
|||
import fetch, { Response } from "node-fetch";
|
||||
import { EmbedOptions, FetchFunction } from "../..";
|
||||
import { Response } from "node-fetch";
|
||||
import { EmbeddingsProviderName, EmbedOptions, FetchFunction } from "../..";
|
||||
import { withExponentialBackoff } from "../../util/withExponentialBackoff";
|
||||
import BaseEmbeddingsProvider from "./BaseEmbeddingsProvider";
|
||||
|
||||
class HuggingFaceTEIEmbeddingsProvider extends BaseEmbeddingsProvider {
|
||||
private maxBatchSize = 32;
|
||||
static providerName: EmbeddingsProviderName = "huggingface-tei";
|
||||
maxBatchSize = 32;
|
||||
|
||||
static defaultOptions: Partial<EmbedOptions> | undefined = {
|
||||
apiBase: "http://localhost:8080",
|
||||
|
|
@ -17,7 +18,7 @@ class HuggingFaceTEIEmbeddingsProvider extends BaseEmbeddingsProvider {
|
|||
if (!this.options.apiBase?.endsWith("/")) {
|
||||
this.options.apiBase += "/";
|
||||
}
|
||||
this.doInfoRequest().then(response => {
|
||||
this.doInfoRequest().then((response) => {
|
||||
this.options.model = response.model_id;
|
||||
this.maxBatchSize = response.max_client_batch_size;
|
||||
});
|
||||
|
|
@ -26,7 +27,9 @@ class HuggingFaceTEIEmbeddingsProvider extends BaseEmbeddingsProvider {
|
|||
async embed(chunks: string[]) {
|
||||
const promises = [];
|
||||
for (let i = 0; i < chunks.length; i += this.maxBatchSize) {
|
||||
promises.push(this.doEmbedRequest(chunks.slice(i, i + this.maxBatchSize)));
|
||||
promises.push(
|
||||
this.doEmbedRequest(chunks.slice(i, i + this.maxBatchSize)),
|
||||
);
|
||||
}
|
||||
const results = await Promise.all(promises);
|
||||
return results.flat();
|
||||
|
|
@ -37,11 +40,11 @@ class HuggingFaceTEIEmbeddingsProvider extends BaseEmbeddingsProvider {
|
|||
this.fetch(new URL("embed", this.options.apiBase), {
|
||||
method: "POST",
|
||||
body: JSON.stringify({
|
||||
inputs: batch
|
||||
inputs: batch,
|
||||
}),
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
},
|
||||
}),
|
||||
);
|
||||
if (!resp.ok) {
|
||||
|
|
@ -75,9 +78,9 @@ class TEIEmbedError extends Error {
|
|||
}
|
||||
|
||||
type TEIEmbedErrorResponse = {
|
||||
error: string
|
||||
error_type: string
|
||||
}
|
||||
error: string;
|
||||
error_type: string;
|
||||
};
|
||||
|
||||
type TEIInfoResponse = {
|
||||
model_id: string;
|
||||
|
|
@ -86,7 +89,7 @@ type TEIInfoResponse = {
|
|||
model_type: {
|
||||
embedding: {
|
||||
pooling: string;
|
||||
}
|
||||
};
|
||||
};
|
||||
max_concurrent_requests: number;
|
||||
max_input_length: number;
|
||||
|
|
|
|||
|
|
@ -1,4 +1,8 @@
|
|||
import { EmbedOptions, FetchFunction } from "../../index.js";
|
||||
import {
|
||||
EmbeddingsProviderName,
|
||||
EmbedOptions,
|
||||
FetchFunction,
|
||||
} from "../../index.js";
|
||||
import { withExponentialBackoff } from "../../util/withExponentialBackoff.js";
|
||||
import BaseEmbeddingsProvider, {
|
||||
IBaseEmbeddingsProvider,
|
||||
|
|
@ -10,12 +14,21 @@ async function embedOne(
|
|||
customFetch: FetchFunction,
|
||||
) {
|
||||
const embedding = await withExponentialBackoff<number[]>(async () => {
|
||||
const resp = await customFetch(new URL("api/embeddings", options.apiBase), {
|
||||
let apiBase = options.apiBase!;
|
||||
|
||||
if (!apiBase.endsWith("/")) {
|
||||
apiBase += "/";
|
||||
}
|
||||
|
||||
const resp = await customFetch(new URL("api/embeddings", apiBase), {
|
||||
method: "POST",
|
||||
body: JSON.stringify({
|
||||
model: options.model,
|
||||
prompt: chunk,
|
||||
}),
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
});
|
||||
|
||||
if (!resp.ok) {
|
||||
|
|
@ -35,6 +48,7 @@ async function embedOne(
|
|||
}
|
||||
|
||||
class OllamaEmbeddingsProvider extends BaseEmbeddingsProvider {
|
||||
static providerName: EmbeddingsProviderName = "ollama";
|
||||
static defaultOptions: IBaseEmbeddingsProvider["defaultOptions"] = {
|
||||
apiBase: "http://localhost:11434/",
|
||||
model: "nomic-embed-text",
|
||||
|
|
|
|||
|
|
@ -1,9 +1,10 @@
|
|||
import { Response } from "node-fetch";
|
||||
import { EmbedOptions } from "../../index.js";
|
||||
import { EmbeddingsProviderName, EmbedOptions } from "../../index.js";
|
||||
import { withExponentialBackoff } from "../../util/withExponentialBackoff.js";
|
||||
import BaseEmbeddingsProvider from "./BaseEmbeddingsProvider.js";
|
||||
|
||||
class OpenAIEmbeddingsProvider extends BaseEmbeddingsProvider {
|
||||
static providerName: EmbeddingsProviderName = "openai";
|
||||
// https://platform.openai.com/docs/api-reference/embeddings/create is 2048
|
||||
// but Voyage is 128
|
||||
static maxBatchSize = 128;
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
import path from "path";
|
||||
import { EmbeddingsProviderName } from "../../index.js";
|
||||
// @ts-ignore
|
||||
// prettier-ignore
|
||||
import { type PipelineType } from "../../vendor/modules/@xenova/transformers/src/transformers.js";
|
||||
|
|
@ -37,10 +38,14 @@ class EmbeddingsPipeline {
|
|||
}
|
||||
|
||||
export class TransformersJsEmbeddingsProvider extends BaseEmbeddingsProvider {
|
||||
static providerName: EmbeddingsProviderName = "transformers.js";
|
||||
static maxGroupSize: number = 4;
|
||||
static model: string = "all-MiniLM-L6-v2";
|
||||
|
||||
constructor() {
|
||||
super({ model: "all-MiniLM-L6-v2" }, () => Promise.resolve(null));
|
||||
super({ model: TransformersJsEmbeddingsProvider.model }, () =>
|
||||
Promise.resolve(null),
|
||||
);
|
||||
}
|
||||
|
||||
async embed(chunks: string[]) {
|
||||
|
|
|
|||
|
|
@ -1,12 +1,14 @@
|
|||
import { EmbeddingsProviderName } from "../../index.js";
|
||||
import BaseEmbeddingsProvider from "./BaseEmbeddingsProvider.js";
|
||||
import CohereEmbeddingsProvider from "./CohereEmbeddingsProvider.js";
|
||||
import ContinueProxyEmbeddingsProvider from "./ContinueProxyEmbeddingsProvider.js";
|
||||
import DeepInfraEmbeddingsProvider from "./DeepInfraEmbeddingsProvider.js";
|
||||
import FreeTrialEmbeddingsProvider from "./FreeTrialEmbeddingsProvider.js";
|
||||
import GeminiEmbeddingsProvider from "./GeminiEmbeddingsProvider.js";
|
||||
import HuggingFaceTEIEmbeddingsProvider from "./HuggingFaceTEIEmbeddingsProvider.js";
|
||||
import OllamaEmbeddingsProvider from "./OllamaEmbeddingsProvider.js";
|
||||
import OpenAIEmbeddingsProvider from "./OpenAIEmbeddingsProvider.js";
|
||||
import TransformersJsEmbeddingsProvider from "./TransformersJsEmbeddingsProvider.js";
|
||||
import GeminiEmbeddingsProvider from "./GeminiEmbeddingsProvider.js";
|
||||
|
||||
type EmbeddingsProviderConstructor = new (
|
||||
...args: any[]
|
||||
|
|
@ -26,4 +28,6 @@ export const allEmbeddingsProviders: Record<
|
|||
// eslint-disable-next-line @typescript-eslint/naming-convention
|
||||
"huggingface-tei": HuggingFaceTEIEmbeddingsProvider,
|
||||
gemini: GeminiEmbeddingsProvider,
|
||||
"continue-proxy": ContinueProxyEmbeddingsProvider,
|
||||
deepinfra: DeepInfraEmbeddingsProvider,
|
||||
};
|
||||
|
|
|
|||
|
|
@ -64,11 +64,13 @@ export const DEFAULT_IGNORE_FILETYPES = [
|
|||
".gitkeep",
|
||||
".continueignore",
|
||||
"config.json",
|
||||
".csv",
|
||||
// "*.prompt", // can be incredibly confusing for the LLM to have another set of instructions injected into the prompt
|
||||
];
|
||||
export const defaultIgnoreFile = ignore().add(DEFAULT_IGNORE_FILETYPES);
|
||||
export const DEFAULT_IGNORE_DIRS = [
|
||||
".git",
|
||||
".svn",
|
||||
".vscode",
|
||||
".idea",
|
||||
".vs",
|
||||
|
|
|
|||
|
|
@ -44,6 +44,36 @@ export class SqliteDb {
|
|||
artifactId STRING NOT NULL
|
||||
)`,
|
||||
);
|
||||
// Delete duplicate rows from tag_catalog
|
||||
await db.exec(`
|
||||
DELETE FROM tag_catalog
|
||||
WHERE id NOT IN (
|
||||
SELECT MIN(id)
|
||||
FROM tag_catalog
|
||||
GROUP BY dir, branch, artifactId, path, cacheKey
|
||||
)
|
||||
`);
|
||||
|
||||
// Delete duplicate rows from global_cache
|
||||
await db.exec(`
|
||||
DELETE FROM global_cache
|
||||
WHERE id NOT IN (
|
||||
SELECT MIN(id)
|
||||
FROM global_cache
|
||||
GROUP BY cacheKey, dir, branch, artifactId
|
||||
)
|
||||
`);
|
||||
|
||||
// Add unique constraints if they don't exist
|
||||
await db.exec(
|
||||
`CREATE UNIQUE INDEX IF NOT EXISTS idx_tag_catalog_unique
|
||||
ON tag_catalog(dir, branch, artifactId, path, cacheKey)`,
|
||||
);
|
||||
|
||||
await db.exec(
|
||||
`CREATE UNIQUE INDEX IF NOT EXISTS idx_global_cache_unique
|
||||
ON global_cache(cacheKey, dir, branch, artifactId)`,
|
||||
);
|
||||
}
|
||||
|
||||
private static indexSqlitePath = getIndexSqlitePath();
|
||||
|
|
@ -90,14 +120,24 @@ enum AddRemoveResultType {
|
|||
Remove = "remove",
|
||||
UpdateNewVersion = "updateNewVersion",
|
||||
UpdateOldVersion = "updateOldVersion",
|
||||
UpdateLastUpdated = "updateLastUpdated",
|
||||
Compute = "compute"
|
||||
}
|
||||
|
||||
async function getAddRemoveForTag(
|
||||
tag: IndexTag,
|
||||
currentFiles: LastModifiedMap,
|
||||
readFile: (path: string) => Promise<string>,
|
||||
): Promise<[PathAndCacheKey[], PathAndCacheKey[], MarkCompleteCallback]> {
|
||||
): Promise<
|
||||
[
|
||||
PathAndCacheKey[],
|
||||
PathAndCacheKey[],
|
||||
PathAndCacheKey[],
|
||||
MarkCompleteCallback,
|
||||
]
|
||||
> {
|
||||
const newLastUpdatedTimestamp = Date.now();
|
||||
const files = { ...currentFiles };
|
||||
|
||||
const saved = await getSavedItemsForTag(tag);
|
||||
|
||||
|
|
@ -105,35 +145,41 @@ async function getAddRemoveForTag(
|
|||
const updateNewVersion: PathAndCacheKey[] = [];
|
||||
const updateOldVersion: PathAndCacheKey[] = [];
|
||||
const remove: PathAndCacheKey[] = [];
|
||||
const updateLastUpdated: PathAndCacheKey[] = [];
|
||||
|
||||
for (const item of saved) {
|
||||
const { lastUpdated, ...pathAndCacheKey } = item;
|
||||
|
||||
if (currentFiles[item.path] === undefined) {
|
||||
if (files[item.path] === undefined) {
|
||||
// Was indexed, but no longer exists. Remove
|
||||
remove.push(pathAndCacheKey);
|
||||
} else {
|
||||
// Exists in old and new, so determine whether it was updated
|
||||
if (lastUpdated < currentFiles[item.path]) {
|
||||
if (lastUpdated < files[item.path]) {
|
||||
// Change was made after last update
|
||||
updateNewVersion.push({
|
||||
path: pathAndCacheKey.path,
|
||||
cacheKey: calculateHash(await readFile(pathAndCacheKey.path)),
|
||||
});
|
||||
updateOldVersion.push(pathAndCacheKey);
|
||||
const newHash = calculateHash(await readFile(pathAndCacheKey.path));
|
||||
if (pathAndCacheKey.cacheKey !== newHash) {
|
||||
updateNewVersion.push({
|
||||
path: pathAndCacheKey.path,
|
||||
cacheKey: newHash,
|
||||
});
|
||||
updateOldVersion.push(pathAndCacheKey);
|
||||
} else {
|
||||
updateLastUpdated.push(pathAndCacheKey);
|
||||
}
|
||||
} else {
|
||||
// Already updated, do nothing
|
||||
}
|
||||
|
||||
// Remove so we can check leftovers afterward
|
||||
delete currentFiles[item.path];
|
||||
delete files[item.path];
|
||||
}
|
||||
}
|
||||
|
||||
// Any leftover in current files need to be added
|
||||
add.push(
|
||||
...(await Promise.all(
|
||||
Object.keys(currentFiles).map(async (path) => {
|
||||
Object.keys(files).map(async (path) => {
|
||||
const fileContents = await readFile(path);
|
||||
return { path, cacheKey: calculateHash(fileContents) };
|
||||
}),
|
||||
|
|
@ -143,18 +189,43 @@ async function getAddRemoveForTag(
|
|||
// Create the markComplete callback function
|
||||
const db = await SqliteDb.get();
|
||||
const itemToAction: {
|
||||
[key: string]: [PathAndCacheKey, AddRemoveResultType];
|
||||
} = {};
|
||||
[key in AddRemoveResultType]: PathAndCacheKey[];
|
||||
} = {
|
||||
[AddRemoveResultType.Add]: [],
|
||||
[AddRemoveResultType.Remove]: [],
|
||||
[AddRemoveResultType.UpdateNewVersion]: [],
|
||||
[AddRemoveResultType.UpdateOldVersion]: [],
|
||||
[AddRemoveResultType.UpdateLastUpdated]: [],
|
||||
[AddRemoveResultType.Compute]: [],
|
||||
};
|
||||
|
||||
async function markComplete(items: PathAndCacheKey[], _: IndexResultType) {
|
||||
const actions = items.map(
|
||||
(item) =>
|
||||
itemToAction[
|
||||
JSON.stringify({ path: item.path, cacheKey: item.cacheKey })
|
||||
],
|
||||
);
|
||||
for (const [{ path, cacheKey }, resultType] of actions) {
|
||||
switch (resultType) {
|
||||
async function markComplete(
|
||||
items: PathAndCacheKey[],
|
||||
resultType: IndexResultType,
|
||||
) {
|
||||
const addRemoveResultType =
|
||||
mapIndexResultTypeToAddRemoveResultType(resultType);
|
||||
|
||||
const actionItems = itemToAction[addRemoveResultType];
|
||||
if (!actionItems) {
|
||||
console.warn(`No action items found for result type: ${resultType}`);
|
||||
return;
|
||||
}
|
||||
|
||||
for (const item of items) {
|
||||
const { path, cacheKey } = item;
|
||||
switch (addRemoveResultType) {
|
||||
case AddRemoveResultType.Compute:
|
||||
await db.run(
|
||||
"REPLACE INTO tag_catalog (path, cacheKey, lastUpdated, dir, branch, artifactId) VALUES (?, ?, ?, ?, ?, ?)",
|
||||
path,
|
||||
cacheKey,
|
||||
newLastUpdatedTimestamp,
|
||||
tag.directory,
|
||||
tag.branch,
|
||||
tag.artifactId,
|
||||
);
|
||||
break;
|
||||
case AddRemoveResultType.Add:
|
||||
await db.run(
|
||||
"INSERT INTO tag_catalog (path, cacheKey, lastUpdated, dir, branch, artifactId) VALUES (?, ?, ?, ?, ?, ?)",
|
||||
|
|
@ -182,6 +253,7 @@ async function getAddRemoveForTag(
|
|||
tag.artifactId,
|
||||
);
|
||||
break;
|
||||
case AddRemoveResultType.UpdateLastUpdated:
|
||||
case AddRemoveResultType.UpdateNewVersion:
|
||||
await db.run(
|
||||
`UPDATE tag_catalog SET
|
||||
|
|
@ -208,27 +280,22 @@ async function getAddRemoveForTag(
|
|||
}
|
||||
|
||||
for (const item of updateNewVersion) {
|
||||
itemToAction[JSON.stringify(item)] = [
|
||||
item,
|
||||
AddRemoveResultType.UpdateNewVersion,
|
||||
];
|
||||
itemToAction[AddRemoveResultType.UpdateNewVersion].push(item);
|
||||
}
|
||||
for (const item of add) {
|
||||
itemToAction[JSON.stringify(item)] = [item, AddRemoveResultType.Add];
|
||||
itemToAction[AddRemoveResultType.Add].push(item);
|
||||
}
|
||||
for (const item of updateOldVersion) {
|
||||
itemToAction[JSON.stringify(item)] = [
|
||||
item,
|
||||
AddRemoveResultType.UpdateOldVersion,
|
||||
];
|
||||
itemToAction[AddRemoveResultType.UpdateOldVersion].push(item);
|
||||
}
|
||||
for (const item of remove) {
|
||||
itemToAction[JSON.stringify(item)] = [item, AddRemoveResultType.Remove];
|
||||
itemToAction[AddRemoveResultType.Remove].push(item);
|
||||
}
|
||||
|
||||
return [
|
||||
[...add, ...updateNewVersion],
|
||||
[...remove, ...updateOldVersion],
|
||||
updateLastUpdated,
|
||||
markComplete,
|
||||
];
|
||||
}
|
||||
|
|
@ -255,13 +322,31 @@ function calculateHash(fileContents: string): string {
|
|||
return hash.digest("hex");
|
||||
}
|
||||
|
||||
function mapIndexResultTypeToAddRemoveResultType(
|
||||
resultType: IndexResultType,
|
||||
): AddRemoveResultType {
|
||||
switch (resultType) {
|
||||
case "updateLastUpdated":
|
||||
return AddRemoveResultType.UpdateLastUpdated;
|
||||
case "compute":
|
||||
return AddRemoveResultType.Compute;
|
||||
case "addTag":
|
||||
return AddRemoveResultType.Add;
|
||||
case "del":
|
||||
case "removeTag":
|
||||
return AddRemoveResultType.Remove;
|
||||
default:
|
||||
throw new Error(`Unexpected result type: ${resultType}`);
|
||||
}
|
||||
}
|
||||
|
||||
export async function getComputeDeleteAddRemove(
|
||||
tag: IndexTag,
|
||||
currentFiles: LastModifiedMap,
|
||||
readFile: (path: string) => Promise<string>,
|
||||
repoName: string | undefined,
|
||||
): Promise<[RefreshIndexResults, MarkCompleteCallback]> {
|
||||
const [add, remove, markComplete] = await getAddRemoveForTag(
|
||||
): Promise<[RefreshIndexResults, PathAndCacheKey[], MarkCompleteCallback]> {
|
||||
const [add, remove, lastUpdated, markComplete] = await getAddRemoveForTag(
|
||||
tag,
|
||||
currentFiles,
|
||||
readFile,
|
||||
|
|
@ -305,6 +390,7 @@ export async function getComputeDeleteAddRemove(
|
|||
|
||||
return [
|
||||
results,
|
||||
lastUpdated,
|
||||
async (items, resultType) => {
|
||||
// Update tag catalog
|
||||
markComplete(items, resultType);
|
||||
|
|
@ -347,15 +433,15 @@ export class GlobalCacheCodeBaseIndex implements CodebaseIndex {
|
|||
_: MarkCompleteCallback,
|
||||
repoName: string | undefined,
|
||||
): AsyncGenerator<IndexingProgressUpdate> {
|
||||
const add = [...results.compute, ...results.addTag];
|
||||
const add = results.addTag;
|
||||
const remove = [...results.del, ...results.removeTag];
|
||||
await Promise.all([
|
||||
...add.map(({ cacheKey }) => {
|
||||
return this.computeOrAddTag(cacheKey, tag);
|
||||
}),
|
||||
...remove.map(({ cacheKey }) => {
|
||||
return this.deleteOrRemoveTag(cacheKey, tag);
|
||||
}),
|
||||
...add.map(({ cacheKey }) => {
|
||||
return this.computeOrAddTag(cacheKey, tag);
|
||||
}),
|
||||
]);
|
||||
yield { progress: 1, desc: "Done updating global cache", status: "done" };
|
||||
}
|
||||
|
|
|
|||
|
|
@ -5,6 +5,7 @@ export enum IndexResultType {
|
|||
Delete = "del",
|
||||
AddTag = "addTag",
|
||||
RemoveTag = "removeTag",
|
||||
UpdateLastUpdated = "updateLastUpdated"
|
||||
}
|
||||
|
||||
export type MarkCompleteCallback = (
|
||||
|
|
|
|||
|
|
@ -1,17 +1,15 @@
|
|||
import { EventEmitter } from "events";
|
||||
import { Minimatch } from "minimatch";
|
||||
import path from "node:path";
|
||||
import { FileType, IDE } from "..";
|
||||
import { DEFAULT_IGNORE_DIRS, DEFAULT_IGNORE_FILETYPES } from "./ignore";
|
||||
import { FileType, IDE } from "../index.d.js";
|
||||
import {
|
||||
DEFAULT_IGNORE_DIRS,
|
||||
DEFAULT_IGNORE_FILETYPES,
|
||||
defaultIgnoreDir,
|
||||
defaultIgnoreFile,
|
||||
} from "./ignore.js";
|
||||
|
||||
export interface WalkerOptions {
|
||||
isSymbolicLink?: boolean;
|
||||
path?: string;
|
||||
ignoreFiles?: string[];
|
||||
parent?: Walker | null;
|
||||
includeEmpty?: boolean;
|
||||
follow?: boolean;
|
||||
exact?: boolean;
|
||||
onlyDirs?: boolean;
|
||||
returnRelativePaths?: boolean;
|
||||
additionalIgnoreRules?: string[];
|
||||
|
|
@ -19,315 +17,194 @@ export interface WalkerOptions {
|
|||
|
||||
type Entry = [string, FileType];
|
||||
|
||||
class Walker extends EventEmitter {
|
||||
isSymbolicLink: boolean;
|
||||
path: string;
|
||||
basename: string;
|
||||
ignoreFiles: string[];
|
||||
ignoreRules: { [key: string]: Minimatch[] };
|
||||
parent: Walker | null;
|
||||
includeEmpty: boolean;
|
||||
root: string;
|
||||
follow: boolean;
|
||||
result: Set<string>;
|
||||
entries: Entry[] | null;
|
||||
sawError: boolean;
|
||||
exact: boolean | undefined;
|
||||
onlyDirs: boolean | undefined;
|
||||
// helper struct used for the DFS walk
|
||||
type WalkableEntry = {
|
||||
relPath: string;
|
||||
absPath: string;
|
||||
type: FileType;
|
||||
entry: Entry;
|
||||
};
|
||||
|
||||
// helper struct used for the DFS walk
|
||||
type WalkContext = {
|
||||
walkableEntry: WalkableEntry;
|
||||
ignoreFiles: IgnoreFile[];
|
||||
};
|
||||
|
||||
class IgnoreFile {
|
||||
private _rules: Minimatch[];
|
||||
|
||||
constructor(
|
||||
opts: WalkerOptions = {},
|
||||
protected readonly ide: IDE,
|
||||
public path: string,
|
||||
public content: string,
|
||||
) {
|
||||
super(opts as any);
|
||||
this.isSymbolicLink = opts.isSymbolicLink || false;
|
||||
this.path = opts.path || process.cwd();
|
||||
this.basename = path.basename(this.path);
|
||||
this.ignoreFiles = [...(opts.ignoreFiles || [".ignore"]), ".defaultignore"];
|
||||
this.ignoreRules = {};
|
||||
this.parent = opts.parent || null;
|
||||
this.includeEmpty = !!opts.includeEmpty;
|
||||
this.root = this.parent ? this.parent.root : this.path;
|
||||
this.follow = !!opts.follow;
|
||||
this.result = this.parent ? this.parent.result : new Set();
|
||||
this.entries = null;
|
||||
this.sawError = false;
|
||||
this.exact = opts.exact;
|
||||
this.onlyDirs = opts.onlyDirs;
|
||||
|
||||
if (opts.additionalIgnoreRules) {
|
||||
this.addIgnoreRules(opts.additionalIgnoreRules);
|
||||
}
|
||||
this.path = path;
|
||||
this.content = content;
|
||||
this._rules = this.contentToRules(content);
|
||||
}
|
||||
|
||||
sort(a: string, b: string): number {
|
||||
return a.localeCompare(b, "en");
|
||||
public get rules() {
|
||||
return this._rules;
|
||||
}
|
||||
|
||||
emit(ev: string, data: any): boolean {
|
||||
let ret = false;
|
||||
if (!(this.sawError && ev === "error")) {
|
||||
if (ev === "error") {
|
||||
this.sawError = true;
|
||||
} else if (ev === "done" && !this.parent) {
|
||||
data = (Array.from(data) as any)
|
||||
.map((e: string) => (/^@/.test(e) ? `./${e}` : e))
|
||||
.sort(this.sort);
|
||||
this.result = new Set(data);
|
||||
}
|
||||
|
||||
if (ev === "error" && this.parent) {
|
||||
ret = this.parent.emit("error", data);
|
||||
} else {
|
||||
ret = super.emit(ev, data);
|
||||
}
|
||||
}
|
||||
return ret;
|
||||
}
|
||||
|
||||
start(): this {
|
||||
this.ide
|
||||
.listDir(this.path)
|
||||
.then((entries) => {
|
||||
this.onReaddir(entries);
|
||||
})
|
||||
.catch((err) => {
|
||||
this.emit("error", err);
|
||||
});
|
||||
return this;
|
||||
}
|
||||
|
||||
isIgnoreFile(e: Entry): boolean {
|
||||
const p = e[0];
|
||||
return p !== "." && p !== ".." && this.ignoreFiles.indexOf(p) !== -1;
|
||||
}
|
||||
|
||||
onReaddir(entries: Entry[]): void {
|
||||
this.entries = entries;
|
||||
if (entries.length === 0) {
|
||||
if (this.includeEmpty) {
|
||||
this.result.add(this.path.slice(this.root.length + 1));
|
||||
}
|
||||
this.emit("done", this.result);
|
||||
} else {
|
||||
const hasIg = this.entries.some((e) => this.isIgnoreFile(e));
|
||||
|
||||
if (hasIg) {
|
||||
this.addIgnoreFiles();
|
||||
} else {
|
||||
this.filterEntries();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
addIgnoreFiles(): void {
|
||||
const newIg = this.entries!.filter((e) => this.isIgnoreFile(e));
|
||||
|
||||
let igCount = newIg.length;
|
||||
const then = () => {
|
||||
if (--igCount === 0) {
|
||||
this.filterEntries();
|
||||
}
|
||||
};
|
||||
|
||||
newIg.forEach((e) => this.addIgnoreFile(e, then));
|
||||
}
|
||||
|
||||
addIgnoreFile(file: Entry, then: () => void): void {
|
||||
const ig = path.resolve(this.path, file[0]);
|
||||
this.ide
|
||||
.readFile(ig)
|
||||
.then((data) => {
|
||||
this.onReadIgnoreFile(file, data, then);
|
||||
})
|
||||
.catch((err) => {
|
||||
this.emit("error", err);
|
||||
});
|
||||
}
|
||||
|
||||
onReadIgnoreFile(file: Entry, data: string, then: () => void): void {
|
||||
const mmopt = {
|
||||
private contentToRules(content: string): Minimatch[] {
|
||||
const options = {
|
||||
matchBase: true,
|
||||
dot: true,
|
||||
flipNegate: true,
|
||||
nocase: true,
|
||||
};
|
||||
const rules = data
|
||||
return content
|
||||
.split(/\r?\n/)
|
||||
.filter((line) => !/^#|^$/.test(line.trim()))
|
||||
.map((rule) => {
|
||||
return new Minimatch(rule.trim(), mmopt);
|
||||
});
|
||||
.map((l) => l.trim())
|
||||
.filter((l) => !/^#|^$/.test(l))
|
||||
.map((l) => new Minimatch(l, options));
|
||||
}
|
||||
}
|
||||
|
||||
this.ignoreRules[file[0]] = rules;
|
||||
class DFSWalker {
|
||||
private readonly path: string;
|
||||
private readonly ide: IDE;
|
||||
private readonly options: WalkerOptions;
|
||||
private readonly ignoreFileNames: Set<string>;
|
||||
|
||||
then();
|
||||
constructor(path: string, ide: IDE, options: WalkerOptions) {
|
||||
this.path = path;
|
||||
this.ide = ide;
|
||||
this.options = options;
|
||||
this.ignoreFileNames = new Set<string>(options.ignoreFiles);
|
||||
}
|
||||
|
||||
addIgnoreRules(rules: string[]) {
|
||||
const mmopt = {
|
||||
matchBase: true,
|
||||
dot: true,
|
||||
flipNegate: true,
|
||||
nocase: true,
|
||||
// walk is a depth-first search implementation
|
||||
public async *walk(): AsyncGenerator<string> {
|
||||
const root: WalkContext = {
|
||||
walkableEntry: {
|
||||
relPath: "",
|
||||
absPath: this.path,
|
||||
type: 2 as FileType.Directory,
|
||||
entry: ["", 2 as FileType.Directory],
|
||||
},
|
||||
ignoreFiles: [],
|
||||
};
|
||||
const minimatchRules = rules
|
||||
.filter((line) => !/^#|^$/.test(line.trim()))
|
||||
.map((rule) => {
|
||||
return new Minimatch(rule.trim(), mmopt);
|
||||
});
|
||||
|
||||
this.ignoreRules[".defaultignore"] = minimatchRules;
|
||||
}
|
||||
|
||||
filterEntries(): void {
|
||||
const filtered = this.entries!.map((entry) => {
|
||||
const passFile = this.filterEntry(entry[0]);
|
||||
const passDir = this.filterEntry(entry[0], true);
|
||||
return passFile || passDir ? [entry, passFile, passDir] : false;
|
||||
}).filter((e) => e) as [Entry, boolean, boolean][];
|
||||
let entryCount = filtered.length;
|
||||
if (entryCount === 0) {
|
||||
this.emit("done", this.result);
|
||||
} else {
|
||||
const then = () => {
|
||||
if (--entryCount === 0) {
|
||||
// Otherwise in onlyDirs mode, nothing would be returned
|
||||
if (this.onlyDirs && this.path !== this.root) {
|
||||
this.result.add(this.path.slice(this.root.length + 1));
|
||||
}
|
||||
this.emit("done", this.result);
|
||||
const stack = [root];
|
||||
for (let cur = stack.pop(); cur; cur = stack.pop()) {
|
||||
const walkableEntries = await this.listDirForWalking(cur.walkableEntry);
|
||||
const ignoreFiles = await this.getIgnoreFilesToApplyInDir(
|
||||
cur.ignoreFiles,
|
||||
walkableEntries,
|
||||
);
|
||||
for (const w of walkableEntries) {
|
||||
if (!this.shouldInclude(w, ignoreFiles)) {
|
||||
continue;
|
||||
}
|
||||
if (this.entryIsDirectory(w.entry)) {
|
||||
stack.push({
|
||||
walkableEntry: w,
|
||||
ignoreFiles: ignoreFiles,
|
||||
});
|
||||
if (this.options.onlyDirs) {
|
||||
// when onlyDirs is enabled the walker will only return directory names
|
||||
yield w.relPath;
|
||||
}
|
||||
} else {
|
||||
yield w.relPath;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private async listDirForWalking(
|
||||
walkableEntry: WalkableEntry,
|
||||
): Promise<WalkableEntry[]> {
|
||||
const entries = await this.ide.listDir(walkableEntry.absPath);
|
||||
return entries.map((e) => {
|
||||
return {
|
||||
relPath: path.join(walkableEntry.relPath, e[0]),
|
||||
absPath: path.join(walkableEntry.absPath, e[0]),
|
||||
type: e[1],
|
||||
entry: e,
|
||||
};
|
||||
filtered.forEach((filt) => {
|
||||
const [entry, file, dir] = filt;
|
||||
this.stat(entry, file, dir, then);
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
private async getIgnoreFilesToApplyInDir(
|
||||
parentIgnoreFiles: IgnoreFile[],
|
||||
walkableEntries: WalkableEntry[],
|
||||
): Promise<IgnoreFile[]> {
|
||||
const ignoreFilesInDir = await this.loadIgnoreFiles(walkableEntries);
|
||||
if (ignoreFilesInDir.length === 0) {
|
||||
return parentIgnoreFiles;
|
||||
}
|
||||
return Array.prototype.concat(parentIgnoreFiles, ignoreFilesInDir);
|
||||
}
|
||||
|
||||
entryIsDirectory(entry: Entry) {
|
||||
const Directory = 2 as FileType.Directory;
|
||||
return entry[1] === Directory;
|
||||
private async loadIgnoreFiles(
|
||||
entries: WalkableEntry[],
|
||||
): Promise<IgnoreFile[]> {
|
||||
const ignoreEntries = entries.filter((w) => this.isIgnoreFile(w.entry));
|
||||
const promises = ignoreEntries.map(async (w) => {
|
||||
const content = await this.ide.readFile(w.absPath);
|
||||
return new IgnoreFile(w.relPath, content);
|
||||
});
|
||||
return Promise.all(promises);
|
||||
}
|
||||
|
||||
entryIsSymlink(entry: Entry) {
|
||||
const Directory = 64 as FileType.SymbolicLink;
|
||||
return entry[1] === Directory;
|
||||
private isIgnoreFile(e: Entry): boolean {
|
||||
const p = e[0];
|
||||
return this.ignoreFileNames.has(p);
|
||||
}
|
||||
|
||||
onstat(entry: Entry, file: boolean, dir: boolean, then: () => void): void {
|
||||
const abs = this.path + "/" + entry[0];
|
||||
const isSymbolicLink = this.entryIsSymlink(entry);
|
||||
if (!this.entryIsDirectory(entry)) {
|
||||
if (file && !this.onlyDirs) {
|
||||
this.result.add(abs.slice(this.root.length + 1));
|
||||
}
|
||||
then();
|
||||
} else {
|
||||
if (dir) {
|
||||
this.walker(
|
||||
entry[0],
|
||||
{ isSymbolicLink, exact: this.filterEntry(entry[0] + "/") },
|
||||
then,
|
||||
);
|
||||
} else {
|
||||
then();
|
||||
}
|
||||
private shouldInclude(
|
||||
walkableEntry: WalkableEntry,
|
||||
ignoreFiles: IgnoreFile[],
|
||||
) {
|
||||
if (this.entryIsSymlink(walkableEntry.entry)) {
|
||||
// If called from the root, a symlink either links to a real file in this repository,
|
||||
// and therefore will be walked OR it linksto something outside of the repository and
|
||||
// we do not want to index it
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
stat(entry: Entry, file: boolean, dir: boolean, then: () => void): void {
|
||||
this.onstat(entry, file, dir, then);
|
||||
}
|
||||
|
||||
walkerOpt(entry: string, opts: Partial<WalkerOptions>): WalkerOptions {
|
||||
return {
|
||||
path: this.path + "/" + entry,
|
||||
parent: this,
|
||||
ignoreFiles: this.ignoreFiles,
|
||||
follow: this.follow,
|
||||
includeEmpty: this.includeEmpty,
|
||||
onlyDirs: this.onlyDirs,
|
||||
...opts,
|
||||
};
|
||||
}
|
||||
|
||||
walker(entry: string, opts: Partial<WalkerOptions>, then: () => void): void {
|
||||
new Walker(this.walkerOpt(entry, opts), this.ide).on("done", then).start();
|
||||
}
|
||||
|
||||
filterEntry(
|
||||
entry: string,
|
||||
partial?: boolean,
|
||||
entryBasename?: string,
|
||||
): boolean {
|
||||
let included = true;
|
||||
|
||||
if (this.parent && this.parent.filterEntry) {
|
||||
const parentEntry = this.basename + "/" + entry;
|
||||
const parentBasename = entryBasename || entry;
|
||||
included = this.parent.filterEntry(parentEntry, partial, parentBasename);
|
||||
if (!included && !this.exact) {
|
||||
let relPath = walkableEntry.relPath;
|
||||
if (this.entryIsDirectory(walkableEntry.entry)) {
|
||||
if (defaultIgnoreDir.ignores(walkableEntry.relPath)) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
this.ignoreFiles.forEach((f) => {
|
||||
if (this.ignoreRules[f]) {
|
||||
this.ignoreRules[f].forEach((rule) => {
|
||||
if (rule.negate !== included) {
|
||||
const isRelativeRule =
|
||||
entryBasename &&
|
||||
rule.globParts.some(
|
||||
(part) => part.length <= (part.slice(-1)[0] ? 1 : 2),
|
||||
);
|
||||
|
||||
const match =
|
||||
rule.match("/" + entry) ||
|
||||
rule.match(entry) ||
|
||||
(!!partial &&
|
||||
(rule.match("/" + entry + "/") ||
|
||||
rule.match(entry + "/") ||
|
||||
(rule.negate &&
|
||||
(rule.match("/" + entry, true) ||
|
||||
rule.match(entry, true))) ||
|
||||
(isRelativeRule &&
|
||||
(rule.match("/" + entryBasename + "/") ||
|
||||
rule.match(entryBasename + "/") ||
|
||||
(rule.negate &&
|
||||
(rule.match("/" + entryBasename, true) ||
|
||||
rule.match(entryBasename, true)))))));
|
||||
|
||||
if (match) {
|
||||
included = rule.negate;
|
||||
}
|
||||
}
|
||||
});
|
||||
relPath = `${relPath}/`;
|
||||
} else {
|
||||
if (this.options.onlyDirs) {
|
||||
return false;
|
||||
}
|
||||
});
|
||||
|
||||
if (defaultIgnoreFile.ignores(walkableEntry.relPath)) {
|
||||
return false;
|
||||
}
|
||||
relPath = `/${relPath}`;
|
||||
}
|
||||
let included = true;
|
||||
for (const ignoreFile of ignoreFiles) {
|
||||
for (const r of ignoreFile.rules) {
|
||||
if (r.negate === included) {
|
||||
// no need to test when the file is already NOT to be included unless this is a negate rule and vice versa
|
||||
continue;
|
||||
}
|
||||
if (r.match(relPath)) {
|
||||
included = r.negate;
|
||||
}
|
||||
}
|
||||
}
|
||||
return included;
|
||||
}
|
||||
}
|
||||
|
||||
interface WalkCallback {
|
||||
(err: Error | null, result?: string[]): void;
|
||||
}
|
||||
private entryIsDirectory(entry: Entry) {
|
||||
return entry[1] === (2 as FileType.Directory);
|
||||
}
|
||||
|
||||
async function walkDirWithCallback(
|
||||
opts: WalkerOptions,
|
||||
ide: IDE,
|
||||
callback?: WalkCallback,
|
||||
): Promise<string[] | void> {
|
||||
const p = new Promise<string[]>((resolve, reject) => {
|
||||
new Walker(opts, ide).on("done", resolve).on("error", reject).start();
|
||||
});
|
||||
return callback ? p.then((res) => callback(null, res), callback) : p;
|
||||
private entryIsSymlink(entry: Entry) {
|
||||
return entry[1] === (64 as FileType.SymbolicLink);
|
||||
}
|
||||
}
|
||||
|
||||
const defaultOptions: WalkerOptions = {
|
||||
ignoreFiles: [".gitignore", ".continueignore"],
|
||||
onlyDirs: false,
|
||||
additionalIgnoreRules: [...DEFAULT_IGNORE_DIRS, ...DEFAULT_IGNORE_FILETYPES],
|
||||
};
|
||||
|
||||
|
|
@ -336,40 +213,18 @@ export async function walkDir(
|
|||
ide: IDE,
|
||||
_options?: WalkerOptions,
|
||||
): Promise<string[]> {
|
||||
let entries: string[] = [];
|
||||
const options = { ...defaultOptions, ..._options };
|
||||
return new Promise((resolve, reject) => {
|
||||
walkDirWithCallback(
|
||||
{
|
||||
path,
|
||||
ignoreFiles: options.ignoreFiles,
|
||||
onlyDirs: options.onlyDirs,
|
||||
follow: true,
|
||||
includeEmpty: false,
|
||||
additionalIgnoreRules: options.additionalIgnoreRules,
|
||||
},
|
||||
ide,
|
||||
async (err, result) => {
|
||||
if (err) {
|
||||
reject(err);
|
||||
} else {
|
||||
const relativePaths = result || [];
|
||||
if (options?.returnRelativePaths) {
|
||||
resolve(relativePaths);
|
||||
} else {
|
||||
const pathSep = await ide.pathSep();
|
||||
if (pathSep === "/") {
|
||||
resolve(relativePaths.map((p) => path + pathSep + p));
|
||||
} else {
|
||||
// Need to replace with windows path sep
|
||||
resolve(
|
||||
relativePaths.map(
|
||||
(p) => path + pathSep + p.split("/").join(pathSep),
|
||||
),
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
);
|
||||
});
|
||||
const dfsWalker = new DFSWalker(path, ide, options);
|
||||
let relativePaths: string[] = [];
|
||||
for await (const e of dfsWalker.walk()) {
|
||||
relativePaths.push(e);
|
||||
}
|
||||
const pathSep = await ide.pathSep();
|
||||
const prefix = options.returnRelativePaths ? "" : path + pathSep;
|
||||
|
||||
if (pathSep === "/") {
|
||||
return relativePaths.map((p) => prefix + p);
|
||||
}
|
||||
return relativePaths.map((p) => prefix + p.split("/").join(pathSep));
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,47 @@
|
|||
import llamaTokenizer from "./llamaTokenizer.js";
|
||||
import { Tiktoken, encodingForModel as _encodingForModel } from "js-tiktoken";
|
||||
import workerpool from "workerpool";
|
||||
import * as path from "path";
|
||||
|
||||
export interface AsyncEncoder {
|
||||
encode(text: string): Promise<number[]>;
|
||||
decode(tokens: number[]): string;
|
||||
}
|
||||
|
||||
export class LlamaAsyncEncoder implements AsyncEncoder {
|
||||
private workerPool: workerpool.Pool;
|
||||
|
||||
constructor() {
|
||||
this.workerPool = workerpool.pool(path.join(__dirname, "/llamaTokenizerWorkerPool.mjs"));
|
||||
}
|
||||
|
||||
async encode(text: string): Promise<number[]> {
|
||||
return this.workerPool.exec("encode", [text]);
|
||||
}
|
||||
|
||||
decode(tokens: number[]): string {
|
||||
return llamaTokenizer.decode(tokens);
|
||||
}
|
||||
|
||||
// TODO: this should be called somewhere before exit or potentially with a shutdown hook
|
||||
public async close(): Promise<void> {
|
||||
await this.workerPool.terminate();
|
||||
}
|
||||
}
|
||||
|
||||
// this class does not yet do anything asynchronous
|
||||
export class GPTAsyncEncoder implements AsyncEncoder {
|
||||
private tiktokenEncoding: Tiktoken;
|
||||
|
||||
constructor() {
|
||||
this.tiktokenEncoding = _encodingForModel("gpt-4");
|
||||
}
|
||||
|
||||
async encode(text: string): Promise<number[]> {
|
||||
return this.tiktokenEncoding.encode(text, "all", []);
|
||||
}
|
||||
|
||||
decode(tokens: number[]): string {
|
||||
return this.tiktokenEncoding.decode(tokens);
|
||||
}
|
||||
}
|
||||
|
|
@ -61,6 +61,7 @@ const MODEL_SUPPORTS_IMAGES: string[] = [
|
|||
"llava",
|
||||
"gpt-4-turbo",
|
||||
"gpt-4o",
|
||||
"gpt-4o-mini",
|
||||
"gpt-4-vision",
|
||||
"claude-3",
|
||||
"gemini-ultra",
|
||||
|
|
|
|||
|
|
@ -20,6 +20,7 @@ const CONTEXT_LENGTH_FOR_MODEL: { [name: string]: number } = {
|
|||
"gpt-4-32k": GPT_4_CTX_LEN,
|
||||
"gpt-4-turbo-preview": GPT_4_CTX_LEN,
|
||||
"gpt-4o": GPT_4_CTX_LEN,
|
||||
"gpt-4o-mini": GPT_4_CTX_LEN,
|
||||
"gpt-4-vision": GPT_4_CTX_LEN,
|
||||
"gpt-4-0125-preview": GPT_4_CTX_LEN,
|
||||
"gpt-4-1106-preview": GPT_4_CTX_LEN,
|
||||
|
|
|
|||
|
|
@ -1,29 +1,42 @@
|
|||
import { Tiktoken, encodingForModel as _encodingForModel } from "js-tiktoken";
|
||||
import { ChatMessage, MessageContent, MessagePart } from "../index.js";
|
||||
import {
|
||||
AsyncEncoder,
|
||||
GPTAsyncEncoder,
|
||||
LlamaAsyncEncoder,
|
||||
} from "./asyncEncoder.js";
|
||||
import { autodetectTemplateType } from "./autodetect.js";
|
||||
import { TOKEN_BUFFER_FOR_SAFETY } from "./constants.js";
|
||||
import { stripImages } from "./images.js";
|
||||
import llamaTokenizer from "./llamaTokenizer.js";
|
||||
|
||||
interface Encoding {
|
||||
encode: Tiktoken["encode"];
|
||||
decode: Tiktoken["decode"];
|
||||
}
|
||||
|
||||
class LlamaEncoding implements Encoding {
|
||||
encode(
|
||||
text: string,
|
||||
allowedSpecial?: string[] | "all" | undefined,
|
||||
disallowedSpecial?: string[] | "all" | undefined,
|
||||
): number[] {
|
||||
encode(text: string): number[] {
|
||||
return llamaTokenizer.encode(text);
|
||||
}
|
||||
|
||||
decode(tokens: number[]): string {
|
||||
return llamaTokenizer.decode(tokens);
|
||||
}
|
||||
}
|
||||
|
||||
let gptEncoding: Encoding | null = null;
|
||||
const gptAsyncEncoder = new GPTAsyncEncoder();
|
||||
const llamaEncoding = new LlamaEncoding();
|
||||
const llamaAsyncEncoder = new LlamaAsyncEncoder();
|
||||
|
||||
function asyncEncoderForModel(modelName: string): AsyncEncoder {
|
||||
const modelType = autodetectTemplateType(modelName);
|
||||
if (!modelType || modelType === "none") {
|
||||
return gptAsyncEncoder;
|
||||
}
|
||||
// Temporary due to issues packaging the worker files
|
||||
return process.env.IS_BINARY ? gptAsyncEncoder : llamaAsyncEncoder;
|
||||
}
|
||||
|
||||
function encodingForModel(modelName: string): Encoding {
|
||||
const modelType = autodetectTemplateType(modelName);
|
||||
|
|
@ -46,6 +59,24 @@ function countImageTokens(content: MessagePart): number {
|
|||
throw new Error("Non-image content type");
|
||||
}
|
||||
|
||||
async function countTokensAsync(
|
||||
content: MessageContent,
|
||||
// defaults to llama2 because the tokenizer tends to produce more tokens
|
||||
modelName = "llama2",
|
||||
): Promise<number> {
|
||||
const encoding = asyncEncoderForModel(modelName);
|
||||
if (Array.isArray(content)) {
|
||||
const promises = content.map(async (part) => {
|
||||
if (part.type === "imageUrl") {
|
||||
return countImageTokens(part);
|
||||
}
|
||||
return (await encoding.encode(part.text ?? "")).length;
|
||||
});
|
||||
return (await Promise.all(promises)).reduce((sum, val) => sum + val, 0);
|
||||
}
|
||||
return (await encoding.encode(content ?? "")).length;
|
||||
}
|
||||
|
||||
function countTokens(
|
||||
content: MessageContent,
|
||||
// defaults to llama2 because the tokenizer tends to produce more tokens
|
||||
|
|
@ -79,16 +110,6 @@ function flattenMessages(msgs: ChatMessage[]): ChatMessage[] {
|
|||
return flattened;
|
||||
}
|
||||
|
||||
export function stripImages(content: MessageContent): string {
|
||||
if (Array.isArray(content)) {
|
||||
return content
|
||||
.filter((part) => part.type === "text")
|
||||
.map((part) => part.text)
|
||||
.join("\n");
|
||||
}
|
||||
return content;
|
||||
}
|
||||
|
||||
function countChatMessageTokens(
|
||||
modelName: string,
|
||||
chatMessage: ChatMessage,
|
||||
|
|
@ -360,6 +381,7 @@ function compileChatMessages(
|
|||
export {
|
||||
compileChatMessages,
|
||||
countTokens,
|
||||
countTokensAsync,
|
||||
pruneLinesFromBottom,
|
||||
pruneLinesFromTop,
|
||||
pruneRawPromptFromTop,
|
||||
|
|
|
|||
|
|
@ -0,0 +1,11 @@
|
|||
import { MessageContent } from "../index.js";
|
||||
|
||||
export function stripImages(content: MessageContent): string {
|
||||
if (Array.isArray(content)) {
|
||||
return content
|
||||
.filter((part) => part.type === "text")
|
||||
.map((part) => part.text)
|
||||
.join("\n");
|
||||
}
|
||||
return content;
|
||||
}
|
||||
|
|
@ -1,3 +1,4 @@
|
|||
import { findLlmInfo } from "@continuedev/llm-info";
|
||||
import Handlebars from "handlebars";
|
||||
import {
|
||||
ChatMessage,
|
||||
|
|
@ -35,9 +36,9 @@ import {
|
|||
compileChatMessages,
|
||||
countTokens,
|
||||
pruneRawPromptFromTop,
|
||||
stripImages,
|
||||
} from "./countTokens.js";
|
||||
import CompletionOptionsForModels from "./templates/options.js";
|
||||
import { stripImages } from "./images.js";
|
||||
|
||||
export abstract class BaseLLM implements ILLM {
|
||||
static providerName: ModelProvider;
|
||||
|
|
@ -114,14 +115,17 @@ export abstract class BaseLLM implements ILLM {
|
|||
..._options,
|
||||
};
|
||||
|
||||
this.model = options.model;
|
||||
const llmInfo = findLlmInfo(this.model);
|
||||
|
||||
const templateType =
|
||||
options.template ?? autodetectTemplateType(options.model);
|
||||
|
||||
this.title = options.title;
|
||||
this.uniqueId = options.uniqueId ?? "None";
|
||||
this.model = options.model;
|
||||
this.systemMessage = options.systemMessage;
|
||||
this.contextLength = options.contextLength ?? DEFAULT_CONTEXT_LENGTH;
|
||||
this.contextLength =
|
||||
options.contextLength ?? llmInfo?.contextLength ?? DEFAULT_CONTEXT_LENGTH;
|
||||
this.completionOptions = {
|
||||
...options.completionOptions,
|
||||
model: options.model || "gpt-4",
|
||||
|
|
@ -235,12 +239,16 @@ ${prompt}`;
|
|||
) {
|
||||
let promptTokens = this.countTokens(prompt);
|
||||
let generatedTokens = this.countTokens(completion);
|
||||
Telemetry.capture("tokens_generated", {
|
||||
model: model,
|
||||
provider: this.providerName,
|
||||
promptTokens: promptTokens,
|
||||
generatedTokens: generatedTokens,
|
||||
});
|
||||
Telemetry.capture(
|
||||
"tokens_generated",
|
||||
{
|
||||
model: model,
|
||||
provider: this.providerName,
|
||||
promptTokens: promptTokens,
|
||||
generatedTokens: generatedTokens,
|
||||
},
|
||||
true,
|
||||
);
|
||||
DevDataSqliteDb.logTokensGenerated(
|
||||
model,
|
||||
this.providerName,
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
|
|
@ -0,0 +1,10 @@
|
|||
import workerpool from "workerpool";
|
||||
import llamaTokenizer from "./llamaTokenizer.mjs";
|
||||
|
||||
function encode(segment) {
|
||||
return llamaTokenizer.encode(segment);
|
||||
}
|
||||
|
||||
workerpool.worker({
|
||||
encode,
|
||||
});
|
||||
|
|
@ -4,7 +4,7 @@ import {
|
|||
LLMOptions,
|
||||
ModelProvider,
|
||||
} from "../../index.js";
|
||||
import { stripImages } from "../countTokens.js";
|
||||
import { stripImages } from "../images.js";
|
||||
import { BaseLLM } from "../index.js";
|
||||
import { streamSse } from "../stream.js";
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,8 @@
|
|||
import { ModelProvider } from "../../index.js";
|
||||
import OpenAI from "./OpenAI.js";
|
||||
|
||||
class Azure extends OpenAI {
|
||||
static providerName: ModelProvider = "azure";
|
||||
}
|
||||
|
||||
export default Azure;
|
||||
|
|
@ -10,7 +10,7 @@ import {
|
|||
MessageContent,
|
||||
ModelProvider,
|
||||
} from "../../index.js";
|
||||
import { stripImages } from "../countTokens.js";
|
||||
import { stripImages } from "../images.js";
|
||||
import { BaseLLM } from "../index.js";
|
||||
|
||||
class Bedrock extends BaseLLM {
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
import { BaseLLM } from "../index.js";
|
||||
import { ChatMessage, CompletionOptions, ModelProvider } from "../../index.js";
|
||||
import { stripImages } from "../countTokens.js";
|
||||
import { stripImages } from "../images.js";
|
||||
import { streamSse } from "../stream.js";
|
||||
|
||||
export default class Cloudflare extends BaseLLM {
|
||||
|
|
@ -38,7 +38,6 @@ export default class Cloudflare extends BaseLLM {
|
|||
});
|
||||
|
||||
for await (const value of streamSse(resp)) {
|
||||
console.log(value);
|
||||
if (value.choices?.[0]?.delta?.content) {
|
||||
yield value.choices[0].delta;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ import {
|
|||
LLMOptions,
|
||||
ModelProvider,
|
||||
} from "../../index.js";
|
||||
import { stripImages } from "../countTokens.js";
|
||||
import { stripImages } from "../images.js";
|
||||
import { streamJSON } from "../stream.js";
|
||||
|
||||
class Cohere extends BaseLLM {
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
import { LLMOptions, ModelProvider } from "../../index.js";
|
||||
import { CompletionOptions, LLMOptions, ModelProvider } from "../../index.js";
|
||||
import { streamSse } from "../stream.js";
|
||||
import { osModelsEditPrompt } from "../templates/edit.js";
|
||||
import OpenAI from "./OpenAI.js";
|
||||
|
||||
|
|
@ -13,6 +14,41 @@ class Deepseek extends OpenAI {
|
|||
useLegacyCompletionsEndpoint: false,
|
||||
};
|
||||
protected maxStopWords: number | undefined = 16;
|
||||
|
||||
supportsFim(): boolean {
|
||||
return true;
|
||||
}
|
||||
|
||||
async *_streamFim(
|
||||
prefix: string,
|
||||
suffix: string,
|
||||
options: CompletionOptions,
|
||||
): AsyncGenerator<string> {
|
||||
const endpoint = new URL("beta/completions", this.apiBase);
|
||||
const resp = await this.fetch(endpoint, {
|
||||
method: "POST",
|
||||
body: JSON.stringify({
|
||||
model: options.model,
|
||||
prompt: prefix,
|
||||
suffix,
|
||||
max_tokens: options.maxTokens,
|
||||
temperature: options.temperature,
|
||||
top_p: options.topP,
|
||||
frequency_penalty: options.frequencyPenalty,
|
||||
presence_penalty: options.presencePenalty,
|
||||
stop: options.stop,
|
||||
stream: true,
|
||||
}),
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
Accept: "application/json",
|
||||
Authorization: `Bearer ${this.apiKey}`,
|
||||
},
|
||||
});
|
||||
for await (const chunk of streamSse(resp)) {
|
||||
yield chunk.choices[0].text;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export default Deepseek;
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue