🚢 Preview (#1355)
* 📝 rename googlepalmapi.md to googlegeminiapi.md * 📝 update mistral models * Rename to geminiapi & change filename this time * ✨ global request options (#1153) * ✨ global request options * 🐛 fix jira context provider by injecting fetch * ✨ request options for embeddings providers * ✨ add Cohere as Reranker (#1159) * ♻️ use custom requestOptions with CohereEmbeddingsProvider * Update preIndexedDocs.ts (#1154) Add WordPress and WooCommerce as preIndexedDocs. * 🩹 remove example "outputDir" from default config * Fix slash command params loading (#1084) Existing slash commands expect an object named "params" so mapping to "options" here caused params to be undefined within the run scope. I renamed from 'm' to 's' just to avoid potential confusion with the model property mapping above. * 🐛 don't index if no open workspace folders * 💄 improve onboarding language * 🚸 improve onboarding * 🐛 stop loading when error * 💄 replace text in input box * Respect Retry-After header when available from 429 responses (#1182) * 🩹 remove dead code for exponential backoff This has been replaced by the withExponentialBackoff helper * 🩹 respect Retry-After header when available * 🚸 update inline tips language * ✨ input box history * 📌 update package-locks * 🔊 log errors in prepackage * 🐛 err to string * 📌 pin llama-tokenizer-js * 📌 update lockfile * 🚚 change /docs to docs. * 📦 package win-ca dependencies in binary * 🔥 remove unpopular models from UI * 🍱 new logo in jetbrains * 🎨 use node-fetch everywhere * 🚸 immediately select newly added models * 🚸 spell out Alt instead of using symbol * 🔥 remove config shortcut * 🐛 fix changing model bug * 🩹 de-duplicate before adding models * 🔧 add embeddingsProvider specific request options * 🎨 refactor to always use node-fetch from LLM * 🔥 remove duplicate tokens generated * 🔊 add timestamp to JetBrains logs * 🎨 maxStopWords for Groq * 🐛 fix groq provider calling /completions * 🐛 correctly adhere to LanceDB table name spec * 🐛 fix sqlite NOT NULL constraint failed error with custom model * Fix issue where Accept/Reject All only accepts/rejects a single diff hunk. (#1197) * Fix issues parsing Ollama /api/show endpoint payloads. (#1199) * ✨ model role for inlineEdit * 🩹 various small updates * 🐛 fix openai image support * 🔖 update gradle version * 🍱 update jetbrains icon * 🐛 fix autocomplete in notebook cells * 🔥 remove unused media * 🔥 remove unused files * Fix schema to allow 'AUTODETECT' sentinel for model when provider is 'groq'. (#1203) * 🐛 small improvements * Fix issue with @codebase provider when n becomes odd due to a divide by 2 during the full text search portion of the query. (#1204) * 🐛 add skipLines * ✨ URLContextProvider * 🥅 improved error handling for codebase indexing * 🏷️ use official Git extension types * ➕ declare vscode.git extension dependency * ⚡️ use reranker for docs context provider * 🚸 Use templating in default customCommand * 🎨 use U+23CE * 🚸 disable autocomplete in commit message box * 🩹 add gems to default ignored paths * ⚡️ format markdown blocks as comments in .ipynb completions * 🐛 don't strip port in URL * 🐛 fix "gemini" provider spacing issues * 📦 update posthog version * CON-1067: Failed state seems to be toggling as intended * 🏷️ update types.ts * 🐛 fix copy/paste/cut behavior in VS Code notebooks * ✨ llama3 prompt template * Rework for proper initialization on start up * CON-1067 Clean-up * CON-1067 more clean-up * Add indexingNotLoaded state * CON-1067 communicate progress to frontend * 🐛 fix undefined prefix, suffix and language for `/edit` (#1216) * 🐛 add .bind to fix templating in systemMessage * CON-1067 clean up * 🐛 small improvements to autocomplete * Update DocsContextProvider.ts (#1217) I fixed a bug where you were sending the query variable (which holds the base URL of the doc) to the rerank method, and it made no sense to rerank the chunks based on a URL. So I changed it to extras.fullInput because it should rerank based on the user input, which should provide better results. * 📝 select-provider.md update * 🐛 fix merge errors * Nate/autocomplete-metrics (#1230) * ⚡️ use context.selectedCompletionInfo, deduplicate logs * ⚡️ don't reject if user keeps typing same as completion * ⚡️ vscode autocomplete edge cases * 🚧 WIP on vscode autocomplete * ⚡️ better bracket handlng * ⚡️ improved multi-line detection * Active file default context (#1231) * 🚸 include currently active file by default * 🚸 warn if non-autocomplete model being used * ✨ try hole filling template for gpt * 💄 ui for no context * ⚡️ leave out bottom of excessively large files * 🚧 experimenting with perplexity style streaming * 🐛 fix #1237 * 💚 fix type error * ⚡️ improve LSP usage in autocomplete * 🐛 fix content parsing regression in /edit * add PySide6 docs to preindexed docs (#1236) * CON-232 bring custom docs to top, alphabetize doc results, make scrol… (#1239) * CON-232 bring custom docs to top, alphabetize doc results, make scrollable * CON-232 cleanup --------- Co-authored-by: Justin Milner <jmilner@jmilner-lt2.deka.local> * CON-1067 condense some things * 🚚 [Auxiliary -> Continue] Sidebar * 🔊 log completion options in ~/.continue/sessions * CON-1067 wrong ret val fix * CON-1067: fixes from testing * ⚡️ filter out completions that are only punctuation/space * ⚡️ inject intellisense docs, no multi-line on comments * ⚡️ crawl type definitions for autocomplete * ⚡️ truncate function text * ⚡️ cache LSP calls * ⚡️ find recently edited ranges with perfect prefix match * 🐛 fix gif paths * ⚡️ bring back double new line stop words * 📌 add yarn lock files * 🐛 allow language keywords to be generated * 💄 toggle on help button * 🎨 defaultContext option * 🐛 fix lancedb bug by upgrading * 🐛 fix groq stop tokens * 🐛 preventDefault to avoid double paste * 🚸 don't repeatedly override cmd+J * 🧑💻 fix npm run test in core * 📝 change description * 🐛 silence Ollama invalid server state warning * ⚡️ more accurate getTerminalContents * ⚡️ make getTerminalContents more accurate * 🧑💻 use yarn instead of npm * 👷 fix yarn add --no-save in prepackge * 🐛 correctly read entire notebook file contents * ➕ import handlebars * 🔥 remove unnecessary migrations * ⚡️ improve /comment prompt * Add debug terminal context menu (#1261) * Add --no-dependencies to vsce package (#1255) This is not needed because we bundle first with esbuild, and vsce pack has issues with modern package managers. see: microsoft/vscode-vsce#421 (comment) * ui: change line decoration color to use vscode theme (#1253) * ui: change line decoration color to use vscode theme to give user a more consistent experience by letting the decoration color to user the color defined in the theme. * fix: incorrect color item should be line background not text background because the decoration is for the whole line * 🎨 refactor indexing state into a single object * CON-223 Correct diff streaming abort (#1263) Co-authored-by: Justin Milner <jmilner@jmilner-lt2.deka.local> * 📦 switch to pnpm (#1265) * 🎨 use pnpm instead of yarn * ➕ make dependencies explicit for pnpm * 🐛 add powershell to extension mapping, and default to ext * Add stream support for Bedrock Anthropic * 🎨 make llamatokenizer commonjs compatible * ➕ add esbuild optional deps * 🚚 rename vendor/node_modules -> modules * 🔖 update core version * 🐛 fixes for transformers.js compatibility * 🔖 update core version * 🎨 set modelPath in constructor * 🎨 fix transformers.js import * 🎨 eslint enforce import extensions * 🎨 require -> import * 🚸 notify user if not diff in /commit * 💄 Improve colors of the IntelliJ tool window icon (#1273) Without this fix, the continue icon sticks out from the other toolwindow icons, resulting in an inconsistent appearance of the whole IDE and creates a feeling that the continue plugin "doesn't fit, something must be broken". According to https://plugins.jetbrains.com/docs/intellij/icons.html#new-ui-icon-colors specific colors are needed to work nicely with dark and light modes. Bonus is that the active tool window icon color then changes automatically to white. Co-authored-by: Lukas Baron <LukasBaron@gmail.com> * ✨ send Bearer token to Ollama if apiKey set * 🐛 fix slash command bug * 🧑💻 add onnxruntime (--save-dev) for types * 🐛 don't apply to file when running code block in terminal * 🐛 avoid double paste * ✨ gpt-4o * 🎨 pass uniqueId to constructor * 🚸 focus without scrolling into vie * 🎨 refactoring for continue-proxy LLM (#1277) * 🐛 continue server client fixes * 🎨 refactor OpenAI _getHeaders * 🎨 pass ideSettings to llmFromDescription * ⚡️ improve add docstring command * 💚 ci updates * 🐛 fix repeated paste bug * 🐛 fix build script * 🩹 various small improvements * 📌 pin esbuild 0.17.19 * 🧑💻 pnpm i gui in prepackage * 🐛 show all diff changes in vscode * 🩹 getMetaKeyName * 🐛 fix reading of unopened ipynb files * ⚡️ gpt-4o system prompt * 💄 make font size configurable in config.json ui.fontSize * 🩹 properly dispose of diff handler * 🐛 fix indexing status display * ⚡️ context pruning * 🎨 update free trial models * fix: remove some backup files generated by pkg if present (#1287) * adjust toCopy (#1305) Co-authored-by: Justin Milner <jmilner@jmilner-lt2.deka.local> * Nate/prompt-file (#1308) * ✨ .prompt files * 🧑💻 back to npm : ( * 🎨 nicer dropdown icon * 🎨 full switch back to npm * 🧑💻 uninstall script for fresh start * 🎨 updated package-locks * 🔥 remove hello from continuerc.json * 🔥 remove example files * 🎨 update test prompt * 🎨 update prompt * 🎨 update test prompt * 👷 create out/node_modules in prepackage.js * Log prompt-tokens to devdb and show in 'My Usage' view (#1309) * Also log the number of prompt tokens to the dev-db * Show prompt tokens in 'My Usage' view * 🔥 remove console logs * add lm studio tab-autocomplete walkthrough (#1298) * 📝 update autocomplete LM Studio docs * 💚 fix prepackage.js * ⬆️ upgrade lancedb version * 🚸 make it easier to continue past trial * 🚸 improve model setup process * 📌 update package-lock.jsons * 🐛 fix prompt file loading bug * docs: add setup guide for OpenRouter (#1284) * Update preIndexedDocs.ts (#1292) Add Bootstrap and Alpine.js as pre indexed docs. * add toString (#1324) * add toString * Fix indentation --------- Co-authored-by: Justin Milner <jmilner@jmilner-lt2.deka.local> * don't overwrite all models in onboarding * 🔧 override headers with custom headers * ✨ allow overwriting slash command description * feature flag (#1327) * ➕ feature flags dependency * 🎨 make getHeaders awaitable * 🔒️ token count for abuse monitoring * 👷 remove win32-arm64 target * Update build.js (#1330) Fix error: EBUSY: resource busy or locked, rmdir '...\continue\binary\tmp\continue-node_modules-lancedb' * 📌 pin onnxruntime version to match transformers.js * 🎨 transformers cleanup, extensionVersion header * 🔥 remove lingering pnpm * 🎨 update default models * 🎨 update defaults * 👷 fix version in build post-check * 🎨 test.js in dev * 🧑💻 cleaning * Add Gemini 1.5 Flash as a model (#1337) * 🎨 log dev data for tokens generated * Expose custom context provider registration through a vscode extension api. (#1288) * Fix typo "Experimantal" (#1353) * 🎨 Refactor core (#1281) * 🎨 refactor out react context * 🎨 refactor local storage * 🎨 refactor IdeMessenger * 🔥 removing unused files * 🚚 move scripts * ✅ setup VSCode integration testing * 🎨 fix test-related stuff * 🚧 testing * 🎨 tweak terminal cmd+L * 🧑💻 install biome in core * 🎨 run biome check on core * 🎨 run biome check in vscode * 🎨 run biome check in binary * 🧑💻 global biome config * 🎨 run biome check on core * 🎨 run biome check in vscode * 🎨 fix a few biome warnings * 🚧 WIP on protocols * 🚧 progress, organizing, about to remove webviewCore protocol * 🚧 now running * 🚧 move indexing code to core * 🚧 WIP on JetBrains indexing implementation * 🎨 finish listDir * 🚧 gui fixes * 🏷️ update IMessenger.on return type * 🐛 fix jetbrains types + IDE detection * 🧑💻 set up debugging for binary with TcpMessenger * 👷 fix prepackage.js * 🧑💻 turn off debug mode for intellij * 🐛 merge fixes * 🐛 fixes after refactor * 🐛 merge fixes * 💄 increase font size in JB * 🔀 merge changes * 🩹 small fixes * 🐛 fix jetbrains build * 🩹 more jetbrains fixes * 🎨 jetbrains improvements * 🐛 fix lancedb prob in jetbrains build * 👷 update jetbrains build process * ✨ intellij problems context provider * 👷 add script utils file * 💚 fix jetbrains build regression * 🎨 dynamic import transformers.js * 🩹 small jetbrains updates * ✨ folder context provider in jetbrains * 🎨 many more jetbrains improvements * 🔀 merge fixes * ⚡️ small improvements * 🎨 tell users transformers.js not supported * 🎨 trial updates * 📝 update jetbrains readmej * ⚡️ only use smaller context for local autocomplete models * ⚡️ improve bracket filter * ✨ global .continue/.prompts folder * 💄 improved model setup process * 🥅 display VSCode req errs * ⚡️ improved autocomplete stopping * 🎨 count cmd+I --------- Co-authored-by: Peter Zaback <pzaback@gmail.com> Co-authored-by: Maxime Brunet <max@brnt.mx> Co-authored-by: Jose Vega <bloguea.y.gana@gmail.com> Co-authored-by: Nejc Habjan <hab.nejc@gmail.com> Co-authored-by: Chad Yates <cyates@dynfxdigital.com> Co-authored-by: Justin Milner <jmilner@jmilner-lt2.deka.local> Co-authored-by: 小颚虫 <123357481+5eqn@users.noreply.github.com> Co-authored-by: 5eqn <491100866@qq.com> Co-authored-by: Pixel <54180211+pixelsortr@users.noreply.github.com> Co-authored-by: Justin Milner <42585006+justinmilner1@users.noreply.github.com> Co-authored-by: Devin Gould <djgould0628@gmail.com> Co-authored-by: Dipta Mahardhika <146386738+d-mahard@users.noreply.github.com> Co-authored-by: Ruben Kostandyan <kostard@amazon.com> Co-authored-by: tnglemongrass <113173292+tnglemongrass@users.noreply.github.com> Co-authored-by: Lukas Baron <LukasBaron@gmail.com> Co-authored-by: Fernando <fernando.sanchez.jr@gmail.com> Co-authored-by: Tijs Zwinkels <tijs@tinkertank.eu> Co-authored-by: DJ Johnson <mr.demarcus.johnson@gmail.com> Co-authored-by: sam <1211977+sambarnes@users.noreply.github.com> Co-authored-by: Pratik Parmar <steveparmar6nov2011@gmail.com> Co-authored-by: Sam El-Husseini <sam.elhusseini@gmail.com>
This commit is contained in:
Nate Sesti
GitHub
parent
311fc0a023
commit
08dc5db183
|
|
@ -11,11 +11,15 @@
|
|||
"name": "Core Binary",
|
||||
"skipFiles": ["<node_internals>/**"],
|
||||
"program": "${workspaceFolder}/binary/out/index.js",
|
||||
"preLaunchTask": "binary:esbuild",
|
||||
// "preLaunchTask": "binary:esbuild",
|
||||
"outFiles": ["${workspaceFolder}/binary/out/**/*.js"],
|
||||
"sourceMaps": true,
|
||||
"smartStep": true,
|
||||
"internalConsoleOptions": "openOnSessionStart"
|
||||
"internalConsoleOptions": "openOnSessionStart",
|
||||
"cwd": "${workspaceFolder}/binary",
|
||||
"env": {
|
||||
"CONTINUE_DEVELOPMENT": "true"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "node",
|
||||
|
|
@ -69,18 +73,17 @@
|
|||
// Pass a directory to run tests in
|
||||
"${workspaceFolder}/extensions/vscode/manual-testing-sandbox",
|
||||
"--extensionDevelopmentPath=${workspaceFolder}/extensions/vscode",
|
||||
"--extensionTestsPath=${workspaceFolder}/extensions/vscode/out/test-runner/mochaRunner"
|
||||
"--extensionTestsPath=${workspaceFolder}/extensions/vscode/out/test/runner/mochaRunner"
|
||||
],
|
||||
"outFiles": [
|
||||
// Allows setting breakpoints in test suites across the /src folder
|
||||
"${workspaceFolder}/extensions/vscode/out/test-suites/**/*.js",
|
||||
"${workspaceFolder}/extensions/vscode/out/test/test-suites/**/*.js",
|
||||
// Allows setting breakpoints in mocha test runner file
|
||||
"${workspaceFolder}/extensions/vscode/out/test-runner/**/*.js"
|
||||
"${workspaceFolder}/extensions/vscode/out/test/runner/**/*.js"
|
||||
],
|
||||
"internalConsoleOptions": "openOnSessionStart",
|
||||
"preLaunchTask": "vscode-extension:tests:build",
|
||||
"env": {
|
||||
"CONTINUE_SERVER_URL": "http://localhost:65432",
|
||||
// Avoid timing out when stopping on breakpoints during debugging in VSCode
|
||||
"MOCHA_TIMEOUT": "0"
|
||||
}
|
||||
|
|
|
|||
|
|
@ -2,3 +2,4 @@ node_modules
|
|||
bin
|
||||
data
|
||||
out
|
||||
tmp
|
||||
|
|
@ -26,3 +26,7 @@ The build process is otherwise defined entirely in `build.js`.
|
|||
- tree-sitter-wasms/
|
||||
|
||||
(\*) = need to download for each platform manually
|
||||
|
||||
## Debugging
|
||||
|
||||
To debug the binary with IntelliJ, set `useTcp` to `true` in `CoreMessenger.kt`, and then in VS Code run the "Core Binary" debug script. Instead of starting a subprocess for the binary and communicating over stdin/stdout, the IntelliJ extension will connect over TCP to the server started from the VS Code window. You can place breakpoints anywhere in the `core` or `binary` folders.
|
||||
|
|
|
|||
150
binary/build.js
150
binary/build.js
|
|
@ -1,21 +1,21 @@
|
|||
const esbuild = require("esbuild");
|
||||
const { execSync } = require("child_process");
|
||||
const fs = require("fs");
|
||||
const path = require("path");
|
||||
const ncp = require("ncp").ncp;
|
||||
const { rimrafSync } = require("rimraf");
|
||||
const { validateFilesPresent, execCmdSync } = require("../scripts/util");
|
||||
|
||||
function execCmdSync(cmd) {
|
||||
try {
|
||||
execSync(cmd);
|
||||
} catch (err) {
|
||||
console.error(`Error executing command '${cmd}': `, err.output.toString());
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
// Clean slate
|
||||
const bin = path.join(__dirname, "bin");
|
||||
const out = path.join(__dirname, "out");
|
||||
rimrafSync(bin);
|
||||
rimrafSync(out);
|
||||
rimrafSync(path.join(__dirname, "tmp"));
|
||||
fs.mkdirSync(bin);
|
||||
fs.mkdirSync(out);
|
||||
|
||||
const esbuildOutputFile = "out/index.js";
|
||||
const targets = [
|
||||
let targets = [
|
||||
"darwin-x64",
|
||||
"darwin-arm64",
|
||||
"linux-x64",
|
||||
|
|
@ -44,13 +44,14 @@ const targetToLanceDb = {
|
|||
"linux-arm64": "@lancedb/vectordb-linux-arm64-gnu",
|
||||
"linux-x64": "@lancedb/vectordb-linux-x64-gnu",
|
||||
"win32-x64": "@lancedb/vectordb-win32-x64-msvc",
|
||||
"win32-arm64": "@lancedb/vectordb-win32-x64-msvc", // they don't have a win32-arm64 build
|
||||
};
|
||||
|
||||
async function installNodeModuleInTempDirAndCopyToCurrent(package, toCopy) {
|
||||
console.log(`Copying ${package} to ${toCopy}`);
|
||||
async function installNodeModuleInTempDirAndCopyToCurrent(packageName, toCopy) {
|
||||
console.log(`Copying ${packageName} to ${toCopy}`);
|
||||
// This is a way to install only one package without npm trying to install all the dependencies
|
||||
// Create a temporary directory for installing the package
|
||||
const adjustedName = toCopy.replace(/^@/, "").replace("/", "-");
|
||||
const adjustedName = packageName.replace(/^@/, "").replace("/", "-");
|
||||
const tempDir = path.join(
|
||||
__dirname,
|
||||
"tmp",
|
||||
|
|
@ -58,10 +59,10 @@ async function installNodeModuleInTempDirAndCopyToCurrent(package, toCopy) {
|
|||
);
|
||||
const currentDir = process.cwd();
|
||||
|
||||
// Remove the dir we will be copying to
|
||||
rimrafSync(`node_modules/${toCopy}`);
|
||||
// // Remove the dir we will be copying to
|
||||
// rimrafSync(`node_modules/${toCopy}`);
|
||||
|
||||
// Ensure the temporary directory exists
|
||||
// // Ensure the temporary directory exists
|
||||
fs.mkdirSync(tempDir, { recursive: true });
|
||||
|
||||
try {
|
||||
|
|
@ -69,13 +70,17 @@ async function installNodeModuleInTempDirAndCopyToCurrent(package, toCopy) {
|
|||
process.chdir(tempDir);
|
||||
|
||||
// Initialize a new package.json and install the package
|
||||
execCmdSync(`npm init -y && npm i -f ${package} --no-save`);
|
||||
execCmdSync(`npm init -y && npm i -f ${packageName} --no-save`);
|
||||
|
||||
console.log(
|
||||
`Contents of: ${package}`,
|
||||
`Contents of: ${packageName}`,
|
||||
fs.readdirSync(path.join(tempDir, "node_modules", toCopy)),
|
||||
);
|
||||
|
||||
// Without this it seems the file isn't completely written to disk
|
||||
// Ideally we validate file integrity in the validation at the end
|
||||
await new Promise((resolve) => setTimeout(resolve, 2000));
|
||||
|
||||
// Copy the installed package back to the current directory
|
||||
await new Promise((resolve, reject) => {
|
||||
ncp(
|
||||
|
|
@ -84,7 +89,10 @@ async function installNodeModuleInTempDirAndCopyToCurrent(package, toCopy) {
|
|||
{ dereference: true },
|
||||
(error) => {
|
||||
if (error) {
|
||||
console.error(`[error] Error copying ${package} package`, error);
|
||||
console.error(
|
||||
`[error] Error copying ${packageName} package`,
|
||||
error,
|
||||
);
|
||||
reject(error);
|
||||
} else {
|
||||
resolve();
|
||||
|
|
@ -102,57 +110,13 @@ async function installNodeModuleInTempDirAndCopyToCurrent(package, toCopy) {
|
|||
}
|
||||
|
||||
(async () => {
|
||||
// console.log("[info] Building with ncc...");
|
||||
// execCmdSync(`npx ncc build src/index.ts -o out`);
|
||||
|
||||
// Copy node_modules for pre-built binaries
|
||||
const DYNAMIC_IMPORTS = [
|
||||
// "esbuild",
|
||||
// "@esbuild",
|
||||
// // "@lancedb",
|
||||
// "posthog-node",
|
||||
// "@octokit",
|
||||
];
|
||||
fs.mkdirSync("out/node_modules", { recursive: true });
|
||||
fs.mkdirSync("bin/node_modules", { recursive: true });
|
||||
|
||||
await Promise.all(
|
||||
DYNAMIC_IMPORTS.map(
|
||||
(mod) =>
|
||||
new Promise((resolve, reject) => {
|
||||
ncp(
|
||||
`node_modules/${mod}`,
|
||||
`out/node_modules/${mod}`,
|
||||
function (error) {
|
||||
if (error) {
|
||||
console.error(`[error] Error copying ${mod}`, error);
|
||||
reject(error);
|
||||
} else {
|
||||
resolve();
|
||||
}
|
||||
},
|
||||
);
|
||||
ncp(
|
||||
`node_modules/${mod}`,
|
||||
`bin/node_modules/${mod}`,
|
||||
function (error) {
|
||||
if (error) {
|
||||
console.error(`[error] Error copying ${mod}`, error);
|
||||
reject(error);
|
||||
} else {
|
||||
resolve();
|
||||
}
|
||||
},
|
||||
);
|
||||
}),
|
||||
),
|
||||
);
|
||||
console.log(`[info] Copied ${DYNAMIC_IMPORTS.join(", ")}`);
|
||||
|
||||
console.log("[info] Downloading prebuilt lancedb...");
|
||||
for (const target of targets) {
|
||||
if (targetToLanceDb[target]) {
|
||||
console.log(`[info] Downloading ${target}...`);
|
||||
console.log(`[info] Downloading for ${target}...`);
|
||||
await installNodeModuleInTempDirAndCopyToCurrent(
|
||||
targetToLanceDb[target],
|
||||
"@lancedb",
|
||||
|
|
@ -160,6 +124,38 @@ async function installNodeModuleInTempDirAndCopyToCurrent(package, toCopy) {
|
|||
}
|
||||
}
|
||||
|
||||
// tree-sitter-wasm
|
||||
const treeSitterWasmsDir = path.join(out, "tree-sitter-wasms");
|
||||
fs.mkdirSync(treeSitterWasmsDir);
|
||||
await new Promise((resolve, reject) => {
|
||||
ncp(
|
||||
path.join(
|
||||
__dirname,
|
||||
"..",
|
||||
"core",
|
||||
"node_modules",
|
||||
"tree-sitter-wasms",
|
||||
"out",
|
||||
),
|
||||
treeSitterWasmsDir,
|
||||
{ dereference: true },
|
||||
(error) => {
|
||||
if (error) {
|
||||
console.warn("[error] Error copying tree-sitter-wasm files", error);
|
||||
reject(error);
|
||||
} else {
|
||||
resolve();
|
||||
}
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
fs.copyFileSync(
|
||||
path.join(__dirname, "../core/vendor/tree-sitter.wasm"),
|
||||
path.join(__dirname, "out/tree-sitter.wasm"),
|
||||
);
|
||||
console.log("[info] Copied tree-sitter wasms");
|
||||
|
||||
console.log("[info] Cleaning up artifacts from previous builds...");
|
||||
|
||||
// delete asset backups generated by previous pkg invocations, if present
|
||||
|
|
@ -167,13 +163,13 @@ async function installNodeModuleInTempDirAndCopyToCurrent(package, toCopy) {
|
|||
fs.rmSync(assetPath, { force: true });
|
||||
}
|
||||
|
||||
console.log("[info] Building with esbuild...");
|
||||
// Bundles the extension into one file
|
||||
console.log("[info] Building with esbuild...");
|
||||
await esbuild.build({
|
||||
entryPoints: ["src/index.ts"],
|
||||
bundle: true,
|
||||
outfile: esbuildOutputFile,
|
||||
external: ["esbuild", ...DYNAMIC_IMPORTS, "./xhr-sync-worker.js", "vscode"],
|
||||
external: ["esbuild", "./xhr-sync-worker.js", "vscode"],
|
||||
format: "cjs",
|
||||
platform: "node",
|
||||
sourcemap: true,
|
||||
|
|
@ -208,6 +204,7 @@ async function installNodeModuleInTempDirAndCopyToCurrent(package, toCopy) {
|
|||
);
|
||||
|
||||
// Download and unzip prebuilt sqlite3 binary for the target
|
||||
console.log("[info] Downloading node-sqlite3");
|
||||
const downloadUrl = `https://github.com/TryGhost/node-sqlite3/releases/download/v5.1.7/sqlite3-v5.1.7-napi-v6-${
|
||||
target === "win32-arm64" ? "win32-ia32" : target
|
||||
}.tar.gz`;
|
||||
|
|
@ -240,9 +237,30 @@ async function installNodeModuleInTempDirAndCopyToCurrent(package, toCopy) {
|
|||
force: true,
|
||||
recursive: true,
|
||||
});
|
||||
|
||||
// copy @lancedb to bin folders
|
||||
console.log("[info] Copying @lancedb files to bin");
|
||||
fs.copyFileSync(
|
||||
`node_modules/${targetToLanceDb[target]}/index.node`,
|
||||
`${targetDir}/index.node`,
|
||||
);
|
||||
}
|
||||
// execCmdSync(
|
||||
// `npx pkg out/index.js --target node18-darwin-arm64 --no-bytecode --public-packages "*" --public -o bin/pkg`
|
||||
// );
|
||||
|
||||
const pathsToVerify = [];
|
||||
for (target of targets) {
|
||||
const exe = target.startsWith("win") ? ".exe" : "";
|
||||
const targetDir = `bin/${target}`;
|
||||
pathsToVerify.push(
|
||||
`${targetDir}/continue-binary${exe}`,
|
||||
`${targetDir}/esbuild${exe}`,
|
||||
`${targetDir}/index.node`, // @lancedb
|
||||
`${targetDir}/node_sqlite3.node`,
|
||||
);
|
||||
}
|
||||
validateFilesPresent(pathsToVerify);
|
||||
|
||||
console.log("[info] Done!");
|
||||
})();
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load Diff
|
|
@ -29,6 +29,7 @@
|
|||
"author": "",
|
||||
"license": "Apache-2.0",
|
||||
"devDependencies": {
|
||||
"@biomejs/biome": "1.6.4",
|
||||
"@types/follow-redirects": "^1.14.4",
|
||||
"@types/uuid": "^9.0.8",
|
||||
"@vercel/ncc": "^0.38.1",
|
||||
|
|
@ -44,6 +45,7 @@
|
|||
"follow-redirects": "^1.15.5",
|
||||
"mac-ca": "^2.0.3",
|
||||
"ncp": "^2.0.0",
|
||||
"node-fetch": "^3.3.2",
|
||||
"posthog-node": "^3.6.3",
|
||||
"system-ca": "^1.0.2",
|
||||
"uuid": "^9.0.1",
|
||||
|
|
|
|||
|
|
@ -9,7 +9,6 @@
|
|||
],
|
||||
"assets": [
|
||||
"../../../core/node_modules/sqlite3/**/*",
|
||||
"../../node_modules/@lancedb/vectordb-darwin-arm64/index.node",
|
||||
"../../out/tree-sitter.wasm",
|
||||
"../../out/tree-sitter-wasms/*"
|
||||
],
|
||||
|
|
|
|||
|
|
@ -9,7 +9,6 @@
|
|||
],
|
||||
"assets": [
|
||||
"../../../core/node_modules/sqlite3/**/*",
|
||||
"../../node_modules/@lancedb/vectordb-darwin-x64/index.node",
|
||||
"../../out/tree-sitter.wasm",
|
||||
"../../out/tree-sitter-wasms/*"
|
||||
],
|
||||
|
|
|
|||
|
|
@ -9,7 +9,6 @@
|
|||
],
|
||||
"assets": [
|
||||
"../../../core/node_modules/sqlite3/**/*",
|
||||
"../../node_modules/@lancedb/vectordb-linux-arm64-gnu/index.node",
|
||||
"../../out/tree-sitter.wasm",
|
||||
"../../out/tree-sitter-wasms/*"
|
||||
],
|
||||
|
|
|
|||
|
|
@ -9,7 +9,6 @@
|
|||
],
|
||||
"assets": [
|
||||
"../../../core/node_modules/sqlite3/**/*",
|
||||
"../../node_modules/@lancedb/vectordb-linux-x64-gnu/index.node",
|
||||
"../../out/tree-sitter.wasm",
|
||||
"../../out/tree-sitter-wasms/*"
|
||||
],
|
||||
|
|
|
|||
|
|
@ -9,7 +9,6 @@
|
|||
],
|
||||
"assets": [
|
||||
"../../../core/node_modules/sqlite3/**/*",
|
||||
"../../node_modules/@lancedb/vectordb-win32-x64-msvc/index.node",
|
||||
"../../out/tree-sitter.wasm",
|
||||
"../../out/tree-sitter-wasms/*",
|
||||
"../../node_modules/win-ca/lib/crypt32-ia32.node",
|
||||
|
|
|
|||
|
|
@ -9,7 +9,6 @@
|
|||
],
|
||||
"assets": [
|
||||
"../../../core/node_modules/sqlite3/**/*",
|
||||
"../../node_modules/@lancedb/vectordb-win32-x64-msvc/index.node",
|
||||
"../../out/tree-sitter.wasm",
|
||||
"../../out/tree-sitter-wasms/*",
|
||||
"../../node_modules/win-ca/lib/crypt32-ia32.node",
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
import { TODO } from "core/util";
|
||||
import { MessageIde } from "core/util/messageIde";
|
||||
import { IpcMessenger } from "./messenger";
|
||||
|
||||
export class IpcIde extends MessageIde {
|
||||
constructor(messenger: IpcMessenger) {
|
||||
constructor(messenger: TODO) {
|
||||
super(messenger.request.bind(messenger));
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,28 +1,17 @@
|
|||
import { IProtocol } from "core/protocol";
|
||||
import { IMessenger, type Message } from "core/util/messenger";
|
||||
import * as fs from "node:fs";
|
||||
import { v4 as uuidv4 } from "uuid";
|
||||
|
||||
import { getCoreLogsPath } from "core/util/paths";
|
||||
import * as fs from "fs";
|
||||
import { Message } from "../../core/util/messenger";
|
||||
import { Protocol, ReverseProtocol } from "./protocol";
|
||||
|
||||
export class IpcMessenger {
|
||||
typeListeners = new Map<keyof Protocol, ((message: Message) => any)[]>();
|
||||
export class IpcMessenger<
|
||||
ToProtocol extends IProtocol,
|
||||
FromProtocol extends IProtocol,
|
||||
> implements IMessenger<ToProtocol, FromProtocol>
|
||||
{
|
||||
typeListeners = new Map<keyof ToProtocol, ((message: Message) => any)[]>();
|
||||
idListeners = new Map<string, (message: Message) => any>();
|
||||
|
||||
constructor() {
|
||||
const logger = (message: any, ...optionalParams: any[]) => {
|
||||
const logFilePath = getCoreLogsPath();
|
||||
const timestamp = new Date().toISOString().split(".")[0];
|
||||
const logMessage = `[${timestamp}] ${message} ${optionalParams.join(
|
||||
" ",
|
||||
)}\n`;
|
||||
fs.appendFileSync(logFilePath, logMessage);
|
||||
};
|
||||
console.log = logger;
|
||||
console.error = logger;
|
||||
console.warn = logger;
|
||||
console.log("[info] Starting Continue core...");
|
||||
|
||||
process.stdin.on("data", (data) => {
|
||||
this._handleData(data);
|
||||
});
|
||||
|
|
@ -111,21 +100,25 @@ export class IpcMessenger {
|
|||
lines.forEach((line) => this._handleLine(line));
|
||||
}
|
||||
|
||||
send(messageType: string, message: any, messageId?: string): string {
|
||||
send<T extends keyof FromProtocol>(
|
||||
messageType: T,
|
||||
data: FromProtocol[T][0],
|
||||
messageId?: string,
|
||||
): string {
|
||||
messageId = messageId ?? uuidv4();
|
||||
const data: Message = {
|
||||
messageType,
|
||||
data: message,
|
||||
const msg: Message = {
|
||||
messageType: messageType as string,
|
||||
data,
|
||||
messageId,
|
||||
};
|
||||
// process.send?.(data);
|
||||
process.stdout?.write(JSON.stringify(data) + "\r\n");
|
||||
process.stdout?.write(JSON.stringify(msg) + "\r\n");
|
||||
return messageId;
|
||||
}
|
||||
|
||||
on<T extends keyof Protocol>(
|
||||
on<T extends keyof ToProtocol>(
|
||||
messageType: T,
|
||||
handler: (message: Message<Protocol[T][0]>) => Protocol[T][1],
|
||||
handler: (message: Message<ToProtocol[T][0]>) => ToProtocol[T][1],
|
||||
): void {
|
||||
if (!this.typeListeners.has(messageType)) {
|
||||
this.typeListeners.set(messageType, []);
|
||||
|
|
@ -133,21 +126,21 @@ export class IpcMessenger {
|
|||
this.typeListeners.get(messageType)?.push(handler);
|
||||
}
|
||||
|
||||
invoke<T extends keyof Protocol>(
|
||||
invoke<T extends keyof ToProtocol>(
|
||||
messageType: T,
|
||||
data: Protocol[T][0],
|
||||
): Protocol[T][1] {
|
||||
data: ToProtocol[T][0],
|
||||
): ToProtocol[T][1] {
|
||||
return this.typeListeners.get(messageType)?.[0]?.({
|
||||
messageId: uuidv4(),
|
||||
messageType,
|
||||
messageType: messageType as string,
|
||||
data,
|
||||
});
|
||||
}
|
||||
|
||||
request<T extends keyof ReverseProtocol>(
|
||||
request<T extends keyof FromProtocol>(
|
||||
messageType: T,
|
||||
data: ReverseProtocol[T][0],
|
||||
): Promise<ReverseProtocol[T][1]> {
|
||||
data: FromProtocol[T][0],
|
||||
): Promise<FromProtocol[T][1]> {
|
||||
const messageId = uuidv4();
|
||||
return new Promise((resolve) => {
|
||||
const handler = (msg: Message) => {
|
||||
|
|
@ -0,0 +1,162 @@
|
|||
import { IProtocol } from "core/protocol";
|
||||
import { IMessenger, Message } from "core/util/messenger";
|
||||
import net from "net";
|
||||
import { v4 as uuidv4 } from "uuid";
|
||||
|
||||
export class TcpMessenger<
|
||||
ToProtocol extends IProtocol,
|
||||
FromProtocol extends IProtocol,
|
||||
> implements IMessenger<ToProtocol, FromProtocol>
|
||||
{
|
||||
private port: number = 3000;
|
||||
private socket: net.Socket | null = null;
|
||||
|
||||
typeListeners = new Map<keyof ToProtocol, ((message: Message) => any)[]>();
|
||||
idListeners = new Map<string, (message: Message) => any>();
|
||||
|
||||
constructor() {
|
||||
const server = net.createServer((socket) => {
|
||||
this.socket = socket;
|
||||
socket.on("data", (data: Buffer) => {
|
||||
this._handleData(data);
|
||||
});
|
||||
|
||||
socket.on("end", () => {
|
||||
console.log("Disconnected from server");
|
||||
});
|
||||
|
||||
socket.on("error", (err: any) => {
|
||||
console.error("Client error:", err);
|
||||
});
|
||||
});
|
||||
|
||||
server.listen(this.port, () => {
|
||||
console.log(`Server listening on port ${this.port}`);
|
||||
});
|
||||
}
|
||||
|
||||
private _onErrorHandlers: ((error: Error) => void)[] = [];
|
||||
|
||||
onError(handler: (error: Error) => void) {
|
||||
this._onErrorHandlers.push(handler);
|
||||
}
|
||||
|
||||
public async awaitConnection() {
|
||||
while (!this.socket) {
|
||||
await new Promise((resolve) => setTimeout(resolve, 100));
|
||||
}
|
||||
}
|
||||
|
||||
private _handleLine(line: string) {
|
||||
try {
|
||||
const msg: Message = JSON.parse(line);
|
||||
if (msg.messageType === undefined || msg.messageId === undefined) {
|
||||
throw new Error("Invalid message sent: " + JSON.stringify(msg));
|
||||
}
|
||||
|
||||
// Call handler and respond with return value
|
||||
const listeners = this.typeListeners.get(msg.messageType as any);
|
||||
listeners?.forEach(async (handler) => {
|
||||
try {
|
||||
const response = await handler(msg);
|
||||
if (
|
||||
response &&
|
||||
typeof response[Symbol.asyncIterator] === "function"
|
||||
) {
|
||||
for await (const update of response) {
|
||||
this.send(msg.messageType, update, msg.messageId);
|
||||
}
|
||||
this.send(msg.messageType, { done: true }, msg.messageId);
|
||||
} else {
|
||||
this.send(msg.messageType, response || {}, msg.messageId);
|
||||
}
|
||||
} catch (e: any) {
|
||||
console.warn(`Error running handler for "${msg.messageType}": `, e);
|
||||
this._onErrorHandlers.forEach((handler) => {
|
||||
handler(e);
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
// Call handler which is waiting for the response, nothing to return
|
||||
this.idListeners.get(msg.messageId)?.(msg);
|
||||
} catch (e) {
|
||||
let truncatedLine = line;
|
||||
if (line.length > 200) {
|
||||
truncatedLine =
|
||||
line.substring(0, 100) + "..." + line.substring(line.length - 100);
|
||||
}
|
||||
console.error("Error parsing line: ", truncatedLine, e);
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
private _unfinishedLine: string | undefined = undefined;
|
||||
private _handleData(data: Buffer) {
|
||||
const d = data.toString();
|
||||
const lines = d.split(/\r\n/).filter((line) => line.trim() !== "");
|
||||
if (lines.length === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (this._unfinishedLine) {
|
||||
lines[0] = this._unfinishedLine + lines[0];
|
||||
this._unfinishedLine = undefined;
|
||||
}
|
||||
if (!d.endsWith("\r\n")) {
|
||||
this._unfinishedLine = lines.pop();
|
||||
}
|
||||
lines.forEach((line) => this._handleLine(line));
|
||||
}
|
||||
|
||||
send<T extends keyof FromProtocol>(
|
||||
messageType: T,
|
||||
data: FromProtocol[T][0],
|
||||
messageId?: string,
|
||||
): string {
|
||||
messageId = messageId ?? uuidv4();
|
||||
const msg: Message = {
|
||||
messageType: messageType as string,
|
||||
data,
|
||||
messageId,
|
||||
};
|
||||
|
||||
this.socket?.write(JSON.stringify(msg) + "\r\n");
|
||||
return messageId;
|
||||
}
|
||||
|
||||
on<T extends keyof ToProtocol>(
|
||||
messageType: T,
|
||||
handler: (message: Message<ToProtocol[T][0]>) => ToProtocol[T][1],
|
||||
): void {
|
||||
if (!this.typeListeners.has(messageType)) {
|
||||
this.typeListeners.set(messageType, []);
|
||||
}
|
||||
this.typeListeners.get(messageType)?.push(handler);
|
||||
}
|
||||
|
||||
invoke<T extends keyof ToProtocol>(
|
||||
messageType: T,
|
||||
data: ToProtocol[T][0],
|
||||
): ToProtocol[T][1] {
|
||||
return this.typeListeners.get(messageType)?.[0]?.({
|
||||
messageId: uuidv4(),
|
||||
messageType: messageType as string,
|
||||
data,
|
||||
});
|
||||
}
|
||||
request<T extends keyof FromProtocol>(
|
||||
messageType: T,
|
||||
data: FromProtocol[T][0],
|
||||
): Promise<FromProtocol[T][1]> {
|
||||
const messageId = uuidv4();
|
||||
return new Promise((resolve) => {
|
||||
const handler = (msg: Message) => {
|
||||
resolve(msg.data);
|
||||
this.idListeners.delete(messageId);
|
||||
};
|
||||
this.idListeners.set(messageId, handler);
|
||||
this.send(messageType, data, messageId);
|
||||
});
|
||||
}
|
||||
}
|
||||
|
|
@ -1,33 +1,36 @@
|
|||
process.env.IS_BINARY = "true";
|
||||
import { Command } from "commander";
|
||||
import { Core } from "core/core";
|
||||
import { FromCoreProtocol, ToCoreProtocol } from "core/protocol";
|
||||
import { IMessenger } from "core/util/messenger";
|
||||
import { getCoreLogsPath } from "core/util/paths";
|
||||
import fs from "fs";
|
||||
import fs from "node:fs";
|
||||
import { IpcIde } from "./IpcIde";
|
||||
import { setupCa } from "./ca";
|
||||
import { Core } from "./core";
|
||||
import { IpcMessenger } from "./messenger";
|
||||
import { IpcMessenger } from "./IpcMessenger";
|
||||
import { TcpMessenger } from "./TcpMessenger";
|
||||
import { setupCoreLogging } from "./logging";
|
||||
|
||||
const logFilePath = getCoreLogsPath();
|
||||
fs.appendFileSync(logFilePath, "[info] Starting Continue core...\n");
|
||||
|
||||
const program = new Command();
|
||||
|
||||
program.action(() => {
|
||||
program.action(async () => {
|
||||
try {
|
||||
const messenger = new IpcMessenger();
|
||||
setupCoreLogging();
|
||||
let messenger: IMessenger<ToCoreProtocol, FromCoreProtocol>;
|
||||
if (process.env.CONTINUE_DEVELOPMENT === "true") {
|
||||
messenger = new TcpMessenger<ToCoreProtocol, FromCoreProtocol>();
|
||||
console.log("Waiting for connection");
|
||||
await (
|
||||
messenger as TcpMessenger<ToCoreProtocol, FromCoreProtocol>
|
||||
).awaitConnection();
|
||||
console.log("Connected");
|
||||
} else {
|
||||
messenger = new IpcMessenger<ToCoreProtocol, FromCoreProtocol>();
|
||||
}
|
||||
const ide = new IpcIde(messenger);
|
||||
// const ide = new FileSystemIde();
|
||||
const core = new Core(messenger, ide);
|
||||
|
||||
setupCa();
|
||||
|
||||
// setTimeout(() => {
|
||||
// messenger.mock({
|
||||
// messageId: "2fe7823c-10bd-4771-abb5-781f520039ec",
|
||||
// messageType: "loadSubmenuItems",
|
||||
// data: { title: "issue" },
|
||||
// });
|
||||
// }, 1000);
|
||||
} catch (e) {
|
||||
fs.writeFileSync("./error.log", `${new Date().toISOString()} ${e}\n`);
|
||||
console.log("Error: ", e);
|
||||
|
|
|
|||
|
|
@ -0,0 +1,15 @@
|
|||
import { getCoreLogsPath } from "core/util/paths";
|
||||
import fs from "node:fs";
|
||||
|
||||
export function setupCoreLogging() {
|
||||
const logger = (message: any, ...optionalParams: any[]) => {
|
||||
const logFilePath = getCoreLogsPath();
|
||||
const timestamp = new Date().toISOString().split(".")[0];
|
||||
const logMessage = `[${timestamp}] ${message} ${optionalParams.join(" ")}\n`;
|
||||
fs.appendFileSync(logFilePath, logMessage);
|
||||
};
|
||||
console.log = logger;
|
||||
console.error = logger;
|
||||
console.warn = logger;
|
||||
console.log("[info] Starting Continue core...");
|
||||
}
|
||||
|
|
@ -0,0 +1,26 @@
|
|||
{
|
||||
"$schema": "https://biomejs.dev/schemas/1.6.4/schema.json",
|
||||
"organizeImports": {
|
||||
"enabled": true
|
||||
},
|
||||
"linter": {
|
||||
"enabled": true,
|
||||
"rules": {
|
||||
"recommended": true,
|
||||
"suspicious": {
|
||||
"noExplicitAny": "off",
|
||||
"noConfusingVoidType": "off"
|
||||
},
|
||||
"style": {
|
||||
"noUselessElse": "off",
|
||||
"noNonNullAssertion": "off"
|
||||
},
|
||||
"complexity": {
|
||||
"noForEach": "off"
|
||||
}
|
||||
}
|
||||
},
|
||||
"formatter": {
|
||||
"indentStyle": "space"
|
||||
}
|
||||
}
|
||||
|
|
@ -27,7 +27,7 @@ export async function getTreePathAtCursor(
|
|||
const path = [ast.rootNode];
|
||||
while (path[path.length - 1].childCount > 0) {
|
||||
let foundChild = false;
|
||||
for (let child of path[path.length - 1].children) {
|
||||
for (const child of path[path.length - 1].children) {
|
||||
if (child.startIndex <= cursorIndex && child.endIndex >= cursorIndex) {
|
||||
path.push(child);
|
||||
foundChild = true;
|
||||
|
|
@ -63,7 +63,7 @@ export async function getScopeAroundRange(
|
|||
let node = ast.rootNode;
|
||||
while (node.childCount > 0) {
|
||||
let foundChild = false;
|
||||
for (let child of node.children) {
|
||||
for (const child of node.children) {
|
||||
if (child.startIndex < startIndex && child.endIndex > endIndex) {
|
||||
node = child;
|
||||
foundChild = true;
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ import { DatabaseConnection } from "../indexing/refreshIndex.js";
|
|||
import { getTabAutocompleteCacheSqlitePath } from "../util/paths.js";
|
||||
|
||||
export class AutocompleteLruCache {
|
||||
private static capacity: number = 1000;
|
||||
private static capacity = 1000;
|
||||
|
||||
db: DatabaseConnection;
|
||||
|
||||
|
|
|
|||
|
|
@ -27,7 +27,7 @@ export async function* onlyWhitespaceAfterEndOfLine(
|
|||
|
||||
export async function* noFirstCharNewline(stream: AsyncGenerator<string>) {
|
||||
let first = true;
|
||||
for await (let char of stream) {
|
||||
for await (const char of stream) {
|
||||
if (first) {
|
||||
first = false;
|
||||
if (char === "\n") {
|
||||
|
|
@ -48,6 +48,7 @@ export async function* stopOnUnmatchedClosingBracket(
|
|||
stream: AsyncGenerator<string>,
|
||||
suffix: string,
|
||||
): AsyncGenerator<string> {
|
||||
// Add corresponding open brackets from suffix to stack
|
||||
const stack: string[] = [];
|
||||
for (let i = 0; i < suffix.length; i++) {
|
||||
if (suffix[i] === " ") continue;
|
||||
|
|
@ -57,7 +58,22 @@ export async function* stopOnUnmatchedClosingBracket(
|
|||
}
|
||||
|
||||
let all = "";
|
||||
let seenNonWhitespaceOrClosingBracket = false;
|
||||
for await (let chunk of stream) {
|
||||
// Allow closing brackets before any non-whitespace characters
|
||||
if (!seenNonWhitespaceOrClosingBracket) {
|
||||
const firstNonWhitespaceOrClosingBracketIndex =
|
||||
chunk.search(/[^\s\)\}\]]/);
|
||||
if (firstNonWhitespaceOrClosingBracketIndex !== -1) {
|
||||
yield chunk.slice(0, firstNonWhitespaceOrClosingBracketIndex);
|
||||
chunk = chunk.slice(firstNonWhitespaceOrClosingBracketIndex);
|
||||
seenNonWhitespaceOrClosingBracket = true;
|
||||
} else {
|
||||
yield chunk;
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
all += chunk;
|
||||
for (let i = 0; i < chunk.length; i++) {
|
||||
const char = chunk[i];
|
||||
|
|
|
|||
|
|
@ -8,6 +8,7 @@ import { streamLines } from "../diff/util.js";
|
|||
import {
|
||||
IDE,
|
||||
ILLM,
|
||||
ModelProvider,
|
||||
Position,
|
||||
Range,
|
||||
TabAutocompleteOptions,
|
||||
|
|
@ -103,11 +104,11 @@ function formatExternalSnippet(
|
|||
) {
|
||||
const comment = language.comment;
|
||||
const lines = [
|
||||
comment + " Path: " + getBasename(filepath),
|
||||
`${comment} Path: ${getBasename(filepath)}`,
|
||||
...snippet
|
||||
.trim()
|
||||
.split("\n")
|
||||
.map((line) => comment + " " + line),
|
||||
.map((line) => `${comment} ${line}`),
|
||||
comment,
|
||||
];
|
||||
return lines.join("\n");
|
||||
|
|
@ -281,7 +282,7 @@ export async function getTabCompletion(
|
|||
)
|
||||
.join("\n");
|
||||
if (formattedSnippets.length > 0) {
|
||||
prefix = formattedSnippets + "\n\n" + prefix;
|
||||
prefix = `${formattedSnippets}\n\n${prefix}`;
|
||||
}
|
||||
|
||||
prompt = compiledTemplate({
|
||||
|
|
@ -308,7 +309,7 @@ export async function getTabCompletion(
|
|||
cacheHit = true;
|
||||
completion = cachedCompletion;
|
||||
} else {
|
||||
let stop = [
|
||||
const stop = [
|
||||
...(completionOptions?.stop || []),
|
||||
...multilineStops,
|
||||
...commonStops,
|
||||
|
|
@ -323,7 +324,7 @@ export async function getTabCompletion(
|
|||
(options.multilineCompletions === "always" || completeMultiline);
|
||||
|
||||
// Try to reuse pending requests if what the user typed matches start of completion
|
||||
let generator = generatorReuseManager.getGenerator(
|
||||
const generator = generatorReuseManager.getGenerator(
|
||||
prefix,
|
||||
() =>
|
||||
llm.streamComplete(prompt, {
|
||||
|
|
@ -399,7 +400,7 @@ export async function getTabCompletion(
|
|||
|
||||
export class CompletionProvider {
|
||||
private static debounceTimeout: NodeJS.Timeout | undefined = undefined;
|
||||
private static debouncing: boolean = false;
|
||||
private static debouncing = false;
|
||||
private static lastUUID: string | undefined = undefined;
|
||||
|
||||
constructor(
|
||||
|
|
@ -527,6 +528,7 @@ export class CompletionProvider {
|
|||
return undefined;
|
||||
}
|
||||
|
||||
// Debounce
|
||||
if (CompletionProvider.debouncing) {
|
||||
CompletionProvider.debounceTimeout?.refresh();
|
||||
const lastUUID = await new Promise((resolve) =>
|
||||
|
|
@ -555,6 +557,18 @@ export class CompletionProvider {
|
|||
llm.completionOptions.temperature = 0.01;
|
||||
}
|
||||
|
||||
// Set model-specific options
|
||||
const LOCAL_PROVIDERS: ModelProvider[] = [
|
||||
"ollama",
|
||||
"lmstudio",
|
||||
"llama.cpp",
|
||||
"llamafile",
|
||||
"text-gen-webui",
|
||||
];
|
||||
if (LOCAL_PROVIDERS.includes(llm.providerName)) {
|
||||
options.maxPromptTokens = 500;
|
||||
}
|
||||
|
||||
const outcome = await getTabCompletion(
|
||||
token,
|
||||
options,
|
||||
|
|
|
|||
|
|
@ -14,10 +14,10 @@ import {
|
|||
Typescript,
|
||||
} from "./languages.js";
|
||||
import {
|
||||
AutocompleteSnippet,
|
||||
fillPromptWithSnippets,
|
||||
rankSnippets,
|
||||
removeRangeFromSnippets,
|
||||
type AutocompleteSnippet,
|
||||
} from "./ranking.js";
|
||||
import { RecentlyEditedRange, findMatchingRange } from "./recentlyEdited.js";
|
||||
|
||||
|
|
@ -97,7 +97,7 @@ async function shouldCompleteMultiline(
|
|||
|
||||
let completeMultiline = false;
|
||||
if (treePath) {
|
||||
let cursorLine = fullPrefix.split("\n").length - 1;
|
||||
const cursorLine = fullPrefix.split("\n").length - 1;
|
||||
completeMultiline = shouldCompleteMultilineAst(treePath, cursorLine);
|
||||
}
|
||||
return completeMultiline;
|
||||
|
|
@ -124,14 +124,14 @@ export async function constructAutocompletePrompt(
|
|||
}> {
|
||||
// Construct basic prefix
|
||||
const maxPrefixTokens = options.maxPromptTokens * options.prefixPercentage;
|
||||
let prefix = pruneLinesFromTop(fullPrefix, maxPrefixTokens, modelName);
|
||||
const prefix = pruneLinesFromTop(fullPrefix, maxPrefixTokens, modelName);
|
||||
|
||||
// Construct suffix
|
||||
const maxSuffixTokens = Math.min(
|
||||
options.maxPromptTokens - countTokens(prefix, modelName),
|
||||
options.maxSuffixPercentage * options.maxPromptTokens,
|
||||
);
|
||||
let suffix = pruneLinesFromBottom(fullSuffix, maxSuffixTokens, modelName);
|
||||
const suffix = pruneLinesFromBottom(fullSuffix, maxSuffixTokens, modelName);
|
||||
|
||||
// Find external snippets
|
||||
let snippets: AutocompleteSnippet[] = [];
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
export function isOnlyPunctuationAndWhitespace(completion: string): boolean {
|
||||
const punctuationAndWhitespaceRegex = /^[^\w\d]+$/;
|
||||
const punctuationAndWhitespaceRegex = /^[^\w\d\}\)\]]+$/;
|
||||
return punctuationAndWhitespaceRegex.test(completion);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
import { distance } from "fastest-levenshtein";
|
||||
import { DiffLine } from "../index.js";
|
||||
import { LineStream } from "../diff/util.js";
|
||||
import { DiffLine } from "../index.js";
|
||||
|
||||
export async function* noTopLevelKeywordsMidline(
|
||||
lines: LineStream,
|
||||
|
|
@ -8,7 +8,7 @@ export async function* noTopLevelKeywordsMidline(
|
|||
): LineStream {
|
||||
for await (const line of lines) {
|
||||
for (const keyword of topLevelKeywords) {
|
||||
const indexOf = line.indexOf(keyword + " ");
|
||||
const indexOf = line.indexOf(`${keyword} `);
|
||||
if (indexOf >= 0 && line.slice(indexOf - 1, indexOf).trim() !== "") {
|
||||
yield line.slice(0, indexOf);
|
||||
break;
|
||||
|
|
@ -26,7 +26,7 @@ export async function* avoidPathLine(
|
|||
// Sometimes the model with copy this pattern, which is unwanted
|
||||
for await (const line of stream) {
|
||||
// Also filter lines that are empty comments
|
||||
if (line.startsWith(comment + " Path: ") || line.trim() === comment) {
|
||||
if (line.startsWith(`${comment} Path: `) || line.trim() === comment) {
|
||||
continue;
|
||||
}
|
||||
yield line;
|
||||
|
|
@ -51,6 +51,15 @@ function isBracketEnding(line: string): boolean {
|
|||
.split("")
|
||||
.some((char) => bracketEnding.includes(char));
|
||||
}
|
||||
|
||||
function commonPrefixLength(a: string, b: string): number {
|
||||
let i = 0;
|
||||
while (i < a.length && i < b.length && a[i] === b[i]) {
|
||||
i++;
|
||||
}
|
||||
return i;
|
||||
}
|
||||
|
||||
export async function* stopAtSimilarLine(
|
||||
stream: LineStream,
|
||||
line: string,
|
||||
|
|
@ -64,7 +73,11 @@ export async function* stopAtSimilarLine(
|
|||
}
|
||||
|
||||
let lineQualifies = nextLine.length > 4 && line.length > 4;
|
||||
if (lineQualifies && distance(nextLine.trim(), line) / line.length < 0.1) {
|
||||
if (
|
||||
lineQualifies &&
|
||||
(commonPrefixLength(nextLine.trim(), line.trim()) > 8 ||
|
||||
distance(nextLine.trim(), line) / line.length < 0.1)
|
||||
) {
|
||||
break;
|
||||
}
|
||||
yield nextLine;
|
||||
|
|
@ -123,9 +136,8 @@ export async function* filterCodeBlockLines(rawLines: LineStream): LineStream {
|
|||
if (!seenValidLine) {
|
||||
if (shouldRemoveLineBeforeStart(line)) {
|
||||
continue;
|
||||
} else {
|
||||
seenValidLine = true;
|
||||
}
|
||||
seenValidLine = true;
|
||||
}
|
||||
|
||||
// Filter out ending ```
|
||||
|
|
@ -158,7 +170,9 @@ function isEnglishFirstLine(line: string) {
|
|||
line.startsWith("here's") ||
|
||||
line.startsWith("sure, here") ||
|
||||
line.startsWith("sure thing") ||
|
||||
line.startsWith("sure!")
|
||||
line.startsWith("sure!") ||
|
||||
line.startsWith("to fill") ||
|
||||
line.startsWith("the code should")
|
||||
) {
|
||||
return true;
|
||||
}
|
||||
|
|
@ -169,7 +183,7 @@ function isEnglishFirstLine(line: string) {
|
|||
export async function* filterEnglishLinesAtStart(lines: LineStream) {
|
||||
let i = 0;
|
||||
let wasEnglishFirstLine = false;
|
||||
for await (let line of lines) {
|
||||
for await (const line of lines) {
|
||||
if (i === 0 && line.trim() === "") {
|
||||
continue;
|
||||
}
|
||||
|
|
@ -200,7 +214,7 @@ function isEnglishPostExplanation(line: string): boolean {
|
|||
|
||||
export async function* filterEnglishLinesAtEnd(lines: LineStream) {
|
||||
let finishedCodeBlock = false;
|
||||
for await (let line of lines) {
|
||||
for await (const line of lines) {
|
||||
if (line.trim() === "```") {
|
||||
finishedCodeBlock = true;
|
||||
}
|
||||
|
|
@ -213,7 +227,7 @@ export async function* filterEnglishLinesAtEnd(lines: LineStream) {
|
|||
|
||||
export async function* fixCodeLlamaFirstLineIndentation(lines: LineStream) {
|
||||
let isFirstLine = true;
|
||||
for await (let line of lines) {
|
||||
for await (const line of lines) {
|
||||
if (isFirstLine && line.startsWith(" ")) {
|
||||
yield line.slice(2);
|
||||
isFirstLine = false;
|
||||
|
|
@ -233,8 +247,8 @@ export async function* filterLeadingAndTrailingNewLineInsertion(
|
|||
): AsyncGenerator<DiffLine> {
|
||||
let isFirst = true;
|
||||
let buffer: DiffLine[] = [];
|
||||
for await (let diffLine of diffLines) {
|
||||
let isBlankLineInsertion =
|
||||
for await (const diffLine of diffLines) {
|
||||
const isBlankLineInsertion =
|
||||
diffLine.type === "new" && isUselessLine(diffLine.line);
|
||||
if (isFirst && isBlankLineInsertion) {
|
||||
isFirst = false;
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
import { Range } from "../index.js";
|
||||
import { RangeInFileWithContents } from "../commands/util.js";
|
||||
import { Range } from "../index.js";
|
||||
import { countTokens } from "../llm/countTokens.js";
|
||||
|
||||
export type AutocompleteSnippet = RangeInFileWithContents & {
|
||||
|
|
@ -115,7 +115,7 @@ function mergeOverlappingRangeContents(
|
|||
): string {
|
||||
const firstLines = first.contents.split("\n");
|
||||
const numOverlapping = first.range.end.line - second.range.start.line;
|
||||
return firstLines.slice(-numOverlapping).join("\n") + "\n" + second.contents;
|
||||
return `${firstLines.slice(-numOverlapping).join("\n")}\n${second.contents}`;
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -135,7 +135,6 @@ export function fillPromptWithSnippets(
|
|||
tokensRemaining -= tokenCount;
|
||||
keptSnippets.push(snippet);
|
||||
} else {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -147,18 +146,17 @@ function rangeIntersectionByLines(a: Range, b: Range): Range | null {
|
|||
const endLine = Math.min(a.end.line, b.end.line);
|
||||
if (startLine >= endLine) {
|
||||
return null;
|
||||
} else {
|
||||
return {
|
||||
start: {
|
||||
line: startLine,
|
||||
character: 0,
|
||||
},
|
||||
end: {
|
||||
line: endLine,
|
||||
character: 0,
|
||||
},
|
||||
};
|
||||
}
|
||||
return {
|
||||
start: {
|
||||
line: startLine,
|
||||
character: 0,
|
||||
},
|
||||
end: {
|
||||
line: endLine,
|
||||
character: 0,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -171,7 +169,8 @@ function rangeDifferenceByLines(orig: Range, remove: Range): Range[] {
|
|||
) {
|
||||
// / | | /
|
||||
return [];
|
||||
} else if (
|
||||
}
|
||||
if (
|
||||
orig.start.line <= remove.start.line &&
|
||||
orig.end.line >= remove.end.line
|
||||
) {
|
||||
|
|
@ -187,7 +186,8 @@ function rangeDifferenceByLines(orig: Range, remove: Range): Range[] {
|
|||
end: orig.end,
|
||||
},
|
||||
];
|
||||
} else if (
|
||||
}
|
||||
if (
|
||||
orig.start.line >= remove.start.line &&
|
||||
orig.end.line >= remove.end.line
|
||||
) {
|
||||
|
|
@ -198,7 +198,8 @@ function rangeDifferenceByLines(orig: Range, remove: Range): Range[] {
|
|||
end: orig.end,
|
||||
},
|
||||
];
|
||||
} else if (
|
||||
}
|
||||
if (
|
||||
orig.start.line <= remove.start.line &&
|
||||
orig.end.line <= remove.end.line
|
||||
) {
|
||||
|
|
@ -209,9 +210,8 @@ function rangeDifferenceByLines(orig: Range, remove: Range): Range[] {
|
|||
end: remove.start,
|
||||
},
|
||||
];
|
||||
} else {
|
||||
return [orig];
|
||||
}
|
||||
return [orig];
|
||||
}
|
||||
|
||||
export function removeRangeFromSnippets(
|
||||
|
|
@ -220,7 +220,7 @@ export function removeRangeFromSnippets(
|
|||
range: Range,
|
||||
): Required<AutocompleteSnippet>[] {
|
||||
const finalSnippets: Required<AutocompleteSnippet>[] = [];
|
||||
for (let snippet of snippets) {
|
||||
for (const snippet of snippets) {
|
||||
if (snippet.filepath !== filepath) {
|
||||
finalSnippets.push(snippet);
|
||||
continue;
|
||||
|
|
|
|||
|
|
@ -51,16 +51,14 @@ const starcoder2FimTemplate: AutocompleteTemplate = {
|
|||
const otherFiles =
|
||||
snippets.length === 0
|
||||
? ""
|
||||
: "<file_sep>" +
|
||||
snippets
|
||||
: `<file_sep>${snippets
|
||||
.map((snippet) => {
|
||||
return snippet.contents;
|
||||
// return `${getBasename(snippet.filepath)}\n${snippet.contents}`;
|
||||
})
|
||||
.join("<file_sep>") +
|
||||
"<file_sep>";
|
||||
.join("<file_sep>")}<file_sep>`;
|
||||
|
||||
let prompt = `${otherFiles}<fim_prefix>${prefix}<fim_suffix>${suffix}<fim_middle>`;
|
||||
const prompt = `${otherFiles}<fim_prefix>${prefix}<fim_suffix>${suffix}<fim_middle>`;
|
||||
return prompt;
|
||||
},
|
||||
completionOptions: {
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@ export class ListenableGenerator<T> {
|
|||
private _source: AsyncGenerator<T>;
|
||||
private _buffer: T[] = [];
|
||||
private _listeners: Set<(value: T) => void> = new Set();
|
||||
private _isEnded: boolean = false;
|
||||
private _isEnded = false;
|
||||
|
||||
constructor(
|
||||
source: AsyncGenerator<T>,
|
||||
|
|
@ -55,7 +55,7 @@ export class ListenableGenerator<T> {
|
|||
}
|
||||
while (!this._isEnded) {
|
||||
let resolve: (value: any) => void;
|
||||
let promise = new Promise<T>((res) => {
|
||||
const promise = new Promise<T>((res) => {
|
||||
resolve = res;
|
||||
this._listeners.add(resolve!);
|
||||
});
|
||||
|
|
@ -77,7 +77,7 @@ export class ListenableGenerator<T> {
|
|||
export class GeneratorReuseManager {
|
||||
currentGenerator: ListenableGenerator<string> | undefined;
|
||||
pendingGeneratorPrefix: string | undefined;
|
||||
pendingCompletion: string = "";
|
||||
pendingCompletion = "";
|
||||
|
||||
constructor(private readonly onError: (err: any) => void) {}
|
||||
|
||||
|
|
@ -117,7 +117,7 @@ export class GeneratorReuseManager {
|
|||
}
|
||||
|
||||
let alreadyTyped = prefix.slice(this.pendingGeneratorPrefix?.length) || "";
|
||||
for await (let chunk of this.currentGenerator!.tee()) {
|
||||
for await (let chunk of this.currentGenerator?.tee() ?? []) {
|
||||
if (!chunk) {
|
||||
continue;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,15 @@
|
|||
{
|
||||
"$schema": "https://biomejs.dev/schemas/1.6.4/schema.json",
|
||||
"organizeImports": {
|
||||
"enabled": true
|
||||
},
|
||||
"linter": {
|
||||
"enabled": true,
|
||||
"rules": {
|
||||
"recommended": true
|
||||
}
|
||||
},
|
||||
"formatter": {
|
||||
"indentStyle": "space"
|
||||
}
|
||||
}
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
import { SlashCommand } from "../../index.js";
|
||||
import { streamLines } from "../../diff/util.js";
|
||||
import { SlashCommand } from "../../index.js";
|
||||
import { removeQuotesAndEscapes } from "../../util/index.js";
|
||||
|
||||
const GenerateTerminalCommand: SlashCommand = {
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ const CommitMessageCommand: SlashCommand = {
|
|||
const diff = await ide.getDiff();
|
||||
|
||||
if (!diff || diff.trim() === "") {
|
||||
yield "No changes detected. Make sure you are in a git repository with current changes."
|
||||
yield "No changes detected. Make sure you are in a git repository with current changes.";
|
||||
return;
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -34,7 +34,7 @@ const DraftIssueCommand: SlashCommand = {
|
|||
{ maxTokens: 20 },
|
||||
);
|
||||
|
||||
title = removeQuotesAndEscapes(title.trim()) + "\n\n";
|
||||
title = `${removeQuotesAndEscapes(title.trim())}\n\n`;
|
||||
yield title;
|
||||
|
||||
let body = "";
|
||||
|
|
|
|||
|
|
@ -14,8 +14,8 @@ import {
|
|||
getMarkdownLanguageTagForFile,
|
||||
} from "../../util/index.js";
|
||||
import {
|
||||
RangeInFileWithContents,
|
||||
contextItemToRangeInFileWithContents,
|
||||
type RangeInFileWithContents,
|
||||
} from "../util.js";
|
||||
|
||||
const PROMPT = `Take the file prefix and suffix into account, but only rewrite the code_to_edit as specified in the user_request. The code you write in modified_code_to_edit will replace the code between the code_to_edit tags. Do NOT preface your answer or write anything other than code. The </modified_code_to_edit> tag should be written to indicate the end of the modified code section. Do not ever use nested tags.
|
||||
|
|
@ -60,7 +60,7 @@ export async function getPromptParts(
|
|||
input: string,
|
||||
tokenLimit: number | undefined,
|
||||
) {
|
||||
let maxTokens = Math.floor(model.contextLength / 2);
|
||||
const maxTokens = Math.floor(model.contextLength / 2);
|
||||
|
||||
const TOKENS_TO_BE_CONSIDERED_LARGE_RANGE = tokenLimit ?? 1200;
|
||||
// if (model.countTokens(rif.contents) > TOKENS_TO_BE_CONSIDERED_LARGE_RANGE) {
|
||||
|
|
@ -75,9 +75,9 @@ export async function getPromptParts(
|
|||
BUFFER_FOR_FUNCTIONS +
|
||||
maxTokens;
|
||||
|
||||
let fullFileContentsList = fullFileContents.split("\n");
|
||||
let maxStartLine = rif.range.start.line;
|
||||
let minEndLine = rif.range.end.line;
|
||||
const fullFileContentsList = fullFileContents.split("\n");
|
||||
const maxStartLine = rif.range.start.line;
|
||||
const minEndLine = rif.range.end.line;
|
||||
let curStartLine = 0;
|
||||
let curEndLine = fullFileContentsList.length - 1;
|
||||
|
||||
|
|
@ -134,7 +134,7 @@ export async function getPromptParts(
|
|||
rif.contents = rif.contents.substring(1);
|
||||
}
|
||||
while (rif.contents.endsWith("\n")) {
|
||||
fileSuffix = "\n" + fileSuffix;
|
||||
fileSuffix = `\n${fileSuffix}`;
|
||||
rif.contents = rif.contents.substring(0, rif.contents.length - 1);
|
||||
}
|
||||
}
|
||||
|
|
@ -147,7 +147,7 @@ function compilePrompt(
|
|||
fileSuffix: string,
|
||||
input: string,
|
||||
): string {
|
||||
if (contents.trim() == "") {
|
||||
if (contents.trim() === "") {
|
||||
// Separate prompt for insertion at the cursor, the other tends to cause it to repeat whole file
|
||||
return `\
|
||||
<file_prefix>
|
||||
|
|
@ -165,7 +165,7 @@ Please output the code to be inserted at the cursor in order to fulfill the user
|
|||
}
|
||||
|
||||
let prompt = PROMPT;
|
||||
if (filePrefix.trim() != "") {
|
||||
if (filePrefix.trim() !== "") {
|
||||
prompt += `
|
||||
<file_prefix>
|
||||
${filePrefix}
|
||||
|
|
@ -176,7 +176,7 @@ ${filePrefix}
|
|||
${contents}
|
||||
</code_to_edit>`;
|
||||
|
||||
if (fileSuffix.trim() != "") {
|
||||
if (fileSuffix.trim() !== "") {
|
||||
prompt += `
|
||||
<file_suffix>
|
||||
${fileSuffix}
|
||||
|
|
@ -200,7 +200,7 @@ function isEndLine(line: string) {
|
|||
);
|
||||
}
|
||||
|
||||
function lineToBeIgnored(line: string, isFirstLine: boolean = false): boolean {
|
||||
function lineToBeIgnored(line: string, isFirstLine = false): boolean {
|
||||
return (
|
||||
line.includes("```") ||
|
||||
line.includes("<modified_code_to_edit>") ||
|
||||
|
|
@ -237,14 +237,14 @@ const EditSlashCommand: SlashCommand = {
|
|||
let content = history[history.length - 1].content;
|
||||
if (typeof content !== "string") {
|
||||
content.forEach((part) => {
|
||||
if (part.text && part.text.startsWith("/edit")) {
|
||||
if (part.text?.startsWith("/edit")) {
|
||||
part.text = part.text.replace("/edit", "").trimStart();
|
||||
}
|
||||
});
|
||||
} else {
|
||||
content = input.replace("/edit", "").trimStart();
|
||||
}
|
||||
let userInput = stripImages(content).replace(
|
||||
const userInput = stripImages(content).replace(
|
||||
`\`\`\`${contextItemToEdit.name}\n${contextItemToEdit.content}\n\`\`\`\n`,
|
||||
"",
|
||||
);
|
||||
|
|
@ -253,7 +253,7 @@ const EditSlashCommand: SlashCommand = {
|
|||
contextItemToRangeInFileWithContents(contextItemToEdit);
|
||||
|
||||
await ide.saveFile(rif.filepath);
|
||||
let fullFileContents = await ide.readFile(rif.filepath);
|
||||
const fullFileContents = await ide.readFile(rif.filepath);
|
||||
|
||||
let { filePrefix, contents, fileSuffix, maxTokens } = await getPromptParts(
|
||||
rif,
|
||||
|
|
@ -266,18 +266,18 @@ const EditSlashCommand: SlashCommand = {
|
|||
dedentAndGetCommonWhitespace(contents);
|
||||
contents = dedentedContents;
|
||||
|
||||
let prompt = compilePrompt(filePrefix, contents, fileSuffix, userInput);
|
||||
let fullFileContentsLines = fullFileContents.split("\n");
|
||||
let fullPrefixLines = fullFileContentsLines.slice(
|
||||
const prompt = compilePrompt(filePrefix, contents, fileSuffix, userInput);
|
||||
const fullFileContentsLines = fullFileContents.split("\n");
|
||||
const fullPrefixLines = fullFileContentsLines.slice(
|
||||
0,
|
||||
Math.max(0, rif.range.start.line - 1),
|
||||
);
|
||||
let fullSuffixLines = fullFileContentsLines.slice(rif.range.end.line);
|
||||
const fullSuffixLines = fullFileContentsLines.slice(rif.range.end.line);
|
||||
|
||||
let linesToDisplay: string[] = [];
|
||||
|
||||
async function sendDiffUpdate(lines: string[], final: boolean = false) {
|
||||
let completion = lines.join("\n");
|
||||
async function sendDiffUpdate(lines: string[], final = false) {
|
||||
const completion = lines.join("\n");
|
||||
|
||||
// Don't do this at the very end, just show the inserted code
|
||||
if (final) {
|
||||
|
|
@ -286,9 +286,9 @@ const EditSlashCommand: SlashCommand = {
|
|||
|
||||
// Only recalculate at every new-line, because this is sort of expensive
|
||||
else if (completion.endsWith("\n")) {
|
||||
let contentsLines = rif.contents.split("\n");
|
||||
const contentsLines = rif.contents.split("\n");
|
||||
let rewrittenLines = 0;
|
||||
for (let line of lines) {
|
||||
for (const line of lines) {
|
||||
for (let i = rewrittenLines; i < contentsLines.length; i++) {
|
||||
if (
|
||||
// difflib.SequenceMatcher(
|
||||
|
|
@ -306,22 +306,18 @@ const EditSlashCommand: SlashCommand = {
|
|||
linesToDisplay = contentsLines.slice(rewrittenLines);
|
||||
}
|
||||
|
||||
let newFileContents =
|
||||
fullPrefixLines.join("\n") +
|
||||
"\n" +
|
||||
completion +
|
||||
"\n" +
|
||||
(linesToDisplay.length > 0 ? linesToDisplay.join("\n") + "\n" : "") +
|
||||
fullSuffixLines.join("\n");
|
||||
const newFileContents = `${fullPrefixLines.join("\n")}\n${completion}\n${
|
||||
linesToDisplay.length > 0 ? `${linesToDisplay.join("\n")}\n` : ""
|
||||
}${fullSuffixLines.join("\n")}`;
|
||||
|
||||
let stepIndex = history.length - 1;
|
||||
const stepIndex = history.length - 1;
|
||||
|
||||
await ide.showDiff(rif.filepath, newFileContents, stepIndex);
|
||||
}
|
||||
|
||||
// Important state variables
|
||||
// -------------------------
|
||||
let originalLines = rif.contents === "" ? [] : rif.contents.split("\n");
|
||||
const originalLines = rif.contents === "" ? [] : rif.contents.split("\n");
|
||||
// In the actual file, taking into account block offset
|
||||
let currentLineInFile = rif.range.start.line;
|
||||
let currentBlockLines: string[] = [];
|
||||
|
|
@ -360,9 +356,9 @@ const EditSlashCommand: SlashCommand = {
|
|||
|
||||
// In a block, and have already matched at least one line
|
||||
// Check if the next line matches, for each of the candidates
|
||||
let matchesFound: any[] = [];
|
||||
const matchesFound: any[] = [];
|
||||
let firstValidMatch: any = null;
|
||||
for (let [
|
||||
for (const [
|
||||
index_of_last_matched_line,
|
||||
num_lines_matched,
|
||||
] of indicesOfLastMatchedLines) {
|
||||
|
|
@ -394,7 +390,7 @@ const EditSlashCommand: SlashCommand = {
|
|||
|
||||
// We added some lines to the block that were matched (including maybe some blank lines)
|
||||
// So here we will strip all matching lines from the end of currentBlockLines
|
||||
let linesStripped: string[] = [];
|
||||
const linesStripped: string[] = [];
|
||||
let indexOfLastLineInBlock: number = firstValidMatch[0];
|
||||
while (
|
||||
currentBlockLines.length > 0 &&
|
||||
|
|
@ -418,9 +414,9 @@ const EditSlashCommand: SlashCommand = {
|
|||
}
|
||||
|
||||
// Always look for new matching candidates
|
||||
let newMatches: any[] = [];
|
||||
const newMatches: any[] = [];
|
||||
for (let i = 0; i < originalLinesBelowPreviousBlocks.length; i++) {
|
||||
let ogLine = originalLinesBelowPreviousBlocks[i];
|
||||
const ogLine = originalLinesBelowPreviousBlocks[i];
|
||||
// TODO: It's a bit sus to be disqualifying empty lines.
|
||||
// What you ideally do is find ALL matches, and then throw them out as you check the following lines
|
||||
if (ogLine === line) {
|
||||
|
|
@ -442,17 +438,17 @@ const EditSlashCommand: SlashCommand = {
|
|||
messages[messages.length - 1] = { role: "user", content: prompt };
|
||||
|
||||
let linesOfPrefixCopied = 0;
|
||||
let lines = [];
|
||||
let unfinishedLine: string = "";
|
||||
const lines = [];
|
||||
let unfinishedLine = "";
|
||||
let completionLinesCovered = 0;
|
||||
let repeatingFileSuffix = false;
|
||||
let lineBelowHighlightedRange = fileSuffix.trim().split("\n")[0];
|
||||
const lineBelowHighlightedRange = fileSuffix.trim().split("\n")[0];
|
||||
|
||||
// Use custom templates defined by the model
|
||||
const template = llm.promptTemplates?.edit;
|
||||
let generator: AsyncGenerator<string>;
|
||||
if (template) {
|
||||
let rendered = llm.renderPromptTemplate(
|
||||
const rendered = llm.renderPromptTemplate(
|
||||
template,
|
||||
// typeof template === 'string' ? template : template.prompt,
|
||||
messages.slice(0, messages.length - 1),
|
||||
|
|
@ -498,7 +494,7 @@ const EditSlashCommand: SlashCommand = {
|
|||
);
|
||||
} else {
|
||||
async function* gen() {
|
||||
for await (let chunk of llm.streamChat(messages, {
|
||||
for await (const chunk of llm.streamChat(messages, {
|
||||
temperature: 0.5, // TODO
|
||||
maxTokens: Math.min(
|
||||
maxTokens,
|
||||
|
|
@ -513,7 +509,7 @@ const EditSlashCommand: SlashCommand = {
|
|||
generator = gen();
|
||||
}
|
||||
|
||||
for await (let chunk of generator) {
|
||||
for await (const chunk of generator) {
|
||||
// Stop early if it is repeating the fileSuffix or the step was deleted
|
||||
if (repeatingFileSuffix) {
|
||||
break;
|
||||
|
|
@ -523,7 +519,7 @@ const EditSlashCommand: SlashCommand = {
|
|||
yield undefined;
|
||||
|
||||
// Accumulate lines
|
||||
let chunkLines = chunk.split("\n");
|
||||
const chunkLines = chunk.split("\n");
|
||||
chunkLines[0] = unfinishedLine + chunkLines[0];
|
||||
if (chunk.endsWith("\n")) {
|
||||
unfinishedLine = "";
|
||||
|
|
@ -543,11 +539,11 @@ const EditSlashCommand: SlashCommand = {
|
|||
break;
|
||||
}
|
||||
// Lines that should be ignored, like the <> tags
|
||||
else if (lineToBeIgnored(chunkLines[i], completionLinesCovered === 0)) {
|
||||
if (lineToBeIgnored(chunkLines[i], completionLinesCovered === 0)) {
|
||||
continue; // noice
|
||||
}
|
||||
// Check if we are currently just copying the prefix
|
||||
else if (
|
||||
if (
|
||||
(linesOfPrefixCopied > 0 || completionLinesCovered === 0) &&
|
||||
linesOfPrefixCopied < filePrefix.split("\n").length &&
|
||||
chunkLines[i] === fullPrefixLines[linesOfPrefixCopied]
|
||||
|
|
@ -558,7 +554,7 @@ const EditSlashCommand: SlashCommand = {
|
|||
}
|
||||
// Because really short lines might be expected to be repeated, this is only a !heuristic!
|
||||
// Stop when it starts copying the fileSuffix
|
||||
else if (
|
||||
if (
|
||||
chunkLines[i].trim() === lineBelowHighlightedRange.trim() &&
|
||||
chunkLines[i].trim().length > 4 &&
|
||||
!(
|
||||
|
|
|
|||
|
|
@ -26,7 +26,9 @@ const HttpSlashCommand: SlashCommand = {
|
|||
const reader = response.body.getReader();
|
||||
while (true) {
|
||||
const { done, value } = await reader.read();
|
||||
if (done) {break;}
|
||||
if (done) {
|
||||
break;
|
||||
}
|
||||
const decoded = new TextDecoder("utf-8").decode(value);
|
||||
yield decoded;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -4,9 +4,9 @@ import CommitMessageCommand from "./commit.js";
|
|||
import DraftIssueCommand from "./draftIssue.js";
|
||||
import EditSlashCommand from "./edit.js";
|
||||
import HttpSlashCommand from "./http.js";
|
||||
import ReviewMessageCommand from "./review.js";
|
||||
import ShareSlashCommand from "./share.js";
|
||||
import StackOverflowSlashCommand from "./stackOverflow.js";
|
||||
import ReviewMessageCommand from "./review.js";
|
||||
|
||||
export default [
|
||||
DraftIssueCommand,
|
||||
|
|
|
|||
|
|
@ -1,6 +1,5 @@
|
|||
import { SlashCommand } from "../../index.js";
|
||||
import { ChatMessage, SlashCommand } from "../../index.js";
|
||||
import { stripImages } from "../../llm/countTokens.js";
|
||||
import { ChatMessage } from "../../index.js";
|
||||
|
||||
const prompt = `
|
||||
Review the following code, focusing on Readability, Maintainability, Code Smells, Speed, and Memory Performance. Provide feedback with these guidelines:
|
||||
|
|
@ -11,22 +10,23 @@ const prompt = `
|
|||
Provide Examples: For each issue identified, offer an example of how the code could be improved or rewritten for better clarity, performance, or maintainability.
|
||||
Your response should be structured to first identify the issue, then explain why it’s a problem, and finally, offer a solution with example code.`;
|
||||
|
||||
|
||||
function getLastUserHistory(history: ChatMessage[]): string {
|
||||
const lastUserHistory = history
|
||||
.reverse()
|
||||
.find((message) => message.role === "user");
|
||||
|
||||
if (!lastUserHistory) {return "";}
|
||||
if (!lastUserHistory) {
|
||||
return "";
|
||||
}
|
||||
|
||||
if (lastUserHistory.content instanceof Array) {
|
||||
if (Array.isArray(lastUserHistory.content)) {
|
||||
return lastUserHistory.content.reduce(
|
||||
(acc: string, current: { type: string; text?: string }) => {
|
||||
return current.type === "text" && current.text
|
||||
? acc + current.text
|
||||
: acc;
|
||||
},
|
||||
""
|
||||
"",
|
||||
);
|
||||
}
|
||||
|
||||
|
|
@ -39,8 +39,7 @@ const ReviewMessageCommand: SlashCommand = {
|
|||
name: "review",
|
||||
description: "Review code and give feedback",
|
||||
run: async function* ({ llm, history }) {
|
||||
|
||||
let reviewText = getLastUserHistory(history).replace("\\review", "");
|
||||
const reviewText = getLastUserHistory(history).replace("\\review", "");
|
||||
|
||||
const content = `${prompt} \r\n ${reviewText}`;
|
||||
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
import * as fs from "node:fs";
|
||||
import { homedir } from "node:os";
|
||||
import path from "path";
|
||||
import * as fs from "fs";
|
||||
import { homedir } from "os";
|
||||
import { SlashCommand } from "../../index.js";
|
||||
import { languageForFilepath } from "../../autocomplete/constructPrompt.js";
|
||||
import { SlashCommand } from "../../index.js";
|
||||
import { stripImages } from "../../llm/countTokens.js";
|
||||
|
||||
// If useful elsewhere, helper funcs should move to core/util/index.ts or similar
|
||||
|
|
@ -24,7 +24,8 @@ function asBasicISOString(date: Date): string {
|
|||
|
||||
function reformatCodeBlocks(msgText: string): string {
|
||||
const codeBlockFenceRegex = /```((.*?\.(\w+))\s*.*)\n/g;
|
||||
msgText = msgText.replace(codeBlockFenceRegex,
|
||||
msgText = msgText.replace(
|
||||
codeBlockFenceRegex,
|
||||
(match, metadata, filename, extension) => {
|
||||
const lang = languageForFilepath(filename);
|
||||
return `\`\`\`${extension}\n${lang.comment} ${metadata}\n`;
|
||||
|
|
|
|||
|
|
@ -18,11 +18,11 @@ export function contextItemToRangeInFileWithContents(
|
|||
filepath: item.description.split(" (")[0],
|
||||
range: {
|
||||
start: {
|
||||
line: parseInt(lines[0]),
|
||||
line: Number.parseInt(lines[0]),
|
||||
character: 0,
|
||||
},
|
||||
end: {
|
||||
line: parseInt(lines[1]),
|
||||
line: Number.parseInt(lines[1]),
|
||||
character: 0,
|
||||
},
|
||||
},
|
||||
|
|
|
|||
|
|
@ -1,11 +1,14 @@
|
|||
import { ContinueConfig, ContinueRcJson, IDE, ILLM, IContextProvider } from "../index.js";
|
||||
import { IdeSettings } from "../protocol.js";
|
||||
import { Telemetry } from "../util/posthog.js";
|
||||
import {
|
||||
BrowserSerializedContinueConfig,
|
||||
finalToBrowserConfig,
|
||||
loadFullConfigNode,
|
||||
} from "./load.js";
|
||||
ContinueConfig,
|
||||
ContinueRcJson,
|
||||
IContextProvider,
|
||||
IDE,
|
||||
ILLM,
|
||||
} from "../index.js";
|
||||
import { IdeSettings } from "../protocol/ideWebview.js";
|
||||
import { Telemetry } from "../util/posthog.js";
|
||||
import { finalToBrowserConfig, loadFullConfigNode } from "./load.js";
|
||||
|
||||
export class ConfigHandler {
|
||||
private savedConfig: ContinueConfig | undefined;
|
||||
|
|
@ -14,12 +17,12 @@ export class ConfigHandler {
|
|||
|
||||
constructor(
|
||||
private readonly ide: IDE,
|
||||
private ideSettings: IdeSettings,
|
||||
private ideSettingsPromise: Promise<IdeSettings>,
|
||||
private readonly writeLog: (text: string) => Promise<void>,
|
||||
private readonly onConfigUpdate: () => void,
|
||||
) {
|
||||
this.ide = ide;
|
||||
this.ideSettings = ideSettings;
|
||||
this.ideSettingsPromise = ideSettingsPromise;
|
||||
this.writeLog = writeLog;
|
||||
this.onConfigUpdate = onConfigUpdate;
|
||||
try {
|
||||
|
|
@ -30,7 +33,7 @@ export class ConfigHandler {
|
|||
}
|
||||
|
||||
updateIdeSettings(ideSettings: IdeSettings) {
|
||||
this.ideSettings = ideSettings;
|
||||
this.ideSettingsPromise = Promise.resolve(ideSettings);
|
||||
this.reloadConfig();
|
||||
}
|
||||
|
||||
|
|
@ -67,7 +70,7 @@ export class ConfigHandler {
|
|||
this.savedConfig = await loadFullConfigNode(
|
||||
this.ide,
|
||||
workspaceConfigs,
|
||||
this.ideSettings,
|
||||
await this.ideSettingsPromise,
|
||||
ideInfo.ideType,
|
||||
uniqueId,
|
||||
this.writeLog,
|
||||
|
|
|
|||
|
|
@ -32,7 +32,7 @@ import { AllEmbeddingsProviders } from "../indexing/embeddings/index.js";
|
|||
import { BaseLLM } from "../llm/index.js";
|
||||
import CustomLLMClass from "../llm/llms/CustomLLM.js";
|
||||
import { llmFromDescription } from "../llm/llms/index.js";
|
||||
import { IdeSettings } from "../protocol.js";
|
||||
import { IdeSettings } from "../protocol/ideWebview.js";
|
||||
import { fetchwithRequestOptions } from "../util/fetchWithOptions.js";
|
||||
import { copyOf } from "../util/index.js";
|
||||
import mergeJson from "../util/merge.js";
|
||||
|
|
@ -43,6 +43,7 @@ import {
|
|||
getConfigJsonPathForRemote,
|
||||
getConfigTsPath,
|
||||
getContinueDotEnv,
|
||||
readAllGlobalPromptFiles,
|
||||
} from "../util/paths.js";
|
||||
import {
|
||||
defaultContextProvidersJetBrains,
|
||||
|
|
@ -55,7 +56,7 @@ const { execSync } = require("child_process");
|
|||
|
||||
function resolveSerializedConfig(filepath: string): SerializedContinueConfig {
|
||||
let content = fs.readFileSync(filepath, "utf8");
|
||||
let config = JSON.parse(content) as SerializedContinueConfig;
|
||||
const config = JSON.parse(content) as SerializedContinueConfig;
|
||||
if (config.env && Array.isArray(config.env)) {
|
||||
const env = {
|
||||
...process.env,
|
||||
|
|
@ -64,7 +65,7 @@ function resolveSerializedConfig(filepath: string): SerializedContinueConfig {
|
|||
|
||||
config.env.forEach((envVar) => {
|
||||
if (envVar in env) {
|
||||
content = content.replaceAll(
|
||||
content = (content as any).replaceAll(
|
||||
new RegExp(`"${envVar}"`, "g"),
|
||||
`"${env[envVar]}"`,
|
||||
);
|
||||
|
|
@ -157,6 +158,10 @@ async function serializedToIntermediateConfig(
|
|||
)
|
||||
.flat()
|
||||
.filter(({ path }) => path.endsWith(".prompt"));
|
||||
|
||||
// Also read from ~/.continue/.prompts
|
||||
promptFiles.push(...readAllGlobalPromptFiles());
|
||||
|
||||
for (const file of promptFiles) {
|
||||
slashCommands.push(slashCommandFromPromptFile(file.path, file.content));
|
||||
}
|
||||
|
|
@ -216,7 +221,7 @@ async function intermediateToFinalConfig(
|
|||
{
|
||||
...desc,
|
||||
model: modelName,
|
||||
title: llm.title + " - " + modelName,
|
||||
title: `${llm.title} - ${modelName}`,
|
||||
},
|
||||
ide.readFile.bind(ide),
|
||||
uniqueId,
|
||||
|
|
@ -313,14 +318,20 @@ async function intermediateToFinalConfig(
|
|||
const { provider, ...options } = embeddingsProviderDescription;
|
||||
const embeddingsProviderClass = AllEmbeddingsProviders[provider];
|
||||
if (embeddingsProviderClass) {
|
||||
config.embeddingsProvider = new embeddingsProviderClass(
|
||||
options,
|
||||
(url: string | URL, init: any) =>
|
||||
fetchwithRequestOptions(url, init, {
|
||||
...config.requestOptions,
|
||||
...options.requestOptions,
|
||||
}),
|
||||
);
|
||||
if (
|
||||
embeddingsProviderClass.name === "_TransformersJsEmbeddingsProvider"
|
||||
) {
|
||||
config.embeddingsProvider = new embeddingsProviderClass();
|
||||
} else {
|
||||
config.embeddingsProvider = new embeddingsProviderClass(
|
||||
options,
|
||||
(url: string | URL, init: any) =>
|
||||
fetchwithRequestOptions(url, init, {
|
||||
...config.requestOptions,
|
||||
...options.requestOptions,
|
||||
}),
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -431,14 +442,13 @@ async function buildConfigTs() {
|
|||
try {
|
||||
if (process.env.IS_BINARY === "true") {
|
||||
execSync(
|
||||
escapeSpacesInPath(path.dirname(process.execPath)) +
|
||||
`/esbuild${
|
||||
getTarget().startsWith("win32") ? ".exe" : ""
|
||||
} ${escapeSpacesInPath(
|
||||
getConfigTsPath(),
|
||||
)} --bundle --outfile=${escapeSpacesInPath(
|
||||
getConfigJsPath(),
|
||||
)} --platform=node --format=cjs --sourcemap --external:fetch --external:fs --external:path --external:os --external:child_process`,
|
||||
`${escapeSpacesInPath(path.dirname(process.execPath))}/esbuild${
|
||||
getTarget().startsWith("win32") ? ".exe" : ""
|
||||
} ${escapeSpacesInPath(
|
||||
getConfigTsPath(),
|
||||
)} --bundle --outfile=${escapeSpacesInPath(
|
||||
getConfigJsPath(),
|
||||
)} --platform=node --format=cjs --sourcemap --external:fetch --external:fs --external:path --external:os --external:child_process`,
|
||||
);
|
||||
} else {
|
||||
// Dynamic import esbuild so potentially disastrous errors can be caught
|
||||
|
|
@ -456,7 +466,7 @@ async function buildConfigTs() {
|
|||
}
|
||||
} catch (e) {
|
||||
console.log(
|
||||
"Build error. Please check your ~/.continue/config.ts file: " + e,
|
||||
`Build error. Please check your ~/.continue/config.ts file: ${e}`,
|
||||
);
|
||||
return undefined;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -413,7 +413,6 @@ declare global {
|
|||
subprocess(command: string): Promise<[string, string]>;
|
||||
getProblems(filepath?: string | undefined): Promise<Problem[]>;
|
||||
getBranch(dir: string): Promise<string>;
|
||||
getStats(directory: string): Promise<{ [path: string]: number }>;
|
||||
getTags(artifactId: string): Promise<IndexTag[]>;
|
||||
getRepoName(dir: string): Promise<string | undefined>;
|
||||
}
|
||||
|
|
@ -706,7 +705,7 @@ declare global {
|
|||
inlineEdit?: string;
|
||||
}
|
||||
|
||||
interface ExperimantalConfig {
|
||||
interface ExperimentalConfig {
|
||||
contextMenuPrompts?: ContextMenuConfig;
|
||||
modelRoles?: ModelRoles;
|
||||
}
|
||||
|
|
@ -729,7 +728,7 @@ declare global {
|
|||
tabAutocompleteOptions?: Partial<TabAutocompleteOptions>;
|
||||
ui?: ContinueUIConfig;
|
||||
reranker?: RerankerDescription;
|
||||
experimental?: ExperimantalConfig;
|
||||
experimental?: ExperimentalConfig;
|
||||
}
|
||||
|
||||
export type ConfigMergeType = "merge" | "overwrite";
|
||||
|
|
@ -775,7 +774,7 @@ declare global {
|
|||
/** Options for the reranker */
|
||||
reranker?: RerankerDescription | Reranker;
|
||||
/** Experimental configuration */
|
||||
experimental?: ExperimantalConfig;
|
||||
experimental?: ExperimentalConfig;
|
||||
}
|
||||
|
||||
export interface ContinueConfig {
|
||||
|
|
@ -794,7 +793,7 @@ declare global {
|
|||
tabAutocompleteOptions?: Partial<TabAutocompleteOptions>;
|
||||
ui?: ContinueUIConfig;
|
||||
reranker?: Reranker;
|
||||
experimental?: ExperimantalConfig;
|
||||
experimental?: ExperimentalConfig;
|
||||
}
|
||||
|
||||
export interface BrowserSerializedContinueConfig {
|
||||
|
|
@ -811,7 +810,7 @@ declare global {
|
|||
embeddingsProvider?: string;
|
||||
ui?: ContinueUIConfig;
|
||||
reranker?: RerankerDescription;
|
||||
experimental?: ExperimantalConfig;
|
||||
experimental?: ExperimentalConfig;
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
import {
|
||||
import type {
|
||||
ContextItem,
|
||||
ContextProviderDescription,
|
||||
ContextProviderExtras,
|
||||
|
|
|
|||
|
|
@ -1,4 +1,3 @@
|
|||
import { BaseContextProvider } from "../index.js";
|
||||
import {
|
||||
ContextItem,
|
||||
ContextProviderDescription,
|
||||
|
|
@ -7,6 +6,7 @@ import {
|
|||
LoadSubmenuItemsArgs,
|
||||
} from "../../index.js";
|
||||
import { CodeSnippetsCodebaseIndex } from "../../indexing/CodeSnippetsIndex.js";
|
||||
import { BaseContextProvider } from "../index.js";
|
||||
|
||||
class CodeContextProvider extends BaseContextProvider {
|
||||
static description: ContextProviderDescription = {
|
||||
|
|
@ -21,7 +21,9 @@ class CodeContextProvider extends BaseContextProvider {
|
|||
extras: ContextProviderExtras,
|
||||
): Promise<ContextItem[]> {
|
||||
// Assume the query is the id as returned by loadSubmenuItems
|
||||
return [await CodeSnippetsCodebaseIndex.getForId(parseInt(query, 10))];
|
||||
return [
|
||||
await CodeSnippetsCodebaseIndex.getForId(Number.parseInt(query, 10)),
|
||||
];
|
||||
}
|
||||
|
||||
async loadSubmenuItems(
|
||||
|
|
|
|||
|
|
@ -1,10 +1,10 @@
|
|||
import { BaseContextProvider } from "../index.js";
|
||||
import {
|
||||
ContextItem,
|
||||
ContextProviderDescription,
|
||||
ContextProviderExtras,
|
||||
} from "../../index.js";
|
||||
import { getBasename } from "../../util/index.js";
|
||||
import { BaseContextProvider } from "../index.js";
|
||||
|
||||
// import { getHighlightsThatFit, ILLMContextSizer } from "llm-code-highlighter/dist/index.continue";
|
||||
|
||||
|
|
|
|||
|
|
@ -1,10 +1,10 @@
|
|||
import { BaseContextProvider } from "../index.js";
|
||||
import {
|
||||
ContextItem,
|
||||
ContextProviderDescription,
|
||||
ContextProviderExtras,
|
||||
} from "../../index.js";
|
||||
import { getBasename } from "../../util/index.js";
|
||||
import { BaseContextProvider } from "../index.js";
|
||||
|
||||
// import { getOutlines } from "llm-code-highlighter/dist/index.continue";
|
||||
|
||||
|
|
|
|||
|
|
@ -1,9 +1,9 @@
|
|||
import { BaseContextProvider } from "../index.js";
|
||||
import {
|
||||
ContextItem,
|
||||
ContextProviderDescription,
|
||||
ContextProviderExtras,
|
||||
} from "../../index.js";
|
||||
import { BaseContextProvider } from "../index.js";
|
||||
import { retrieveContextItemsFromEmbeddings } from "../retrieval/retrieval.js";
|
||||
|
||||
class CodebaseContextProvider extends BaseContextProvider {
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
import {
|
||||
import type {
|
||||
ContextItem,
|
||||
ContextProviderDescription,
|
||||
ContextProviderExtras,
|
||||
|
|
|
|||
|
|
@ -1,4 +1,3 @@
|
|||
import { BaseContextProvider } from "../index.js";
|
||||
import {
|
||||
ContextItem,
|
||||
ContextProviderDescription,
|
||||
|
|
@ -6,6 +5,7 @@ import {
|
|||
ContextSubmenuItem,
|
||||
LoadSubmenuItemsArgs,
|
||||
} from "../../index.js";
|
||||
import { BaseContextProvider } from "../index.js";
|
||||
|
||||
class DatabaseContextProvider extends BaseContextProvider {
|
||||
static description: ContextProviderDescription = {
|
||||
|
|
@ -27,12 +27,12 @@ class DatabaseContextProvider extends BaseContextProvider {
|
|||
return contextItems;
|
||||
}
|
||||
|
||||
let [connectionName, table] = query.split(".");
|
||||
const [connectionName, table] = query.split(".");
|
||||
|
||||
const getDatabaseAdapter = await require("dbinfoz");
|
||||
|
||||
for (const connection of connections) {
|
||||
if (connection.name == connectionName) {
|
||||
if (connection.name === connectionName) {
|
||||
const adapter = getDatabaseAdapter(
|
||||
connection.connection_type,
|
||||
connection.connection,
|
||||
|
|
@ -45,7 +45,7 @@ class DatabaseContextProvider extends BaseContextProvider {
|
|||
let prompt = `Schema for all tables on ${connection.connection_type} is `;
|
||||
prompt += JSON.stringify(tablesAndSchemas);
|
||||
|
||||
let contextItem = {
|
||||
const contextItem = {
|
||||
name: `${connectionName}-all-tables-schemas`,
|
||||
description: "Schema for all tables.",
|
||||
content: prompt,
|
||||
|
|
@ -60,7 +60,7 @@ class DatabaseContextProvider extends BaseContextProvider {
|
|||
let prompt = `Schema for ${tableName} on ${connection.connection_type} is `;
|
||||
prompt += JSON.stringify(tablesAndSchemas[tableName]);
|
||||
|
||||
let contextItem = {
|
||||
const contextItem = {
|
||||
name: `${connectionName}-${tableName}-schema`,
|
||||
description: `${tableName} Schema`,
|
||||
content: prompt,
|
||||
|
|
@ -89,7 +89,7 @@ class DatabaseContextProvider extends BaseContextProvider {
|
|||
const getDatabaseAdapter = await require("dbinfoz");
|
||||
|
||||
for (const connection of connections) {
|
||||
let adapter = getDatabaseAdapter(
|
||||
const adapter = getDatabaseAdapter(
|
||||
connection.connection_type,
|
||||
connection.connection,
|
||||
);
|
||||
|
|
@ -98,7 +98,7 @@ class DatabaseContextProvider extends BaseContextProvider {
|
|||
);
|
||||
const tables = Object.keys(tablesAndSchemas);
|
||||
|
||||
let contextItem = {
|
||||
const contextItem = {
|
||||
id: `${connection.name}.all`,
|
||||
title: `${connection.name} all table schemas`,
|
||||
description: "",
|
||||
|
|
@ -107,7 +107,7 @@ class DatabaseContextProvider extends BaseContextProvider {
|
|||
contextItems.push(contextItem);
|
||||
|
||||
tables.forEach((tableName) => {
|
||||
let contextItem = {
|
||||
const contextItem = {
|
||||
id: `${connection.name}.${tableName}`,
|
||||
title: `${connection.name}.${tableName} schema`,
|
||||
description: "",
|
||||
|
|
|
|||
|
|
@ -1,9 +1,9 @@
|
|||
import { BaseContextProvider } from "../index.js";
|
||||
import {
|
||||
ContextItem,
|
||||
ContextProviderDescription,
|
||||
ContextProviderExtras,
|
||||
} from "../../index.js";
|
||||
import { BaseContextProvider } from "../index.js";
|
||||
|
||||
class DiffContextProvider extends BaseContextProvider {
|
||||
static description: ContextProviderDescription = {
|
||||
|
|
@ -21,7 +21,10 @@ class DiffContextProvider extends BaseContextProvider {
|
|||
return [
|
||||
{
|
||||
description: "The current git diff",
|
||||
content: `\`\`\`git diff\n${diff}\n\`\`\``,
|
||||
content:
|
||||
diff.trim() === ""
|
||||
? "Git shows no current changes."
|
||||
: `\`\`\`git diff\n${diff}\n\`\`\``,
|
||||
name: "Git Diff",
|
||||
},
|
||||
];
|
||||
|
|
|
|||
|
|
@ -1,4 +1,3 @@
|
|||
import { BaseContextProvider } from "../index.js";
|
||||
import {
|
||||
ContextItem,
|
||||
ContextProviderDescription,
|
||||
|
|
@ -8,6 +7,7 @@ import {
|
|||
} from "../../index.js";
|
||||
import configs from "../../indexing/docs/preIndexedDocs.js";
|
||||
import TransformersJsEmbeddingsProvider from "../../indexing/embeddings/TransformersJsEmbeddingsProvider.js";
|
||||
import { BaseContextProvider } from "../index.js";
|
||||
|
||||
class DocsContextProvider extends BaseContextProvider {
|
||||
static DEFAULT_N_RETRIEVE = 30;
|
||||
|
|
@ -23,6 +23,13 @@ class DocsContextProvider extends BaseContextProvider {
|
|||
query: string,
|
||||
extras: ContextProviderExtras,
|
||||
): Promise<ContextItem[]> {
|
||||
// Not supported in JetBrains IDEs right now
|
||||
if ((await extras.ide.getIdeInfo()).ideType === "jetbrains") {
|
||||
throw new Error(
|
||||
"The @docs context provider is not currently supported in JetBrains IDEs. We'll have an update soon!",
|
||||
);
|
||||
}
|
||||
|
||||
const { retrieveDocs } = await import("../../indexing/docs/db");
|
||||
const embeddingsProvider = new TransformersJsEmbeddingsProvider();
|
||||
const [vector] = await embeddingsProvider.embed([extras.fullInput]);
|
||||
|
|
@ -104,9 +111,13 @@ class DocsContextProvider extends BaseContextProvider {
|
|||
|
||||
// Sort submenuItems such that the objects with titles which don't occur in configs occur first, and alphabetized
|
||||
submenuItems.sort((a, b) => {
|
||||
const aTitleInConfigs = !!configs.find(config => config.title === a.title);
|
||||
const bTitleInConfigs = !!configs.find(config => config.title === b.title);
|
||||
|
||||
const aTitleInConfigs = !!configs.find(
|
||||
(config) => config.title === a.title,
|
||||
);
|
||||
const bTitleInConfigs = !!configs.find(
|
||||
(config) => config.title === b.title,
|
||||
);
|
||||
|
||||
// Primary criterion: Items not in configs come first
|
||||
if (!aTitleInConfigs && bTitleInConfigs) {
|
||||
return -1;
|
||||
|
|
@ -114,7 +125,7 @@ class DocsContextProvider extends BaseContextProvider {
|
|||
return 1;
|
||||
} else {
|
||||
// Secondary criterion: Alphabetical order when both items are in the same category
|
||||
return a.title.toString().localeCompare(b.title.toString());
|
||||
return a.title.toString().localeCompare(b.title.toString());
|
||||
}
|
||||
});
|
||||
|
||||
|
|
|
|||
|
|
@ -1,4 +1,3 @@
|
|||
import { BaseContextProvider } from "../index.js";
|
||||
import {
|
||||
ContextItem,
|
||||
ContextProviderDescription,
|
||||
|
|
@ -7,6 +6,7 @@ import {
|
|||
LoadSubmenuItemsArgs,
|
||||
} from "../../index.js";
|
||||
import { getBasename, getLastNPathParts } from "../../util/index.js";
|
||||
import { BaseContextProvider } from "../index.js";
|
||||
|
||||
class FileContextProvider extends BaseContextProvider {
|
||||
static description: ContextProviderDescription = {
|
||||
|
|
|
|||
|
|
@ -1,9 +1,9 @@
|
|||
import { BaseContextProvider } from "../index.js";
|
||||
import {
|
||||
ContextItem,
|
||||
ContextProviderDescription,
|
||||
ContextProviderExtras,
|
||||
} from "../../index.js";
|
||||
import { BaseContextProvider } from "../index.js";
|
||||
|
||||
interface Directory {
|
||||
name: string;
|
||||
|
|
@ -14,21 +14,21 @@ interface Directory {
|
|||
function splitPath(path: string, withRoot?: string): string[] {
|
||||
let parts = path.includes("/") ? path.split("/") : path.split("\\");
|
||||
if (withRoot !== undefined) {
|
||||
let rootParts = splitPath(withRoot);
|
||||
const rootParts = splitPath(withRoot);
|
||||
parts = parts.slice(rootParts.length - 1);
|
||||
}
|
||||
return parts;
|
||||
}
|
||||
|
||||
function formatFileTree(tree: Directory, indentation: string = ""): string {
|
||||
function formatFileTree(tree: Directory, indentation = ""): string {
|
||||
let result = "";
|
||||
for (let file of tree.files) {
|
||||
for (const file of tree.files) {
|
||||
result += `${indentation}${file}\n`;
|
||||
}
|
||||
|
||||
for (let directory of tree.directories) {
|
||||
for (const directory of tree.directories) {
|
||||
result += `${indentation}${directory.name}/\n`;
|
||||
result += formatFileTree(directory, indentation + " ");
|
||||
result += formatFileTree(directory, `${indentation} `);
|
||||
}
|
||||
|
||||
return result;
|
||||
|
|
@ -47,9 +47,9 @@ class FileTreeContextProvider extends BaseContextProvider {
|
|||
extras: ContextProviderExtras,
|
||||
): Promise<ContextItem[]> {
|
||||
const workspaceDirs = await extras.ide.getWorkspaceDirs();
|
||||
let trees = [];
|
||||
const trees = [];
|
||||
|
||||
for (let workspaceDir of workspaceDirs) {
|
||||
for (const workspaceDir of workspaceDirs) {
|
||||
const contents = await extras.ide.listWorkspaceContents(workspaceDir);
|
||||
|
||||
const subDirTree: Directory = {
|
||||
|
|
@ -58,11 +58,11 @@ class FileTreeContextProvider extends BaseContextProvider {
|
|||
directories: [],
|
||||
};
|
||||
|
||||
for (let file of contents) {
|
||||
for (const file of contents) {
|
||||
const parts = splitPath(file, workspaceDir);
|
||||
|
||||
let currentTree = subDirTree;
|
||||
for (let part of parts.slice(0, -1)) {
|
||||
for (const part of parts.slice(0, -1)) {
|
||||
if (!currentTree.directories.some((d) => d.name === part)) {
|
||||
currentTree.directories.push({
|
||||
name: part,
|
||||
|
|
@ -82,9 +82,9 @@ class FileTreeContextProvider extends BaseContextProvider {
|
|||
|
||||
return [
|
||||
{
|
||||
content:
|
||||
"Here is a file tree of the current workspace:\n\n" +
|
||||
trees.join("\n\n"),
|
||||
content: `Here is a file tree of the current workspace:\n\n${trees.join(
|
||||
"\n\n",
|
||||
)}`,
|
||||
name: "File Tree",
|
||||
description: "File Tree",
|
||||
},
|
||||
|
|
|
|||
|
|
@ -1,4 +1,3 @@
|
|||
import { BaseContextProvider } from "../index.js";
|
||||
import {
|
||||
ContextItem,
|
||||
ContextProviderDescription,
|
||||
|
|
@ -7,6 +6,7 @@ import {
|
|||
LoadSubmenuItemsArgs,
|
||||
} from "../../index.js";
|
||||
import { getBasename, getLastNPathParts } from "../../util/index.js";
|
||||
import { BaseContextProvider } from "../index.js";
|
||||
|
||||
class FolderContextProvider extends BaseContextProvider {
|
||||
static description: ContextProviderDescription = {
|
||||
|
|
|
|||
|
|
@ -1,4 +1,3 @@
|
|||
import { BaseContextProvider } from "../index.js";
|
||||
import {
|
||||
ContextItem,
|
||||
ContextProviderDescription,
|
||||
|
|
@ -6,6 +5,7 @@ import {
|
|||
ContextSubmenuItem,
|
||||
LoadSubmenuItemsArgs,
|
||||
} from "../../index.js";
|
||||
import { BaseContextProvider } from "../index.js";
|
||||
|
||||
class GitHubIssuesContextProvider extends BaseContextProvider {
|
||||
static description: ContextProviderDescription = {
|
||||
|
|
@ -49,7 +49,7 @@ class GitHubIssuesContextProvider extends BaseContextProvider {
|
|||
issue.data.body || "No description",
|
||||
...comments.data.map((comment) => comment.body),
|
||||
];
|
||||
content += "\n\n" + parts.join("\n\n---\n\n");
|
||||
content += `\n\n${parts.join("\n\n---\n\n")}`;
|
||||
|
||||
return [
|
||||
{
|
||||
|
|
|
|||
|
|
@ -1,6 +1,10 @@
|
|||
import { AxiosInstance, AxiosError } from "axios";
|
||||
import {BaseContextProvider} from "../index.js";
|
||||
import { ContextProviderExtras, ContextItem, ContextProviderDescription } from "../../index.js";
|
||||
import { AxiosError, AxiosInstance } from "axios";
|
||||
import {
|
||||
ContextItem,
|
||||
ContextProviderDescription,
|
||||
ContextProviderExtras,
|
||||
} from "../../index.js";
|
||||
import { BaseContextProvider } from "../index.js";
|
||||
|
||||
interface RemoteBranchInfo {
|
||||
branch: string | null;
|
||||
|
|
@ -57,9 +61,7 @@ const trimFirstElement = (args: Array<string>): string => {
|
|||
};
|
||||
|
||||
const getSubprocess = async (extras: ContextProviderExtras) => {
|
||||
const workingDir = await extras.ide
|
||||
.getWorkspaceDirs()
|
||||
.then(trimFirstElement);
|
||||
const workingDir = await extras.ide.getWorkspaceDirs().then(trimFirstElement);
|
||||
|
||||
return (command: string) =>
|
||||
extras.ide
|
||||
|
|
@ -67,13 +69,12 @@ const getSubprocess = async (extras: ContextProviderExtras) => {
|
|||
.then(trimFirstElement);
|
||||
};
|
||||
|
||||
|
||||
class GitLabMergeRequestContextProvider extends BaseContextProvider {
|
||||
static description: ContextProviderDescription = {
|
||||
title: "gitlab-mr",
|
||||
displayTitle: "GitLab Merge Request",
|
||||
description: "Reference comments in a GitLab Merge Request",
|
||||
type: "normal"
|
||||
type: "normal",
|
||||
};
|
||||
|
||||
private async getApi(): Promise<AxiosInstance> {
|
||||
|
|
@ -82,85 +83,88 @@ class GitLabMergeRequestContextProvider extends BaseContextProvider {
|
|||
const domain = this.options.domain ?? "gitlab.com";
|
||||
const token = this.options.token;
|
||||
|
||||
if(!token) {
|
||||
if (!token) {
|
||||
throw new Error("GitLab Private Token is required!");
|
||||
}
|
||||
|
||||
|
||||
return Axios.create({
|
||||
baseURL: `https://${domain ?? "gitlab.com"}/api/v4`,
|
||||
headers: {
|
||||
"PRIVATE-TOKEN": token,
|
||||
},
|
||||
});
|
||||
};
|
||||
|
||||
|
||||
private async getRemoteBranchName(extras: ContextProviderExtras): Promise<RemoteBranchInfo> {
|
||||
}
|
||||
|
||||
private async getRemoteBranchName(
|
||||
extras: ContextProviderExtras,
|
||||
): Promise<RemoteBranchInfo> {
|
||||
const subprocess = await getSubprocess(extras);
|
||||
|
||||
|
||||
const branchName = await subprocess("git branch --show-current");
|
||||
|
||||
|
||||
const branchRemote = await subprocess(
|
||||
`git config branch.${branchName}.remote`
|
||||
`git config branch.${branchName}.remote`,
|
||||
);
|
||||
|
||||
|
||||
const branchInfo = await subprocess("git branch -vv");
|
||||
|
||||
|
||||
const currentBranchInfo = branchInfo
|
||||
.split("\n")
|
||||
.find((line) => line.startsWith("*"));
|
||||
|
||||
|
||||
const remoteMatches = RegExp(
|
||||
`\\[${branchRemote}/(?<remote_branch>[^\\]]+)\\]`
|
||||
`\\[${branchRemote}/(?<remote_branch>[^\\]]+)\\]`,
|
||||
).exec(currentBranchInfo!);
|
||||
|
||||
|
||||
console.dir({ remoteMatches });
|
||||
|
||||
const remoteBranch = remoteMatches?.groups?.["remote_branch"] ?? null;
|
||||
|
||||
|
||||
const remoteBranch = remoteMatches?.groups?.remote_branch ?? null;
|
||||
|
||||
const remoteUrl = await subprocess(`git remote get-url ${branchRemote}`);
|
||||
|
||||
const urlMatches = RegExp(":(?<project>.*).git").exec(remoteUrl);
|
||||
|
||||
const project = urlMatches?.groups?.["project"] ?? null;
|
||||
|
||||
|
||||
const urlMatches = /:(?<project>.*).git/.exec(remoteUrl);
|
||||
|
||||
const project = urlMatches?.groups?.project ?? null;
|
||||
|
||||
return {
|
||||
branch: remoteBranch,
|
||||
project,
|
||||
};
|
||||
};
|
||||
|
||||
async getContextItems(query: string, extras: ContextProviderExtras): Promise<ContextItem[]> {
|
||||
}
|
||||
|
||||
async getContextItems(
|
||||
query: string,
|
||||
extras: ContextProviderExtras,
|
||||
): Promise<ContextItem[]> {
|
||||
const { branch, project } = await this.getRemoteBranchName(extras);
|
||||
|
||||
|
||||
const api = await this.getApi();
|
||||
|
||||
const result = [] as Array<ContextItem>;
|
||||
|
||||
|
||||
try {
|
||||
const mergeRequests = await api
|
||||
.get<Array<GitLabMergeRequest>>(
|
||||
`/projects/${encodeURIComponent(project!)}/merge_requests`,
|
||||
{
|
||||
params: {
|
||||
source_branch: branch,
|
||||
state: "opened",
|
||||
const mergeRequests = await api
|
||||
.get<Array<GitLabMergeRequest>>(
|
||||
`/projects/${encodeURIComponent(project!)}/merge_requests`,
|
||||
{
|
||||
params: {
|
||||
source_branch: branch,
|
||||
state: "opened",
|
||||
},
|
||||
},
|
||||
}
|
||||
)
|
||||
.then((x) => x.data);
|
||||
)
|
||||
.then((x) => x.data);
|
||||
|
||||
const subprocess = await getSubprocess(extras);
|
||||
|
||||
const subprocess = await getSubprocess(extras);
|
||||
|
||||
for (const mergeRequest of mergeRequests) {
|
||||
const parts = [
|
||||
`# GitLab Merge Request\ntitle: "${mergeRequest.title}"\ndescription: "${mergeRequest.description ?? "None"}"`,
|
||||
`# GitLab Merge Request\ntitle: "${
|
||||
mergeRequest.title
|
||||
}"\ndescription: "${mergeRequest.description ?? "None"}"`,
|
||||
"## Comments",
|
||||
];
|
||||
|
||||
|
||||
const comments = await api.get<Array<GitLabComment>>(
|
||||
`/projects/${mergeRequest.project_id}/merge_requests/${mergeRequest.iid}/notes`,
|
||||
{
|
||||
|
|
@ -168,89 +172,106 @@ class GitLabMergeRequestContextProvider extends BaseContextProvider {
|
|||
sort: "asc",
|
||||
order_by: "created_at",
|
||||
},
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
const filteredComments = comments.data.filter(
|
||||
(x) => x.type === "DiffNote"
|
||||
(x) => x.type === "DiffNote",
|
||||
);
|
||||
|
||||
|
||||
const locations = {} as Record<string, Array<GitLabComment>>;
|
||||
|
||||
|
||||
for (const comment of filteredComments) {
|
||||
const filename = comment.position?.new_path ?? "general";
|
||||
|
||||
|
||||
if (!locations[filename]) {
|
||||
locations[filename] = [];
|
||||
}
|
||||
|
||||
|
||||
locations[filename].push(comment);
|
||||
}
|
||||
|
||||
if (extras.selectedCode.length && this.options.filterComments) {
|
||||
const toRemove = Object.keys(locations).filter(filename => !extras.selectedCode.find(selection => selection.filepath.endsWith(filename)) && filename !== "general");
|
||||
const toRemove = Object.keys(locations).filter(
|
||||
(filename) =>
|
||||
!extras.selectedCode.find((selection) =>
|
||||
selection.filepath.endsWith(filename),
|
||||
) && filename !== "general",
|
||||
);
|
||||
for (const filepath of toRemove) {
|
||||
delete locations[filepath];
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
const commentFormatter = async (comment: GitLabComment) => {
|
||||
const commentLabel = comment.body.includes("```suggestion") ? "Code Suggestion" : "Comment";
|
||||
let result = `#### ${commentLabel}\nauthor: "${comment.author.name}"\ndate: "${comment.created_at}"\nresolved: ${
|
||||
comment.resolved ? "Yes" : "No"
|
||||
}`;
|
||||
|
||||
const commentLabel = comment.body.includes("```suggestion")
|
||||
? "Code Suggestion"
|
||||
: "Comment";
|
||||
let result = `#### ${commentLabel}\nauthor: "${
|
||||
comment.author.name
|
||||
}"\ndate: "${comment.created_at}"\nresolved: ${
|
||||
comment.resolved ? "Yes" : "No"
|
||||
}`;
|
||||
|
||||
if (comment.position?.new_line) {
|
||||
result += `\nline: ${comment.position.new_line}`;
|
||||
|
||||
if (comment.position.head_sha) {
|
||||
const sourceLines = await subprocess(`git show ${comment.position.head_sha}:${comment.position.new_path}`).then(result => result.split("\n")).catch(ex => []);
|
||||
|
||||
const line = comment.position.new_line <= sourceLines.length ? sourceLines[comment.position.new_line - 1] : null;
|
||||
if (comment.position.head_sha) {
|
||||
const sourceLines = await subprocess(
|
||||
`git show ${comment.position.head_sha}:${comment.position.new_path}`,
|
||||
)
|
||||
.then((result) => result.split("\n"))
|
||||
.catch((ex) => []);
|
||||
|
||||
const line =
|
||||
comment.position.new_line <= sourceLines.length
|
||||
? sourceLines[comment.position.new_line - 1]
|
||||
: null;
|
||||
|
||||
if (line) {
|
||||
result += `\nsource: \`${line}\``;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
result += `\n\n${comment.body}`;
|
||||
|
||||
|
||||
return result;
|
||||
};
|
||||
|
||||
|
||||
for (const [filename, locationComments] of Object.entries(locations)) {
|
||||
if (filename !== "general") {
|
||||
parts.push(`### File ${filename}`);
|
||||
locationComments.sort(
|
||||
(a, b) => a.position!.new_line - b.position!.new_line
|
||||
(a, b) =>
|
||||
(a.position?.new_line ?? 0) - (b.position?.new_line ?? 0),
|
||||
);
|
||||
} else {
|
||||
parts.push("### General");
|
||||
}
|
||||
|
||||
const commentSections = await Promise.all(locationComments.map(commentFormatter));
|
||||
|
||||
const commentSections = await Promise.all(
|
||||
locationComments.map(commentFormatter),
|
||||
);
|
||||
parts.push(...commentSections);
|
||||
}
|
||||
|
||||
const content = parts.join("\n\n");
|
||||
|
||||
const content = parts.join("\n\n");
|
||||
|
||||
result.push(
|
||||
{
|
||||
result.push({
|
||||
name: mergeRequest.title,
|
||||
content,
|
||||
description: "Comments from the Merge Request for this branch.",
|
||||
},
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
} catch (ex) {
|
||||
let content = "# GitLab Merge Request\n\nError getting merge request. ";
|
||||
if (ex instanceof AxiosError) {
|
||||
if (ex.response) {
|
||||
const errorMessage = ex.response?.data ? ex.response.data.message ?? JSON.stringify(ex.response?.data) : `${ex.response.status}: ${ex.response.statusText}`;
|
||||
content += `GitLab Error: ${errorMessage}`;
|
||||
const errorMessage = ex.response?.data
|
||||
? ex.response.data.message ?? JSON.stringify(ex.response?.data)
|
||||
: `${ex.response.status}: ${ex.response.statusText}`;
|
||||
content += `GitLab Error: ${errorMessage}`;
|
||||
} else {
|
||||
content += `GitLab Request Error ${ex.request}`;
|
||||
}
|
||||
|
|
@ -259,19 +280,15 @@ class GitLabMergeRequestContextProvider extends BaseContextProvider {
|
|||
content += `Unknown error: ${ex.message ?? JSON.stringify(ex)}`;
|
||||
}
|
||||
|
||||
|
||||
result.push(
|
||||
{
|
||||
name: "GitLab Merge Request",
|
||||
content,
|
||||
description: "Error getting the Merge Request for this branch.",
|
||||
},
|
||||
);
|
||||
result.push({
|
||||
name: "GitLab Merge Request",
|
||||
content,
|
||||
description: "Error getting the Merge Request for this branch.",
|
||||
});
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
export default GitLabMergeRequestContextProvider;
|
||||
export default GitLabMergeRequestContextProvider;
|
||||
|
|
|
|||
|
|
@ -1,9 +1,9 @@
|
|||
import { BaseContextProvider } from "../index.js";
|
||||
import {
|
||||
ContextItem,
|
||||
ContextProviderDescription,
|
||||
ContextProviderExtras,
|
||||
} from "../../index.js";
|
||||
import { BaseContextProvider } from "../index.js";
|
||||
|
||||
class GoogleContextProvider extends BaseContextProvider {
|
||||
static description: ContextProviderDescription = {
|
||||
|
|
@ -40,16 +40,16 @@ class GoogleContextProvider extends BaseContextProvider {
|
|||
|
||||
const results = await response.text();
|
||||
|
||||
let jsonResults = JSON.parse(results);
|
||||
const jsonResults = JSON.parse(results);
|
||||
let content = `Google Search: ${query}\n\n`;
|
||||
let answerBox = jsonResults["answerBox"];
|
||||
const answerBox = jsonResults.answerBox;
|
||||
|
||||
if (answerBox) {
|
||||
content += `Answer Box (${answerBox["title"]}): ${answerBox["answer"]}\n\n`;
|
||||
content += `Answer Box (${answerBox.title}): ${answerBox.answer}\n\n`;
|
||||
}
|
||||
|
||||
for (let result of jsonResults["organic"]) {
|
||||
content += `${result["title"]}\n${result["link"]}\n${result["snippet"]}\n\n`;
|
||||
for (const result of jsonResults.organic) {
|
||||
content += `${result.title}\n${result.link}\n${result.snippet}\n\n`;
|
||||
}
|
||||
|
||||
return [
|
||||
|
|
|
|||
|
|
@ -1,9 +1,9 @@
|
|||
import { BaseContextProvider } from "../index.js";
|
||||
import {
|
||||
ContextItem,
|
||||
ContextProviderDescription,
|
||||
ContextProviderExtras,
|
||||
} from "../../index.js";
|
||||
import { BaseContextProvider } from "../index.js";
|
||||
|
||||
class HttpContextProvider extends BaseContextProvider {
|
||||
static description: ContextProviderDescription = {
|
||||
|
|
|
|||
|
|
@ -67,7 +67,8 @@ export class JiraClient {
|
|||
private authHeader;
|
||||
constructor(options: JiraClientOptions) {
|
||||
this.options = {
|
||||
issueQuery: "assignee = currentUser() AND resolution = Unresolved order by updated DESC",
|
||||
issueQuery:
|
||||
"assignee = currentUser() AND resolution = Unresolved order by updated DESC",
|
||||
apiVersion: "3",
|
||||
requestOptions: {},
|
||||
...options,
|
||||
|
|
@ -75,9 +76,9 @@ export class JiraClient {
|
|||
this.baseUrl = `https://${this.options.domain}/rest/api/${this.options.apiVersion}`;
|
||||
this.authHeader = this.options.username
|
||||
? {
|
||||
Authorization:
|
||||
"Basic " +
|
||||
btoa(this.options.username + ":" + this.options.password),
|
||||
Authorization: `Basic ${btoa(
|
||||
`${this.options.username}:${this.options.password}`,
|
||||
)}`,
|
||||
}
|
||||
: {
|
||||
Authorization: `Bearer ${this.options.password}`,
|
||||
|
|
@ -154,7 +155,7 @@ export class JiraClient {
|
|||
},
|
||||
);
|
||||
|
||||
if (response.status != 200) {
|
||||
if (response.status !== 200) {
|
||||
console.warn(
|
||||
"Unable to get jira tickets. Response code from API is",
|
||||
response.status,
|
||||
|
|
|
|||
|
|
@ -1,4 +1,3 @@
|
|||
import { BaseContextProvider } from "../../index.js";
|
||||
import {
|
||||
ContextItem,
|
||||
ContextProviderDescription,
|
||||
|
|
@ -6,6 +5,7 @@ import {
|
|||
ContextSubmenuItem,
|
||||
LoadSubmenuItemsArgs,
|
||||
} from "../../../index.js";
|
||||
import { BaseContextProvider } from "../../index.js";
|
||||
import { JiraClient } from "./JiraClient.js";
|
||||
|
||||
class JiraIssuesContextProvider extends BaseContextProvider {
|
||||
|
|
|
|||
|
|
@ -1,4 +1,3 @@
|
|||
import { BaseContextProvider } from "../index.js";
|
||||
import {
|
||||
ContextItem,
|
||||
ContextProviderDescription,
|
||||
|
|
@ -6,6 +5,7 @@ import {
|
|||
ContextSubmenuItem,
|
||||
LoadSubmenuItemsArgs,
|
||||
} from "../../index.js";
|
||||
import { BaseContextProvider } from "../index.js";
|
||||
|
||||
class LocalsProvider extends BaseContextProvider {
|
||||
static description: ContextProviderDescription = {
|
||||
|
|
@ -17,22 +17,22 @@ class LocalsProvider extends BaseContextProvider {
|
|||
|
||||
async getContextItems(
|
||||
query: string,
|
||||
extras: ContextProviderExtras
|
||||
extras: ContextProviderExtras,
|
||||
): Promise<ContextItem[]> {
|
||||
// Assuming that the query is a number
|
||||
const localVariables = await extras.ide.getDebugLocals(Number(query));
|
||||
const threadIndex = Number(query);
|
||||
const thread = (await extras.ide.getAvailableThreads()).find(
|
||||
(thread) => thread.id == threadIndex
|
||||
(thread) => thread.id === threadIndex,
|
||||
);
|
||||
const callStacksSources = await extras.ide.getTopLevelCallStackSources(
|
||||
threadIndex,
|
||||
this.options?.stackDepth || 3
|
||||
this.options?.stackDepth || 3,
|
||||
);
|
||||
const callStackContents = callStacksSources.reduce(
|
||||
(acc, source, index) =>
|
||||
acc + `\n\ncall stack ${index}\n` + "```\n" + source + "\n```",
|
||||
""
|
||||
`${acc}\n\ncall stack ${index}\n\`\`\`\n${source}\n\`\`\``,
|
||||
"",
|
||||
);
|
||||
return [
|
||||
{
|
||||
|
|
@ -47,7 +47,7 @@ class LocalsProvider extends BaseContextProvider {
|
|||
}
|
||||
|
||||
async loadSubmenuItems(
|
||||
args: LoadSubmenuItemsArgs
|
||||
args: LoadSubmenuItemsArgs,
|
||||
): Promise<ContextSubmenuItem[]> {
|
||||
const threads = await args.ide.getAvailableThreads();
|
||||
|
||||
|
|
|
|||
|
|
@ -1,13 +1,13 @@
|
|||
//os.platform()
|
||||
//os.arch()
|
||||
|
||||
import { BaseContextProvider } from "../index.js";
|
||||
import os from "os";
|
||||
import {
|
||||
ContextItem,
|
||||
ContextProviderDescription,
|
||||
ContextProviderExtras,
|
||||
} from "../../index.js";
|
||||
import os from "os";
|
||||
import { BaseContextProvider } from "../index.js";
|
||||
|
||||
class OSContextProvider extends BaseContextProvider {
|
||||
static description: ContextProviderDescription = {
|
||||
|
|
|
|||
|
|
@ -1,10 +1,10 @@
|
|||
import { BaseContextProvider } from "../index.js";
|
||||
import {
|
||||
ContextItem,
|
||||
ContextProviderDescription,
|
||||
ContextProviderExtras,
|
||||
} from "../../index.js";
|
||||
import { getBasename } from "../../util/index.js";
|
||||
import { BaseContextProvider } from "../index.js";
|
||||
|
||||
class OpenFilesContextProvider extends BaseContextProvider {
|
||||
static description: ContextProviderDescription = {
|
||||
|
|
|
|||
|
|
@ -1,4 +1,3 @@
|
|||
import { BaseContextProvider } from "../index.js";
|
||||
import {
|
||||
ContextItem,
|
||||
ContextProviderDescription,
|
||||
|
|
@ -6,6 +5,7 @@ import {
|
|||
ContextSubmenuItem,
|
||||
LoadSubmenuItemsArgs,
|
||||
} from "../../index.js";
|
||||
import { BaseContextProvider } from "../index.js";
|
||||
|
||||
class PostgresContextProvider extends BaseContextProvider {
|
||||
static description: ContextProviderDescription = {
|
||||
|
|
@ -18,18 +18,6 @@ class PostgresContextProvider extends BaseContextProvider {
|
|||
static ALL_TABLES = "__all_tables";
|
||||
static DEFAULT_SAMPLE_ROWS = 3;
|
||||
|
||||
constructor(options: {
|
||||
host: string;
|
||||
port: number;
|
||||
user: string;
|
||||
password: string;
|
||||
database: string;
|
||||
schema?: string;
|
||||
sampleRows?: number;
|
||||
}) {
|
||||
super(options);
|
||||
}
|
||||
|
||||
private async getPool() {
|
||||
const pg = await require("pg");
|
||||
return new pg.Pool({
|
||||
|
|
@ -43,7 +31,7 @@ class PostgresContextProvider extends BaseContextProvider {
|
|||
|
||||
private async getTableNames(pool: any): Promise<string[]> {
|
||||
const schema = this.options.schema ?? "public";
|
||||
var tablesInfoQuery = `
|
||||
let tablesInfoQuery = `
|
||||
SELECT table_schema, table_name
|
||||
FROM information_schema.tables`;
|
||||
if (schema != null) {
|
||||
|
|
@ -56,7 +44,7 @@ FROM information_schema.tables`;
|
|||
}
|
||||
|
||||
async getContextItems(
|
||||
query: string = "",
|
||||
query = "",
|
||||
_: ContextProviderExtras = {} as ContextProviderExtras,
|
||||
): Promise<ContextItem[]> {
|
||||
const pool = await this.getPool();
|
||||
|
|
@ -78,7 +66,7 @@ FROM information_schema.tables`;
|
|||
`Table name must be in format schema.table_name, got ${tableName}`,
|
||||
);
|
||||
}
|
||||
var schemaQuery = `
|
||||
const schemaQuery = `
|
||||
SELECT column_name, data_type, character_maximum_length
|
||||
FROM INFORMATION_SCHEMA.COLUMNS
|
||||
WHERE table_schema = '${tableName.split(".")[0]}'
|
||||
|
|
@ -96,7 +84,7 @@ FROM ${tableName}
|
|||
LIMIT ${sampleRows}`);
|
||||
|
||||
// Create prompt from the table schema and sample rows
|
||||
var prompt = `Postgres schema for database ${this.options.database} table ${tableName}:\n`;
|
||||
let prompt = `Postgres schema for database ${this.options.database} table ${tableName}:\n`;
|
||||
prompt += `${JSON.stringify(tableSchema, null, 2)}\n\n`;
|
||||
prompt += `Sample rows: ${JSON.stringify(sampleRowResults, null, 2)}`;
|
||||
|
||||
|
|
|
|||
|
|
@ -1,10 +1,10 @@
|
|||
import { BaseContextProvider } from "../index.js";
|
||||
import {
|
||||
ContextItem,
|
||||
ContextProviderDescription,
|
||||
ContextProviderExtras,
|
||||
} from "../../index.js";
|
||||
import { getBasename } from "../../util/index.js";
|
||||
import { BaseContextProvider } from "../index.js";
|
||||
|
||||
class ProblemsContextProvider extends BaseContextProvider {
|
||||
static description: ContextProviderDescription = {
|
||||
|
|
|
|||
|
|
@ -1,9 +1,9 @@
|
|||
import { BaseContextProvider } from "../index.js";
|
||||
import {
|
||||
ContextItem,
|
||||
ContextProviderDescription,
|
||||
ContextProviderExtras,
|
||||
} from "../../index.js";
|
||||
import { BaseContextProvider } from "../index.js";
|
||||
|
||||
class SearchContextProvider extends BaseContextProvider {
|
||||
static description: ContextProviderDescription = {
|
||||
|
|
|
|||
|
|
@ -1,9 +1,9 @@
|
|||
import { BaseContextProvider } from "../index.js";
|
||||
import {
|
||||
ContextItem,
|
||||
ContextProviderDescription,
|
||||
ContextProviderExtras,
|
||||
} from "../../index.js";
|
||||
import { BaseContextProvider } from "../index.js";
|
||||
|
||||
class TerminalContextProvider extends BaseContextProvider {
|
||||
static description: ContextProviderDescription = {
|
||||
|
|
|
|||
|
|
@ -64,7 +64,7 @@ export class LLMReranker implements Reranker {
|
|||
return 0.0;
|
||||
}
|
||||
|
||||
let answer = completion
|
||||
const answer = completion
|
||||
.trim()
|
||||
.toLowerCase()
|
||||
.replace(/"/g, "")
|
||||
|
|
@ -72,14 +72,14 @@ export class LLMReranker implements Reranker {
|
|||
|
||||
if (answer === "yes") {
|
||||
return 1.0;
|
||||
} else if (answer === "no") {
|
||||
return 0.0;
|
||||
} else {
|
||||
console.warn(
|
||||
`Unexpected response from single token reranker: "${answer}". Expected "yes" or "no".`,
|
||||
);
|
||||
}
|
||||
if (answer === "no") {
|
||||
return 0.0;
|
||||
}
|
||||
console.warn(
|
||||
`Unexpected response from single token reranker: "${answer}". Expected "yes" or "no".`,
|
||||
);
|
||||
return 0.0;
|
||||
}
|
||||
|
||||
async rerank(query: string, chunks: Chunk[]): Promise<number[]> {
|
||||
|
|
|
|||
|
|
@ -29,6 +29,16 @@ export async function retrieveContextItemsFromEmbeddings(
|
|||
return [];
|
||||
}
|
||||
|
||||
// transformers.js not supported in JetBrains IDEs right now
|
||||
if (
|
||||
extras.embeddingsProvider.id === "all-MiniLM-L6-v2" &&
|
||||
(await extras.ide.getIdeInfo()).ideType === "jetbrains"
|
||||
) {
|
||||
throw new Error(
|
||||
"The transformers.js context provider is not currently supported in JetBrains. For now, you can use Ollama to set up local embeddings, or use our 'free-trial' embeddings provider. See here to learn more: https://docs.continue.dev/walkthroughs/codebase-embeddings#embeddings-providers",
|
||||
);
|
||||
}
|
||||
|
||||
const nFinal = options?.nFinal || RETRIEVAL_PARAMS.nFinal;
|
||||
const useReranking = extras.reranker !== undefined;
|
||||
const nRetrieve =
|
||||
|
|
@ -60,7 +70,7 @@ export async function retrieveContextItemsFromEmbeddings(
|
|||
const retrievalResults: Chunk[] = [];
|
||||
|
||||
// Source: Full-text search
|
||||
let ftsResults = await retrieveFts(
|
||||
const ftsResults = await retrieveFts(
|
||||
extras.fullInput,
|
||||
nRetrieve / 2,
|
||||
tags,
|
||||
|
|
@ -99,7 +109,7 @@ export async function retrieveContextItemsFromEmbeddings(
|
|||
const lanceDbIndex = new LanceDbIndex(extras.embeddingsProvider, (path) =>
|
||||
extras.ide.readFile(path),
|
||||
);
|
||||
let vecResults = await lanceDbIndex.retrieve(
|
||||
const vecResults = await lanceDbIndex.retrieve(
|
||||
extras.fullInput,
|
||||
nRetrieve,
|
||||
tags,
|
||||
|
|
|
|||
|
|
@ -21,7 +21,7 @@ export interface EmbeddingsCacheResponse<T extends ArtifactType> {
|
|||
export interface IContinueServerClient {
|
||||
connected: boolean;
|
||||
url: URL | undefined;
|
||||
getUserToken(): Promise<string | undefined>;
|
||||
getUserToken(): string | undefined;
|
||||
getConfig(): Promise<{ configJson: string; configJs: string }>;
|
||||
getFromIndexCache<T extends ArtifactType>(
|
||||
keys: string[],
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
import {
|
||||
import type {
|
||||
ArtifactType,
|
||||
EmbeddingsCacheResponse,
|
||||
IContinueServerClient,
|
||||
|
|
@ -9,7 +9,7 @@ export class ContinueServerClient implements IContinueServerClient {
|
|||
|
||||
constructor(
|
||||
serverUrl: string | undefined,
|
||||
private readonly userToken: Promise<string | undefined>,
|
||||
private readonly userToken: string | undefined,
|
||||
) {
|
||||
try {
|
||||
this.url =
|
||||
|
|
@ -22,7 +22,7 @@ export class ContinueServerClient implements IContinueServerClient {
|
|||
}
|
||||
}
|
||||
|
||||
getUserToken(): Promise<string | undefined> {
|
||||
getUserToken(): string | undefined {
|
||||
return this.userToken;
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -1,26 +1,35 @@
|
|||
import { ContextItemId, IDE } from "core";
|
||||
import { CompletionProvider } from "core/autocomplete/completionProvider";
|
||||
import { ConfigHandler } from "core/config/handler";
|
||||
import { addModel, addOpenAIKey, deleteModel } from "core/config/util";
|
||||
import { indexDocs } from "core/indexing/docs";
|
||||
import TransformersJsEmbeddingsProvider from "core/indexing/embeddings/TransformersJsEmbeddingsProvider";
|
||||
import { CodebaseIndexer, PauseToken } from "core/indexing/indexCodebase";
|
||||
import { logDevData } from "core/util/devdata";
|
||||
import { fetchwithRequestOptions } from "core/util/fetchWithOptions";
|
||||
import historyManager from "core/util/history";
|
||||
import { Message } from "core/util/messenger";
|
||||
import { Telemetry } from "core/util/posthog";
|
||||
import { streamDiffLines } from "core/util/verticalEdit";
|
||||
import { v4 as uuidv4 } from "uuid";
|
||||
import { IpcMessenger } from "./messenger";
|
||||
import { Protocol } from "./protocol";
|
||||
import { ContextItemId, IDE } from ".";
|
||||
import { CompletionProvider } from "./autocomplete/completionProvider";
|
||||
import { ConfigHandler } from "./config/handler";
|
||||
import {
|
||||
setupLocalAfterFreeTrial,
|
||||
setupLocalMode,
|
||||
setupOptimizedExistingUserMode,
|
||||
setupOptimizedMode,
|
||||
} from "./config/onboarding";
|
||||
import { addModel, addOpenAIKey, deleteModel } from "./config/util";
|
||||
import { ContinueServerClient } from "./continueServer/stubs/client";
|
||||
import { indexDocs } from "./indexing/docs";
|
||||
import TransformersJsEmbeddingsProvider from "./indexing/embeddings/TransformersJsEmbeddingsProvider";
|
||||
import { CodebaseIndexer, PauseToken } from "./indexing/indexCodebase";
|
||||
import { FromCoreProtocol, ToCoreProtocol } from "./protocol";
|
||||
import { GlobalContext } from "./util/GlobalContext";
|
||||
import { logDevData } from "./util/devdata";
|
||||
import { DevDataSqliteDb } from "./util/devdataSqlite";
|
||||
import { fetchwithRequestOptions } from "./util/fetchWithOptions";
|
||||
import historyManager from "./util/history";
|
||||
import type { IMessenger, Message } from "./util/messenger";
|
||||
import { editConfigJson, getConfigJsonPath } from "./util/paths";
|
||||
import { Telemetry } from "./util/posthog";
|
||||
import { streamDiffLines } from "./util/verticalEdit";
|
||||
|
||||
export class Core {
|
||||
private messenger: IpcMessenger;
|
||||
private readonly ide: IDE;
|
||||
private readonly configHandler: ConfigHandler;
|
||||
private readonly codebaseIndexer: CodebaseIndexer;
|
||||
private readonly completionProvider: CompletionProvider;
|
||||
// implements IMessenger<ToCoreProtocol, FromCoreProtocol>
|
||||
configHandler: ConfigHandler;
|
||||
codebaseIndexerPromise: Promise<CodebaseIndexer>;
|
||||
completionProvider: CompletionProvider;
|
||||
continueServerClientPromise: Promise<ContinueServerClient>;
|
||||
|
||||
private abortedMessageIds: Set<string> = new Set();
|
||||
|
||||
|
|
@ -34,10 +43,19 @@ export class Core {
|
|||
return await this.configHandler.llmFromTitle(this.selectedModelTitle);
|
||||
}
|
||||
|
||||
constructor(messenger: IpcMessenger, ide: IDE) {
|
||||
this.messenger = messenger;
|
||||
this.ide = ide;
|
||||
invoke<T extends keyof ToCoreProtocol>(
|
||||
messageType: T,
|
||||
data: ToCoreProtocol[T][0],
|
||||
): ToCoreProtocol[T][1] {
|
||||
return this.messenger.invoke(messageType, data);
|
||||
}
|
||||
|
||||
// TODO: It shouldn't actually need an IDE type, because this can happen
|
||||
// through the messenger (it does in the case of any non-VS Code IDEs already)
|
||||
constructor(
|
||||
private readonly messenger: IMessenger<ToCoreProtocol, FromCoreProtocol>,
|
||||
private readonly ide: IDE,
|
||||
) {
|
||||
const ideSettingsPromise = messenger.request("getIdeSettings", undefined);
|
||||
this.configHandler = new ConfigHandler(
|
||||
this.ide,
|
||||
|
|
@ -45,13 +63,40 @@ export class Core {
|
|||
async (text: string) => {},
|
||||
(() => this.messenger.send("configUpdate", undefined)).bind(this),
|
||||
);
|
||||
this.codebaseIndexer = new CodebaseIndexer(
|
||||
this.configHandler,
|
||||
this.ide,
|
||||
new PauseToken(false),
|
||||
undefined, // TODO
|
||||
Promise.resolve(undefined), // TODO
|
||||
|
||||
// Codebase Indexer and ContinueServerClient depend on IdeSettings
|
||||
const indexingPauseToken = new PauseToken(
|
||||
new GlobalContext().get("indexingPaused") === true,
|
||||
);
|
||||
let codebaseIndexerResolve: (_: any) => void | undefined;
|
||||
this.codebaseIndexerPromise = new Promise(
|
||||
async (resolve) => (codebaseIndexerResolve = resolve),
|
||||
);
|
||||
|
||||
let continueServerClientResolve: (_: any) => void | undefined;
|
||||
this.continueServerClientPromise = new Promise(
|
||||
(resolve) => (continueServerClientResolve = resolve),
|
||||
);
|
||||
|
||||
ideSettingsPromise.then((ideSettings) => {
|
||||
const continueServerClient = new ContinueServerClient(
|
||||
ideSettings.remoteConfigServerUrl,
|
||||
ideSettings.userToken,
|
||||
);
|
||||
continueServerClientResolve(continueServerClient);
|
||||
|
||||
codebaseIndexerResolve(
|
||||
new CodebaseIndexer(
|
||||
this.configHandler,
|
||||
this.ide,
|
||||
new PauseToken(false),
|
||||
continueServerClient,
|
||||
),
|
||||
);
|
||||
this.ide
|
||||
.getWorkspaceDirs()
|
||||
.then((dirs) => this.refreshCodebaseIndex(dirs));
|
||||
});
|
||||
|
||||
const getLlm = async () => {
|
||||
const config = await this.configHandler.loadConfig();
|
||||
|
|
@ -110,13 +155,46 @@ export class Core {
|
|||
|
||||
// Edit config
|
||||
on("config/addModel", (msg) => {
|
||||
addModel(msg.data.model);
|
||||
const model = msg.data.model;
|
||||
const newConfigString = addModel(model);
|
||||
this.configHandler.reloadConfig();
|
||||
this.ide.openFile(getConfigJsonPath());
|
||||
|
||||
// Find the range where it was added and highlight
|
||||
const lines = newConfigString.split("\n");
|
||||
let startLine: number | undefined;
|
||||
let endLine: number | undefined;
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
const line = lines[i];
|
||||
|
||||
if (!startLine) {
|
||||
if (line.trim() === `"title": "${model.title}",`) {
|
||||
startLine = i - 1;
|
||||
}
|
||||
} else {
|
||||
if (line.startsWith(" }")) {
|
||||
endLine = i;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (startLine && endLine) {
|
||||
this.ide.showLines(
|
||||
getConfigJsonPath(),
|
||||
startLine,
|
||||
endLine,
|
||||
// "#fff1"
|
||||
);
|
||||
}
|
||||
});
|
||||
on("config/addOpenAiKey", (msg) => {
|
||||
addOpenAIKey(msg.data);
|
||||
this.configHandler.reloadConfig();
|
||||
});
|
||||
on("config/deleteModel", (msg) => {
|
||||
deleteModel(msg.data.title);
|
||||
this.configHandler.reloadConfig();
|
||||
});
|
||||
on("config/reload", (msg) => {
|
||||
this.configHandler.reloadConfig();
|
||||
|
|
@ -134,6 +212,8 @@ export class Core {
|
|||
new TransformersJsEmbeddingsProvider(),
|
||||
)) {
|
||||
}
|
||||
this.ide.infoPopup(`🎉 Successfully indexed ${msg.data.title}`);
|
||||
this.messenger.send("refreshSubmenuItems", undefined);
|
||||
});
|
||||
on("context/loadSubmenuItems", async (msg) => {
|
||||
const config = await this.config();
|
||||
|
|
@ -147,36 +227,42 @@ export class Core {
|
|||
return items || [];
|
||||
});
|
||||
on("context/getContextItems", async (msg) => {
|
||||
const { name, query, fullInput, selectedCode } = msg.data;
|
||||
const config = await this.config();
|
||||
const llm = await this.getSelectedModel();
|
||||
const provider = config.contextProviders?.find(
|
||||
(provider) => provider.description.title === msg.data.name,
|
||||
(provider) => provider.description.title === name,
|
||||
);
|
||||
if (!provider) return [];
|
||||
|
||||
const id: ContextItemId = {
|
||||
providerTitle: provider.description.title,
|
||||
itemId: uuidv4(),
|
||||
};
|
||||
const items = await provider.getContextItems(msg.data.query, {
|
||||
llm,
|
||||
embeddingsProvider: config.embeddingsProvider,
|
||||
fullInput: msg.data.fullInput,
|
||||
ide,
|
||||
selectedCode: msg.data.selectedCode,
|
||||
reranker: config.reranker,
|
||||
fetch: (url, init) =>
|
||||
fetchwithRequestOptions(url, init, config.requestOptions),
|
||||
});
|
||||
try {
|
||||
const id: ContextItemId = {
|
||||
providerTitle: provider.description.title,
|
||||
itemId: uuidv4(),
|
||||
};
|
||||
const items = await provider.getContextItems(query, {
|
||||
llm,
|
||||
embeddingsProvider: config.embeddingsProvider,
|
||||
fullInput,
|
||||
ide,
|
||||
selectedCode,
|
||||
reranker: config.reranker,
|
||||
fetch: (url, init) =>
|
||||
fetchwithRequestOptions(url, init, config.requestOptions),
|
||||
});
|
||||
|
||||
Telemetry.capture("useContextProvider", {
|
||||
name: provider.description.title,
|
||||
});
|
||||
Telemetry.capture("useContextProvider", {
|
||||
name: provider.description.title,
|
||||
});
|
||||
|
||||
return items.map((item) => ({
|
||||
...item,
|
||||
id,
|
||||
}));
|
||||
return items.map((item) => ({
|
||||
...item,
|
||||
id,
|
||||
}));
|
||||
} catch (e) {
|
||||
this.ide.errorPopup(`Error getting context items from ${name}: ${e}`);
|
||||
return [];
|
||||
}
|
||||
});
|
||||
|
||||
on("config/getBrowserSerialized", (msg) => {
|
||||
|
|
@ -186,7 +272,7 @@ export class Core {
|
|||
async function* llmStreamChat(
|
||||
configHandler: ConfigHandler,
|
||||
abortedMessageIds: Set<string>,
|
||||
msg: Message<Protocol["llm/streamChat"][0]>,
|
||||
msg: Message<ToCoreProtocol["llm/streamChat"][0]>,
|
||||
) {
|
||||
const model = await configHandler.llmFromTitle(msg.data.title);
|
||||
const gen = model.streamChat(
|
||||
|
|
@ -222,7 +308,7 @@ export class Core {
|
|||
configHandler: ConfigHandler,
|
||||
abortedMessageIds: Set<string>,
|
||||
|
||||
msg: Message<Protocol["llm/streamComplete"][0]>,
|
||||
msg: Message<ToCoreProtocol["llm/streamComplete"][0]>,
|
||||
) {
|
||||
const model = await configHandler.llmFromTitle(msg.data.title);
|
||||
const gen = model.streamComplete(
|
||||
|
|
@ -254,10 +340,26 @@ export class Core {
|
|||
llmStreamComplete(this.configHandler, this.abortedMessageIds, msg),
|
||||
);
|
||||
|
||||
on("llm/complete", async (msg) => {
|
||||
const model = await this.configHandler.llmFromTitle(msg.data.title);
|
||||
const completion = await model.complete(
|
||||
msg.data.prompt,
|
||||
msg.data.completionOptions,
|
||||
);
|
||||
return completion;
|
||||
});
|
||||
on("llm/listModels", async (msg) => {
|
||||
const config = await this.configHandler.loadConfig();
|
||||
const model =
|
||||
config.models.find((model) => model.title === msg.data.title) ??
|
||||
config.models.find((model) => model.title?.startsWith(msg.data.title));
|
||||
return model?.listModels();
|
||||
});
|
||||
|
||||
async function* runNodeJsSlashCommand(
|
||||
configHandler: ConfigHandler,
|
||||
abortedMessageIds: Set<string>,
|
||||
msg: Message<Protocol["command/run"][0]>,
|
||||
msg: Message<ToCoreProtocol["command/run"][0]>,
|
||||
) {
|
||||
const {
|
||||
input,
|
||||
|
|
@ -329,7 +431,7 @@ export class Core {
|
|||
async function* streamDiffLinesGenerator(
|
||||
configHandler: ConfigHandler,
|
||||
abortedMessageIds: Set<string>,
|
||||
msg: Message<Protocol["streamDiffLines"][0]>,
|
||||
msg: Message<ToCoreProtocol["streamDiffLines"][0]>,
|
||||
) {
|
||||
const data = msg.data;
|
||||
const llm = await configHandler.llmFromTitle(msg.data.modelTitle);
|
||||
|
|
@ -355,13 +457,57 @@ export class Core {
|
|||
on("streamDiffLines", (msg) =>
|
||||
streamDiffLinesGenerator(this.configHandler, this.abortedMessageIds, msg),
|
||||
);
|
||||
|
||||
on("completeOnboarding", (msg) => {
|
||||
const mode = msg.data.mode;
|
||||
Telemetry.capture("onboardingSelection", {
|
||||
mode,
|
||||
});
|
||||
if (mode === "custom" || mode === "localExistingUser") {
|
||||
return;
|
||||
}
|
||||
editConfigJson(
|
||||
mode === "local"
|
||||
? setupLocalMode
|
||||
: mode === "localAfterFreeTrial"
|
||||
? setupLocalAfterFreeTrial
|
||||
: mode === "optimized"
|
||||
? setupOptimizedMode
|
||||
: setupOptimizedExistingUserMode,
|
||||
);
|
||||
this.configHandler.reloadConfig();
|
||||
});
|
||||
|
||||
on("stats/getTokensPerDay", async (msg) => {
|
||||
const rows = await DevDataSqliteDb.getTokensPerDay();
|
||||
return rows;
|
||||
});
|
||||
on("stats/getTokensPerModel", async (msg) => {
|
||||
const rows = await DevDataSqliteDb.getTokensPerModel();
|
||||
return rows;
|
||||
});
|
||||
on("index/forceReIndex", async (msg) => {
|
||||
const dirs = msg.data ? [msg.data] : await this.ide.getWorkspaceDirs();
|
||||
this.refreshCodebaseIndex(dirs);
|
||||
});
|
||||
on("index/setPaused", (msg) => {
|
||||
new GlobalContext().update("indexingPaused", msg.data);
|
||||
indexingPauseToken.paused = msg.data;
|
||||
});
|
||||
}
|
||||
|
||||
public invoke<T extends keyof Protocol>(
|
||||
method: keyof Protocol,
|
||||
data: Protocol[T][0],
|
||||
): Protocol[T][1] {
|
||||
const response = this.messenger.invoke(method, data);
|
||||
return response;
|
||||
private indexingCancellationController: AbortController | undefined;
|
||||
|
||||
private async refreshCodebaseIndex(dirs: string[]) {
|
||||
if (this.indexingCancellationController) {
|
||||
this.indexingCancellationController.abort();
|
||||
}
|
||||
this.indexingCancellationController = new AbortController();
|
||||
for await (const update of (await this.codebaseIndexerPromise).refresh(
|
||||
dirs,
|
||||
this.indexingCancellationController.signal,
|
||||
)) {
|
||||
this.messenger.request("indexProgress", update);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -53,7 +53,7 @@ export async function* streamDiff(
|
|||
|
||||
// Once at the edge, only one choice
|
||||
if (newLineResult.done === true && oldLines.length > 0) {
|
||||
for (let oldLine of oldLines) {
|
||||
for (const oldLine of oldLines) {
|
||||
yield { type: "old", line: oldLine };
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -10,11 +10,7 @@ function linesMatchPerfectly(lineA: string, lineB: string): boolean {
|
|||
|
||||
const END_BRACKETS = ["}", "});", "})"];
|
||||
|
||||
function linesMatch(
|
||||
lineA: string,
|
||||
lineB: string,
|
||||
linesBetween: number = 0,
|
||||
): boolean {
|
||||
function linesMatch(lineA: string, lineB: string, linesBetween = 0): boolean {
|
||||
// Require a perfect (without padding) match for these lines
|
||||
// Otherwise they are edit distance 1 from empty lines and other single char lines (e.g. each other)
|
||||
if (["}", "*", "});", "})"].includes(lineA.trim())) {
|
||||
|
|
@ -37,7 +33,7 @@ function linesMatch(
|
|||
export function matchLine(
|
||||
newLine: string,
|
||||
oldLines: string[],
|
||||
permissiveAboutIndentation: boolean = false,
|
||||
permissiveAboutIndentation = false,
|
||||
): [number, boolean, string] {
|
||||
// Only match empty lines if it's the next one:
|
||||
if (newLine.trim() === "" && oldLines[0]?.trim() === "") {
|
||||
|
|
@ -53,7 +49,8 @@ export function matchLine(
|
|||
|
||||
if (linesMatchPerfectly(newLine, oldLines[i])) {
|
||||
return [i, true, newLine];
|
||||
} else if (linesMatch(newLine, oldLines[i], i)) {
|
||||
}
|
||||
if (linesMatch(newLine, oldLines[i], i)) {
|
||||
// This is a way to fix indentation, but only for sufficiently long lines to avoid matching whitespace or short lines
|
||||
if (
|
||||
newLine.trimStart() === oldLines[i].trimStart() &&
|
||||
|
|
|
|||
|
|
@ -374,6 +374,13 @@ export interface IndexTag extends BranchAndDir {
|
|||
artifactId: string;
|
||||
}
|
||||
|
||||
export enum FileType {
|
||||
Unkown = 0,
|
||||
File = 1,
|
||||
Directory = 2,
|
||||
SymbolicLink = 64,
|
||||
}
|
||||
|
||||
export interface IDE {
|
||||
getIdeInfo(): Promise<IdeInfo>;
|
||||
getDiff(): Promise<string>;
|
||||
|
|
@ -415,9 +422,14 @@ export interface IDE {
|
|||
subprocess(command: string): Promise<[string, string]>;
|
||||
getProblems(filepath?: string | undefined): Promise<Problem[]>;
|
||||
getBranch(dir: string): Promise<string>;
|
||||
getStats(directory: string): Promise<{ [path: string]: number }>;
|
||||
getTags(artifactId: string): Promise<IndexTag[]>;
|
||||
getRepoName(dir: string): Promise<string | undefined>;
|
||||
errorPopup(message: string): Promise<void>;
|
||||
infoPopup(message: string): Promise<void>;
|
||||
|
||||
getGitRootPath(dir: string): Promise<string | undefined>;
|
||||
listDir(dir: string): Promise<[string, FileType][]>;
|
||||
getLastModified(files: string[]): Promise<{ [path: string]: number }>;
|
||||
}
|
||||
|
||||
// Slash Commands
|
||||
|
|
@ -715,7 +727,7 @@ interface ModelRoles {
|
|||
inlineEdit?: string;
|
||||
}
|
||||
|
||||
interface ExperimantalConfig {
|
||||
interface ExperimentalConfig {
|
||||
contextMenuPrompts?: ContextMenuConfig;
|
||||
modelRoles?: ModelRoles;
|
||||
defaultContext?: "activeFile"[];
|
||||
|
|
@ -739,7 +751,7 @@ export interface SerializedContinueConfig {
|
|||
tabAutocompleteOptions?: Partial<TabAutocompleteOptions>;
|
||||
ui?: ContinueUIConfig;
|
||||
reranker?: RerankerDescription;
|
||||
experimental?: ExperimantalConfig;
|
||||
experimental?: ExperimentalConfig;
|
||||
}
|
||||
|
||||
export type ConfigMergeType = "merge" | "overwrite";
|
||||
|
|
@ -785,7 +797,7 @@ export interface Config {
|
|||
/** Options for the reranker */
|
||||
reranker?: RerankerDescription | Reranker;
|
||||
/** Experimental configuration */
|
||||
experimental?: ExperimantalConfig;
|
||||
experimental?: ExperimentalConfig;
|
||||
}
|
||||
|
||||
export interface ContinueConfig {
|
||||
|
|
@ -804,7 +816,7 @@ export interface ContinueConfig {
|
|||
tabAutocompleteOptions?: Partial<TabAutocompleteOptions>;
|
||||
ui?: ContinueUIConfig;
|
||||
reranker?: Reranker;
|
||||
experimental?: ExperimantalConfig;
|
||||
experimental?: ExperimentalConfig;
|
||||
}
|
||||
|
||||
export interface BrowserSerializedContinueConfig {
|
||||
|
|
@ -821,5 +833,5 @@ export interface BrowserSerializedContinueConfig {
|
|||
embeddingsProvider?: string;
|
||||
ui?: ContinueUIConfig;
|
||||
reranker?: RerankerDescription;
|
||||
experimental?: ExperimantalConfig;
|
||||
experimental?: ExperimentalConfig;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
import fs from "fs";
|
||||
import path from "path";
|
||||
import {
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import type {
|
||||
ChunkWithoutID,
|
||||
ContextItem,
|
||||
ContextSubmenuItem,
|
||||
|
|
@ -16,10 +16,10 @@ import {
|
|||
} from "../util/treeSitter.js";
|
||||
import { DatabaseConnection, SqliteDb, tagToString } from "./refreshIndex.js";
|
||||
import {
|
||||
CodebaseIndex,
|
||||
IndexResultType,
|
||||
MarkCompleteCallback,
|
||||
RefreshIndexResults,
|
||||
type CodebaseIndex,
|
||||
} from "./types.js";
|
||||
|
||||
export class CodeSnippetsCodebaseIndex implements CodebaseIndex {
|
||||
|
|
|
|||
|
|
@ -1,16 +1,21 @@
|
|||
import { BranchAndDir, Chunk, IndexTag, IndexingProgressUpdate } from "../index.js";
|
||||
import {
|
||||
BranchAndDir,
|
||||
Chunk,
|
||||
IndexTag,
|
||||
IndexingProgressUpdate,
|
||||
} from "../index.js";
|
||||
import { RETRIEVAL_PARAMS } from "../util/parameters.js";
|
||||
import { ChunkCodebaseIndex } from "./chunk/ChunkCodebaseIndex.js";
|
||||
import { DatabaseConnection, SqliteDb, tagToString } from "./refreshIndex.js";
|
||||
import {
|
||||
CodebaseIndex,
|
||||
IndexResultType,
|
||||
MarkCompleteCallback,
|
||||
RefreshIndexResults,
|
||||
type CodebaseIndex,
|
||||
} from "./types.js";
|
||||
|
||||
export class FullTextSearchCodebaseIndex implements CodebaseIndex {
|
||||
artifactId: string = "sqliteFts";
|
||||
artifactId = "sqliteFts";
|
||||
|
||||
private async _createTables(db: DatabaseConnection) {
|
||||
await db.exec(`CREATE VIRTUAL TABLE IF NOT EXISTS fts USING fts5(
|
||||
|
|
@ -47,7 +52,7 @@ export class FullTextSearchCodebaseIndex implements CodebaseIndex {
|
|||
[item.path, item.cacheKey],
|
||||
);
|
||||
|
||||
for (let chunk of chunks) {
|
||||
for (const chunk of chunks) {
|
||||
const { lastID } = await db.run(
|
||||
"INSERT INTO fts (path, content) VALUES (?, ?)",
|
||||
[item.path, chunk.content],
|
||||
|
|
|
|||
|
|
@ -32,7 +32,7 @@ interface LanceDbRow {
|
|||
|
||||
export class LanceDbIndex implements CodebaseIndex {
|
||||
get artifactId(): string {
|
||||
return "vectordb::" + this.embeddingsProvider.id;
|
||||
return `vectordb::${this.embeddingsProvider.id}`;
|
||||
}
|
||||
|
||||
static MAX_CHUNK_SIZE = MAX_CHUNK_SIZE;
|
||||
|
|
@ -79,7 +79,7 @@ export class LanceDbIndex implements CodebaseIndex {
|
|||
const content = contents[i];
|
||||
const chunks: Chunk[] = [];
|
||||
|
||||
for await (let chunk of chunkDocument(
|
||||
for await (const chunk of chunkDocument(
|
||||
items[i].path,
|
||||
content,
|
||||
LanceDbIndex.MAX_CHUNK_SIZE,
|
||||
|
|
@ -258,7 +258,7 @@ export class LanceDbIndex implements CodebaseIndex {
|
|||
}
|
||||
|
||||
// Add tag - retrieve the computed info from lance sqlite cache
|
||||
for (let { path, cacheKey } of results.addTag) {
|
||||
for (const { path, cacheKey } of results.addTag) {
|
||||
const stmt = await sqlite.prepare(
|
||||
"SELECT * FROM lance_db_cache WHERE cacheKey = ? AND path = ?",
|
||||
cacheKey,
|
||||
|
|
@ -279,7 +279,7 @@ export class LanceDbIndex implements CodebaseIndex {
|
|||
table = await db.createTable(tableName, lanceRows);
|
||||
needToCreateTable = false;
|
||||
} else if (lanceRows.length > 0) {
|
||||
await table!.add(lanceRows);
|
||||
await table?.add(lanceRows);
|
||||
}
|
||||
|
||||
markComplete([{ path, cacheKey }], IndexResultType.AddTag);
|
||||
|
|
@ -287,15 +287,15 @@ export class LanceDbIndex implements CodebaseIndex {
|
|||
|
||||
// Delete or remove tag - remove from lance table)
|
||||
if (!needToCreateTable) {
|
||||
for (let { path, cacheKey } of [...results.removeTag, ...results.del]) {
|
||||
for (const { path, cacheKey } of [...results.removeTag, ...results.del]) {
|
||||
// This is where the aforementioned lowercase conversion problem shows
|
||||
await table!.delete(`cachekey = '${cacheKey}' AND path = '${path}'`);
|
||||
await table?.delete(`cachekey = '${cacheKey}' AND path = '${path}'`);
|
||||
}
|
||||
}
|
||||
markComplete(results.removeTag, IndexResultType.RemoveTag);
|
||||
|
||||
// Delete - also remove from sqlite cache
|
||||
for (let { path, cacheKey } of results.del) {
|
||||
for (const { path, cacheKey } of results.del) {
|
||||
await sqlite.run(
|
||||
"DELETE FROM lance_db_cache WHERE cacheKey = ? AND path = ?",
|
||||
cacheKey,
|
||||
|
|
|
|||
|
|
@ -4,15 +4,15 @@ import { MAX_CHUNK_SIZE } from "../../llm/constants.js";
|
|||
import { getBasename } from "../../util/index.js";
|
||||
import { DatabaseConnection, SqliteDb, tagToString } from "../refreshIndex.js";
|
||||
import {
|
||||
CodebaseIndex,
|
||||
IndexResultType,
|
||||
MarkCompleteCallback,
|
||||
RefreshIndexResults,
|
||||
type CodebaseIndex,
|
||||
} from "../types.js";
|
||||
import { chunkDocument } from "./chunk.js";
|
||||
|
||||
export class ChunkCodebaseIndex implements CodebaseIndex {
|
||||
static artifactId: string = "chunks";
|
||||
static artifactId = "chunks";
|
||||
artifactId: string = ChunkCodebaseIndex.artifactId;
|
||||
|
||||
constructor(
|
||||
|
|
@ -101,7 +101,7 @@ export class ChunkCodebaseIndex implements CodebaseIndex {
|
|||
const item = results.compute[i];
|
||||
|
||||
// Insert chunks
|
||||
for await (let chunk of chunkDocument(
|
||||
for await (const chunk of chunkDocument(
|
||||
item.path,
|
||||
contents[i],
|
||||
MAX_CHUNK_SIZE,
|
||||
|
|
|
|||
|
|
@ -20,7 +20,7 @@ export function* basicChunker(
|
|||
}
|
||||
|
||||
if (lineTokens < maxChunkSize) {
|
||||
chunkContent += line + "\n";
|
||||
chunkContent += `${line}\n`;
|
||||
chunkTokens += lineTokens + 1;
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -38,7 +38,7 @@ export async function* chunkDocument(
|
|||
digest: string,
|
||||
): AsyncGenerator<Chunk> {
|
||||
let index = 0;
|
||||
for await (let chunkWithoutId of chunkDocumentWithoutId(
|
||||
for await (const chunkWithoutId of chunkDocumentWithoutId(
|
||||
filepath,
|
||||
contents,
|
||||
maxChunkSize,
|
||||
|
|
|
|||
|
|
@ -6,9 +6,8 @@ import { getParserForFile } from "../../util/treeSitter.js";
|
|||
function collapsedReplacement(node: SyntaxNode): string {
|
||||
if (node.type === "statement_block") {
|
||||
return "{ ... }";
|
||||
} else {
|
||||
return "...";
|
||||
}
|
||||
return "...";
|
||||
}
|
||||
|
||||
function firstChild(
|
||||
|
|
@ -19,9 +18,8 @@ function firstChild(
|
|||
return (
|
||||
node.children.find((child) => grammarName.includes(child.type)) || null
|
||||
);
|
||||
} else {
|
||||
return node.children.find((child) => child.type === grammarName) || null;
|
||||
}
|
||||
return node.children.find((child) => child.type === grammarName) || null;
|
||||
}
|
||||
|
||||
function collapseChildren(
|
||||
|
|
@ -138,12 +136,10 @@ function constructFunctionDefinitionChunk(
|
|||
// If inside a class, include the class header
|
||||
const classNode = node.parent.parent;
|
||||
const classBlock = node.parent;
|
||||
return (
|
||||
code.slice(classNode.startIndex, classBlock.startIndex) +
|
||||
"...\n\n" +
|
||||
" ".repeat(node.startPosition.column) + // ...
|
||||
funcText
|
||||
);
|
||||
return `${code.slice(
|
||||
classNode.startIndex,
|
||||
classBlock.startIndex,
|
||||
)}...\n\n${" ".repeat(node.startPosition.column)}${funcText}`;
|
||||
}
|
||||
return funcText;
|
||||
}
|
||||
|
|
@ -208,7 +204,7 @@ export async function* codeChunker(
|
|||
return;
|
||||
}
|
||||
|
||||
let parser = await getParserForFile(filepath);
|
||||
const parser = await getParserForFile(filepath);
|
||||
if (parser === undefined) {
|
||||
console.warn(`Failed to load parser for file ${filepath}: `);
|
||||
return;
|
||||
|
|
|
|||
|
|
@ -74,7 +74,8 @@ export async function* markdownChunker(
|
|||
},
|
||||
};
|
||||
return;
|
||||
} else if (hLevel > 4) {
|
||||
}
|
||||
if (hLevel > 4) {
|
||||
const header = findHeader(content.split("\n"));
|
||||
|
||||
for (const chunk of basicChunker(content, maxChunkSize)) {
|
||||
|
|
@ -89,7 +90,7 @@ export async function* markdownChunker(
|
|||
return;
|
||||
}
|
||||
|
||||
const h = "#".repeat(hLevel + 1) + " ";
|
||||
const h = `${"#".repeat(hLevel + 1)} `;
|
||||
const lines = content.split("\n");
|
||||
const sections = [];
|
||||
|
||||
|
|
@ -131,7 +132,7 @@ export async function* markdownChunker(
|
|||
hLevel + 1,
|
||||
)) {
|
||||
yield {
|
||||
content: section.header + "\n" + chunk.content,
|
||||
content: `${section.header}\n${chunk.content}`,
|
||||
startLine: section.startLine + chunk.startLine,
|
||||
endLine: section.startLine + chunk.endLine,
|
||||
otherMetadata: {
|
||||
|
|
|
|||
|
|
@ -22,18 +22,18 @@ function breakdownArticleComponent(
|
|||
article: ArticleComponent,
|
||||
subpath: string,
|
||||
): Chunk[] {
|
||||
let chunks: Chunk[] = [];
|
||||
const chunks: Chunk[] = [];
|
||||
|
||||
let lines = article.body.split("\n");
|
||||
const lines = article.body.split("\n");
|
||||
let startLine = 0;
|
||||
let endLine = 0;
|
||||
let content = "";
|
||||
let index = 0;
|
||||
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
let line = lines[i];
|
||||
const line = lines[i];
|
||||
if (content.length + line.length <= MAX_CHUNK_SIZE) {
|
||||
content += line + "\n";
|
||||
content += `${line}\n`;
|
||||
endLine = i;
|
||||
} else {
|
||||
chunks.push({
|
||||
|
|
@ -45,12 +45,12 @@ function breakdownArticleComponent(
|
|||
},
|
||||
index: index,
|
||||
filepath: new URL(
|
||||
subpath + `#${cleanFragment(article.title)}`,
|
||||
`${subpath}#${cleanFragment(article.title)}`,
|
||||
url,
|
||||
).toString(),
|
||||
digest: subpath,
|
||||
});
|
||||
content = line + "\n";
|
||||
content = `${line}\n`;
|
||||
startLine = i;
|
||||
endLine = i;
|
||||
index += 1;
|
||||
|
|
@ -68,7 +68,7 @@ function breakdownArticleComponent(
|
|||
},
|
||||
index: index,
|
||||
filepath: new URL(
|
||||
subpath + `#${cleanFragment(article.title)}`,
|
||||
`${subpath}#${cleanFragment(article.title)}`,
|
||||
url,
|
||||
).toString(),
|
||||
digest: subpath,
|
||||
|
|
@ -82,8 +82,8 @@ function breakdownArticleComponent(
|
|||
export function chunkArticle(articleResult: Article): Chunk[] {
|
||||
let chunks: Chunk[] = [];
|
||||
|
||||
for (let article of articleResult.article_components) {
|
||||
let articleChunks = breakdownArticleComponent(
|
||||
for (const article of articleResult.article_components) {
|
||||
const articleChunks = breakdownArticleComponent(
|
||||
articleResult.url,
|
||||
article,
|
||||
articleResult.subpath,
|
||||
|
|
@ -122,14 +122,14 @@ export function stringToArticle(
|
|||
): Article | undefined {
|
||||
try {
|
||||
const dom = new JSDOM(html);
|
||||
let reader = new Readability(dom.window.document);
|
||||
let article = reader.parse();
|
||||
const reader = new Readability(dom.window.document);
|
||||
const article = reader.parse();
|
||||
|
||||
if (!article) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
let article_components = extractTitlesAndBodies(article.content);
|
||||
const article_components = extractTitlesAndBodies(article.content);
|
||||
|
||||
return {
|
||||
url,
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
import { Octokit } from "@octokit/rest";
|
||||
import cheerio from "cheerio";
|
||||
import fetch from "node-fetch";
|
||||
import { URL } from "url";
|
||||
import { URL } from "node:url";
|
||||
|
||||
const IGNORE_PATHS_ENDING_IN = [
|
||||
"favicon.ico",
|
||||
|
|
@ -25,7 +25,7 @@ async function crawlGithubRepo(baseUrl: URL) {
|
|||
|
||||
const [_, owner, repo] = baseUrl.pathname.split("/");
|
||||
|
||||
let dirContentsConfig = {
|
||||
const dirContentsConfig = {
|
||||
owner: owner,
|
||||
repo: repo,
|
||||
};
|
||||
|
|
@ -63,13 +63,12 @@ async function getLinksFromUrl(url: string, path: string) {
|
|||
html: "",
|
||||
links: [],
|
||||
};
|
||||
} else {
|
||||
console.error(error);
|
||||
return {
|
||||
html: "",
|
||||
links: [],
|
||||
};
|
||||
}
|
||||
console.error(error);
|
||||
return {
|
||||
html: "",
|
||||
links: [],
|
||||
};
|
||||
}
|
||||
|
||||
const html = await response.text();
|
||||
|
|
@ -156,7 +155,7 @@ export async function* crawlPage(url: URL): AsyncGenerator<PageData> {
|
|||
};
|
||||
}
|
||||
|
||||
for (let link of links) {
|
||||
for (const link of links) {
|
||||
if (!paths.includes(link)) {
|
||||
paths.push(link);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
import { Database, open } from "sqlite";
|
||||
import { open, type Database } from "sqlite";
|
||||
import sqlite3 from "sqlite3";
|
||||
import { Chunk } from "../../index.js";
|
||||
import { getDocsSqlitePath, getLanceDbPath } from "../../util/paths.js";
|
||||
|
|
@ -45,7 +45,7 @@ export async function retrieveDocs(
|
|||
vector: number[],
|
||||
nRetrieve: number,
|
||||
embeddingsProviderId: string,
|
||||
nested: boolean = false,
|
||||
nested = false,
|
||||
): Promise<Chunk[]> {
|
||||
const lancedb = await import("vectordb");
|
||||
const db = await getDBDocs();
|
||||
|
|
@ -69,7 +69,9 @@ export async function retrieveDocs(
|
|||
const tableNames = await lance.tableNames();
|
||||
if (!tableNames.includes(DOCS_TABLE_NAME)) {
|
||||
const downloaded = await downloadDocs();
|
||||
if (downloaded) {return downloaded;}
|
||||
if (downloaded) {
|
||||
return downloaded;
|
||||
}
|
||||
}
|
||||
|
||||
const table = await lance.openTable(DOCS_TABLE_NAME);
|
||||
|
|
@ -83,7 +85,9 @@ export async function retrieveDocs(
|
|||
|
||||
if ((!docs || docs.length === 0) && !nested) {
|
||||
const downloaded = await downloadDocs();
|
||||
if (downloaded) {return downloaded;}
|
||||
if (downloaded) {
|
||||
return downloaded;
|
||||
}
|
||||
}
|
||||
|
||||
return docs.map((doc) => ({
|
||||
|
|
|
|||
|
|
@ -1,4 +1,8 @@
|
|||
import { Chunk, EmbeddingsProvider, IndexingProgressUpdate } from "../../index.js";
|
||||
import {
|
||||
Chunk,
|
||||
EmbeddingsProvider,
|
||||
IndexingProgressUpdate,
|
||||
} from "../../index.js";
|
||||
|
||||
import { Article, chunkArticle, pageToArticle } from "./article.js";
|
||||
import { crawlPage } from "./crawl.js";
|
||||
|
|
@ -28,7 +32,9 @@ export async function* indexDocs(
|
|||
|
||||
for await (const page of crawlPage(baseUrl)) {
|
||||
const article = pageToArticle(page);
|
||||
if (!article) {continue;}
|
||||
if (!article) {
|
||||
continue;
|
||||
}
|
||||
|
||||
articles.push(article);
|
||||
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@ class FreeTrialEmbeddingsProvider extends BaseEmbeddingsProvider {
|
|||
};
|
||||
|
||||
get id(): string {
|
||||
return FreeTrialEmbeddingsProvider.defaultOptions!.model!;
|
||||
return FreeTrialEmbeddingsProvider.defaultOptions?.model!;
|
||||
}
|
||||
|
||||
async embed(chunks: string[]) {
|
||||
|
|
|
|||
|
|
@ -19,7 +19,7 @@ async function embedOne(
|
|||
);
|
||||
const resp = await fetchWithBackoff();
|
||||
if (!resp.ok) {
|
||||
throw new Error("Failed to embed chunk: " + (await resp.text()));
|
||||
throw new Error(`Failed to embed chunk: ${await resp.text()}`);
|
||||
}
|
||||
|
||||
return (await resp.json()).embedding;
|
||||
|
|
|
|||
|
|
@ -1,25 +1,31 @@
|
|||
// prettier-ignore
|
||||
// @ts-ignore
|
||||
import { PipelineType, env, pipeline } from "../../vendor/modules/@xenova/transformers/src/transformers.js";
|
||||
|
||||
import path from "path";
|
||||
// @ts-ignore
|
||||
// prettier-ignore
|
||||
import { type PipelineType } from "../../vendor/modules/@xenova/transformers/src/transformers.js";
|
||||
import BaseEmbeddingsProvider from "./BaseEmbeddingsProvider.js";
|
||||
|
||||
env.allowLocalModels = true;
|
||||
env.allowRemoteModels = false;
|
||||
env.localModelPath = path.join(__dirname, "..", "models");
|
||||
|
||||
class EmbeddingsPipeline {
|
||||
static task: PipelineType = "feature-extraction";
|
||||
static model = "all-MiniLM-L6-v2";
|
||||
static instance: any | null = null;
|
||||
|
||||
static async getInstance() {
|
||||
if (this.instance === null) {
|
||||
this.instance = await pipeline(this.task, this.model);
|
||||
if (EmbeddingsPipeline.instance === null) {
|
||||
// @ts-ignore
|
||||
// prettier-ignore
|
||||
const { env, pipeline } = await import("../../vendor/modules/@xenova/transformers/src/transformers.js");
|
||||
|
||||
env.allowLocalModels = true;
|
||||
env.allowRemoteModels = false;
|
||||
env.localModelPath = path.join(__dirname, "..", "models");
|
||||
|
||||
EmbeddingsPipeline.instance = await pipeline(
|
||||
EmbeddingsPipeline.task,
|
||||
EmbeddingsPipeline.model,
|
||||
);
|
||||
}
|
||||
|
||||
return this.instance;
|
||||
return EmbeddingsPipeline.instance;
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -34,11 +40,11 @@ export class TransformersJsEmbeddingsProvider extends BaseEmbeddingsProvider {
|
|||
}
|
||||
|
||||
get id(): string {
|
||||
return "sentence-transformers/all-MiniLM-L6-v2";
|
||||
return EmbeddingsPipeline.model;
|
||||
}
|
||||
|
||||
async embed(chunks: string[]) {
|
||||
let extractor = await EmbeddingsPipeline.getInstance();
|
||||
const extractor = await EmbeddingsPipeline.getInstance();
|
||||
|
||||
if (!extractor) {
|
||||
throw new Error("TransformerJS embeddings pipeline is not initialized");
|
||||
|
|
@ -48,17 +54,17 @@ export class TransformersJsEmbeddingsProvider extends BaseEmbeddingsProvider {
|
|||
return [];
|
||||
}
|
||||
|
||||
let outputs = [];
|
||||
const outputs = [];
|
||||
for (
|
||||
let i = 0;
|
||||
i < chunks.length;
|
||||
i += TransformersJsEmbeddingsProvider.MaxGroupSize
|
||||
) {
|
||||
let chunkGroup = chunks.slice(
|
||||
const chunkGroup = chunks.slice(
|
||||
i,
|
||||
i + TransformersJsEmbeddingsProvider.MaxGroupSize,
|
||||
);
|
||||
let output = await extractor(chunkGroup, {
|
||||
const output = await extractor(chunkGroup, {
|
||||
pooling: "mean",
|
||||
normalize: true,
|
||||
});
|
||||
|
|
|
|||
|
|
@ -1,10 +1,10 @@
|
|||
import path from "path";
|
||||
import path from "node:path";
|
||||
import {
|
||||
env,
|
||||
pipeline,
|
||||
} from "../../vendor/node_modules/@xenova/transformers/types/transformers";
|
||||
import TransformersJsEmbeddingsProvider from "./TransformersJsEmbeddingsProvider";
|
||||
const { parentPort } = require("worker_threads");
|
||||
const { parentPort } = require("node:worker_threads");
|
||||
|
||||
env.allowLocalModels = true;
|
||||
env.allowRemoteModels = false;
|
||||
|
|
@ -19,11 +19,14 @@ class EmbeddingsPipeline {
|
|||
static instance = null;
|
||||
|
||||
static async getInstance() {
|
||||
if (this.instance === null) {
|
||||
this.instance = await pipeline(this.task, this.model);
|
||||
if (EmbeddingsPipeline.instance === null) {
|
||||
EmbeddingsPipeline.instance = await pipeline(
|
||||
EmbeddingsPipeline.task,
|
||||
EmbeddingsPipeline.model,
|
||||
);
|
||||
}
|
||||
|
||||
return this.instance;
|
||||
return EmbeddingsPipeline.instance;
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -35,17 +38,17 @@ parentPort.on("message", async (chunks) => {
|
|||
throw new Error("TransformerJS embeddings pipeline is not initialized");
|
||||
}
|
||||
|
||||
let outputs = [];
|
||||
const outputs = [];
|
||||
for (
|
||||
let i = 0;
|
||||
i < chunks.length;
|
||||
i += TransformersJsEmbeddingsProvider.MaxGroupSize
|
||||
) {
|
||||
let chunkGroup = chunks.slice(
|
||||
const chunkGroup = chunks.slice(
|
||||
i,
|
||||
i + TransformersJsEmbeddingsProvider.MaxGroupSize,
|
||||
);
|
||||
let output = await extractor(chunkGroup, {
|
||||
const output = await extractor(chunkGroup, {
|
||||
pooling: "mean",
|
||||
normalize: true,
|
||||
});
|
||||
|
|
|
|||
|
|
@ -81,6 +81,7 @@ export const DEFAULT_IGNORE_DIRS = [
|
|||
".continue",
|
||||
"__pycache__",
|
||||
"site-packages",
|
||||
".gradle",
|
||||
".cache",
|
||||
"gems",
|
||||
];
|
||||
|
|
|
|||
|
|
@ -91,13 +91,17 @@ export class CodebaseIndexer {
|
|||
status: "starting",
|
||||
};
|
||||
|
||||
for (let directory of workspaceDirs) {
|
||||
const stats = await this.ide.getStats(directory);
|
||||
for (const directory of workspaceDirs) {
|
||||
// const scheme = vscode.workspace.workspaceFolders?.[0].uri.scheme;
|
||||
// const files = await this.listWorkspaceContents(directory);
|
||||
|
||||
const files = await this.ide.listWorkspaceContents(directory);
|
||||
const stats = await this.ide.getLastModified(files);
|
||||
const branch = await this.ide.getBranch(directory);
|
||||
const repoName = await this.ide.getRepoName(directory);
|
||||
let completedIndexes = 0;
|
||||
|
||||
for (let codebaseIndex of indexesToBuild) {
|
||||
for (const codebaseIndex of indexesToBuild) {
|
||||
// TODO: IndexTag type should use repoName rather than directory
|
||||
const tag: IndexTag = {
|
||||
directory,
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
import crypto from "crypto";
|
||||
import * as fs from "fs";
|
||||
import { Database, open } from "sqlite";
|
||||
import crypto from "node:crypto";
|
||||
import * as fs from "node:fs";
|
||||
import { open, type Database } from "sqlite";
|
||||
import sqlite3 from "sqlite3";
|
||||
import { IndexTag, IndexingProgressUpdate } from "../index.js";
|
||||
import { getIndexSqlitePath } from "../util/paths.js";
|
||||
|
|
@ -106,7 +106,7 @@ async function getAddRemoveForTag(
|
|||
const updateOldVersion: PathAndCacheKey[] = [];
|
||||
const remove: PathAndCacheKey[] = [];
|
||||
|
||||
for (let item of saved) {
|
||||
for (const item of saved) {
|
||||
const { lastUpdated, ...pathAndCacheKey } = item;
|
||||
|
||||
if (currentFiles[item.path] === undefined) {
|
||||
|
|
@ -207,22 +207,22 @@ async function getAddRemoveForTag(
|
|||
}
|
||||
}
|
||||
|
||||
for (let item of updateNewVersion) {
|
||||
for (const item of updateNewVersion) {
|
||||
itemToAction[JSON.stringify(item)] = [
|
||||
item,
|
||||
AddRemoveResultType.UpdateNewVersion,
|
||||
];
|
||||
}
|
||||
for (let item of add) {
|
||||
for (const item of add) {
|
||||
itemToAction[JSON.stringify(item)] = [item, AddRemoveResultType.Add];
|
||||
}
|
||||
for (let item of updateOldVersion) {
|
||||
for (const item of updateOldVersion) {
|
||||
itemToAction[JSON.stringify(item)] = [
|
||||
item,
|
||||
AddRemoveResultType.UpdateOldVersion,
|
||||
];
|
||||
}
|
||||
for (let item of remove) {
|
||||
for (const item of remove) {
|
||||
itemToAction[JSON.stringify(item)] = [item, AddRemoveResultType.Remove];
|
||||
}
|
||||
|
||||
|
|
@ -272,7 +272,7 @@ export async function getComputeDeleteAddRemove(
|
|||
const addTag: PathAndCacheKey[] = [];
|
||||
const removeTag: PathAndCacheKey[] = [];
|
||||
|
||||
for (let { path, cacheKey } of add) {
|
||||
for (const { path, cacheKey } of add) {
|
||||
const existingTags = await getTagsFromGlobalCache(cacheKey, tag.artifactId);
|
||||
if (existingTags.length > 0) {
|
||||
addTag.push({ path, cacheKey });
|
||||
|
|
@ -281,7 +281,7 @@ export async function getComputeDeleteAddRemove(
|
|||
}
|
||||
}
|
||||
|
||||
for (let { path, cacheKey } of remove) {
|
||||
for (const { path, cacheKey } of remove) {
|
||||
const existingTags = await getTagsFromGlobalCache(cacheKey, tag.artifactId);
|
||||
if (existingTags.length > 1) {
|
||||
removeTag.push({ path, cacheKey });
|
||||
|
|
@ -310,14 +310,14 @@ export async function getComputeDeleteAddRemove(
|
|||
markComplete(items, resultType);
|
||||
|
||||
// Update the global cache
|
||||
let results: any = {
|
||||
const results: any = {
|
||||
compute: [],
|
||||
del: [],
|
||||
addTag: [],
|
||||
removeTag: [],
|
||||
};
|
||||
results[resultType] = items;
|
||||
for await (let _ of globalCacheIndex.update(
|
||||
for await (const _ of globalCacheIndex.update(
|
||||
tag,
|
||||
results,
|
||||
() => {},
|
||||
|
|
@ -334,7 +334,7 @@ export class GlobalCacheCodeBaseIndex implements CodebaseIndex {
|
|||
constructor(db: DatabaseConnection) {
|
||||
this.db = db;
|
||||
}
|
||||
artifactId: string = "globalCache";
|
||||
artifactId = "globalCache";
|
||||
|
||||
static async create(): Promise<GlobalCacheCodeBaseIndex> {
|
||||
return new GlobalCacheCodeBaseIndex(await SqliteDb.get());
|
||||
|
|
|
|||
|
|
@ -305,7 +305,7 @@ function autodetectPromptTemplates(
|
|||
}
|
||||
|
||||
if (editTemplate !== null) {
|
||||
templates["edit"] = editTemplate;
|
||||
templates.edit = editTemplate;
|
||||
}
|
||||
|
||||
return templates;
|
||||
|
|
@ -316,6 +316,5 @@ export {
|
|||
autodetectTemplateFunction,
|
||||
autodetectTemplateType,
|
||||
llmCanGenerateInParallel,
|
||||
modelSupportsImages
|
||||
modelSupportsImages,
|
||||
};
|
||||
|
||||
|
|
|
|||
|
|
@ -11,7 +11,7 @@ export function constructMessages(history: ChatHistory): ChatMessage[] {
|
|||
: [{ type: "text", text: historyItem.message.content } as MessagePart];
|
||||
|
||||
const ctxItems = historyItem.contextItems.map((ctxItem) => {
|
||||
return { type: "text", text: ctxItem.content + "\n" } as MessagePart;
|
||||
return { type: "text", text: `${ctxItem.content}\n` } as MessagePart;
|
||||
});
|
||||
|
||||
content = [...ctxItems, ...content];
|
||||
|
|
|
|||
|
|
@ -10,13 +10,16 @@ interface Encoding {
|
|||
}
|
||||
|
||||
class LlamaEncoding implements Encoding {
|
||||
encode(text: string, allowedSpecial?: string[] | "all" | undefined, disallowedSpecial?: string[] | "all" | undefined): number[] {
|
||||
encode(
|
||||
text: string,
|
||||
allowedSpecial?: string[] | "all" | undefined,
|
||||
disallowedSpecial?: string[] | "all" | undefined,
|
||||
): number[] {
|
||||
return llamaTokenizer.encode(text);
|
||||
}
|
||||
decode(tokens: number[]): string {
|
||||
return llamaTokenizer.decode(tokens);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
let gptEncoding: Encoding | null = null;
|
||||
|
|
@ -39,15 +42,14 @@ function encodingForModel(modelName: string): Encoding {
|
|||
function countImageTokens(content: MessagePart): number {
|
||||
if (content.type === "imageUrl") {
|
||||
return 85;
|
||||
} else {
|
||||
throw new Error("Non-image content type");
|
||||
}
|
||||
throw new Error("Non-image content type");
|
||||
}
|
||||
|
||||
function countTokens(
|
||||
content: MessageContent,
|
||||
// defaults to llama2 because the tokenizer tends to produce more tokens
|
||||
modelName: string = "llama2",
|
||||
modelName = "llama2",
|
||||
): number {
|
||||
const encoding = encodingForModel(modelName);
|
||||
if (Array.isArray(content)) {
|
||||
|
|
@ -69,7 +71,7 @@ function flattenMessages(msgs: ChatMessage[]): ChatMessage[] {
|
|||
flattened.length > 0 &&
|
||||
flattened[flattened.length - 1].role === msg.role
|
||||
) {
|
||||
flattened[flattened.length - 1].content += "\n\n" + (msg.content || "");
|
||||
flattened[flattened.length - 1].content += `\n\n${msg.content || ""}`;
|
||||
} else {
|
||||
flattened.push(msg);
|
||||
}
|
||||
|
|
@ -83,9 +85,8 @@ export function stripImages(content: MessageContent): string {
|
|||
.filter((part) => part.type === "text")
|
||||
.map((part) => part.text)
|
||||
.join("\n");
|
||||
} else {
|
||||
return content;
|
||||
}
|
||||
return content;
|
||||
}
|
||||
|
||||
function countChatMessageTokens(
|
||||
|
|
@ -181,10 +182,9 @@ function pruneRawPromptFromBottom(
|
|||
|
||||
function summarize(message: MessageContent): string {
|
||||
if (Array.isArray(message)) {
|
||||
return stripImages(message).substring(0, 100) + "...";
|
||||
} else {
|
||||
return message.substring(0, 100) + "...";
|
||||
return `${stripImages(message).substring(0, 100)}...`;
|
||||
}
|
||||
return `${message.substring(0, 100)}...`;
|
||||
}
|
||||
|
||||
function pruneChatHistory(
|
||||
|
|
@ -215,7 +215,7 @@ function pruneChatHistory(
|
|||
for (let i = 0; i < longerThanOneThird.length; i++) {
|
||||
// Prune line-by-line from the top
|
||||
const message = longerThanOneThird[i];
|
||||
let content = stripImages(message.content);
|
||||
const content = stripImages(message.content);
|
||||
const deltaNeeded = totalTokens - contextLength;
|
||||
const delta = Math.min(deltaNeeded, distanceFromThird[i]);
|
||||
message.content = pruneStringFromTop(
|
||||
|
|
@ -283,7 +283,7 @@ function pruneChatHistory(
|
|||
|
||||
function compileChatMessages(
|
||||
modelName: string,
|
||||
msgs: ChatMessage[] | undefined = undefined,
|
||||
msgs: ChatMessage[] | undefined,
|
||||
contextLength: number,
|
||||
maxTokens: number,
|
||||
supportsImages: boolean,
|
||||
|
|
@ -364,6 +364,5 @@ export {
|
|||
pruneLinesFromTop,
|
||||
pruneRawPromptFromTop,
|
||||
pruneStringFromBottom,
|
||||
pruneStringFromTop
|
||||
pruneStringFromTop,
|
||||
};
|
||||
|
||||
|
|
|
|||
|
|
@ -96,14 +96,14 @@ export abstract class BaseLLM implements ILLM {
|
|||
|
||||
private _llmOptions: LLMOptions;
|
||||
|
||||
constructor(options: LLMOptions) {
|
||||
this._llmOptions = options;
|
||||
constructor(_options: LLMOptions) {
|
||||
this._llmOptions = _options;
|
||||
|
||||
// Set default options
|
||||
options = {
|
||||
const options = {
|
||||
title: (this.constructor as typeof BaseLLM).providerName,
|
||||
...(this.constructor as typeof BaseLLM).defaultOptions,
|
||||
...options,
|
||||
..._options,
|
||||
};
|
||||
|
||||
const templateType =
|
||||
|
|
@ -142,7 +142,7 @@ export abstract class BaseLLM implements ILLM {
|
|||
this.apiKey = options.apiKey;
|
||||
this.apiBase = options.apiBase;
|
||||
if (this.apiBase && !this.apiBase.endsWith("/")) {
|
||||
this.apiBase = this.apiBase + "/";
|
||||
this.apiBase = `${this.apiBase}/`;
|
||||
}
|
||||
|
||||
this.engine = options.engine;
|
||||
|
|
@ -284,7 +284,9 @@ ${prompt}`;
|
|||
"Failed to connect to local Ollama instance. To start Ollama, first download it at https://ollama.ai.",
|
||||
);
|
||||
}
|
||||
throw new Error(`${e}`);
|
||||
throw new Error(
|
||||
`${e.message}\n\nCode: ${e.code}\nError number: ${e.errno}\nSyscall: ${e.erroredSysCall}\nType: ${e.type}\n\n${e.stack}`,
|
||||
);
|
||||
}
|
||||
};
|
||||
return withExponentialBackoff<Response>(
|
||||
|
|
@ -297,7 +299,7 @@ ${prompt}`;
|
|||
private _parseCompletionOptions(options: LLMFullCompletionOptions) {
|
||||
const log = options.log ?? true;
|
||||
const raw = options.raw ?? false;
|
||||
delete options.log;
|
||||
options.log = undefined;
|
||||
|
||||
const completionOptions: CompletionOptions = mergeJson(
|
||||
this.completionOptions,
|
||||
|
|
@ -310,7 +312,7 @@ ${prompt}`;
|
|||
private _formatChatMessages(messages: ChatMessage[]): string {
|
||||
const msgsCopy = messages ? messages.map((msg) => ({ ...msg })) : [];
|
||||
let formatted = "";
|
||||
for (let msg of msgsCopy) {
|
||||
for (const msg of msgsCopy) {
|
||||
if ("content" in msg && Array.isArray(msg.content)) {
|
||||
const content = stripImages(msg.content);
|
||||
msg.content = content;
|
||||
|
|
@ -321,16 +323,16 @@ ${prompt}`;
|
|||
}
|
||||
|
||||
async *streamComplete(
|
||||
prompt: string,
|
||||
_prompt: string,
|
||||
options: LLMFullCompletionOptions = {},
|
||||
) {
|
||||
const { completionOptions, log, raw } =
|
||||
this._parseCompletionOptions(options);
|
||||
|
||||
prompt = pruneRawPromptFromTop(
|
||||
let prompt = pruneRawPromptFromTop(
|
||||
completionOptions.model,
|
||||
this.contextLength,
|
||||
prompt,
|
||||
_prompt,
|
||||
completionOptions.maxTokens ?? DEFAULT_MAX_TOKENS,
|
||||
);
|
||||
|
||||
|
|
@ -362,14 +364,14 @@ ${prompt}`;
|
|||
return { prompt, completion, completionOptions };
|
||||
}
|
||||
|
||||
async complete(prompt: string, options: LLMFullCompletionOptions = {}) {
|
||||
async complete(_prompt: string, options: LLMFullCompletionOptions = {}) {
|
||||
const { completionOptions, log, raw } =
|
||||
this._parseCompletionOptions(options);
|
||||
|
||||
prompt = pruneRawPromptFromTop(
|
||||
let prompt = pruneRawPromptFromTop(
|
||||
completionOptions.model,
|
||||
this.contextLength,
|
||||
prompt,
|
||||
_prompt,
|
||||
completionOptions.maxTokens ?? DEFAULT_MAX_TOKENS,
|
||||
);
|
||||
|
||||
|
|
@ -405,13 +407,13 @@ ${prompt}`;
|
|||
}
|
||||
|
||||
async *streamChat(
|
||||
messages: ChatMessage[],
|
||||
_messages: ChatMessage[],
|
||||
options: LLMFullCompletionOptions = {},
|
||||
): AsyncGenerator<ChatMessage, PromptLog> {
|
||||
const { completionOptions, log, raw } =
|
||||
this._parseCompletionOptions(options);
|
||||
|
||||
messages = this._compileChatMessages(completionOptions, messages);
|
||||
const messages = this._compileChatMessages(completionOptions, _messages);
|
||||
|
||||
const prompt = this.templateMessages
|
||||
? this.templateMessages(messages)
|
||||
|
|
@ -462,6 +464,7 @@ ${prompt}`;
|
|||
};
|
||||
}
|
||||
|
||||
// biome-ignore lint/correctness/useYield: Purposefully not implemented
|
||||
protected async *_streamComplete(
|
||||
prompt: string,
|
||||
options: CompletionOptions,
|
||||
|
|
@ -511,41 +514,40 @@ ${prompt}`;
|
|||
template: PromptTemplate,
|
||||
history: ChatMessage[],
|
||||
otherData: Record<string, string>,
|
||||
canPutWordsInModelsMouth: boolean = false,
|
||||
canPutWordsInModelsMouth = false,
|
||||
): string | ChatMessage[] {
|
||||
if (typeof template === "string") {
|
||||
let data: any = {
|
||||
const data: any = {
|
||||
history: history,
|
||||
...otherData,
|
||||
};
|
||||
if (history.length > 0 && history[0].role == "system") {
|
||||
data["system_message"] = history.shift()!.content;
|
||||
if (history.length > 0 && history[0].role === "system") {
|
||||
data.system_message = history.shift()!.content;
|
||||
}
|
||||
|
||||
const compiledTemplate = Handlebars.compile(template);
|
||||
return compiledTemplate(data);
|
||||
} else {
|
||||
const rendered = template(history, {
|
||||
...otherData,
|
||||
supportsCompletions: this.supportsCompletions() ? "true" : "false",
|
||||
supportsPrefill: this.supportsPrefill() ? "true" : "false",
|
||||
});
|
||||
if (
|
||||
typeof rendered !== "string" &&
|
||||
rendered[rendered.length - 1]?.role === "assistant" &&
|
||||
!canPutWordsInModelsMouth
|
||||
) {
|
||||
// Some providers don't allow you to put words in the model's mouth
|
||||
// So we have to manually compile the prompt template and use
|
||||
// raw /completions, not /chat/completions
|
||||
const templateMessages = autodetectTemplateFunction(
|
||||
this.model,
|
||||
this.providerName,
|
||||
autodetectTemplateType(this.model),
|
||||
);
|
||||
return templateMessages(rendered);
|
||||
}
|
||||
return rendered;
|
||||
}
|
||||
const rendered = template(history, {
|
||||
...otherData,
|
||||
supportsCompletions: this.supportsCompletions() ? "true" : "false",
|
||||
supportsPrefill: this.supportsPrefill() ? "true" : "false",
|
||||
});
|
||||
if (
|
||||
typeof rendered !== "string" &&
|
||||
rendered[rendered.length - 1]?.role === "assistant" &&
|
||||
!canPutWordsInModelsMouth
|
||||
) {
|
||||
// Some providers don't allow you to put words in the model's mouth
|
||||
// So we have to manually compile the prompt template and use
|
||||
// raw /completions, not /chat/completions
|
||||
const templateMessages = autodetectTemplateFunction(
|
||||
this.model,
|
||||
this.providerName,
|
||||
autodetectTemplateType(this.model),
|
||||
);
|
||||
return templateMessages(rendered);
|
||||
}
|
||||
return rendered;
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,4 +1,3 @@
|
|||
import { BaseLLM } from "../index.js";
|
||||
import {
|
||||
ChatMessage,
|
||||
CompletionOptions,
|
||||
|
|
@ -6,6 +5,7 @@ import {
|
|||
ModelProvider,
|
||||
} from "../../index.js";
|
||||
import { stripImages } from "../countTokens.js";
|
||||
import { BaseLLM } from "../index.js";
|
||||
import { streamSse } from "../stream.js";
|
||||
|
||||
class Anthropic extends BaseLLM {
|
||||
|
|
@ -20,10 +20,6 @@ class Anthropic extends BaseLLM {
|
|||
apiBase: "https://api.anthropic.com/v1/",
|
||||
};
|
||||
|
||||
constructor(options: LLMOptions) {
|
||||
super(options);
|
||||
}
|
||||
|
||||
private _convertArgs(options: CompletionOptions) {
|
||||
const finalOptions = {
|
||||
top_k: options.topK,
|
||||
|
|
@ -44,25 +40,23 @@ class Anthropic extends BaseLLM {
|
|||
.map((message) => {
|
||||
if (typeof message.content === "string") {
|
||||
return message;
|
||||
} else {
|
||||
return {
|
||||
...message,
|
||||
content: message.content.map((part) => {
|
||||
if (part.type === "text") {
|
||||
return part;
|
||||
} else {
|
||||
return {
|
||||
type: "image",
|
||||
source: {
|
||||
type: "base64",
|
||||
media_type: "image/jpeg",
|
||||
data: part.imageUrl?.url.split(",")[1],
|
||||
},
|
||||
};
|
||||
}
|
||||
}),
|
||||
};
|
||||
}
|
||||
return {
|
||||
...message,
|
||||
content: message.content.map((part) => {
|
||||
if (part.type === "text") {
|
||||
return part;
|
||||
}
|
||||
return {
|
||||
type: "image",
|
||||
source: {
|
||||
type: "base64",
|
||||
media_type: "image/jpeg",
|
||||
data: part.imageUrl?.url.split(",")[1],
|
||||
},
|
||||
};
|
||||
}),
|
||||
};
|
||||
});
|
||||
return messages;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -2,9 +2,9 @@ import {
|
|||
BedrockRuntimeClient,
|
||||
InvokeModelWithResponseStreamCommand,
|
||||
} from "@aws-sdk/client-bedrock-runtime";
|
||||
import * as fs from "fs";
|
||||
import os from "os";
|
||||
import { join as joinPath } from "path";
|
||||
import * as fs from "node:fs";
|
||||
import os from "node:os";
|
||||
import { join as joinPath } from "node:path";
|
||||
import { promisify } from "util";
|
||||
import {
|
||||
ChatMessage,
|
||||
|
|
@ -69,25 +69,23 @@ class Bedrock extends BaseLLM {
|
|||
.map((message) => {
|
||||
if (typeof message.content === "string") {
|
||||
return message;
|
||||
} else {
|
||||
return {
|
||||
...message,
|
||||
content: message.content.map((part) => {
|
||||
if (part.type === "text") {
|
||||
return part;
|
||||
} else {
|
||||
return {
|
||||
type: "image",
|
||||
source: {
|
||||
type: "base64",
|
||||
media_type: "image/jpeg",
|
||||
data: part.imageUrl?.url.split(",")[1],
|
||||
},
|
||||
};
|
||||
}
|
||||
}),
|
||||
};
|
||||
}
|
||||
return {
|
||||
...message,
|
||||
content: message.content.map((part) => {
|
||||
if (part.type === "text") {
|
||||
return part;
|
||||
}
|
||||
return {
|
||||
type: "image",
|
||||
source: {
|
||||
type: "base64",
|
||||
media_type: "image/jpeg",
|
||||
data: part.imageUrl?.url.split(",")[1],
|
||||
},
|
||||
};
|
||||
}),
|
||||
};
|
||||
});
|
||||
return messages;
|
||||
}
|
||||
|
|
@ -141,7 +139,9 @@ class Bedrock extends BaseLLM {
|
|||
region: this.region,
|
||||
};
|
||||
|
||||
let accessKeyId: string, secretAccessKey: string, sessionToken: string;
|
||||
let accessKeyId: string;
|
||||
let secretAccessKey: string;
|
||||
let sessionToken: string;
|
||||
|
||||
try {
|
||||
const data = await readFile(
|
||||
|
|
@ -159,9 +159,8 @@ class Bedrock extends BaseLLM {
|
|||
}
|
||||
return await this.fetch(new URL(`${this.apiBase}${path}`), {
|
||||
method: "POST",
|
||||
headers: aws4.sign(opts, { accessKeyId, secretAccessKey, sessionToken })[
|
||||
"headers"
|
||||
],
|
||||
headers: aws4.sign(opts, { accessKeyId, secretAccessKey, sessionToken })
|
||||
.headers,
|
||||
body: body,
|
||||
});
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,5 +1,4 @@
|
|||
import socketIOClient, { Socket } from "socket.io-client";
|
||||
import { BaseLLM } from "../index.js";
|
||||
import {
|
||||
ChatMessage,
|
||||
CompletionOptions,
|
||||
|
|
@ -7,6 +6,7 @@ import {
|
|||
ModelProvider,
|
||||
} from "../../index.js";
|
||||
import { stripImages } from "../countTokens.js";
|
||||
import { BaseLLM } from "../index.js";
|
||||
|
||||
interface IFlowiseApiOptions {
|
||||
/** Sampling temperature to use */
|
||||
|
|
@ -67,7 +67,7 @@ class Flowise extends BaseLLM {
|
|||
};
|
||||
|
||||
protected additionalFlowiseConfiguration: IFlowiseKeyValueProperty[] = [];
|
||||
protected timeout: number = 5000;
|
||||
protected timeout = 5000;
|
||||
protected additionalHeaders: IFlowiseKeyValueProperty[] = [];
|
||||
|
||||
constructor(options: IFlowiseProviderLLMOptions) {
|
||||
|
|
@ -92,7 +92,7 @@ class Flowise extends BaseLLM {
|
|||
};
|
||||
|
||||
if (this.apiKey) {
|
||||
headers["Authorization"] = `Bearer ${this.apiKey}`;
|
||||
headers.Authorization = `Bearer ${this.apiKey}`;
|
||||
}
|
||||
|
||||
for (const additionalHeader of this.additionalHeaders) {
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue