{"id":400370,"date":"2024-06-29T15:13:32","date_gmt":"2024-06-29T15:13:32","guid":{"rendered":"http:\/\/savepearlharbor.com\/?p=400370"},"modified":"-0001-11-30T00:00:00","modified_gmt":"2024-06-29T15:13:32","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=400370","title":{"rendered":"How to write simple intelligent code search with Open AI<\/span>"},"content":{"rendered":"
<\/div>\n
\n
\n
\n
\n

In this article, we will briefly review a technology that underlies ChatGPT \u2014 embeddings. Also we\u2019ll write a simple intelligent search in a codebase of a project.<\/p>\n

<\/figcaption><\/figure>\n

Embedding is the process of converting words or text into a set of numbers called a numeric vector. Vectors can be compared with each other to determine how similar two texts or words are in meaning.<\/p>\n

For example, we take two numerical vectors (embeddings) of the words «donate» and «give». The words are different but the meaning is similar, i.e. they are interconnected, and the result of both will be giving something to someone.<\/p>\n

To get a similar result in the context of a code, we can convert the words into embeddings and compare their similarity measures<\/a>. The value of measure will range from 0 to 1, where 1 is the maximum similarity and 0 is no correlation. Cosine similarity<\/a> can be taken as such a comparison function. Imagine that we have done all the necessary operations with embeddings and now analyze the result. We also compared two embeddings of the words «give» and «red». For the words «give» and «donate» the function returned the number 0.80, and for «give» and «red» \u2014 only 0.20. Thus, we can conclude that «give» and «donate» are closer than «give» and «red».<\/p>\n

In the codebase of a project, embeddings can be used to search in a code or documentation. For example, you can make an embedding (vector) out of the search query and then measure similarity of both to find relevant functions or classes.<\/p>\n

So, to make it you need an Open AI account and an API token. If you don’t have an account yet, you can register on the official Open AI website<\/u><\/a>. After registering and verifying your account, go to the API Keys<\/u><\/a> profile section and generate an API token.<\/p>\n

To start they give $18. That was enough to make an example for this article (below) and conduct further testing of the service.<\/p>\n

Take a TypeScript project as a codebase. I recommend taking a small one so as not to wait for generating embeddings long. Also you can use an example<\/u><\/a>. You also need Python 3+ versions and the library from Open AI. No worries if you don’t know any of these languages. The code examples below are simple and don’t require a deep understanding of both.<\/p>\n

Let’s get started. First, you need to write a code to extract various pieces of code from the project, such as functions. TypeScript provides a convenient compiler API<\/a> for working with an AST<\/a>, which simplifies the task. Install csv-stringify<\/code> library to generate CSV:<\/p>\n

$ npm install csv-stringify<\/code><\/pre>\n
Then create the file code-to-csv.js<\/code> and write the extraction of information from the code:<\/p>\n
const path = require('path'); const ts = require('typescript'); const csv = require('csv-stringify\/sync');   const cwd = process.cwd(); const configJSON = require(path.join(cwd, 'tsconfig.json')); const config = ts.parseJsonConfigFileContent(configJSON, ts.sys, cwd); const program = ts.createProgram(     config.fileNames,      config.options,      ts.createCompilerHost(config.options) ); const checker = program.getTypeChecker();  const rows = [];  const addRow = (fileName, name, code, docs = '') => rows.push({     file_name: path.relative(cwd, fileName),     name,     code,     docs });  function addFunction(fileName, node) {     const symbol = checker.getSymbolAtLocation(node.name);     if (symbol) {         const name = symbol.getName();         const docs = getDocs(symbol);         const code = node.getText();         addRow(fileName, name, code, docs);     } }  function addClass(fileName, node) {     const symbol = checker.getSymbolAtLocation(node.name);     if (symbol) {         const name = symbol.getName();         const docs = getDocs(symbol);         const code = `class ${name} {}`;         addRow(fileName, name, code, docs);         node.members.forEach(m => addClassMember(fileName, name, m));     } }  function addClassMember(fileName, className, node) {     const symbol = checker.getSymbolAtLocation(node.name);     if (symbol) {         const name = className + ':' + symbol.getName();         const docs = getDocs(symbol);         const code = node.getText();         addRow(fileName, name, code, docs);     } }  function addInterface(fileName, node) {     const symbol = checker.getSymbolAtLocation(node.name);     if (symbol) {         const name = symbol.getName();         const docs = getDocs(symbol);         const code = `interface ${name} {}`;         addRow(fileName, name, code, docs);         node.members.forEach(m => addInterfaceMember(fileName, name, m));     } }  function addInterfaceMember(fileName, interfaceName, node) {     if (!ts.isPropertySignature(node) || !ts.isMethodSignature(node)) {         return;     }     const symbol = checker.getSymbolAtLocation(node.name);     if (symbol) {         const name = interfaceName + ':' + symbol.getName();         const docs = getDocs(symbol);         const code = node.getText();         addRow(fileName, name, code, docs);     } }  function getDocs(symbol) {     return ts.displayPartsToString(symbol.getDocumentationComment(checker)); }  for (const fileName of config.fileNames) {     const sourceFile = program.getSourceFile(fileName);     const visitNode = node => {         if (ts.isFunctionDeclaration(node)) {             addFunction(fileName, node);         } else if (ts.isClassDeclaration(node)) {             addClass(fileName, node);         } else if (ts.isInterfaceDeclaration(node)) {             addInterface(fileName, node);         }         ts.forEachChild(node, visitNode);     };     ts.forEachChild(sourceFile, visitNode); }  for (const row of rows) {     row.combined = '';     if (row.docs) {         row.combined += `Code documentation: ${row.docs}; `;     }     row.combined += `Code: ${row.code}; Name: ${row.name};`; }  const output = csv.stringify(rows, {     header: true });  console.log(output);<\/code><\/pre>\n
The script collects all the fragments we need and prints a CSV table to the console. CSV table consists of the columns file_name, name, code, docs, combined<\/strong>.<\/p>\n