Fix parsing for real this time

Fixes #48 (again)

.gitignore

@@ -0,0 +1 @@
+tags/temp/

README.md

@@ -19,12 +19,10 @@ You can either clone / download the files manually as described [below](#install
 For example, editing `atago (azur lane)`, it would be replaced with e.g. `taihou (azur lane), lane)`, since the script currently doesn't see the second part of the bracket as the same tag. So in those cases you should delete the old tag beforehand.
 
 ### Wildcard & Embedding support
-Autocompletion also works with wildcard files used by [this script](https://github.com/jtkelm2/stable-diffusion-webui-1/blob/master/scripts/wildcards.py) of the same name (demo video further down). This enables you to either insert categories to be replaced by the script, or even replace them with the actual wildcard file content in the same step.
+Autocompletion also works with wildcard files used by [this script](https://github.com/jtkelm2/stable-diffusion-webui-1/blob/master/scripts/wildcards.py) of the same name (demo video further down). This enables you to either insert categories to be replaced by the script, or even replace them with the actual wildcard file content in the same step. Wildcards are searched for in every extension folder as well as the `scripts/wildcards` folder to support legacy versions. This means that you can combine wildcards from multiple extensions. Nested folders are also supported if you have grouped your wildcards in that way.
 
 It also scans the embeddings folder and displays completion hints for the names of all .pt and .bin files inside if you start typing `<`. Note that some normal tags also use < in Kaomoji (like ">_<" for example), so the results will contain both.
 
-Both are now enabled by default and scan the `/embeddings` and `/scripts/wildcards` folders automatically.
-
 ## Screenshots
 Demo video (with keyboard navigation):
 
@@ -40,14 +38,27 @@ Dark and Light mode supported, including tag colors:
 ![tagtypes_light](https://user-images.githubusercontent.com/34448969/195180061-ceebcc25-9e4c-424f-b0c9-ba8e8f4f17f4.png)
 
 ## Installation
-Simply copy the `javascript`, `scripts` and `tags` folder into your web UI installation root. It will run automatically the next time the web UI is started.
+### As an extension (recommended)
+Either clone the repo into your extensions folder:
+```bash
+git clone "https://github.com/DominikDoom/a1111-sd-webui-tagcomplete.git" extensions/tag-autocomplete
+```
+(The second argument specifies the name of the folder, you can choose whatever you like).
 
-The tags folder contains `config.json` and the tag data the script uses for autocompletion. By default, Danbooru and e621 tags are included.
+Or create a folder there manually and place the `javascript`, `scripts` and `tags` folders in it.
+
+### In the root folder (old)
+Copy the `javascript`, `scripts` and `tags` folder into your web UI installation root. It will run automatically the next time the web UI is started.
+
+---
+
+In both configurations, the tags folder contains `config.json` and the tag data the script uses for autocompletion. By default, Danbooru and e621 tags are included.
 After scanning for embeddings and wildcards, the script will also create a `temp` directory here which lists the found files so they can be accessed in the browser side of the script. You can delete the temp folder without consequences as it will be recreated on the next startup.
+
 ### Important:
 The script needs **all three folders** to work properly.
 
-### Config
+## Config
 The config contains the following settings and defaults:
 ```json
 {
@@ -58,8 +69,9 @@ The config contains the following settings and defaults:
 		"negativePrompts": true
 	},
 	"maxResults": 5,
-	"showAllResults": false,
 	"resultStepLength": 500,
+	"showAllResults": false,
+	"useLeftRightArrowKeys": false,
 	"replaceUnderscores": true,
 	"escapeParentheses": true,
 	"useWildcards": true,
@@ -101,6 +113,7 @@ The config contains the following settings and defaults:
 | maxResults | How many results to show max. For the default tag set, the results are ordered by occurence count. For embeddings and wildcards it will show all results in a scrollable list. |
 | resultStepLength | Allows to load results in smaller batches of the specified size for better performance in long lists or if showAllResults is true. |
 | showAllResults | If true, will ignore maxResults and show all results in a scrollable list. **Warning:** can lag your browser for long lists. |
+| useLeftRightArrowKeys | If true, left and right arrows will select the first/last result in the popup instead of moving the cursor in the textbox. |
 | replaceUnderscores | If true, undescores are replaced with spaces on clicking a tag. Might work better for some models. |
 | escapeParentheses | If true, escapes tags containing () so they don't contribute to the web UI's prompt weighting functionality. |
 | useWildcards | Used to toggle the wildcard completion functionality. |
@@ -135,7 +148,7 @@ Methods 1 & 2 can also be mixed, in which case translations in the extra file wi
 The extra files can also be used to just add new / custom tags not included in the main set, provided `onlyTranslationExtraFile` is false.
 If an extra tag doesn't match any existing tag, it will be added to the list as a new tag instead.
 
-### CSV tag data
+## CSV tag data
 The script expects a CSV file with tags saved in the following way:
 ```csv
 1girl,0

javascript/tagAutocomplete.js

@@ -92,15 +92,24 @@ function parseCSV(str) {
 
 // Load file
 function readFile(filePath) {
-    let request = new XMLHttpRequest();
-    request.open("GET", filePath, false);
-    request.send(null);
-    return request.responseText;
+    return new Promise(function (resolve, reject) {
+        let request = new XMLHttpRequest();
+        request.open("GET", filePath, true);
+        request.onload = function () {
+            var status = request.status;
+            if (status == 200) {
+                resolve(request.responseText);
+            } else {
+                reject(status);
+            }
+        };
+        request.send(null);
+    });
 }
 
 // Load CSV
-function loadCSV(path) {
-    let text = readFile(path);
+async function loadCSV(path) {
+    let text = await readFile(path);
     return parseCSV(text);
 }
 
@@ -220,6 +229,7 @@ function escapeRegExp(string) {
     return string.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); // $& means the whole matched string
 }
 
+const TAG_REGEX = /[([]([^,()[\]:| ]+)(?::(?:\d+(?:\.\d+)?|\.\d+))?[)\]]|((?:\b|<)[^,|\n\r ]*(?:>|\b)?)/g
 let hideBlocked = false;
 
 // On click, insert the tag into the prompt textbox with respect to the cursor position
@@ -255,16 +265,16 @@ function insertTextAtCursor(textArea, result, tagword) {
     let editStart = Math.max(cursorPos - tagword.length, 0);
     let editEnd = Math.min(cursorPos + tagword.length, prompt.length);
     let surrounding = prompt.substring(editStart, editEnd);
-    let match = surrounding.match(new RegExp(escapeRegExp(`${tagword}`)));
+    let match = surrounding.match(new RegExp(escapeRegExp(`${tagword}`), "i"));
     let afterInsertCursorPos = editStart + match.index + sanitizedText.length;
 
     var optionalComma = "";
     if (tagType !== "wildcardFile") {
-        optionalComma = surrounding.match(new RegExp(escapeRegExp(`${tagword},`))) !== null ? "" : ", ";
+        optionalComma = surrounding.match(new RegExp(`${escapeRegExp(tagword)}[,:]`, "i")) !== null ? "" : ", ";
     }
 
     // Replace partial tag word with new text, add comma if needed
-    let insert = surrounding.replace(tagword, sanitizedText + optionalComma);
+    let insert = surrounding.replace(match, sanitizedText + optionalComma);
 
     // Add back start
     var newPrompt = prompt.substring(0, editStart) + insert + prompt.substring(editEnd);
@@ -277,7 +287,9 @@ function insertTextAtCursor(textArea, result, tagword) {
     textArea.dispatchEvent(new Event("input", { bubbles: true }));
 
     // Update previous tags with the edited prompt to prevent re-searching the same term
-    let tags = newPrompt.match(/[^, ]+/g);
+    let tags = [...newPrompt.matchAll(TAG_REGEX)]
+        .filter(x => x[2] !== undefined ? x[2].length > 0 : x)
+        .flatMap(x => x[1] || x[2])
     previousTags = tags;
 
     // Hide results after inserting
@@ -366,13 +378,13 @@ function updateSelectionStyle(textArea, newIndex, oldIndex) {
 }
 
 var wildcardFiles = [];
-var wildcards = {};
+var wildcardExtFiles = [];
 var embeddings = [];
 var allTags = [];
 var results = [];
 var tagword = "";
 var resultCount = 0;
-function autocomplete(textArea, prompt, fixedTag = null) {
+async function autocomplete(textArea, prompt, fixedTag = null) {
     // Return if the function is deactivated in the UI
     if (!acActive) return;
 
@@ -384,7 +396,10 @@ function autocomplete(textArea, prompt, fixedTag = null) {
 
     if (fixedTag === null) {
         // Match tags with RegEx to get the last edited one
-        let tags = prompt.match(/[^, ]+/g);
+        // We also match for the weighting format (e.g. "tag:1.0") here, and flatMap it to just the tag part if it exists
+        let tags = [...prompt.matchAll(TAG_REGEX)]
+            .filter(x => x[2] !== undefined ? x[2].length > 0 : x)
+            .flatMap(x => x[1] || x[2])
         let diff = difference(tags, previousTags)
         previousTags = tags;
 
@@ -405,24 +420,37 @@ function autocomplete(textArea, prompt, fixedTag = null) {
         tagword = fixedTag;
     }
 
-    tagword = tagword.toLowerCase();
+    tagword = tagword.toLowerCase().replace(/[\n\r]/g, "");
 
-    if (acConfig.useWildcards && [...tagword.matchAll(/\b__([^,_ ]+)__([^, ]*)\b/g)].length > 0) {
+    if (acConfig.useWildcards && [...tagword.matchAll(/\b__([^, ]+)__([^, ]*)\b/g)].length > 0) {
         // Show wildcards from a file with that name
-        wcMatch = [...tagword.matchAll(/\b__([^,_ ]+)__([^, ]*)\b/g)]
+        wcMatch = [...tagword.matchAll(/\b__([^, ]+)__([^, ]*)\b/g)]
         let wcFile = wcMatch[0][1];
         let wcWord = wcMatch[0][2];
-        results = wildcards[wcFile].filter(x => (wcWord !== null) ? x.toLowerCase().includes(wcWord) : x) // Filter by tagword
+
+        var wcPair;
+
+        // Look in normal wildcard files
+        if (wcFound = wildcardFiles.find(x => x[1].toLowerCase() === wcFile))
+            wcPair = wcFound;
+        else // Look in extensions wildcard files
+            wcPair = wildcardExtFiles.find(x => x[1].toLowerCase() === wcFile);
+
+        let wildcards = (await readFile(`file/${wcPair[0]}/${wcPair[1]}.txt`)).split("\n")
+            .filter(x => x.trim().length > 0); // Remove empty lines
+
+        results = wildcards.filter(x => (wcWord !== null && wcWord.length > 0) ? x.toLowerCase().includes(wcWord) : x) // Filter by tagword
             .map(x => [wcFile + ": " + x.trim(), "wildcardTag"]); // Mark as wildcard
     } else if (acConfig.useWildcards && (tagword.startsWith("__") && !tagword.endsWith("__") || tagword === "__")) {
         // Show available wildcard files
         let tempResults = [];
         if (tagword !== "__") {
-            tempResults = wildcardFiles.filter(x => x.toLowerCase().includes(tagword.replace("__", ""))) // Filter by tagword
+            let lmb = (x) => x[1].toLowerCase().includes(tagword.replace("__", ""))
+            tempResults = wildcardFiles.filter(lmb).concat(wildcardExtFiles.filter(lmb)) // Filter by tagword
         } else {
-            tempResults = wildcardFiles;
+            tempResults = wildcardFiles.concat(wildcardExtFiles);
         }
-        results = tempResults.map(x => ["Wildcards: " + x.trim(), "wildcardFile"]); // Mark as wildcard
+        results = tempResults.map(x => ["Wildcards: " + x[1].trim(), "wildcardFile"]); // Mark as wildcard
     } else if (acConfig.useEmbeddings && tagword.match(/<[^,> ]*>?/g)) {
         // Show embeddings
         let tempResults = [];
@@ -461,11 +489,14 @@ function autocomplete(textArea, prompt, fixedTag = null) {
     addResultsToList(textArea, results, tagword, true);
 }
 
+var oldSelectedTag = null;
 function navigateInList(textArea, event) {
     // Return if the function is deactivated in the UI
     if (!acActive) return;
 
-    validKeys = ["ArrowUp", "ArrowDown", "ArrowLeft", "ArrowRight", "Enter", "Escape"];
+    validKeys = ["ArrowUp", "ArrowDown", "PageUp", "PageDown", "Home", "End", "Enter", "Tab", "Escape"];
+    if (acConfig.useLeftRightArrowKeys)
+        validKeys.push("ArrowLeft", "ArrowRight");
 
     if (!validKeys.includes(event.key)) return;
     if (!isVisible(textArea)) return
@@ -489,6 +520,26 @@ function navigateInList(textArea, event) {
                 selectedTag = (selectedTag + 1) % resultCount;
             }
             break;
+        case "PageUp":
+            if (selectedTag === null || selectedTag === 0) {
+                selectedTag = resultCount - 1;
+            } else {
+                selectedTag = (Math.max(selectedTag - 5, 0) + resultCount) % resultCount;
+            }
+            break;
+        case "PageDown":
+            if (selectedTag === null || selectedTag === resultCount - 1) {
+                selectedTag = 0;
+            } else {
+                selectedTag = Math.min(selectedTag + 5, resultCount - 1) % resultCount;
+            }
+            break;
+        case "Home":
+            selectedTag = 0;
+            break;
+        case "End":
+            selectedTag = resultCount - 1;
+            break;
         case "ArrowLeft":
             selectedTag = 0;
             break;
@@ -500,11 +551,17 @@ function navigateInList(textArea, event) {
                 insertTextAtCursor(textArea, results[selectedTag], tagword);
             }
             break;
+        case "Tab":
+            if (selectedTag === null) {
+                selectedTag = 0;
+            }
+            insertTextAtCursor(textArea, results[selectedTag], tagword);
+            break;
         case "Escape":
             hideResults(textArea);
             break;
     }
-    if (selectedTag == resultCount - 1
+    if (selectedTag === resultCount - 1
         && (event.key === "ArrowUp" || event.key === "ArrowDown" || event.key === "ArrowLeft" || event.key === "ArrowRight")) {
         addResultsToList(textArea, results, tagword, false);
     }
@@ -518,11 +575,14 @@ function navigateInList(textArea, event) {
 }
 
 var styleAdded = false;
-onUiUpdate(function () {
+onUiUpdate(async function () {
+    // Get our tag base path from the temp file
+    let tagBasePath = await readFile("file/tmp/tagAutocompletePath.txt");
+
     // Load config
     if (acConfig === null) {
         try {
-            acConfig = JSON.parse(readFile("file/tags/config.json"));
+            acConfig = JSON.parse(await readFile(`file/${tagBasePath}/config.json`));
             if (acConfig.translation.onlyShowTranslation) {
                 acConfig.translation.searchByTranslation = true; // if only show translation, enable search by translation is necessary
             }
@@ -534,14 +594,14 @@ onUiUpdate(function () {
     // Load main tags and translations
     if (allTags.length === 0) {
         try {
-            allTags = loadCSV(`file/tags/${acConfig.tagFile}`);
+            allTags = await loadCSV(`file/${tagBasePath}/${acConfig.tagFile}`);
         } catch (e) {
             console.error("Error loading tags file: " + e);
             return;
         }
         if (acConfig.extra.extraFile) {
             try {
-                extras = loadCSV(`file/tags/${acConfig.extra.extraFile}`);
+                extras = await loadCSV(`file/${tagBasePath}/${acConfig.extra.extraFile}`);
                 if (acConfig.extra.onlyTranslationExtraFile) {
                     // This works purely on index, so it's not very robust. But a lot faster.
                     for (let i = 0, n = extras.length; i < n; i++) {
@@ -570,28 +630,45 @@ onUiUpdate(function () {
     // Load wildcards
     if (wildcardFiles.length === 0 && acConfig.useWildcards) {
         try {
-            wildcardFiles = readFile("file/tags/temp/wc.txt").split("\n")
+            let wcFileArr = (await readFile(`file/${tagBasePath}/temp/wc.txt`)).split("\n");
+            let wcBasePath = wcFileArr[0].trim(); // First line should be the base path
+            wildcardFiles = wcFileArr.slice(1)
                 .filter(x => x.trim().length > 0) // Remove empty lines
-                .map(x => x.trim().replace(".txt", "")); // Remove file extension & newlines
+                .map(x => [wcBasePath, x.trim().replace(".txt", "")]); // Remove file extension & newlines
 
-            wildcardFiles.forEach(fName => {
-                try {
-                    wildcards[fName] = readFile(`file/scripts/wildcards/${fName}.txt`).split("\n")
-                        .filter(x => x.trim().length > 0); // Remove empty lines
-                } catch (e) {
-                    console.log(`Could not load wildcards for ${fName}`);
+            // To support multiple sources, we need to separate them using the provided "-----" strings
+            let wcExtFileArr = (await readFile(`file/${tagBasePath}/temp/wce.txt`)).split("\n");
+            let splitIndices = [];
+            for (let index = 0; index < wcExtFileArr.length; index++) {
+                if (wcExtFileArr[index].trim() === "-----") {
+                    splitIndices.push(index);
                 }
-            });
+            }
+            // For each group, add them to the wildcardFiles array with the base path as the first element
+            for (let i = 0; i < splitIndices.length; i++) {
+                let start = splitIndices[i - 1] || 0;
+                if (i > 0) start++; // Skip the "-----" line
+                let end = splitIndices[i];
+
+                let wcExtFile = wcExtFileArr.slice(start, end);
+                let base = wcExtFile[0].trim() + "/";
+                wcExtFile = wcExtFile.slice(1)
+                    .filter(x => x.trim().length > 0) // Remove empty lines
+                    .map(x => x.trim().replace(base, "").replace(".txt", "")); // Remove file extension & newlines;
+
+                wcExtFile = wcExtFile.map(x => [base, x]);
+                wildcardExtFiles.push(...wcExtFile);
+            }
         } catch (e) {
-            console.error("Error loading wildcardNames.txt: " + e);
+            console.error("Error loading wildcards: " + e);
         }
     }
     // Load embeddings
     if (embeddings.length === 0 && acConfig.useEmbeddings) {
         try {
-            embeddings = readFile("file/tags/temp/emb.txt").split("\n")
+            embeddings = (await readFile(`file/${tagBasePath}/temp/emb.txt`)).split("\n")
                 .filter(x => x.trim().length > 0) // Remove empty lines
-                .map(x => x.replace(".bin", "").replace(".pt", "")); // Remove file extensions
+                .map(x => x.replace(".bin", "").replace(".pt", "").replace(".png", "")); // Remove file extensions
         } catch (e) {
             console.error("Error loading embeddings.txt: " + e);
         }
@@ -636,7 +713,7 @@ onUiUpdate(function () {
             hideResults(area);
 
             // Add autocomplete event listener
-            area.addEventListener('input', debounce(() => autocomplete(area, area.value), 100));
+            area.addEventListener('input', debounce(() => autocomplete(area, area.value), acConfig.delayTime));
             // Add focusout event listener
             area.addEventListener('focusout', debounce(() => hideResults(area), 400));
             // Add up and down arrow event listener

scripts/tag_autocomplete_helper.py

@@ -2,25 +2,75 @@
 # to a temporary file to expose it to the javascript side
 
 from pathlib import Path
+from modules import scripts
+
+# Webui root path
+FILE_DIR = Path().absolute()
+
+# The extension base path
+EXT_PATH = FILE_DIR.joinpath('extensions')
+
+# Tags base path
+
+
+def get_tags_base_path():
+    script_path = Path(scripts.basedir())
+    if (script_path.is_relative_to(EXT_PATH)):
+        return script_path.joinpath('tags')
+    else:
+        return FILE_DIR.joinpath('tags')
+
+
+TAGS_PATH = get_tags_base_path()
 
 # The path to the folder containing the wildcards and embeddings
-FILE_DIR = Path().absolute()
 WILDCARD_PATH = FILE_DIR.joinpath('scripts/wildcards')
 EMB_PATH = FILE_DIR.joinpath('embeddings')
-# The path to the temporary file
-TEMP_PATH = FILE_DIR.joinpath('tags/temp')
+
+
+def find_ext_wildcard_paths():
+    """Returns the path to the extension wildcards folder"""
+    found = list(EXT_PATH.glob('*/wildcards/'))
+    return found
+
+
+# The path to the extension wildcards folder
+WILDCARD_EXT_PATHS = find_ext_wildcard_paths()
+
+# The path to the temporary files
+STATIC_TEMP_PATH = FILE_DIR.joinpath('tmp') # In the webui root, on windows it exists by default, on linux it doesn't
+TEMP_PATH = TAGS_PATH.joinpath('temp') # Extension specific temp files
 
 
 def get_wildcards():
     """Returns a list of all wildcards. Works on nested folders."""
     wildcard_files = list(WILDCARD_PATH.rglob("*.txt"))
-    resolved = [str(w.relative_to(WILDCARD_PATH)) for w in wildcard_files]
+    resolved = [w.relative_to(WILDCARD_PATH).as_posix(
+    ) for w in wildcard_files if w.name != "put wildcards here.txt"]
     return resolved
 
 
+def get_ext_wildcards():
+    """Returns a list of all extension wildcards. Works on nested folders."""
+    wildcard_files = []
+
+    for path in WILDCARD_EXT_PATHS:
+        wildcard_files.append(path.relative_to(FILE_DIR).as_posix())
+        wildcard_files.extend(p.relative_to(path).as_posix() for p in path.rglob("*.txt") if p.name != "put wildcards here.txt")
+        wildcard_files.append("-----")
+
+    return wildcard_files
+
+
 def get_embeddings():
     """Returns a list of all embeddings"""
-    return [str(e.relative_to(EMB_PATH)) for e in EMB_PATH.glob("**/*") if e.suffix in {".bin", ".pt"}]
+    return [str(e.relative_to(EMB_PATH)) for e in EMB_PATH.glob("**/*") if e.suffix in {".bin", ".pt", ".png"}]
+
+
+def write_tag_base_path():
+    """Writes the tag base path to a fixed location temporary file"""
+    with open(STATIC_TEMP_PATH.joinpath('tagAutocompletePath.txt'), 'w', encoding="utf-8") as f:
+        f.write(TAGS_PATH.relative_to(FILE_DIR).as_posix())
 
 
 def write_to_temp_file(name, data):
@@ -29,20 +79,35 @@ def write_to_temp_file(name, data):
         f.write(('\n'.join(data)))
 
 
+# Write the tag base path to a fixed location temporary file
+# to enable the javascript side to find our files regardless of extension folder name
+if not STATIC_TEMP_PATH.exists():
+    STATIC_TEMP_PATH.mkdir(exist_ok=True)
+
+write_tag_base_path()
+
 # Check if the temp path exists and create it if not
 if not TEMP_PATH.exists():
     TEMP_PATH.mkdir(parents=True, exist_ok=True)
-    # Set up files to ensure the script doesn't fail to load them
-    # even if no wildcards or embeddings are found
-    write_to_temp_file('wc.txt', [])
-    write_to_temp_file('emb.txt', [])
+
+# Set up files to ensure the script doesn't fail to load them
+# even if no wildcards or embeddings are found
+write_to_temp_file('wc.txt', [])
+write_to_temp_file('wce.txt', [])
+write_to_temp_file('emb.txt', [])
 
 # Write wildcards to wc.txt if found
 if WILDCARD_PATH.exists():
-    wildcards = get_wildcards()
+    wildcards = [WILDCARD_PATH.relative_to(FILE_DIR).as_posix()] + get_wildcards()
     if wildcards:
         write_to_temp_file('wc.txt', wildcards)
 
+# Write extension wildcards to wce.txt if found
+if WILDCARD_EXT_PATHS is not None:
+    wildcards_ext = get_ext_wildcards()
+    if wildcards_ext:
+        write_to_temp_file('wce.txt', wildcards_ext)
+
 # Write embeddings to emb.txt if found
 if EMB_PATH.exists():
     embeddings = get_embeddings()

tags/config.json

@@ -7,7 +7,9 @@
 	},
 	"maxResults": 5,
 	"resultStepLength": 500,
+	"delayTime": 100,
 	"showAllResults": false,
+	"useLeftRightArrowKeys": false,
 	"replaceUnderscores": true,
 	"escapeParentheses": true,
 	"useWildcards": true,