[GH-ISSUE #10 ] [Bug] Can&#39;t visit more than ten Websites with DuckDuckGo #5

Author

Owner

@NXTler commented on GitHub (Mar 7, 2025):

Tinkered a bit with the WebSearch Selenium code and could confirm that this is due to DuckDuckGo only giving 10 results initially.
I was able to work around this issue by enabling autoloading in the link parameters and then scroll to the bottom with some waiting time.

Here is how the modified link looks now: https://duckduckgo.com/?kl=wt-wt&kp=-2&kav=1&kf=-1&kac=-1&k1=-1&kv=n&q=

kav=-1 enables Auto-load Results
kf=-1 turns off Favicons, resulting in faster loading speed
kac=-1 turns off Auto-suggestions, they are not needed so why not disable them
k1=-1 disables Advertisements, probably helps with getting higher quality results
kv=-1 disables page numbers, also not needed

Here is the scrolling and waiting code I inserted after await driver.wait(until.elementLocated(By.id('web_content_wrapper')), config.TIMEOUT);:

        await driver.executeScript('window.scrollTo(0, document.body.scrollHeight)');
        await driver.sleep(1000);

This is just a quick and dirty fix to the issue, we probably need to introduce a max links parameter and repeatedly scroll until the desired amount is reached. Though, we would need to adjust how a timeout is detected.

@NXTler commented on GitHub (Mar 7, 2025): Tinkered a bit with the WebSearch Selenium code and could confirm that this is due to DuckDuckGo only giving 10 results initially. I was able to work around this issue by enabling autoloading in the link parameters and then scroll to the bottom with some waiting time. Here is how the modified link looks now: `https://duckduckgo.com/?kl=wt-wt&kp=-2&kav=1&kf=-1&kac=-1&k1=-1&kv=n&q=` - `kav=-1` enables Auto-load Results - `kf=-1` turns off Favicons, resulting in faster loading speed - `kac=-1` turns off Auto-suggestions, they are not needed so why not disable them - `k1=-1` disables Advertisements, probably helps with getting higher quality results - `kv=-1` disables page numbers, also not needed Here is the scrolling and waiting code I inserted after `await driver.wait(until.elementLocated(By.id('web_content_wrapper')), config.TIMEOUT);`: ``` await driver.executeScript('window.scrollTo(0, document.body.scrollHeight)'); await driver.sleep(1000); ``` This is just a quick and dirty fix to the issue, we probably need to introduce a max links parameter and repeatedly scroll until the desired amount is reached. Though, we would need to adjust how a timeout is detected.

bsakaguchi commented

Author

Owner

@Cohee1207 commented on GitHub (Mar 7, 2025):

kf=-1 turns off Favicons, resulting in faster loading speed
kac=-1 turns off Auto-suggestions, they are not needed so why not disable them
k1=-1 disables Advertisements, probably helps with getting higher quality results

Good suggestions about those. I think I could add those even without the scrolling hack.

Additional features I can suggest too:

kbh=-1 disable page titles (unused)
ko=-1 disable page header (faster load)

@Cohee1207 commented on GitHub (Mar 7, 2025): > kf=-1 turns off Favicons, resulting in faster loading speed > kac=-1 turns off Auto-suggestions, they are not needed so why not disable them > k1=-1 disables Advertisements, probably helps with getting higher quality results Good suggestions about those. I think I could add those even without the scrolling hack. Additional features I can suggest too: * `kbh=-1` disable page titles (unused) * `ko=-1` disable page header (faster load)

bsakaguchi commented

Author

Owner

@NXTler commented on GitHub (Mar 7, 2025):

Are instant answers used? Else we could turn off headers completely with -2.

@NXTler commented on GitHub (Mar 7, 2025): Are instant answers used? Else we could turn off headers completely with -2.

bsakaguchi commented

Author

Owner

@NXTler commented on GitHub (Mar 7, 2025):

Where did you find the kbh parameter, haven't seen it documented anywhere?
We might be able to disable some more stuff, as I don't feel like spamming DuckDuckGo with useless requests.

@NXTler commented on GitHub (Mar 7, 2025): Where did you find the kbh parameter, haven't seen it documented anywhere? We might be able to disable some more stuff, as I don't feel like spamming DuckDuckGo with useless requests.

bsakaguchi commented

Author

Owner

@Cohee1207 commented on GitHub (Mar 7, 2025):

Where did you find the kbh parameter, haven't seen it documented anywhere?

You can set it via DDG preferences page.

@Cohee1207 commented on GitHub (Mar 7, 2025): > Where did you find the kbh parameter, haven't seen it documented anywhere? You can set it via DDG preferences page.

bsakaguchi commented

Author

Owner

@NXTler commented on GitHub (Mar 8, 2025):

Found a few that might be interesting:

kz=-1 disables the quick answer boxes
kat=-1 turns off location specific results
ksn=5 increases the text snipped size to 5 lines
kbg=-1 hides the AI chat Icon
kbe=0 turns off AI answers
kpsb=-1 hides the always private reminder

@NXTler commented on GitHub (Mar 8, 2025): Found a few that might be interesting: - `kz=-1` disables the quick answer boxes - `kat=-1` turns off location specific results - `ksn=5` increases the text snipped size to 5 lines - `kbg=-1` hides the AI chat Icon - `kbe=0` turns off AI answers - `kpsb=-1` hides the always private reminder

bsakaguchi commented

Author

Owner

@NXTler commented on GitHub (Mar 11, 2025):

Here is what I came up with for now:

        for (let NumberOfLinks = 0, Attempts = 0; 30 >= NumberOfLinks && 3 >= Attempts;) {
            await driver.executeScript('window.scrollTo(0, document.body.scrollHeight)');
            await driver.sleep(1000);
            const links = await driver.findElements(By.css('[data-testid="result-title-a"]'));
            if (NumberOfLinks >= links.length) {
                Attempts++
            } else {
                NumberOfLinks = links.length;
            }
        }

I'm no developer, so I don't know if this is a good/proper approach. 30 is just a placeholder right now and should be replaced with the visit count set by the extension, however I have no clue how to do that.

This is the link I'm currently using for testing, and it seems very fast:
https://duckduckgo.com/?kl=wt-wt&kp=-2&kav=1&kf=-1&kac=-1&kbh=-1&ko=-1&k1=-1&kv=n&kz=-1&kat=-1&kbg=-1&kbe=0&kpsb=-1&q=

@NXTler commented on GitHub (Mar 11, 2025): Here is what I came up with for now: ``` for (let NumberOfLinks = 0, Attempts = 0; 30 >= NumberOfLinks && 3 >= Attempts;) { await driver.executeScript('window.scrollTo(0, document.body.scrollHeight)'); await driver.sleep(1000); const links = await driver.findElements(By.css('[data-testid="result-title-a"]')); if (NumberOfLinks >= links.length) { Attempts++ } else { NumberOfLinks = links.length; } } ``` I'm no developer, so I don't know if this is a good/proper approach. `30` is just a placeholder right now and should be replaced with the visit count set by the extension, however I have no clue how to do that. This is the link I'm currently using for testing, and it seems very fast: `https://duckduckgo.com/?kl=wt-wt&kp=-2&kav=1&kf=-1&kac=-1&kbh=-1&ko=-1&k1=-1&kv=n&kz=-1&kat=-1&kbg=-1&kbe=0&kpsb=-1&q=`

bsakaguchi commented

https://github.com/SillyTavern/Extension-WebSearch/blob/main/settings.html#L172

Author

Owner

@Cohee1207 commented on GitHub (Mar 11, 2025):

We're still talking about the plugin, but the real limit is here:

@Cohee1207 commented on GitHub (Mar 11, 2025): We're still talking about the plugin, but the real limit is here: https://github.com/SillyTavern/Extension-WebSearch/blob/main/settings.html#L172

bsakaguchi commented

Author

Owner

@NXTler commented on GitHub (Mar 11, 2025):

I already saw that, but it seemed to work fine when I entered 30 in the settings.
I mean, the plugin also needs to know how much it should load, that's why I did it like this.

@NXTler commented on GitHub (Mar 11, 2025): I already saw that, but it seemed to work fine when I entered 30 in the settings. I mean, the plugin also needs to know how much it should load, that's why I did it like this.

bsakaguchi commented

Author

Owner

@NXTler commented on GitHub (Mar 12, 2025):

Worked a bit more on it, the visit count isn't limited anymore and directly controls how many links the previous code aims for. The whole thing seems fully functional. Is it okay when I do the PRs only fixing this for DuckDuckGo?

@NXTler commented on GitHub (Mar 12, 2025): Worked a bit more on it, the visit count isn't limited anymore and directly controls how many links the previous code aims for. The whole thing seems fully functional. Is it okay when I do the PRs only fixing this for DuckDuckGo?

bsakaguchi commented

Author

Owner

@Cohee1207 commented on GitHub (Mar 12, 2025):

Technically this was and still is a provider limitation. You can uncap the limit on the UI, but all the other providers except DDG via a plugin would not benefit from that.

@Cohee1207 commented on GitHub (Mar 12, 2025): Technically this was and still is a provider limitation. You can uncap the limit on the UI, but all the other providers except DDG via a plugin would not benefit from that.

bsakaguchi commented