[GH-ISSUE #9 ] [Feature] Regex for website scraping #4

Author

Owner

@NXTler commented on GitHub (Mar 3, 2025):

In case someone is searching for a quick and dirty solution - you can replace at const text line 708 in index.js with something like

const rawtext = await extractTextFromHTML(data, 'p'); // Only extract text from <p> tags
const text = rawtext
 .replace(/\n\n\n\n+/g, '\n\n\n') //removes excessive new lines
 .replace(/\s\s+/g, ' ') //removes double spaces
 .replace(/([\u2700-\u27BF]|[\uE000-\uF8FF]|\uD83C[\uDC00-\uDFFF]|\uD83D[\uDC00-\uDFFF]|[\u2011-\u26FF]|\uD83E[\uDD10-\uDDFF])/g, ''); //removes emojis

This cleans up the output a bit, but I'm unsure if you actually get more information with this.

@NXTler commented on GitHub (Mar 3, 2025): In case someone is searching for a quick and dirty solution - you can replace at `const text` line 708 in index.js with something like ``` const rawtext = await extractTextFromHTML(data, 'p'); // Only extract text from <p> tags const text = rawtext .replace(/\n\n\n\n+/g, '\n\n\n') //removes excessive new lines .replace(/\s\s+/g, ' ') //removes double spaces .replace(/([\u2700-\u27BF]|[\uE000-\uF8FF]|\uD83C[\uDC00-\uDFFF]|\uD83D[\uDC00-\uDFFF]|[\u2011-\u26FF]|\uD83E[\uDD10-\uDDFF])/g, ''); //removes emojis ``` This cleans up the output a bit, but I'm unsure if you actually get more information with this.

bsakaguchi commented

github.com/SillyTavern/SillyTavern@938c8a9a36/public/scripts/utils.js (L1515)

Author

Owner

@Cohee1207 commented on GitHub (Mar 3, 2025):

There's already a method that post-processes text in core code. So if you need specific processing, add it there.

@Cohee1207 commented on GitHub (Mar 3, 2025): There's already a method that post-processes text in core code. So if you need specific processing, add it there. https://github.com/SillyTavern/SillyTavern/blob/938c8a9a364919433ac1cb0e6d82710116dabf86/public/scripts/utils.js#L1515

bsakaguchi commented

Author

Owner

@NXTler commented on GitHub (Mar 3, 2025):

Thanks, I will experiment a bit and try to get more info into the budget with this. When I have something working I might commit it.

@NXTler commented on GitHub (Mar 3, 2025): Thanks, I will experiment a bit and try to get more info into the budget with this. When I have something working I might commit it.

bsakaguchi commented

Author

Owner

@Cohee1207 commented on GitHub (Mar 3, 2025):

.replace(/\n\n\n\n+/g, '\n\n\n') //removes excessive new lines
.replace(/\s\s+/g, ' ') //removes double spaces

These look good generally.

.replace(/([\u2700-\u27BF]|[\uE000-\uF8FF]|\uD83C[\uDC00-\uDFFF]|\uD83D[\uDC00-\uDFFF]|[\u2011-\u26FF]|\uD83E[\uDD10-\uDDFF])/g, ''); //removes emojis

This is questionable. I think if you're scraping articles about emojis, losing emojis would be detrimental.

@Cohee1207 commented on GitHub (Mar 3, 2025): > .replace(/\n\n\n\n+/g, '\n\n\n') //removes excessive new lines > .replace(/\s\s+/g, ' ') //removes double spaces These look good generally. > .replace(/([\u2700-\u27BF]|[\uE000-\uF8FF]|\uD83C[\uDC00-\uDFFF]|\uD83D[\uDC00-\uDFFF]|[\u2011-\u26FF]|\uD83E[\uDD10-\uDDFF])/g, ''); //removes emojis This is questionable. I think if you're scraping articles about emojis, losing emojis would be detrimental.

bsakaguchi commented

Author

Owner

@NXTler commented on GitHub (Mar 3, 2025):

For the last one I just search how to remove emojis in js and haven't spend much time testing it, I'm also unsure if that's the correct way.

@NXTler commented on GitHub (Mar 3, 2025): For the last one I just search how to remove emojis in js and haven't spend much time testing it, I'm also unsure if that's the correct way.

bsakaguchi commented

Author

Owner

@NXTler commented on GitHub (Mar 3, 2025):

Okay, this is how it looks now:

        const rawtext = await extractTextFromHTML(data, 'p'); // Only extract text from <p> tags

        const text = rawtext
            .replace(/^\n+|\n+$/g, '')                                                // Remove leading and trailing new lines
            .replace(/\n{4,}/g, '\n\n\n')                                             // Remove excessive new lines
            .replace(/(?<!.) +(?=(?:|["'([])(?:\w+ ){2})| +(?!.)/g, '')               // Remove leading and trailing spaces
            .replace(/(?<! )  (?! )/g, ' ')                                           // Remove double spaces
            .replace(/(?<= \w+) +(?=[.,?!:;])/g, '')                                  // Remove leading spaces from Punctiation
            .replace(/(?<= \w+)[.,!?:;](?=\w+(?: |, ))/g, '$& ')                      // Add missing spaces between sentences
            .replace(/(\p{Emoji_Presentation}|\p{Extended_Pictographic})(?=\1)/g, '') // Remove spammed emojis
            ;

This really cleans up the output without changing the actual information that much. Emojis are still a bit weird, as they sometimes have or not have leading/trailing spaces.

EDIT: fixed some issues with the replacements, should work in more cases now while preserving more information.
EDIT: Further improved leading space detection

@NXTler commented on GitHub (Mar 3, 2025): Okay, this is how it looks now: ``` const rawtext = await extractTextFromHTML(data, 'p'); // Only extract text from <p> tags const text = rawtext .replace(/^\n+|\n+$/g, '') // Remove leading and trailing new lines .replace(/\n{4,}/g, '\n\n\n') // Remove excessive new lines .replace(/(?<!.) +(?=(?:|["'([])(?:\w+ ){2})| +(?!.)/g, '') // Remove leading and trailing spaces .replace(/(?<! ) (?! )/g, ' ') // Remove double spaces .replace(/(?<= \w+) +(?=[.,?!:;])/g, '') // Remove leading spaces from Punctiation .replace(/(?<= \w+)[.,!?:;](?=\w+(?: |, ))/g, '$& ') // Add missing spaces between sentences .replace(/(\p{Emoji_Presentation}|\p{Extended_Pictographic})(?=\1)/g, '') // Remove spammed emojis ; ``` This really cleans up the output without changing the actual information that much. Emojis are still a bit weird, as they sometimes have or not have leading/trailing spaces. EDIT: fixed some issues with the replacements, should work in more cases now while preserving more information. EDIT: Further improved leading space detection

bsakaguchi commented

Author

Owner

@NXTler commented on GitHub (Mar 3, 2025):

Seems to work correctly with the character budget.

@NXTler commented on GitHub (Mar 3, 2025): Seems to work correctly with the character budget.

bsakaguchi commented

Author

Owner

@NXTler commented on GitHub (Mar 13, 2025):

Okay, after some edits, I would call these expressions good. They differ quite a lot from the ones in postProcessText, so how should go about adding these? Or should I rather create a new function for this?

@NXTler commented on GitHub (Mar 13, 2025): Okay, after some edits, I would call these expressions good. They differ quite a lot from the ones in `postProcessText`, so how should go about adding these? Or should I rather create a new function for this?

bsakaguchi commented

Author

Owner

@Cohee1207 commented on GitHub (Mar 13, 2025):

I'm not sure about a new function, especially around removing emojis. The rest of the regex look fine.

@Cohee1207 commented on GitHub (Mar 13, 2025): I'm not sure about a new function, especially around removing emojis. The rest of the regex look fine.

bsakaguchi commented

Author

Owner

@NXTler commented on GitHub (Mar 13, 2025):

It doesn't remove emojis completely, it simply collapses multiple into one to handle unnecessary spamming from some websites.

@NXTler commented on GitHub (Mar 13, 2025): It doesn't remove emojis completely, it simply collapses multiple into one to handle unnecessary spamming from some websites.

bsakaguchi commented

Author

Owner

@Cohee1207 commented on GitHub (Mar 13, 2025):

You mean if the same emoji is repeated multiple times? You can also target the emoji with a unicode group instead. This example will match all emojis in a string:

/(\p{Emoji_Presentation}|\p{Extended_Pictographic})/gu

@Cohee1207 commented on GitHub (Mar 13, 2025): You mean if the same emoji is repeated multiple times? You can also target the emoji with a unicode group instead. This example will match all emojis in a string: `/(\p{Emoji_Presentation}|\p{Extended_Pictographic})/gu`

bsakaguchi commented

Author

Owner

@NXTler commented on GitHub (Mar 13, 2025):

Yes, that's what I meant. If one emoji is repeated multiple times, it will be reduced to only one instance.
I wasn't aware of that functionality. In that case, the regex would look like this:
.replace(/(\p{Emoji_Presentation}|\p{Extended_Pictographic})(?=\1)/gu, '') // Remove spammed emojis

So, how should I add these to postProcessText? I have some different approaches for some Regexs (see removing double spaces and leading and trailing spaces) to retain more information.

@NXTler commented on GitHub (Mar 13, 2025): Yes, that's what I meant. If one emoji is repeated multiple times, it will be reduced to only one instance. I wasn't aware of that functionality. In that case, the regex would look like this: `.replace(/(\p{Emoji_Presentation}|\p{Extended_Pictographic})(?=\1)/gu, '') // Remove spammed emojis` So, how should I add these to `postProcessText`? I have some different approaches for some Regexs (see removing double spaces and leading and trailing spaces) to retain more information.

bsakaguchi commented