[GH-ISSUE #296] update danbooru.csv? #207

New issue

Closed

opened 2026-10-30 21:20:58 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-30 21:20:58 +00:00

Owner

Originally created by @dopaminerush32 on GitHub (Jul 30, 2024).
Original GitHub issue: https://github.com/DominikDoom/a1111-sd-webui-tagcomplete/issues/296

Hello, I know you recently updated with 2023 data a few months ago. Would it be possible to update it again with this dataset? https://huggingface.co/datasets/isek-ai/danbooru-tags-2024. I am not sure how easy it would be to convert. Thank you in advance.

Originally created by @dopaminerush32 on GitHub (Jul 30, 2024). Original GitHub issue: https://github.com/DominikDoom/a1111-sd-webui-tagcomplete/issues/296 Hello, I know you recently updated with 2023 data a few months ago. Would it be possible to update it again with this dataset? https://huggingface.co/datasets/isek-ai/danbooru-tags-2024. I am not sure how easy it would be to convert. Thank you in advance.

bsakaguchi closed this issue

2026-10-30 21:20:58 +00:00

bsakaguchi commented

2026-10-30 21:20:59 +00:00

Author

Owner

@DominikDoom commented on GitHub (Jul 31, 2024):

Sorry, this isn't in a format I can use or convert.

The dataset you linked does not list the tags themselves with the info I need like post count, aliases, tag category etc.
Instead, it is actually a collection of posts/images with the tags for that post listed (intended for model training on these image-tag-pairs, but useless for me). Even if I were to download the full 1.47 GB of data and count & separate each tag, I would not be able to get the rest of the required data like aliases from it, as danbooru's API does not list this info for the API endpoint they used. Obviously that is also way too big compared to just a tag/alias dump.

For comparison, the dataset I used for the 2023 update was also mainly a post/image dataset like your link is, but crucially it also included this metadata folder https://huggingface.co/datasets/nyanko7/danbooru2023/tree/main/metadata in which the actual tags.tar.xz I could use was in (that comes from another API endpoint). Sadly, even though that dataset was updated to almost the same image count as the 2024 one (7.59 vs 7.66M rows), the metadata wasn't updated with it.

I could also query this data myself from the API with a bit of patience and effort, but see no reason to at the moment. The tags itself don't change that much from year to year. Maybe at the end of 2024 so the full year of changes is included, but it's too much effort for me to do randomly in the middle.

Of course, if you find another dataset that already has the current tags separate, feel free to reopen or create a new issue.

@DominikDoom commented on GitHub (Jul 31, 2024): Sorry, this isn't in a format I can use or convert. The dataset you linked does not list the tags themselves with the info I need like post count, aliases, tag category etc. Instead, it is actually a collection of posts/images with the tags for that post listed (intended for model training on these image-tag-pairs, but useless for me). Even if I were to download the full 1.47 GB of data and count & separate each tag, I would not be able to get the rest of the required data like aliases from it, as danbooru's API does not list this info for the API endpoint they used. Obviously that is also way too big compared to just a tag/alias dump. For comparison, the dataset I used for the 2023 update was also mainly a post/image dataset like your link is, but crucially it also included this metadata folder https://huggingface.co/datasets/nyanko7/danbooru2023/tree/main/metadata in which the actual tags.tar.xz I could use was in (that comes from another API endpoint). Sadly, even though that dataset was updated to almost the same image count as the 2024 one (7.59 vs 7.66M rows), the metadata wasn't updated with it. I could also query this data myself from the API with a bit of patience and effort, but see no reason to at the moment. The tags itself don't change that much from year to year. Maybe at the end of 2024 so the full year of changes is included, but it's too much effort for me to do randomly in the middle. Of course, if you find another dataset that already has the current tags separate, feel free to reopen or create a new issue.

bsakaguchi commented

2026-10-30 21:20:59 +00:00

Author

Owner

@dopaminerush32 commented on GitHub (Aug 8, 2024):

Thank you for your efforts

@dopaminerush32 commented on GitHub (Aug 8, 2024): Thank you for your efforts