[PR #60] [MERGED] sorts messages in ChromaDB export by 'date' order #171

New issue

Closed

opened 2026-10-15 18:05:17 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-15 18:05:17 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/SillyTavern/SillyTavern-Extras/pull/60
Author: @johnbenac
Created: 6/22/2023
Status: ✅ Merged
Merged: 6/22/2023
Merged by: @Cohee1207

Base: main ← Head: main

📝 Commits (1)

add7daa sorts messages in ChromaDB export by 'date' order

📊 Changes

1 file changed (+5 additions, -4 deletions)

View changed files

📝 server.py (+5 -4)

📄 Description

Added a line that sorts the exported chromaDB export by the order in which the conversation happened

This is my first pull request, on any project, ever. Apologies for any mistakes.

From the Discord:


When it exports the ChromaDB from the SmartContext in extras, it would be nice if the exported file had the messages in the order in which they were made. Right now, I'm getting this order for a chromadb that I exported:

msg-0, msg-21, msg-2, msg-3, msg-4, msg-5, msg-6, msg-1, msg-8, msg-9, msg-10, msg-11, msg-12, msg-13, msg-14, msg-15, msg-16, msg-17, msg-18, msg-19, msg-20, msg-26, msg-22, msg-23, msg-24, msg-25, msg-7


I dont know if that's determined by the ChromaDB codebase, or if it could just be a small tweak to the SillyTavern codebase. 

I'm wanting to experiment with transforming (with AI) chromadb databased and sideloading them into conversations with different characters to created a "shared universe" across group chats. Having a slightly more interpretable chromaDB export file would help. In the meantime, I may (have the AI) write a little script to reorder them chronologically, but if you all could make a small tweak to make such a script unnecessary, that would be nice.
JoJorge — Yesterday at 11:34 PM
Here is the script. It works, and the output chromaDB looks a little more like a transcript. 

Even if the wonky order is part of the ChromaDB code, you might use somethign like this to process the exporting file before sending it to the user.

import json
import sys

def sort_chat_data(file_path):
    with open(file_path, 'r') as file:
        data = json.load(file)
        content = data['content']
        sorted_content = sorted(content, key=lambda x: x['metadata']['date'])

    new_data = data.copy()
    new_data['content'] = sorted_content

    new_file_path = file_path.replace('.json', '-sorted.json')
    with open(new_file_path, 'w') as sorted_file:
        json.dump(new_data, sorted_file, indent=2)

if __name__ == "__main__":
    if len(sys.argv) != 2:
        raise ValueError("Please provide the JSON file path as the only argument.")
    sort_chat_data(sys.argv[1])
Attachment file type: code
[sort_chromadb_json.py](https://cdn.discordapp.com/attachments/1101440561442992229/1121282160196866048/sort_chromadb_json.py)
660 bytes
RossAscends (SillyTavern UI Dev) — Yesterday at 11:50 PM
drop into PR
```end of discord chat

---

<sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

## 📋 Pull Request Information **Original PR:** https://github.com/SillyTavern/SillyTavern-Extras/pull/60 **Author:** [@johnbenac](https://github.com/johnbenac) **Created:** 6/22/2023 **Status:** ✅ Merged **Merged:** 6/22/2023 **Merged by:** [@Cohee1207](https://github.com/Cohee1207) **Base:** `main` ← **Head:** `main` --- ### 📝 Commits (1) - [`add7daa`](https://github.com/SillyTavern/SillyTavern-Extras/commit/add7daa61b94e66468421ee60f1be18c2b3c4b0e) sorts messages in ChromaDB export by 'date' order ### 📊 Changes **1 file changed** (+5 additions, -4 deletions) <details> <summary>View changed files</summary> 📝 `server.py` (+5 -4) </details> ### 📄 Description Added a line that sorts the exported chromaDB export by the order in which the conversation happened This is my first pull request, on any project, ever. Apologies for any mistakes. From the Discord: ```start of discord chat When it exports the ChromaDB from the SmartContext in extras, it would be nice if the exported file had the messages in the order in which they were made. Right now, I'm getting this order for a chromadb that I exported: msg-0, msg-21, msg-2, msg-3, msg-4, msg-5, msg-6, msg-1, msg-8, msg-9, msg-10, msg-11, msg-12, msg-13, msg-14, msg-15, msg-16, msg-17, msg-18, msg-19, msg-20, msg-26, msg-22, msg-23, msg-24, msg-25, msg-7 I dont know if that's determined by the ChromaDB codebase, or if it could just be a small tweak to the SillyTavern codebase. I'm wanting to experiment with transforming (with AI) chromadb databased and sideloading them into conversations with different characters to created a "shared universe" across group chats. Having a slightly more interpretable chromaDB export file would help. In the meantime, I may (have the AI) write a little script to reorder them chronologically, but if you all could make a small tweak to make such a script unnecessary, that would be nice. JoJorge — Yesterday at 11:34 PM Here is the script. It works, and the output chromaDB looks a little more like a transcript. Even if the wonky order is part of the ChromaDB code, you might use somethign like this to process the exporting file before sending it to the user. import json import sys def sort_chat_data(file_path): with open(file_path, 'r') as file: data = json.load(file) content = data['content'] sorted_content = sorted(content, key=lambda x: x['metadata']['date']) new_data = data.copy() new_data['content'] = sorted_content new_file_path = file_path.replace('.json', '-sorted.json') with open(new_file_path, 'w') as sorted_file: json.dump(new_data, sorted_file, indent=2) if __name__ == "__main__": if len(sys.argv) != 2: raise ValueError("Please provide the JSON file path as the only argument.") sort_chat_data(sys.argv[1]) Attachment file type: code [sort_chromadb_json.py](https://cdn.discordapp.com/attachments/1101440561442992229/1121282160196866048/sort_chromadb_json.py) 660 bytes RossAscends (SillyTavern UI Dev) — Yesterday at 11:50 PM drop into PR ```end of discord chat --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-15 18:05:17 +00:00