From 5603f9e004b41cb27342388a4e95a3c20d580e68 Mon Sep 17 00:00:00 2001 From: martintomov <101264514+martintomov@users.noreply.github.com> Date: Mon, 19 Aug 2024 00:43:45 +0300 Subject: [PATCH 1/4] issue #76, and RunPod cloud training setup #90 (#80) * issue #76, load_checkpoint_and_dispatch() 'force_hooks' https://github.com/ostris/ai-toolkit/issues/76 * RunPod cloud config https://github.com/ostris/ai-toolkit/issues/90 * change 2x A40 to 1x A40 and price per hour referring to https://github.com/ostris/ai-toolkit/issues/90#issuecomment-2294894929 * include missed FLUX.1-schnell setup guide in last commit * huggingface-cli login required auth --- README.md | 43 +++++++++++++++++++++++++++++++++++++++++++ requirements.txt | 3 ++- 2 files changed, 45 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 45779dff..9c2e9e70 100644 --- a/README.md +++ b/README.md @@ -115,8 +115,51 @@ Please do not open a bug report unless it is a bug in the code. You are welcome and ask for help there. However, please refrain from PMing me directly with general question or support. Ask in the discord and I will answer when I can. +### Training in RunPod cloud +Example RunPod template: **runpod/pytorch:2.2.0-py3.10-cuda12.1.1-devel-ubuntu22.04** +> You need a minimum of 24GB VRAM, pick a GPU by your preference. + +#### Example config ($0.5/hr): +- 1x A40 (48 GB VRAM) +- 19 vCPU 100 GB RAM + +#### Custom overrides (you need some storage to clone FLUX.1, store datasets, store trained models and samples): +- ~120 GB Disk +- ~120 GB Pod Volume +- Start Jupyter Notebook + +### 1. Setup +``` +git clone https://github.com/ostris/ai-toolkit.git +cd ai-toolkit +git submodule update --init --recursive +python -m venv venv +source venv/bin/activate +pip install torch +pip install -r requirements.txt +pip install --upgrade accelerate transformers diffusers huggingface_hub #Optional, run it if you run into issues +``` +### 2. Upload your dataset +- Create a new folder in the root, name it `dataset` or whatever you like +- Drag and drop your .jpg and .txt files inside the newly created dataset folder + +### 3. Login into Hugging Face with an Access Token +- Get a READ token from [here](https://huggingface.co/settings/tokens) +- Run ```huggingface-cli login``` and paste your token + +### 4. Training +- Copy an example config file located at ```config/examples``` to the config folder and rename it to ```whatever_you_want.yml``` +- Edit the config following the comments in the file +- Change ```folder_path: "/path/to/images/folder"``` to your dataset path like ```folder_path: "/workspace/ai-toolkit/your-dataset"``` +- Run the file: ```python run.py config/whatever_you_want.yml``` + +### Screenshot from RunPod +RunPod Training Screenshot + + --- diff --git a/requirements.txt b/requirements.txt index 23781266..119eeee0 100644 --- a/requirements.txt +++ b/requirements.txt @@ -27,4 +27,5 @@ hf_transfer lpips pytorch_fid optimum-quanto -sentencepiece \ No newline at end of file +sentencepiece +huggingface_hub From dc6f36cd826875b75043a6b509125eff5526fcfe Mon Sep 17 00:00:00 2001 From: Jaret Burkett Date: Sun, 18 Aug 2024 16:09:52 -0600 Subject: [PATCH 2/4] Testing github bug reporting stuff --- .github/ISSUE_TEMPLATE/before_reporting.md | 11 +++++++++++ .github/ISSUE_TEMPLATE/bug_report.md | 22 ++++++++++++++++++++++ .github/ISSUE_TEMPLATE/config.yml | 5 +++++ 3 files changed, 38 insertions(+) create mode 100644 .github/ISSUE_TEMPLATE/before_reporting.md create mode 100644 .github/ISSUE_TEMPLATE/bug_report.md create mode 100644 .github/ISSUE_TEMPLATE/config.yml diff --git a/.github/ISSUE_TEMPLATE/before_reporting.md b/.github/ISSUE_TEMPLATE/before_reporting.md new file mode 100644 index 00000000..15ae6416 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/before_reporting.md @@ -0,0 +1,11 @@ +--- +name: Before Reporting an Issue +about: Read this before submitting an issue +title: '' +labels: '' +assignees: '' +--- + +# BEFORE REPORTING AN ISSUE! + +Reporting an "issue" is only for bugs. You must first verify it is an actual bug by asking [in the discord](https://discord.com/invite/nuR9zZ2nsh). If you have a feature request or question, [ask in the discord](https://discord.com/invite/nuR9zZ2ns). Github issues is ONLY for verified bugs in the code. There are many members happy to help you out [in the discord](https://discord.com/invite/nuR9zZ2ns), so please check there first. diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md new file mode 100644 index 00000000..d0b968c5 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -0,0 +1,22 @@ +name: Bug Report +description: File a bug report +title: "[Bug]: " +labels: + - bug +body: + - type: markdown + attributes: + value: > + Thanks for taking the time to fill out this bug report! + + Before submitting, please make sure you've read our [important information about reporting issues](link-to-before_reporting.md). + - type: checkboxes + id: terms + attributes: + label: Preflight Checklist + description: Please ensure you've completed these before submitting + options: + - label: I have verified this is an actual bug by asking in the Discord + required: true + - label: I understand that GitHub issues are ONLY for verified bugs in the code + required: true \ No newline at end of file diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 00000000..24cabf10 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,5 @@ +blank_issues_enabled: false +contact_links: + - name: Read Before Reporting an Issue + url: https://github.com/ostris/ai-toolkit/blob/main/.github/ISSUE_TEMPLATE/before_reporting.md + about: Please read this important information before submitting an issue. \ No newline at end of file From 6e7d721382fd32a36946b00e683fe29ade20c982 Mon Sep 17 00:00:00 2001 From: Jaret Burkett Date: Sun, 18 Aug 2024 16:20:08 -0600 Subject: [PATCH 3/4] More issues testing --- .github/ISSUE_TEMPLATE/before_reporting.md | 11 ------ .github/ISSUE_TEMPLATE/bug_report.md | 40 ++++++++++------------ .github/ISSUE_TEMPLATE/config.yml | 6 ++-- 3 files changed, 22 insertions(+), 35 deletions(-) delete mode 100644 .github/ISSUE_TEMPLATE/before_reporting.md diff --git a/.github/ISSUE_TEMPLATE/before_reporting.md b/.github/ISSUE_TEMPLATE/before_reporting.md deleted file mode 100644 index 15ae6416..00000000 --- a/.github/ISSUE_TEMPLATE/before_reporting.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -name: Before Reporting an Issue -about: Read this before submitting an issue -title: '' -labels: '' -assignees: '' ---- - -# BEFORE REPORTING AN ISSUE! - -Reporting an "issue" is only for bugs. You must first verify it is an actual bug by asking [in the discord](https://discord.com/invite/nuR9zZ2nsh). If you have a feature request or question, [ask in the discord](https://discord.com/invite/nuR9zZ2ns). Github issues is ONLY for verified bugs in the code. There are many members happy to help you out [in the discord](https://discord.com/invite/nuR9zZ2ns), so please check there first. diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md index d0b968c5..0c5f8e65 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.md +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -1,22 +1,20 @@ +--- name: Bug Report -description: File a bug report -title: "[Bug]: " -labels: - - bug -body: - - type: markdown - attributes: - value: > - Thanks for taking the time to fill out this bug report! - - Before submitting, please make sure you've read our [important information about reporting issues](link-to-before_reporting.md). - - type: checkboxes - id: terms - attributes: - label: Preflight Checklist - description: Please ensure you've completed these before submitting - options: - - label: I have verified this is an actual bug by asking in the Discord - required: true - - label: I understand that GitHub issues are ONLY for verified bugs in the code - required: true \ No newline at end of file +about: For bugs only +title: '' +labels: '' +assignees: '' +--- + +## This is for bugs only + +Did you already ask [in the discord](https://discord.com/invite/nuR9zZ2nsh)? + +Yes/No + +You verified that this is a bug and not a feature request or question by asking [in the discord](https://discord.com/invite/nuR9zZ2nsh)? + +Yes/No + +## Describe the bug + diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml index 24cabf10..63453d51 100644 --- a/.github/ISSUE_TEMPLATE/config.yml +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -1,5 +1,5 @@ blank_issues_enabled: false contact_links: - - name: Read Before Reporting an Issue - url: https://github.com/ostris/ai-toolkit/blob/main/.github/ISSUE_TEMPLATE/before_reporting.md - about: Please read this important information before submitting an issue. \ No newline at end of file + - name: Ask in the Discord BEFORE opening an issue + url: https://discord.com/invite/nuR9zZ2nsh + about: Please ask in the discord before opening a github issue. \ No newline at end of file From 169dbd22ba3e3d6f1d8ed63c14b4ef3d8d1ab305 Mon Sep 17 00:00:00 2001 From: Jaret Burkett Date: Sun, 18 Aug 2024 16:21:48 -0600 Subject: [PATCH 4/4] Finaized bug reports --- .github/ISSUE_TEMPLATE/bug_report.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md index 0c5f8e65..8127969a 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.md +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -1,6 +1,6 @@ --- name: Bug Report -about: For bugs only +about: For bugs only. Not for feature requests or questions. title: '' labels: '' assignees: ''