crawl4ai

mirror of https://github.com/unclecode/crawl4ai.git synced 2026-06-10 15:58:15 +00:00

Files

unclecode f77c0a856f fix(security): SSRF protection on all crawl/md/llm URL entry points

Reported by secsys_codex (2026-04-18): /md, /crawl, /llm endpoints
pass user URLs to crawler.arun() with no private IP validation.

- Add validate_url_destination() to utils.py with opt-out via
  CRAWL4AI_ALLOW_INTERNAL_URLS=true env var for users who need
  to crawl internal services.
- Integrate into validate_url_scheme() (covers all server.py endpoints).
- Add validation at all 4 URL entry points in api.py (handle_llm_qa,
  handle_markdown_request, create_new_task, handle_crawl_request).
- raw: URLs bypass check (inline HTML, no network fetch).
- 16 adversarial + source coverage tests added.
- secsys_codex added to SECURITY-CREDITS.md.

DO NOT PUSH until release day.

2026-04-20 09:42:43 +00:00

demo_monitor_dashboard.py

Add demo and test scripts for monitor dashboard activity

2025-10-17 22:43:06 +08:00

requirements.txt

feat(docker): implement smart browser pool with 10x memory efficiency

2025-10-17 20:38:39 +08:00

run_security_tests.py

Fix critical RCE and LFI vulnerabilities in Docker API deployment

2026-01-12 04:14:37 +00:00

test_1_basic.py

feat(docker): implement smart browser pool with 10x memory efficiency