UncleCode f78c46446b feat(deep-crawling): improve URL normalization and domain filtering ...

Enhance URL handling in deep crawling with:
- New URL normalization functions for consistent URL formats
- Improved domain filtering with subdomain support
- Added URLPatternFilter to public API
- Better URL deduplication in BFS strategy

These changes improve crawling accuracy and reduce duplicate visits.

2025-03-06 22:45:57 +08:00

..

assets

feat(core): release version 0.5.0 with deep crawling and CLI

2025-02-21 19:55:02 +08:00

deprecated

docs: update README badges and Docker section, reorganize documentation structure

2024-12-31 19:45:02 +08:00

examples

Merge branch 'vr0.5.0.post1' into next

2025-03-05 14:17:19 +08:00

md_v2

refactor(llm): rename LlmConfig to LLMConfig for consistency

2025-03-05 14:17:04 +08:00

releases_review

feat(proxy): add proxy rotation strategy

2025-02-09 18:49:10 +08:00

snippets/deep_crawl

feat(deep-crawling): improve URL normalization and domain filtering

2025-03-06 22:45:57 +08:00