mirror of
https://github.com/unclecode/crawl4ai.git
synced 2026-06-12 00:38:00 +00:00
85847ff13f
1. Make active_crawls into a dict instead of set and remove jobs array. Effective lookup and storage of active crawls and crawl control. 2. Put a lock on active_crawls, so similtanious push and pop by coroutines doesn't cause a race condition 3. Move the depth check logic outside the child link for loop, as source_url doesn't change in the loop.