Skip to content

fix: allow single-label hostnames in deep crawling - #2210

Open
rozin-not-found wants to merge 2 commits into
unclecode:mainfrom
rozin-not-found:fix-issue-2079
Open

fix: allow single-label hostnames in deep crawling#2210
rozin-not-found wants to merge 2 commits into
unclecode:mainfrom
rozin-not-found:fix-issue-2079

Conversation

@rozin-not-found

Copy link
Copy Markdown

Summary

Fixes #2079.

This PR updates the URL validation logic in the deep crawling strategies to accept valid single-label hostnames. Previously, can_process_url() strictly required a period (.) in the network location (netloc). This incorrectly rejected local development servers (e.g., http://localhost:8080) and internal Docker services. This strict check has been removed while maintaining the essential scheme and netloc presence validations.

List of files changed and why

  • crawl4ai/deep_crawling/bfs_strategy.py: Removed the "." not in parsed.netloc check to allow dotless domains.
  • crawl4ai/deep_crawling/bff_strategy.py: Removed the identical dot requirement for consistency across strategies.

How Has This Been Tested?

  • Created a local reproduction script to test can_process_url() against single-label hostnames like http://localhost:8080.
  • Verified the methods now accept these URLs without raising a ValueError, while still properly rejecting completely invalid formats.

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added/updated unit tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: BFSDeepCrawlStrategy.can_process_url() rejects valid single-label hostnames (netloc without a dot)

1 participant