Repository navigation
ThrottlingRequestManager has no effect behind RequestManagerTandem #2238
Description
Activity
The bug generalizes past the three call sites: every
isinstance(self._request_manager, ThrottlingRequestManager)check breaks the moment the manager gets wrapped, so to_tandem() silently disables throttling, the robots warning, and 429 recording in one stroke. The durable fix is to stop asking for the concrete class and expose the capability on the protocol — athrottlerproperty that unwraps through any wrapper — so decoration layers stay transparent:@Property
def throttler(self): return self._inner.throttler # Tandem delegates inwardI would pair that with a mechanical eval gate: a regression test that builds every documented manager combination and asserts throttling stays effective through each one. Want me to take a crack at it? Happy to open a PR with working code.
BasicCrawlerfinds the throttler with a bareisinstance(self._request_manager, ThrottlingRequestManager)check in three places: therespect_robots_txt_filestartup warning (_basic_crawler.py:708), the 429 recording (:1644), and the robots.txt crawl-delay (:1710).RequestLoader.to_tandem()wraps the manager in aRequestManagerTandem, soawait loader.to_tandem(throttler)silently drops the per-domain backoff and the crawl-delay. Both warnings then say the crawler is not usingThrottlingRequestManager, while one is configured.RequestManagerTandemexposes no accessor for the manager it wraps, so a fix needs new public API. #2158 left this out of scope on purpose.✍️ Drafted by Claude Code