Web access — fetch / search / traverse / news / rss

The web-access tool namespace — HTTP fetch with full method/header control, DuckDuckGo web and news search, breadth-first link traversal, and RSS/Atom feeds.

The web-access tools cover everything from a single HTTP call to a bounded crawl. They’re the right reach for WHOIS/abuse-contact lookups during incident work, vendor-page checks, and general research.

fetch_url

A full HTTP client, not just a GET — set method, headers, query params, JSON body, and HTML parsing.

fetch_url(url, method="GET", headers=None, params=None, json_data=None,
          parse_html=False, max_bytes=None)
ParamNotes
methodGET (default), POST, etc.
headers / paramsRequest headers and query params
json_dataJSON request body for POST/PUT
parse_htmlTrue returns parsed/cleaned HTML rather than raw bytes
max_bytesCap the response size
web_search(query, max_results=5)

DuckDuckGo web search — the general “look this up” tool.

search_news

search_news(query, max_results=10)

Recent news articles specifically, rather than general web results.

web_traverse

Breadth-first crawl from a starting URL, with guardrails so it stays bounded.

web_traverse(start_url, max_depth=1, max_pages=5, max_links_per_page=30,
             same_domain=True, delay_seconds=0)

same_domain=True (default) keeps it on one site; max_depth / max_pages / max_links_per_page bound the crawl; delay_seconds throttles politely. Use this instead of hand-looping fetch_url when you need to follow links.

fetch_rss

fetch_rss(feed_url, max_items=10)

Fetch and parse an RSS or Atom feed.