Raw LLM Responses

the token burn on garbage HTML is the part that really hurts. the actual scraping failure is one thing but feeding that noise into an LLM that happily processes it token by token is where the financial damage happens. two things that would have saved you here: 1. circuit breaker on your agent loop. set a max token budget per URL and a max retry count. if the extracted text has more HTML tags than actual words, bail out instead of re-trying. a simple heuristic like checking the ratio of angle brackets to alphanumeric characters catches 90% of captcha/garbage pages before they hit the LLM. 2. pre-filtering before the LLM. tools like trafilatura or readability-lx can extract clean text from HTML without any AI involvement. run that first, check if you got meaningful content, THEN send to the LLM for structured extraction. cuts your token costs by like 80% on pages that do render correctly too. the build it yourself trap is absolutely real. i went down the same road and eventually landed on firecrawl for most things. not sponsored or anything, just the pain of maintaining custom scrapers vs paying 20 dollars a month for something that handles the edge cases is not even close anymore.