Trying to figure out why a thrown error is not caught

128
February 01, 2020, at 09:40 AM

Here's the complete code I'm trying to run. I will, however, censor the url of the website for personal reasons. I am trying to scrap titles from a very slow website which occasionally sends error status codes in the 4xx range because that, so to handle that I throw an error then retry fetching the same pages after a couple of seconds. The problem is that this error is never caught by the "catch" block. Any idea what I am doing wrong?

const URL = "https://webpage.com/page=";
const SELECTOR = ".post-title.entry-title>a";
const MAX_CONCURRENT_FETCH = 5;
const NB_OF_PAGES = 125;
const ERROR_WAIT_TIME = 20000;
const titles = [];
const parser = new DOMParser();

function fetchPages() {
  helper(1, MAX_CONCURRENT_FETCH);
  function helper(first, last) {
    const requests = [];
    console.log("\n" + "*".repeat(40));
    for (let i = first; i <= last; i++) {
      requests.push(fetch(URL + i));
      console.log(`Fetching page: ${i}`);
    }
    console.log("*".repeat(40) + "\n");
    try {
      Promise.all(requests).then(responses => {
        responses.forEach(async(response, i) => {
          /* if no code errors, parse the page, extract the titles then add them to the "titles" array */
          if (response.ok) {
            const htmlPage = await response.text();
            console.log("\n" + "*".repeat(40));
            console.log(`Extracting titles from page: ${first+i}`);
            console.log("*".repeat(40) + "\n");
            const htmlObject = parser.parseFromString(htmlPage, "text/html");
            htmlObject.querySelectorAll(SELECTOR).forEach(node => {
              titles.push(node.textContent);
              console.log(`Title: ${node.textContent}`);
            });
          }
          /* code error, get out of the forEach method by throwing an error */
          else {
            console.log("*".repeat(40) + "\n");
            console.log("Throwing error...");
            throw {
              response: response
            }
          }
        });
        /* keep fetching until the last page */
        if (last + 1 < NB_OF_PAGES)
          helper(last + 1, last + MAX_CONCURRENT_FETCH);
        /*  once all pages have been fetched, show the result on screen */
        else showResult();
      });
    }
    /* catch the error that was throw inside the forEach method, show the status code error and how long before the pages will be fetched again */
    catch (err) {
      console.log("Error captured...");
      console.log(`Status Code: ${err.response.status}, retrying in ${ERROR_WAIT_TIME/1000} seconds.`);
      console.log("*".repeat(40) + "\n");
      setTimeout(() => helper(first, last), ERROR_WAIT_TIME);
    }
  }
}
function showResult() {
  const uniqueTitles = [...new Set(titles)].sort();
  const titlesUl = document.createElement("ul");
  uniqueTitles.forEach(title => {
    const titleLi = document.createElement("li");
    titleLi.textContent = title;
    titlesUl.appendChild(titleLi);
  });
  document.body.innerHTML = "";
  document.body.appendChild(titlesUl);
}
fetchPages();
Answer 1

The try...catch statement contains Promises.all(). This is a non-blocking asynchronous function. The try...catch only applies to initializing the Promises.all() function, it does not catch resolved Promises.

Also note that you are sending all fetch requests in parallel, this may overload the server somewhat. It would be more polite to fire requests one by one using async await.

Rent Charter Buses Company
READ ALSO
how to test window.scroll in jest

how to test window.scroll in jest

I am calling $('all-products-tab-buttons

150
Installing dependencies in React

Installing dependencies in React

If I downloaded a dependency using "npm install pkgname --save" but never used it will it slow my speed on production build react?

143
Element sometimes appears and sometimes does not, how to continue script either way?

Element sometimes appears and sometimes does not, how to continue script either way?

My selenium (python) script is doing some work on a website that is a dashboard Ie the actual webpage/link does not change as I interact with elements on this dashboard

51
Behance application not exist

Behance application not exist

I want to get my user's products and show in websiteBut i can't get behance api

49