Out of curiosity I have no idea how to approach this. Are there indexes of websites available to the public that you are suggesting that can be searched via grep or some simple scripting?
Or are you suggesting writing a web crawler to build our own index?
Or are you suggesting finding curated links to sites?
I still have a paperback book that claimed to be a comprehensive, not curated list of internet sites. It didn't even come with electronic media as became customary with computer books later on.
There was a post a month or two ago that talked about the design of the Yahoo homepage back then, and looking at the link again, it's striking how much more pleasurable an experience it is.
i might be wrong but i didn't think OP was suggesting to re-index the web. that's a totally different beast from finding an answer to a question/problem or learning a new topic.
depending on what i am looking for there are a few avenues i would explore if google wasn't available. github, gitlab, & stackoverflow/stackexchange for code related stuff.
wikipedia for general topics. youtube for how-tos. twitter for the news. newsletters and podcasts for links to new articles.
