content: What we really need is to decentralize this process, so that organizations with smaller budgets could create their own search engines.  One possibility is to decentralize the crawling to some kind of a P2P algorithm. You can also have open data sharing between search engines (ex. dump ZIM / WARC files per domain), so every engine doesn't have to crawl every site.  I recommended starting with Voat itself. Also, I'm a PostgreSQL junkie.  I wanna build everything on top of PostgreSQL.  🐘