Command for fetching Hacker News threads from the search API

[simonw]

I want to be able to fetch every item for a domain, e.g. https://news.ycombinator.com/from?site=simonwillison.net

[simonw]

That page doesn't have an API but does look easy to scrape.

The other option here is the HN Search API powered by Algolia, documented at https://hn.algolia.com/api

[simonw]

https://hn.algolia.com/api/v1/search_by_date?query=simonwillison.net&restrictSearchableAttributes=url looks like it does what I want.

https://hn.algolia.com/api/v1/search_by_date?query=simonwillison.net&restrictSearchableAttributes=url&hitsPerPage=1000 - returns 1000 at once.

Otherwise you have to paginate using &page=2 etc - up to nbPages pages.

https://www.algolia.com/doc/api-reference/api-parameters/hitsPerPage/ says 1000 is the maximum.

[simonw]

Prototype:

curl "https://hn.algolia.com/api/v1/search_by_date?query=simonwillison.net&restrictSearchableAttributes=url&hitsPerPage=1000" | \
  jq .hits | sqlite-utils insert hn.db items - --pk objectID --alter

[simonw]

I'm going to build a general-purpose hacker-new-to-sqlite search ... command, where one of the options is to search within the URL.