List of Links that aren't working

[calebpeffer]

Hi all, Caleb from @mendableai here.

We will start collecting links that have issues here so we can fix them. If you'd like to add a link to the list, please follow this format.

Link: https://news.ycombinator.com/
Endpoint: /scrape, /crawl
Issue: Tables don't seem to be parsed correctly. The tables return as HTML instead of as a formatted table.

We will fix them as quickly as possible.

[calebpeffer]

Link: https://privacy.patreon.com/policies
Endpoint: /scrape /crawl
Issue: fails all fallbacks

[rafaelsideguide]

I tested the system once and it failed, but it worked when I tried again. Non-deterministic behavior may be related to the fallback conditions?

[nickscamara]

Yep. It seems to be a bit better now. Tried a few times and even triggering most fallbacks, it was still able to scrape the content.

[rafaelsideguide]

Works with /scrape. "No pages found" for /crawl though

[calebpeffer]

Link: https://scholarship.law.uci.edu/cgi/viewcontent.cgi?article=1584&context=ucilr
Endpoint: /scrape /crawl
Issue: All content is Null

[nickscamara]

#29 fixes this.

[calebpeffer]

Link: https://docs.streamlit.io/
Endpoint: /crawl
Issue:

[nickscamara]

@calebpeffer hmm interesting. I believe we could add a dynamic routes and dynamic paths like [route]/path/path2/[dnyamic].cache in the excludes and includes to be able to filter out unwanted paths/files.

[igalmarino]

Link: https://support.greenpay.me/portal/es/kb/support
Endpoint: /crawl
Issue: It only gets the links on the header, but not the rest on the body of the page.

[igalmarino]

i do not know if im doing this right.

Do i need a new version of firecrawl-py to test these fixes.

https://pypi.org/project/firecrawl-py/

Or is there any other way to test it?

Tnanks

[nickscamara]

No need for a new version, we pushed a server change so it should just work with what you currently had.

[igalmarino]

I have been testing it and still only get 7 results (there are mora than 40 links on the page)

[nickscamara]

@igalmarino Pushing a change that might help with this... Will be up in 1 hour. Can you let me know if it works? Thank you!

[igalmarino]

It is working now. Thank you !

[calebpeffer]

Link: https://apps.dos.ny.gov/publicinquiry/
Endpoint: /crawl
Issue: this site displays with a filter and javascript, this is a nasty one.

Note: some other websites that may or may not work in the same space:

1. Alabama - https://www.sos.alabama.gov/
2. Alaska - https://www.commerce.alaska.gov/web/cbpl/Corporations.aspx
3. Arizona - https://azsos.gov/business
4. Arkansas - https://www.sos.arkansas.gov/business-commercial-services-bcs
5. California - https://www.sos.ca.gov/business-programs/business-entities
6. Colorado - https://www.sos.state.co.us/

Note: Email michael@smobi.com once complete

[nickscamara]

Yea, first url is really weird...

[calebpeffer]

Links: https://investors.snapone.com/static-files/c0d11e9f-f69b-48e2-b9a1-3cdb4992ed38
https://investors.snapone.com/static-files/365445cc-1182-4707-ab07-552c8c0a0bdf
https://investor.shutterstock.com/static-files/f5b52aee-0070-4922-acb6-9886feee789d
https://investor.shutterstock.com/static-files/79474632-2f5f-46a7-b481-e5cbd5274915

Endpoint: /scrape

returning
%PDF-1.7 %���� 1 0 obj <>/Metadata 115 0 R/ViewerPreferences 116 0 R>> endobj 2 0 obj <> endobj 3 0 obj <>/ExtGState<>/ProcSet[/PDF/Text/ImageB/ImageC/ImageI] >>/MediaBox[ 0 0 612 792] /Contents 4 0 R/Group<>/Tabs/S/StructParents 0>> endobj 4 0 obj <> stream x��][S��w��C=v�0EI%�T��a����abf�C���g��ж������h�u�]U*]2S�_�R����z�iq�n��v�������k����_�g?o�����/������鷏k���rq����o_5__������6]#��0��n�������������'�!��Xs���
id rather it fail gracefully

Extra context:

[nickscamara]

#29 fixes this.

[calebpeffer]

Link: https://www.rareskills.io/zk-book
Endpoint: /scrape, /crawl
receive a 502, could you please help? from Dmitry Ukhanov on discord

[nickscamara]

Fixed ✅

[nickscamara]

DM'd him!

[calebpeffer]

Link: https://developers.notion.com/reference/post-page
endpoint: /scrape /crawl

[nickscamara]

Fixed ✅ @calebpeffer can you let them know? and see if it works better now for them?

[garrettfrohman]

Link: https://docs2.hubitat.com/
Endpoint: /crawl
Error: No pages found

[nickscamara]

@garrettfrohman Fixed ✅

[lodex23]

Link: https://www.bax-shop.be/nl/speakers/speakerset
Endpoint: /scrape, /crawl
Issue: Request Time out everytime i try it, also when i try other urls that belong to the same website

[immartian]

link: https://www.wsj.com/tech/elon-musks-twitter-takeover-is-now-the-worst-buyout-for-banks-since-the-financial-crisis-3f4272cb
endpoint: /scrape, /crawl
Issue: Request Time out always, same to every url from the website

[wukimidaire]

Link: https://mendable.ai
endpoint: /scrape, /crawl
Issue: No pages found, Please try a different URL or check for compatibility issues with the current website.

[talaus-tadata]

Link: https://docs.temporal.io/
Endpoint: /crawl
Issue: No content found. This could be due to an incorrect URL or a compatibility issue on certain websites. Please verify the URL, and if correct, help us improve by reporting this issue.

[NioCarSan]

https://www.idealista.com/sala-de-prensa/informes-precio-vivienda/venta/cataluna/barcelona-provincia/castelldefels/

issue: No pages found

Please try a different URL or check for compatibility issues with the current website.

[wukimidaire]

https://www.elmos.be

issue: no pages found

Please try a different URL or check for compatibility issues with the current website.

[ad-angelo]

Hi guys !

Link : https://www.laredoute.fr/ppdp/prod-542712900.aspx
Endpoint : /scrape
Issue : Blocked by robot checking

Returned content :

Un cœur bat en vous.
====================

Merci de confirmer que vous n’êtes pas un robot.
------------------------------------------------

Verifying you are human. This may take a few seconds.

Waiting for www.laredoute.fr to respond...

Please enable Cookies and reload the page.

*   Ray ID : 8bc3885abeaa0a0d
*   Client IP : 193.142.36.20
*   Error reason : Country challenge

[jeremyssocial]

Link: https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/add-type?view=powershell-7.4
Endpoint: /scrape
Issue: No content found This could be due to an incorrect URL or a compatibility issue on certain websites. Please verify the URL, and if correct, help us improve by reporting this issue.

[jeremyssocial]

Link: https://support.microsoft.com/de-de/windows/tastenkombinationen-in-windows-dcc61a57-8ff0-cffe-9796-cb9706c75eec
Endpoint: /scrape
Issue: No content found This could be due to an incorrect URL or a compatibility issue on certain websites. Please verify the URL, and if correct, help us improve by reporting this issue.

[ad-angelo]

Link : https://www.footlocker.fr/fr/product/nike-air-max-tuned-1-drift-homme-chaussures/314216761404.html
End-point : /scrape
Issue : No content - code 403

[bjoaquinc]

Link : https://learn.microsoft.com/en-us/javascript/api/office-scripts/overview?view=office-scripts
End-point : /scrape
Issue : Failed to scrape URL. Error: {'success': True, 'warning': 'No page found'}. I added a waitFor: 2000 as well

[ericciarla]

Link: https://insider.fitt.co/newsletter/
Endpoint: /crawl
Wont gather pages - notify rexhunter8@gmail.com

[ericciarla]

Link: https://www.welcometothejungle.com/fr/companies/veesion-1/jobs
Endpoint: /crawl
I'm working on trying to crawl all job posting in the list of this website (https://www.welcometothejungle.com/fr/companies/veesion-1/jobs), but the output of the crawler is only 1 page. Email arthur@repio.co when done

[wonhs91]

Link: https://www.wsj.com/politics
Endpoint: /scrape /crawl
The page returned an error while being scraped. Unauthorized

[TheRayFitzgerald]

Link: https://opencorporates.com/companies/us_or/138287693, https://opencorporates.com/companies/us_tx/0802555441, https://opencorporates.com/companies/us_ca/202464211766
Endpoint: /scrape
Issue: Scraping OpenCorporates pages fails intermittently with the following error: Expecting value: line 2 column 1 (char 1)