Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Suggestion: Not parseable resources ->parseable resources #40

Open
ghost opened this issue Sep 7, 2017 · 1 comment
Open

Suggestion: Not parseable resources ->parseable resources #40

ghost opened this issue Sep 7, 2017 · 1 comment
Assignees
@c4software

Comments

@ghost
Copy link

ghost commented Sep 7, 2017

I took a peak at your source code. One source for crawling issues is that you currently define in the code not_parseable_ressources. Instead, if you define parseable resources and limit those to only truly parseable resources that are are supported in the sitemap and may contain plain text html links, you can limit issues with unknown extensions. Also you might take a look at using mime types instead of file extensions. I am not sure how that works in Python though.
@c4software c4software self-assigned this Sep 7, 2017
@c4software
Copy link
Owner

The not parseable ressource is more like a self guard to avoid some nasty case. But, its not a bad idea i will take a look.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@c4software c4software

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@c4software