Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add basic web interface #14

Open
wvengen opened this issue Jan 30, 2024 · 3 comments
Open

Add basic web interface #14

wvengen opened this issue Jan 30, 2024 · 3 comments
Labels
API API & Web enhancement New feature or request

Comments

@wvengen
Copy link
Member

wvengen commented Jan 30, 2024

Scrapyd has a basic web interface. It would be useful to have something here too, for easy access.
Granted, you can already inspect everything with docker or kubectl, but a basic web interface still has its uses.

The idea is to have a simple interface:

  • show projects, versions, spiders
  • list jobs
  • schedule and cancel jobs
  • view job logs (after Expose logs #12)
@wvengen wvengen added the enhancement New feature or request label Jan 30, 2024
@wvengen
Copy link
Member Author

wvengen commented Jan 30, 2024

This can be very very simple.
SInce there is an API already, it does make sense to make use of it.
This means it would be a basic client-side application accessing the API. (Or add JSON and HTML responses to the API endpoints, if you can do that without polluting the API code, that may be an option. Don't over-engineer here.)
Don't use some large framework, but keep it small and simple.

@wvengen wvengen added the API API & Web label Jan 31, 2024
@wvengen wvengen moved this to Todo in Hack day 9 Feb 2024 Jan 31, 2024
@JeremyVriens
Copy link

Can we also use https://github.com/my8100/scrapydweb for this? If we're adhering to the same API contracts, I think this should work out-of-the-box for scrapyd_k8s as well.

@wvengen
Copy link
Member Author

wvengen commented Oct 31, 2024

Well, yes, and no. That project is meant for multiple hosts running scrapyd, with possibly different environment. It would be nice if scrapyd-k8s would work nicely in that (that would be a different issue, perhaps, or maybe not). And it may be a bit overkill.

I was thinking of something a bit more light-weight, that would be something like what scrapyd offers, but a little nicer, and more tuned to scrapyd-k8s. I would think of it something like a client-side app, talking to the existing API, so except for some asset serving, it wouldn't actually involve more code in scrapyd-k8s core (or go real-time with a websocket connection, that's not possible with regular scrapyd servers, but that can quickly get complex).

The goal is that users can get an easy overview of what is happening with scrapyd-k8s, without having to setup more things.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
API API & Web enhancement New feature or request
Projects
No open projects
Status: Todo
Development

No branches or pull requests

2 participants