Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add XLoader image to run in it's own container, separate from the CKAN UI container #76

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

kowh-ai
Copy link
Contributor

@kowh-ai kowh-ai commented Jul 23, 2024

Fixes #66

This PR attempts to create an XLoader image to be used as a replacement for DataPusher.

However it currently has a problem in that the callback to the ckan container uses the resource data URL. This means if the URL is something like http://localhost:5000/dataset/testset20/resource/26004e4f-d13c-49f3-b370-bc38806050fa XLoader will try to connect to the ckan container running on localhost:5000 which is the xloader container rather than the ckan container

There needs to be a modification to the XLoader extension itself to allow similar environment variable to DataPusher ie: ckan.datapusher.callback_url_base. This would allow control of over what URL xloader would use to communicate back to the ckan container

It seems these are currently the only xloader configuration parameters available:
ckanext.xloader.jobs_db.uri
ckanext.xloader.api_token
ckanext.xloader.formats
ckanext.xloader.max_content_length
ckanext.xloader.use_type_guessing
ckanext.xloader.strict_type_guessing
ckanext.xloader.max_type_guessing_length
ckanext.xloader.parse_dates_dayfirst
ckanext.xloader.parse_dates_yearfirst
ckanext.xloader.job_timeout
ckanext.xloader.ignore_hash
ckanext.xloader.max_excerpt_lines
ckanext.xloader.ssl_verify
ckanext.xloader.clean_datastore_tables

Dockerfile, Makefile and setup process
@kowh-ai kowh-ai changed the title Add XLoader image to run in it's own container seperate from the CKAN UI container Add XLoader image to run in it's own container, separate from the CKAN UI container Jul 23, 2024
@kowh-ai kowh-ai marked this pull request as draft July 23, 2024 12:50
@amercader
Copy link
Member

This shouldn't be difficult to implement in xloader, I've put down the work needed here:

ckan/ckanext-xloader#226

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Investigate best practices for creating an XLoader Docker image + subsequent CKAN (worker) container
2 participants