Cannot find service token even though it's present in redis

[rahulanand16nov]

Hi there! I was using a docker image of Apisonator to avoid reliance on SaaS for integration testing of WASM filters, being developed under the GSoC'21 program. I am using internal APIs to initialize service ids, tokens, and applications. Even though all calls are successful and registered by Apisonator and Redis, authorize endpoint is not able to resolve the service token.

Script to reproduce the error:

echo "Start Redis"
docker run -p 6379:6379 -d --name my-redis redis --databases 2

echo "Start Apisonator"
docker run -e CONFIG_QUEUES_MASTER_NAME=redis://redis:6379/0 \
        -e CONFIG_REDIS_PROXY=redis://redis:6379/1 -e CONFIG_INTERNAL_API_USER=root \
        -e CONFIG_INTERNAL_API_PASSWORD=root -p 3000:3000 -d --link my-redis:redis \
        --name apisonator quay.io/3scale/apisonator 3scale_backend start

echo "Wait for redis and apisontor to launch"
sleep 5

echo "Create a service"
curl -d '{"service":{"id":"my_service_id","state":"active"}}' http://root:root@0.0.0.0:3000/internal/services/ | jq '.'

echo "Create a service id and token pair"
curl -d '{"service_tokens":{"my_service_token":{"service_id":"my_service_id"}}}' http://root:root@0.0.0.0:3000/internal/service_tokens/ | jq '.'

echo "Add application"
curl -d '{"application":{"service_id":"my_service_id","id":"my_app_id","plan_id":"my_plan_id","state":"active"}}' http://root:root@0.0.0.0:3000/internal/services/my_service_id/applications/my_app_id | jq '.'

echo "Check if service exists or not (Should return back service in JSON format)"
curl http://root:root@0.0.0.0:3000/internal/services/my_service_id | jq '.'

echo "Check if pair exists or not (should return 200 OK)"
curl --head http://root:root@0.0.0.0:3000/internal/service_tokens/my_service_token/my_service_id/

echo "Check pair without head (returns 'not found')"
curl http://root:root@0.0.0.0:3000/internal/service_tokens/my_service_token/my_service_id/ | jq '.'

echo "Use Authorize endpoint (returns 'service_token_invalid'):"
curl "http://0.0.0.0:3000/transactions/authorize.xml?service_token=my_service_token&service_id=my_service_id&user_key=my_user_key"

sleep 2

echo "Clean up"
docker rm my-redis -f
docker rm apisonator -f

Apisonator logs:

172.17.0.1 - root [12/Jul/2021 11:50:59 UTC] "POST /internal/services/ HTTP/1.1" 201 169 0.030991 0 0 0 0 2 1 - -

172.17.0.1 - root [12/Jul/2021 11:50:59 UTC] "POST /internal/service_tokens/ HTTP/1.1" 201 20 0.0023496 0 0 0 0 2 1 - -

172.17.0.1 - root [12/Jul/2021 11:50:59 UTC] "POST /internal/services/my_service_id/applications/my_app_id HTTP/1.1" 201 180 0.0138289 0 0 0 1 3 1 - -

172.17.0.1 - root [12/Jul/2021 11:50:59 UTC] "GET /internal/services/my_service_id HTTP/1.1" 200 167 0.0052454 0 0 0 3 5 1 - -

172.17.0.1 - root [12/Jul/2021 11:51:00 UTC] "HEAD /internal/service_tokens/my_service_token/my_service_id/ HTTP/1.1" 200 - 0.0069157 0 0 0 4 6 1 - -

172.17.0.1 - root [12/Jul/2021 11:51:00 UTC] "GET /internal/service_tokens/my_service_token/my_service_id/ HTTP/1.1" 404 42 0.002605 0 0 0 4 6 1 - -

172.17.0.1 - - [12/Jul/2021 11:51:00 UTC] "GET /transactions/authorize.xml?service_token=my_service_token&service_id=my_service_id&user_key=my_user_key HTTP/1.1" 403 - 0.0026236 0 0 0 6 10 3 - -

Redis keys dump (using docker exec -it my-redis redis-cli; select 1; keys *):

1) "application/service_id:my_service_id/id:my_app_id/state"
 2) "service/id:my_service_id/state"
 3) "service/id:my_service_id/referrer_filters_required"
 4) "services_set"
 5) "application/service_id:my_service_id/id:my_app_id/plan_id"
 6) "service_id:my_service_id/applications"
 7) "service/provider_key:/ids"
 8) "service_token/token:my_service_token/service_id:my_service_id"
 9) "application/service_id:my_service_id/id:my_app_id/user_required"
10) "service/id:my_service_id/provider_key"
11) "provider_keys_set"

Checking for the service pair with '--head' makes sense because there is a path listed for it and not for GET. But @unleashed asked me to mention it in this Issue.

apisonator/app/api/internal/service_tokens.rb

Lines 6 to 8 in ea23574

	head '/:token/:service_id/' do |token, service_id|
	ServiceToken.exists?(token, service_id) ? 200 : 404
	end

I am not sure why authorize endpoint is not able to resolve the service token which is required for integration tests for the wasm-filters.

Please let me know if I missed anything, thanks!

Update: I mistakenly added the provider key into the JSON data sent for creating a service and error message change from "invalid service token" to "user key missing/not found" (which makes sense as I haven't initialized any user key). So, I think, a more relevant error should be used (pertaining to provider key); OR, @unleashed mentioned that the provider key is deprecated so maybe there shouldn't be any reliance on it?