Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue with URL encoding of parameters in WMS #846

Open
ThomasG77 opened this issue Nov 30, 2022 · 8 comments
Open

Issue with URL encoding of parameters in WMS #846

ThomasG77 opened this issue Nov 30, 2022 · 8 comments
Labels

Comments

@ThomasG77
Copy link

ThomasG77 commented Nov 30, 2022

Why? See below two URLs

Unworking URL. Obtained with Owslib that encode characters /,, et : and it's normal

http://ogc.geo-ide.developpement-durable.gouv.fr/wxs?map=/opt/data/carto/geoide-catalogue/1.4/org_38178/908a2fc2-6752-4eae-952a-142393e657b7.internet.map&service=WMS&version=1.3.0&request=GetMap&layers=N_PERIM_MAET_ZINF_S_R11&styles=&width=800&height=448&crs=EPSG%3A4326&bbox=48.1107%2C1.44041%2C49.2484%2C3.56583&format=image%2Fpng&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=XML

vs

Working URL

http://ogc.geo-ide.developpement-durable.gouv.fr/wxs?map=/opt/data/carto/geoide-catalogue/1.4/org_38178/908a2fc2-6752-4eae-952a-142393e657b7.internet.map&service=WMS&version=1.3.0&request=GetMap&layers=N_PERIM_MAET_ZINF_S_R11&styles=&width=800&height=448&crs=EPSG:4326&bbox=48.1107,1.44041,49.2484,3.56583&format=image/png&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=XML

The issue seems to be on the server side of the service I consume that does not decode the encoded parameters.

Code to reproduce how I got the following URL

from owslib.wms import WebMapService

wms = WebMapService('https://ogc.geo-ide.developpement-durable.gouv.fr/wxs?map=/opt/data/carto/geoide-catalogue/1.4/org_38178/908a2fc2-6752-4eae-952a-142393e657b7.internet.map', version='1.3.0')

response = wms.getmap(layers=['N_PERIM_MAET_ZINF_S_R11',],
    bbox=(1.44041, 48.1107, 3.56583, 49.2484),
    format='image/png',
    size=(800, 428),
    srs='EPSG:4326',
)
with open(f"N_PERIM_MAET_ZINF_S_R11.png", 'wb') as out:
    out.write(response.read())

Possible approach

To solve the issue, I want to use the safe option of https://docs.python.org/3/library/urllib.parse.html#urllib.parse.urlencode

getmap method would get a new safe option, empty by default (like the normal urlencode behavior) but could be changed at https://github.com/geopython/OWSLib/blob/master/owslib/map/wms111.py#L258 and https://github.com/geopython/OWSLib/blob/master/owslib/map/wms111.py#L258 and https://github.com/geopython/OWSLib/blob/master/owslib/map/wms130.py#L304
where

data = urlencode(request)

would be replaced by

data = urlencode(request,safe)

So, it can help solve the issue when you do not control the server encode/decode behavior while not breaking anything. I would be able to provide to the new safe option the value :/, in my case (already tested with hardcoded value while debugging) .

PS: I can do a PR but need to confirm the logic could be fine on owslib project. Moreover, I've only look at the WMS code part but this could be applied maybe to other types of webservices with the same issue.

@sbrunner
Copy link
Contributor

When you create the WebMapService the map path should be encoded:

wms = WebMapService('https://ogc.geo-ide.developpement-durable.gouv.fr/wxs?map=%2Fopt%2Fdata%2Fcarto%2Fgeoide-catalogue%2F1.4%2Forg_38178%2F908a2fc2-6752-4eae-952a-142393e657b7.internet.map', version='1.3.0')

And you should be directly in https because the http to https redirect will break the URL...

@ThomasG77
Copy link
Author

ThomasG77 commented Nov 30, 2022

From a correct code perspective, the suggestion is surely right but it does not solve my sample code at all as it gives the same issue after change.

Did I miss something obvious?!

Ok. I understand that changing http to https in the samples URLs solve the issue but it's owlib that return/do http calls instead of https and I can't change the behavior without fixing issue at owlib level.

From my understanding, the issue is related to the fact owslib internally use http scheme from xlink:href in the capabilities from https://ogc.geo-ide.developpement-durable.gouv.fr/wxs?map=/opt/data/carto/geoide-catalogue/1.4/org_38178/908a2fc2-6752-4eae-952a-142393e657b7.internet.map&service=WMS&request=GetCapabilities so although the redirection http to https create "garbage", I do not see how a more obvious way to solve the issue as my first proposition.

@sbrunner
Copy link
Contributor

With the version 0.25.0 it's working...

@sbrunner
Copy link
Contributor

I think that a better solution is to have an option to skip this kind of code...

OWSLib/owslib/map/wms130.py

Lines 283 to 285 in 018c7f8

base_url = next((m.get('url') for m in
self.getOperationByName('GetMap').methods if
m.get('type').lower() == method.lower()))

and use directly self.url :-)

@ThomasG77
Copy link
Author

ThomasG77 commented Nov 30, 2022

Using 0.27.2 and not working. Tested also with 0.26.0 and breaking too.

I've tested 0.25.0 and working with both cases e.g map url part encoded and unencoded...
Can you confirm/infirm regression?

Diagnostic: side effect due to another fix e.g

b3acd1b#diff-a8ea809e6cb216f4c6c765a0d19397bc6d463c883e37e16d9f4a08db87f2cdedL18-L200

Tested that changing back owslib/util.py in latest code solve the issue but will establish again the issue fixed on the WFS side (the cause of the code breaking change)

@sbrunner
Copy link
Contributor

sbrunner commented Nov 30, 2022

Finally, when I see the full code this looks relay strange

data = urlencode(request)

The result is given to openURL, used here:
https://github.com/geopython/OWSLib/blob/018c7f83fb042a6390d65aaa683110743609f3e3/owslib/util.py#LL200C33-L200C33
and given to requests here:
req = requests.request(method.upper(), url_base, headers=headers, **rkwargs)

In requests documentation, we see that prams should be a dictionary
https://tedboy.github.io/requests/generated/generated/requests.Request.html
And there is a duplication
Then this code
u = openURL(base_url, data, method, timeout=timeout or self.timeout, auth=self.auth, headers=self.headers)

should be:

        u = openURL(base_url, request, method, timeout=timeout or self.timeout, auth=self.auth, headers=self.headers)

The same issue is in many place in the code...

@yagossc
Copy link

yagossc commented Sep 16, 2024

Is this still a problem?

I'm using __version__ = '0.31.0'

I create a wfs instance passing a URL that uses HTTPS:

WebFeatureService("https://domain/path/etc", version="2.0.0")

However, when I call the method getfeature(typename="blabla", ...) the request is made using HTTP.

The server I'm trying to reach only allows https requests and returns redirects otherwise (i.e., 302 Found).

Is there a workaround somewhere that I couldn't find? This is the only issue I found mentioning this specific case.

Copy link

This Issue has been inactive for 90 days. In order to manage maintenance burden, it will be automatically closed in 7 days.

@github-actions github-actions bot added the stale label Dec 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants