How to run unitest when using opendal services-oss/s3 with mock server

[Lzzzzzt]

As title, I am developing the dragonflyoss/client#496, I use opendal to access oss/s3, now I need to add unitests for it.

How could I design the unitest, it seems that opendal will only use the 80/443 port, and if i bind mock server with 80 port, it returns 501/502, but when I use reqwest, the server works fine.

[Xuanwo]

How could I design the unitest, it seems that opendal will only use the 80/443 port, and if i bind mock server with 80 port, it returns 501/502, but when I use reqwest, the server works fine.

Hi, please try set endpoint to full uri like https://x.x.x.x:4433 or http://x.x.x.x:8080.

• If port is omitted, we will use the default port based on scheme.
• If scheme is omitted, we will use https as default.

[Lzzzzzt]

since oss use the virtual host style, if the mock server is listening on 127.0.0.1:8080, then bucket is 127, endpoint is http://0.0.1:8080, is that right?

[Xuanwo]

since oss use the virtual host style, if the mock server is listening on 127.0.0.1:8080, then bucket is 127, endpoint is http://0.0.1:8080, is that right?

Hi, this approach doesn't work. 127.0.0.1 itself is an IP address, we can't split it in this way. So we can't use virtual host style with an IP address; it must be a hostname. Neither <bucket>.127.0.0.1 nor 0.0.1 are valid URLs.

To mock the OSS service, we need to modify /etc/hosts or use reqwest::ClientBuilder::resolve_to_addrs in the HTTP client to map domains <bucket>.<hostname> to 127.0.0.1.

After those setup, we can set oss config endpoint to http://<hostname>:port. OpenDAL will send requests to http://<bucket>.<hostname>:port.

Instead of adding those tricks, I encourage you to test against real services like MinIO.

[Lzzzzzt]

thank you very much, i’ll try it

[Lzzzzzt]

i try to use

    let mut client_builder = reqwest::Client::builder()
        .use_preconfigured_tls(client_config_builder)
        .timeout(timeout)
        .default_headers(header);

    #[cfg(test)]
    {
        use std::net::SocketAddr;

        client_builder = client_builder.resolve(
            "bucket.test.oss.com",
            SocketAddr::new([127, 0, 0, 1].into(), 8080),
        );
    }

    let client = client_builder.build()?;

but still doesn't work

Builder {
    root: Some(
        "/",
    ),
    bucket: "bucket",
    endpoint: Some(
        "http://test.oss.com:8080",
    ),
    allow_anonymous: false,
    ..
}
Some(
    BackendError(
        BackendError {
            message: "Unexpected (temporary) at stat, context: { uri: http://bucket.test.oss.com/test.txt, response: Parts { status: 502, version: HTTP/1.1, headers: {\"connection\": \"close\", \"content-length\": \"0\"} }, service: oss, path: test.txt }",
            status_code: 500,
            header: {},
        },
    ),
)

while reqwest work fine

        let c = reqwest::Client::builder()
            .resolve_to_addrs(
                "bucket.test.oss.com",
                &[SocketAddr::new([127, 0, 0, 1].into(), 8080)],
            )
            .build()
            .unwrap();

        let r = c
            .head("http://bucket.test.oss.com:8080/test.txt")
            .send()
            .await;

        println!("{r:#?}");

output:

Ok(
    Response {
        url: Url {
            scheme: "http",
            cannot_be_a_base: false,
            username: "",
            password: None,
            host: Some(
                Domain(
                    "bucket.test.oss.com",
                ),
            ),
            port: Some(
                8080,
            ),
            path: "/test.txt",
            query: None,
            fragment: None,
        },
        status: 200,
        headers: {
            "connection": "close",
            "content-type": "text/plain",
            "content-length": "24",
            "x-oss-request-id": "6693AAD8F15BB239",
            "accept-ranges": "bytes",
            "etag": "\"6BD3A4527FA7993747C1\"",
            "last-modified": "Sun, 07 Jul 2024 04:12:50 GMT",
            "x-oss-object-type": "Normal",
            "x-oss-hash-crc64ecma": "11089307625453",
            "x-oss-storage-class": "Standard",
            "x-oss-version-id": "CAEQNxiBgICOgcqthBkiIDQ4ZGQ0ODQzMTExNDQ3MDE5NTVl***************",
            "content-md5": "a9OkUn+nmTdHwZ/Y6A==",
            "date": "Mon, 15 Jul 2024 02:39:08 GMT",
        },
    },
)

[Lzzzzzt]

it seems that oss will not use the port

opendal/core/src/services/oss/backend.rs

Lines 166 to 191 in b9f7c21

	let uri = ep.parse::<Uri>().map_err(|err| {
	Error::new(ErrorKind::ConfigInvalid, "endpoint is invalid")
	.with_context("service", Scheme::Oss)
	.with_context("endpoint", &ep)
	.set_source(err)
	})?;
	let host = uri.host().ok_or_else(|| {
	Error::new(ErrorKind::ConfigInvalid, "endpoint host is empty")
	.with_context("service", Scheme::Oss)
	.with_context("endpoint", &ep)
	})?;
	let full_host = format!("{bucket}.{host}");
	let endpoint = match uri.scheme_str() {
	Some(scheme_str) => match scheme_str {
	"http" | "https" => format!("{scheme_str}://{full_host}"),
	_ => {
	return Err(Error::new(
	ErrorKind::ConfigInvalid,
	"endpoint protocol is invalid",
	)
	.with_context("service", Scheme::Oss));
	}
	},
	None => format!("https://{full_host}"),
	};
	(endpoint, full_host)

[Xuanwo]

Wow, you catch it! Would you like to give this a fix?