Skip to content

Commit

Permalink
Only fetch checksums for final gradle releases during build
Browse files Browse the repository at this point in the history
This is a follow up to #2775
I frequently experience build failures due to connection errors.

After some debugging with wireshark, netstat and
`-Djdk.httpclient.HttpClient.log=all` I suspect that I'm getting rate
throttled.

Somewhere between 150 and 200 requests, the connections either get
closed by the remote or almost stall.

To mitigate the issue, this excludes nightly, snapshot, rc and broken
gradle releases. The biggest impact has cutting out RC releases:

total versions: 386
without nightly: 385
without RCs: 173
without broken: 172

Note that this still doesn't completely fix the issue for me, but it
helps reduce the chance for it happening.
  • Loading branch information
mfussenegger committed Oct 20, 2023
1 parent fe0ac4c commit 282f01d
Showing 1 changed file with 12 additions and 30 deletions.
42 changes: 12 additions & 30 deletions org.eclipse.jdt.ls.core/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -52,12 +52,12 @@
import groovy.json.JsonOutput
import java.net.http.HttpClient
import java.net.http.HttpClient.Redirect
import java.net.http.HttpRequest
import java.net.http.HttpResponse.BodyHandlers
import java.util.concurrent.CompletableFuture
import java.util.stream.Collectors;
def checksumsFile = new File(project.basedir.absolutePath, "gradle/checksums/checksums.json")
if (System.getProperty("eclipse.jdt.ls.skipGradleChecksums") != null && checksumsFile.exists()) {
println "Skipping gradle wrapper validation checksums ..."
Expand All @@ -77,42 +77,24 @@
String wrapperChecksumUrl;
String sha256
}
HttpClient client = HttpClient.newHttpClient()
HttpClient client = HttpClient.newBuilder()
.followRedirects(Redirect.NORMAL)
.build();
def futures = []
versions.each {
if (it.wrapperChecksumUrl == null) {
if (it.wrapperChecksumUrl == null || it.nightly || it.snapshot || it.rcFor != "" || it.broken) {
return
}
HttpRequest request = HttpRequest.newBuilder().uri(URI.create(it.wrapperChecksumUrl)).build()
futures.add(client.sendAsync(request, BodyHandlers.ofString()).thenApply { response ->
if (response.statusCode() == 301) {
String newUrl = response.headers().firstValue("location").orElse(null)
if (newUrl != null) {
HttpRequest newRequest = HttpRequest.newBuilder()
.uri(URI.create(newUrl))
.build()
try {
String sha256 = client.send(newRequest, BodyHandlers.ofString()).body()
return new Checksum(wrapperChecksumUrl: it.wrapperChecksumUrl, sha256: sha256)
} catch (IOException | InterruptedException e) {
return null
}
} else {
return null
}
} else {
// Return the body of the original response
return new Checksum(wrapperChecksumUrl: it.wrapperChecksumUrl, sha256: response.body())
}
})
future = client.sendAsync(request, BodyHandlers.ofString()).thenApply { response ->
return new Checksum(wrapperChecksumUrl: it.wrapperChecksumUrl, sha256: response.body())
}
futures.add(future)
}
def checksums = CompletableFuture.allOf(futures.toArray(new CompletableFuture[0]))
.thenApply { v ->
futures.stream().map({ f ->
f.join()
}).collect(Collectors.toList())
}.get()
CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])).get()
def checksums = futures.stream().map({ f -> f.join() }).collect(Collectors.toList())
def json = JsonOutput.toJson(checksums)
checksumsFile.write(JsonOutput.prettyPrint(json))
Expand Down

0 comments on commit 282f01d

Please sign in to comment.