Skip to content

Commit

Permalink
Share a client per instance instead of a singleton
Browse files Browse the repository at this point in the history
  • Loading branch information
cerbero90 committed Feb 12, 2024
1 parent 84650a2 commit dbb4eb9
Show file tree
Hide file tree
Showing 19 changed files with 242 additions and 172 deletions.
28 changes: 22 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ $lazyCollection = LazyJsonPages::from($source)
Framework-agnostic package to load items from any paginated JSON API into a [Laravel lazy collection](https://laravel.com/docs/collections#lazy-collections) via async HTTP requests.

> [!TIP]
> Need to read large JSON with no pagination in a memory-efficient way? Consider using [🐼 Lazy JSON](https://github.com/cerbero90/lazy-json) or [🧩 JSON Parser](https://github.com/cerbero90/json-parser) instead.
> Do you need to read large JSON with no pagination in a memory-efficient way? Consider using [🐼 Lazy JSON](https://github.com/cerbero90/lazy-json) or [🧩 JSON Parser](https://github.com/cerbero90/json-parser) instead.

## 📦 Install
Expand Down Expand Up @@ -49,7 +49,7 @@ composer require cerbero/lazy-json-pages

### 👣 Basics

Depending on our coding style, we can call Lazy JSON Pages in 3 different ways:
Depending on our coding style, we can initialize Lazy JSON Pages in 3 different ways:

```php
use Cerbero\LazyJsonPages\LazyJsonPages;
Expand Down Expand Up @@ -123,7 +123,9 @@ class CustomSource extends Source
}
```

The parent class `Source` gives us access to the property `$source`, which is the custom source for our use case.
The parent class `Source` gives us access to 2 properties:
- `$source`: the custom source for our use case
- `$client`: the Guzzle HTTP client

The methods to implement respectively turn our custom source into a PSR-7 request and a PSR-7 response. Please refer to the [already existing sources](https://github.com/cerbero90/json-parser/tree/master/src/Sources) to see some implementations.

Expand Down Expand Up @@ -263,8 +265,9 @@ class CustomPagination extends Pagination
}
```

The parent class `Pagination` gives us access to 2 properties:
The parent class `Pagination` gives us access to 3 properties:
- `$source`: the [source](#-sources) pointing to the paginated JSON API
- `$client`: the Guzzle HTTP client
- `$config`: the configuration that we generated by chaining methods like `totalPages()`

The method `getIterator()` defines the logic to extract paginated items in a memory-efficient way. Please refer to the [already existing paginations](https://github.com/cerbero90/json-parser/tree/master/src/Paginations) to see some implementations.
Expand All @@ -280,8 +283,21 @@ If you find yourself implementing the same custom pagination in different projec

### 🚀 Requests optimization

> [!WARNING]
> The documentation of this feature is a work in progress.
Paginated APIs differ from each other, so Lazy JSON Pages lets us tweak our HTTP requests specifically for our use case.

Internally, Lazy JSON Pages uses [Guzzle](https://docs.guzzlephp.org) as its HTTP client. We can customize the client behavior by adding as many [middleware](https://docs.guzzlephp.org/en/stable/handlers-and-middleware.html#middleware) as we need:

```php
LazyJsonPages::from($source)
->middleware('log_requests', $logRequests)
->middleware('cache_responses', $cacheResponses);
```

If we a middleware to be added every time we invoke Lazy JSON Pages, we can add a global middleware:

```php
LazyJsonPages::globalMiddleware('fire_events', $fireEvents);
```


### 💢 Errors handling
Expand Down
3 changes: 1 addition & 2 deletions src/Concerns/SendsAsyncRequests.php
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@

namespace Cerbero\LazyJsonPages\Concerns;

use Cerbero\LazyJsonPages\Services\Client;
use Generator;
use GuzzleHttp\Pool;
use Illuminate\Support\LazyCollection;
Expand Down Expand Up @@ -44,7 +43,7 @@ protected function fetchPagesAsynchronously(LazyCollection $chunkedPages, UriInt
*/
protected function pool(UriInterface $uri, array $pages): Pool
{
return new Pool(Client::instance(), $this->yieldRequests($uri, $pages), [
return new Pool($this->client, $this->yieldRequests($uri, $pages), [
'concurrency' => $this->config->async,
'fulfilled' => fn(ResponseInterface $response, int $page) => $this->book->addPage($page, $response),
'rejected' => fn(Throwable $e, int $page) => $this->book->addFailedPage($page) && throw $e,
Expand Down
3 changes: 1 addition & 2 deletions src/Concerns/YieldsItemsByCursor.php
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@

namespace Cerbero\LazyJsonPages\Concerns;

use Cerbero\LazyJsonPages\Services\Client;
use Closure;
use Generator;
use Psr\Http\Message\ResponseInterface;
Expand All @@ -28,7 +27,7 @@ protected function yieldItemsByCursor(Closure $callback): Generator

while ($cursor = $this->toPage($generator->getReturn(), onlyNumerics: false)) {
$uri = $this->uriForPage($request->getUri(), (string) $cursor);
$response = Client::instance()->send($request->withUri($uri));
$response = $this->client->send($request->withUri($uri));

yield from $generator = $callback($response);
}
Expand Down
69 changes: 46 additions & 23 deletions src/LazyJsonPages.php
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
use Cerbero\LazyJson\Pointers\DotsConverter;
use Cerbero\LazyJsonPages\Dtos\Config;
use Cerbero\LazyJsonPages\Paginations\AnyPagination;
use Cerbero\LazyJsonPages\Services\Client;
use Cerbero\LazyJsonPages\Services\ClientFactory;
use Cerbero\LazyJsonPages\Sources\AnySource;
use Closure;
use GuzzleHttp\RequestOptions;
Expand All @@ -19,9 +19,27 @@
final class LazyJsonPages
{
/**
* The source of the paginated API.
* The HTTP client factory.
*/
private readonly AnySource $source;
private readonly ClientFactory $factory;

/**
* The Guzzle HTTP request options.
*
* @var array<string, mixed>
*/
private array $options = [
RequestOptions::CONNECT_TIMEOUT => 5,
RequestOptions::READ_TIMEOUT => 5,
RequestOptions::TIMEOUT => 5,
];

/**
* The Guzzle client middleware.
*
* @var array<string, callable>
*/
private array $middleware = [];

/**
* The raw configuration of the API pagination.
Expand All @@ -31,13 +49,12 @@ final class LazyJsonPages
private array $config = [];

/**
* The Guzzle HTTP request options.
* Add a global middleware.
*/
private array $requestOptions = [
RequestOptions::CONNECT_TIMEOUT => 5,
RequestOptions::READ_TIMEOUT => 5,
RequestOptions::TIMEOUT => 5,
];
public static function globalMiddleware(string $name, callable $middleware): void
{
ClientFactory::globalMiddleware($name, $middleware);
}

/**
* Instantiate the class statically.
Expand All @@ -50,9 +67,9 @@ public static function from(mixed $source): self
/**
* Instantiate the class.
*/
public function __construct(mixed $source)
public function __construct(private readonly mixed $source)
{
$this->source = new AnySource($source);
$this->factory = new ClientFactory();
}

/**
Expand Down Expand Up @@ -178,7 +195,7 @@ public function async(int $requests): self
*/
public function connectionTimeout(float|int $seconds): self
{
$this->requestOptions[RequestOptions::CONNECT_TIMEOUT] = max(0, $seconds);
$this->options[RequestOptions::CONNECT_TIMEOUT] = max(0, $seconds);

return $this;
}
Expand All @@ -188,8 +205,8 @@ public function connectionTimeout(float|int $seconds): self
*/
public function requestTimeout(float|int $seconds): self
{
$this->requestOptions[RequestOptions::TIMEOUT] = max(0, $seconds);
$this->requestOptions[RequestOptions::READ_TIMEOUT] = max(0, $seconds);
$this->options[RequestOptions::TIMEOUT] = max(0, $seconds);
$this->options[RequestOptions::READ_TIMEOUT] = max(0, $seconds);

return $this;
}
Expand All @@ -214,6 +231,16 @@ public function backoff(Closure $callback): self
return $this;
}

/**
* Add an HTTP client middleware.
*/
public function middleware(string $name, callable $middleware): self
{
$this->middleware[$name] = $middleware;

return $this;
}

/**
* Retrieve a lazy collection yielding the paginated items.
*
Expand All @@ -222,16 +249,12 @@ public function backoff(Closure $callback): self
*/
public function collect(string $dot = '*'): LazyCollection
{
Client::configure($this->requestOptions);

$config = new Config(...$this->config, itemsPointer: DotsConverter::toPointer($dot));
return new LazyCollection(function() use ($dot) {
$config = new Config(...$this->config, itemsPointer: DotsConverter::toPointer($dot));
$client = $this->factory->options($this->options)->middleware($this->middleware)->make();
$source = new AnySource($this->source, $client);

return new LazyCollection(function() use ($config) {
try {
yield from new AnyPagination($this->source, $config);
} finally {
Client::reset();
}
yield from new AnyPagination($source, $client, $config);
});
}
}
2 changes: 1 addition & 1 deletion src/Paginations/AnyPagination.php
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ public function getIterator(): Traversable
protected function matchingPagination(): Pagination
{
foreach ($this->supportedPaginations as $class) {
$pagination = new $class($this->source, $this->config);
$pagination = new $class($this->source, $this->client, $this->config);

if ($pagination->matches()) {
return $pagination;
Expand Down
2 changes: 1 addition & 1 deletion src/Paginations/CustomPagination.php
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,6 @@ public function getIterator(): Traversable
throw new InvalidPaginationException($this->config->pagination);
}

yield from new $this->config->pagination($this->source, $this->config);
yield from new ($this->config->pagination)($this->source, $this->client, $this->config);
}
}
2 changes: 2 additions & 0 deletions src/Paginations/Pagination.php
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
use Cerbero\LazyJsonPages\Dtos\Config;
use Cerbero\LazyJsonPages\Services\Book;
use Cerbero\LazyJsonPages\Sources\AnySource;
use GuzzleHttp\Client;
use IteratorAggregate;
use Traversable;

Expand Down Expand Up @@ -39,6 +40,7 @@ abstract public function getIterator(): Traversable;
*/
final public function __construct(
protected readonly AnySource $source,
protected readonly Client $client,
protected readonly Config $config,
) {
$this->book = new Book();
Expand Down
4 changes: 2 additions & 2 deletions src/Providers/LazyJsonPagesServiceProvider.php
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

namespace Cerbero\LazyJsonPages\Providers;

use Cerbero\LazyJsonPages\Services\Client;
use Cerbero\LazyJsonPages\LazyJsonPages;
use GuzzleHttp\Middleware;
use GuzzleHttp\Promise\PromiseInterface;
use Illuminate\Http\Client\Events\ConnectionFailed;
Expand All @@ -26,7 +26,7 @@ final class LazyJsonPagesServiceProvider extends ServiceProvider
*/
public function boot(): void
{
Client::middleware('laravel_events', Middleware::tap($this->sending(...), $this->sent(...)));
LazyJsonPages::globalMiddleware('laravel_events', Middleware::tap($this->sending(...), $this->sent(...)));
}

/**
Expand Down
100 changes: 0 additions & 100 deletions src/Services/Client.php

This file was deleted.

Loading

0 comments on commit dbb4eb9

Please sign in to comment.