Suppress calls to cloud server for a Recovery Period after request failures.

[drasmart]

Changes

• Introduce PipelineTemporarilyUnavailableException.
  • Treated as [PropertiesNotYetLoadedException] by Pipeline -- does not get thrown from [IPipeline.ElementAvailableProperties], but prevents caching.
  • Treated as SuppressProcessExceptions by WebPipeline and PipelineResultService -- Process will not throw, the error can be read from [IFlowData.Errors].
• Introduce IRecoveryStrategy -- defines for how long to suppress calls after certain amount of "logical failures".
  • InstantRecoveryStrategy -- always allow new calls (pre-change behavior or no recovery period).
  • NoRecoveryStrategy -- does not allow any new calls ever after first failure.
  • SimpleRecoveryStrategy -- disallow new calls for RecoverySeconds period after then most recent failure.
• Introduce IFailHandler -- defines when to suppress calls.
  • SimpleFailHandler -- simply wraps provided IRecoveryStrategy.
  • WindowedFailHandler -- wait for certain amount (failuresToEnterRecovery) to accumulate within certain amount of time (failuresWindow) before forwarding the failure to the underling IRecoveryStrategy. Also calls Reset upon first success after exiting Recovery Period.
• Add new CloudRequestEngine constructor.
  • Endpoint- and key- related string parameters grouped into a single EndpointsAndKeys struct (saved to read-only field).
  • Required IFailHandler parameter.
  • Old constructor (without throttling strategy) now forwards to this (new) one.
  • CloudRequestEngineBuilder now also uses this (new) constructor.
• Use [Lazy] for initialization and storing values for [IFlowElement.EvidenceKeyFilter] and [ICloudRequestEngine.PublicProperties].
  • Simplifies update to AsyncLazyFailable planned for v4.5
• Introduce CloudRequestEngineTemporarilyUnavailableException (subclass of PipelineTemporarilyUnavailableException).
  • Thrown by CloudRequestEngine when IFailHandler prevents (likely temporarily) sending requests to cloud server due to recent failures.
• Expose SimpleThrottlingStrategy.RecoverySeconds on CloudRequestEngineBuilder.
  • If non-positive value provided (default: 60), NoThrottlingStrategy is used.
• Expose WindowedFailHandler.FailuresToEnterRecovery (min: 1, max: 100, default: 10) and WindowedFailHandler.FailuresWindowSeconds (required to be positive, default: 100) on CloudRequestEngineBuilder.
• Mark constructor-initialized fields (_loggerFactory, _dataLogger, _httpClient) on CloudRequestEngineBuilder as read-only.
• Add safety checks to PipelineCapabilities
  • Set _availableElementProperties to [ImmutableDictionary<TKey,TValue>.Empty] singleton if flowData has errors.
  • (Short-circuit) forward to _baseCaps[key] from string this[string key] indexer (like done in explicitly implemented methods).
• Dispose of (not-to-be-returned) flowData when WebPipeline.Process throws.
• Add new unit tests
  • for implementations of IFailThrottlingStrategy.
  • for using CloudRequestEngine with SimpleThrottlingStrategy.
• Also includes formatting changes from original patch file.

Why

• CloudRequestEngine graceful non-blocking degradation  #132

Test results

• Unit tests run ✅ https://github.com/postindustria-tech/pipeline-dotnet/actions/runs/11368373783
• .nupkg built by 📦 https://github.com/postindustria-tech/pipeline-dotnet/actions/runs/11368708022

---

Manual tests of Cloud/Framework-Web example (v.4.4.110-alpha.1+pi.wip)

with .nupkg from 📦 https://github.com/postindustria-tech/pipeline-dotnet/actions/runs/11297040712

---

Block-listing in Charles

Reloading the page within recovery period ( "RecoveryMilliseconds": 60000 )

^ did not produce new requests (only the initial 2 blocked ones from the first page load attempt can be seen)

After the recovery period

^ only 2 new requests were made.

---

Timeout due to latency by Charles

Conditions:

"TimeOutSeconds": 2
Round-trip latency (ms): [2500]

It takes around 6~10 second for the page to show error on [4.4.103]

While it still takes more than full awaiting of TimeOutSeconds (e.g. ~2.5s) for an error to propagate when outside Recovery Period on [4.4.110-alpha.1+pi.wip] :

Within Recovery Period, error page loading time is much shorter (under 200ms locally) :

^ and the page does not fail, as though SuppressProcessExceptions is true.

---

Manual tests of Cloud/Framework-Web example (v.4.4.113-alpha.1+pi.wip)

with .nupkg from 📦 https://github.com/postindustria-tech/pipeline-dotnet/actions/runs/11335248015

---

Default settings

"FailuresToEnterRecovery": 10,
"FailuresWindowSeconds": 60,

Time	Description	Image
0 ms	6 requests fail (timed out). Exception page shown.
17.84 s	4 requests fail (timed out). Recovery started. Exception page shown.
37.16 s	(Recovery) No requests sent. Default.aspx rendered.
59.21 s	(Recovery) No requests sent. Default.aspx rendered.
1.28 min	(Recovery) No requests sent. Default.aspx rendered.
1.46 min	(Recovery Period ended) 2 requests sent. Device detected. Default.aspx rendered.

---

Modified FailuresToEnterRecovery

"FailuresToEnterRecovery": 4,

Description	Image
4 requests fail (timed out). Recovery started. Exception page shown.
(Recovery) No requests sent. Default.aspx rendered.

---

Modified FailuresToEnterRecovery and FailuresWindowSeconds

"FailuresToEnterRecovery": 7,
"FailuresWindowSeconds": 30,

Time	Description	Image
0	6 requests fail (timed out). Exception page shown.
51.07 s	5 requests fail (timed out). Exception page shown.
1.38 min	5 requests fail (timed out). Exception page shown.
1.68 min	2 requests fail (timed out). Recovery started. Exception page shown.
1.94 min	(Recovery) No requests sent. Default.aspx rendered.

---

Manual tests of Cloud/Framework-Web example (v.4.4.115-alpha.1+pi.wip)

with .nupkg from 📦 https://github.com/postindustria-tech/pipeline-dotnet/actions/runs/11349595554

---

Modified RecoverySeconds

"RecoverySeconds": 20,

Time	Description	Image
41.41 s	Within recovery due to error 19.1968s ago. Default.aspx rendered.
51.07 s	Recovery Period ended. Requests timed out. Exception page shown.

---

Manual tests of Cloud/Framework-Web example (v.4.4.121-alpha.1+pi.wip)

ASP.NET Core integration

with

• .nupkg from 📦 https://github.com/postindustria-tech/pipeline-dotnet/actions/runs/11368708022
• changes for Enable Cloud/GettingStarted-Web to display suppressed errors. device-detection-dotnet-examples#350

Unsuppressed CloudRequestException due to timeout :

Suppressed CloudRequestEngineTemporarilyUnavailableException displayed by Index.cshtml :

[justadreamer]

Will need to also port this functionality to Java, Python, PHP, Node

[justadreamer]

@BohdanVV Let's change the default timeout from 100 seconds to 2 seconds.

[jwrosewell]

@justadreamer @BohdanVV Please can you ensure that the error message and exception will be generated when in recovery mode is clear in the documentation and in the related specifications before this is marked as ready for review?