What are the ways to make sure features can be still served if FH server goes down?

[IrinaSouth]

For example, could we store values somewhere like S3 bucket? Or are there any other ways?

[rvowles]

While our recommended way to deal with this is to use Fastly, if you are running your own FeatureHub service and don't wish to use Fastly and are happy with non-realtime events, the way that all of our SDKs are designed is to separate the Repository (which holds the features and which your application talks to) from the way that data is received. The functions of the repository are here: https://docs.featurehub.io/featurehub/latest/sdks-development.html#_a_feature_repository

For example, you could write a webhook listener that updates your s3 bucket locations with the updated features as they come in. You could then easily grab the feature state from the bucket, hand it over to the repository and then connect to FeatureHub. If FeatureHub goes away at anytime, the SDK clients will continue to request data as long as they don't get a 4xx response from the server.

We built an example application for servers that would only start if FeatureHub was up and was able to swap over to something else in the meantime. The process uses the Android SDK (which is a simple GET based API with a URL that could be easily emulated by s3) to grab the initial state and then fall back to SSE once started.

That example is here: https://github.com/featurehub-io/featurehub-java-sdk/blob/main/examples/migration-check/src/main/java/io/featurehub/migrationcheck/Main.java

[robcleghorn]

Not sure I follow here. We're fine with non-realtime events, so we setup a webhook to say a lamda, which persists the feature value in an S3 bucket, this part is fine. Can we then use the SDK client to query the value from this S3 bucket, or are we now down a customer path and new need to write our own client to get the value from S3?

[rvowles]

If you could give me an SDK you are using then I could give you a better example? You are definitely not on your own.

[robcleghorn]

Java, C#, Node, JS are the main ones we'll use.

[rvowles]

So they are all (except Golang) basically the same.

When you get the webhook, you can just save the data into a bucket with a filename like .json - and ensure its the same format as the Edge API (I think its pretty close, I can't remember exactly).

To make the SDK work for you, you just basically need to implement a simple interface that is the new Edge connector. You can if you want just talk directly to S3 or use it to call the proper endpoint and if that fails, call S3, your choice. All it does is pass the data it receives lock-stock into the repository. From there on, its just fully standard use case.

For each SDK, they have a "config" which ties together the "repository" (where the state is stored and evaluated from) and the "edge service provider" which goes and gets the data, and is responsible for doing it regularly. In JS for example, you can implement your own EdgeServiceProvider, it is just a function call:

(repository: InternalFeatureRepository, config: FeatureHubConfig) => EdgeService; 

So you get passed the new repository and a reference back to the config, the config will tell you what the API key is - and its just a two part of the environment id (which you need) and the service account key (which you don't).

EdgeService is just this:

export interface EdgeService {
  contextChange(header: string): Promise<void>;
  clientEvaluated(): boolean;

  // do we need a new one if the header changes?
  requiresReplacementOnHeaderChange(): boolean;

  close(): void;

  poll(): Promise<void>;
}

You are just going to ignore all the rest of the methods and implement a poll - which would just be an axios.get('https://your-s3-bucket/.json'), and then pass that to the repository.notify method.

C#, Node/JS and Java all work the same.