Provide Labels to Loki from Faro Web SDK

[matiduck]

I would like to send a Logs from Angular application using Grafana Agent and store them in Loki.
I initialized Faro SDK in application as described here

When I try to send log request (using pushLog with some context set) I constantly got an error:

error: at least one label pair is required per stream

My Grafana Agent is configured in flow mode and looks exactly like configuration from example here (without sourcemaps)

Could you advise how I could set up labels from Faro SDK? I have multiple small angular ui and I wanted all of them report to single Grafana Agent instance that push logs to Loki with appropriate labels.

Thank you in advance for you help!

[codecapitano]

Hi @matiduck

Did you maybe try out to send the data via curl so we can now were in the path the problem happens?
Can you provide an example of the data your sending with push log?

Cheers,
Marco

[matiduck]

Hi @codecapitano

Thanks for you response!
As for example of the data - I tried to send out the following log using faro api:

faro.api.pushLog(['User clicked'], {
      level: LogLevel.WARN,
      context: {
        service: "testService"
      }
    });

which produces following body to /collect endpoint:

{"meta":{"sdk":{"version":"1.3.5"},"app":{"name":"frontend","version":"1.0.0","environment":"dev"},"view":{"name":"default"},"browser":{"name":"Chrome","version":"120.0.0.0","os":"Windows 10","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36","language":"en-GB","mobile":false,"brands":[{"brand":"Not_A Brand","version":"8"},{"brand":"Chromium","version":"120"},{"brand":"Google Chrome","version":"120"}]},"page":{"url":"http://localhost:4200/"},"session":{"id":"6wB08uSfj1"}},"logs":[{"message":"User clicked","level":"warn","context":{"service":"testService"},"timestamp":"2024-01-23T09:32:41.435Z"}],"measurements":[{"type":"web-vitals","values":{"lcp":648.2000000001863},"timestamp":"2024-01-23T09:32:42.672Z","context":{}}]}

I also set up loki.echo component in grafana agent to see how this call is parsed and I got following log:

{"ts":"2024-01-23T09:33:38.4194152Z","level":"info","component":"loki.echo.test","receiver":"loki.echo.test","entry":"timestamp=\"2024-01-23 09:33:38.33 +0000 UTC\" kind=measurement type=web-vitals cls=0.000000 sdk_version=1.3.5 app_name=frontend app_version=1.0.0 app_environment=dev session_id=6wB08uSfj1 page_url=http://localhost:4200/ browser_name=Chrome browser_version=120.0.0.0 browser_os=\"Windows 10\" browser_mobile=false view_name=default","labels":"{}"}

As you can see, the labels array is empty in agent log because there is no entry in a request body which holds information about them. Is there any place in Faro SDK that I could specify labels?

[codecapitano]

Hi @matiduck

Attributes send via context are simply parts of the log line, no log labels.

Faro itself does not provide a n option to create labels.
Grafana Cloud (Frontend Observability) provides to define custom log labels, which are real Loki labels.
For the Agent it does not provide a similar feature.

I started the Faro demo, which uses the Agent to send a similar log message as in you example and attached a few screenshots to clarify how context is parsed.

Test log

faro.api.pushLog(['Context test'], {
    level: LogLevel.WARN,
    context: {
      service: 'testService',
    },
  });

[PaulSender]

Seeing a similar issue except I'm not seeing the log come through at all from faro -> loki grafana.

When I do
console.log('hello world') or faro.api.pushLog(['hello world'], { level: LogLevel.WARN, context: { service: 'testService', }, });

The log I see in Loki is:

level=error msg="final error sending batch" component_path=/ component_id=loki.write.logs component=client host=loki-write.loki.svc.cluster.local:3100 status=400 tenant="" error="server returned HTTP status 400 Bad Request (400): error at least one label pair is required per stream"

I'm sending these events through faro-web-sdk -> grafana Alloy which sends them to Loki. What am I missing? Shouldn't base console.log work out of the box?

[PaulSender]

The issue was with the Alloy/Agent set up. I needed to add external_labels to the loki.write block of my set up.

Documentation for loki.write states that external_labels is not required which I guess is true is you're only using loki.write but they seem to be very required if you want to see loki logs sent from Faro -> Alloy/Agent -> Loki -> Grafana.

adding

external_labels = {
          job = "faro",
        }

allowed me to filter logs like: {job="faro"} | logfmt and subsequently {job="faro"} | logfmt |~ <log-mesage> to see the logs I wanted. This also works for RUM events.

Totally get driving people towards Grafana Cloud but there should probably be some up to date guide/documentation on how to use Faro with a self hosted Grafana set up.

[codecapitano]

Hi @PaulSender I don't think that this is there to drive people towards cloud but rather an issue we need to invetsigate.

I synced with the team yesterday:

They shouldn’t need to ... Adding the extra log labels might be a workaround for now until we can figure it out

Would you mind providing your config so we can dig deeper?

cc @rlankfo

[PaulSender]

Sure, here is my alloy config:

global:
  image:
    registry:<secret>
    pullSecrets:
      - name: <secret>

alloy:
  clustering:
    enabled: true
  resources:
    requests:
      cpu: 250m
      memory: 512Mi
  extraPorts:
    - name: "otlp"
      port: 4318
      targetPort: 4318
      protocol: "TCP"
    - name: "faro"
      port: 12347
      targetPort: 12347
      protocol: "TCP"
  configMap:
    content: |
      logging {
        level = "info"
        format = "logfmt"
      }
      loki.write "logs" {
        external_labels = {
          job = "faro",
        }
        endpoint {
          url = "<valid-url>"
        }
      }
      otelcol.exporter.otlphttp "traces" {
        client {
          endpoint = "http://tempo-gateway:80"
          tls {
            insecure = true
            insecure_skip_verify = true
          }
        }
      }
      faro.receiver "faro" {
        server {
          listen_address = "0.0.0.0"
          cors_allowed_origins = ["*"]
        }
        output {
          logs   = [loki.write.logs.receiver]
          traces = [otelcol.exporter.otlphttp.traces.input]
        }
      }

configReloader:
  enabled: false

controller:
  autoscaling:
    enabled: true
    maxReplicas: 5
    minReplicas: 2
  type: "deployment"
  extraAnnotations: { reloader.stakater.com/auto: "true" }

rbac:
  create: false

And I was initing Faro in a basic way:

initializeFaro({
  // Mandatory, the URL of the Grafana collector
  url: 'https://<alloy-endpoint>/collect',
  app: {
    name: '<secret>',
  },

  instrumentations: [
    ...getWebInstrumentations({
      captureConsole: true,
    }),

    new TracingInstrumentation(),
  ],
});

Note the external logs in Loki.write.

I'd also like to point out that I HAD to do cors_allowed_origins = ["*"] in order to send things from localhost. If I didn't I'd get a cors error. I had it as cors_allowed_origins = ["http//:localhost:1234"] and cors_allowed_origins = ["localhost"] which did not work even though docs said I should only need cors_allowed_origins = ["localhost"].

[raresfelecan]

Seems like if you set extra_labels that match data you send from faro sdk and assign them an empty string value, they will be seen as labels in grafana cloud and populated with the value sent from faro.
E.g:

external_labels = {
  job = "faro",
  app_name = "",
  app_environment = "",
}