7

I am trying to use loki and grafana for generating and visualizing log-based metrics. I have created a Grafana dashboard using the loki filters. While clicking the refresh button, all dashboards fail with the error "too many outstanding requests" and display no data. Refer to the screenshot attached. enter image description here I deployed grafana and loki to the EKS cluster.

Is there a parameter I can adjust to resolve the issue? I examined the pod/deployment configurations but found nothing pertinent.

Please assist.

Shine Kumar K P
  • 81
  • 1
  • 5
  • I also faced this issue, and updated my Loki configuration according to some of the examples at https://github.com/grafana/loki/issues/5123 and it seemed to help. – Ashley Kleynhans Nov 29 '22 at 12:38
  • 4
    Thanks, the issue was fixed after adjusting the "max_outstanding_per_tenant" parameter in Loki configuration. – Shine Kumar K P Dec 05 '22 at 14:31

2 Answers2

0

What worked for me was adding a max_outstanding_requests_per_tenant value to the local-config.yaml file, which is the default Loki config file. The file is stored in the official Docker image at /etc/loki/local-config.yaml. So I mounted a local-config.yaml file to that location:

auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

query_scheduler:
  max_outstanding_requests_per_tenant: 2048
Trevor Arnold
  • 76
  • 4
0

You can change default limits for some Loki internal variables. Here is an example of a tested config that works fine on 4 core virtual machine with simple file storage.

query_scheduler:
  max_outstanding_requests_per_tenant: 4096
frontend:
  max_outstanding_per_tenant: 4096
query_range:
  parallelise_shardable_queries: true
limits_config:
  split_queries_by_interval: 15m
  max_query_parallelism: 32

More details on this issue: https://github.com/grafana/loki/issues/5123

Roman Shishkin
  • 2,097
  • 20
  • 21