Prometheus dynamic labels. 8) and an instance name (node2).
Prometheus dynamic labels. Dec 9, 2023 · You are using Prometheus incorrectly: whenever you find yourself in situation that you want/need to make some calculation over values of labels - you messed up. . So, when you’ll decide to cover a multitude of availabilities, you’ll end Jul 1, 2021 · 1. metricName := "test_purposes" help := "tests" counter = promauto. For example, the following relabeling rule drops all the label names starting with foo: - action: labeldrop regex: "foo. Jun 9, 2020 · Running your software on dynamic infrastructure means that your monitoring platform needs to change dynamically. customLabels []*io_prometheus_client. yml: global: external_labels Nov 21, 2019 · So a Prometheus metric property is mapped to Grafana table column. In return you can declare the keys/labels at the same time. 14. However, these logs also have a variety of other labels attached which I want to use in the alerts labels and annotations, for example they have a reason and a resolution label. Every such time series is stored separately on the Prometheus node in the form of an Jul 26, 2021 · The idea would be to have an upstream-blessed way to add additional labels (that may have to be computed) to the metrics exposed by the middleware. Apr 24, 2020 · Its very simple and only three-step is needed to configure this: Create a YAML or JSON file which consist of the target’s. Go to the "Visualization" tab: Choose the value "Name" under "Value - Stat". Step 4: Configured service discovery result. Ties are resolved by rounding up. Templating labels and annotations. You need to provide the labelname in the metric: private static final Summary latencySummary = Summary. You can even format the legend in order to print the metric name + a custom label. build() . The same expression, but summed by application, could be written like this: Jan 9, 2022 · Thanks for contributing an answer to Stack Overflow! Please be sure to answer the question. Some of them are: I think what you require is file_sd_config. name("latency_seconds") . After unregistering, the // unregistered metrics will cease to show up in the /metrics HTTP // response, but the registry still remembers that those metrics had // been exported before. replacement: 127. Prometheus: PromQL - Adding a label to the metric. It also offers a registry for metrics. Friendly Tips：the $ (POD_NAME) valiable has been defined in helm chart resource file (kube-prometheus-stack). Compared to Prometheus' requirement to register the metric as a static variable, Micrometer allows avoiding that. yml. TL;DR Dynamically route alerts to relevant Slack team channels by labelling Kubernetes resources with team and extracting team label within alert rules. ), the configuration file defines everything related to scraping jobs and their instances, as well as which rule files to load Feb 27, 2024 · Overview. *prometheus. My metrics looks like the below. Apr 9, 2020 · Hello! Warning appeared in Grafana: “dynamic label lookup is disabled for data sources with more than 10000 metrics” at the same time, data is no longer displayed on the chart where this warning is displayed I found P… Nov 2, 2022 · 1. - record: threshold_NodeHighCpuLoad_warning expr: 10 labels: Mar 8, 2019 · I met the same issue when deploying Prometheus. The result of an expression can either be shown as a graph, viewed as tabular data in Prometheus's expression browser, or consumed by external systems via the HTTP API. allyrr asked this question in Q&A. It is recommended to initialize the label values by calling the . Nov 23, 2020 · This can be done in Prometheus config using #metric_relabel_configs: metric_relabel_configs: - source_labels: [vm_name] regex: ^([^. What this solution entails is (1) using the ‘resourceprocessor’ to add our labels as resource attributes, and (2) ‘Prometheus remote write exporter’ taking these resource attributes and converting them to valid Prometheus labels. 1. Let's say: Sep 17, 2021 · Custom last_request_received_time metric. Step 1: Install Prometheus Operator. Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables. Serious Prometheus-fu ahead! At loveholidays, the “you build it, you run it” ethos is deeply ingrained into our engineering culture. Hi 👋 👋 👋 community, There are a bunch of K8S NameSpaces (each for a separate customer). *xyz" Aug 10, 2020 · Here is how I configure Prometheus-Operator resources to scrape metrics from Istio 1. Modified 5 years, 8 months ago. Features. Here’s the query I’m using: PromQL code. For my case, the solution is: Oct 20, 2020 · Prometheus Model: Metrics and Labels. 99. *, so that character’s length is very less in the query. 0 Oct 29, 2021 · 2. The name of the label to drop must be specified as a regular expression in regex. what would be the approach there? To add multiple targets for a job, file_sd_configs can be used. scrape_configs: Sep 25, 2020 · Prometheus metric with dynamic label value. Deploying Prometheus via a helm chart will have an issue as the same delimiters ({{and }}) are used both by helm templating and the alerting templating in Prometheus. label_replace: This function allows you to replace or modify labels of a time series. labelNames("api") . I currently have an alert set up with: Query A: a Prometheus metric with labels like {tenant="tenant1", } and legend set to {{tenant}}. LabelPair. 8) and an instance name (node2). # evaluated at the current time, and the result recorded as a new set of. 215. Jul 26, 2021 · The alerts seem to fire as expected, which is basically as any log comes into Loki with the above labels. The alerting UI shows labels for every alert instance generated during evaluation of that rule. external_labels: env: prod cluster: project-prod-eks The community chart has it in values. } func NewCustomMetricsRegistry(labels map[string]string) *customMetricsRegistry {. If just one label value changes, this creates a new stream. PromQL supports the ability to join two metrics together: You can append a label set from one metric and append it to another at query time. You can use templates to include data from queries and expressions in labels and annotations. ]+)\. The most important of these is the le label which is the “less than or equal to” bucket threshold. For example: - record: job:availability:999 expr: |. You need a custom registry that knows how to add custom labels to metrics :) type customMetricsRegistry struct {. I want to fire an alert only for a set of assets based on asset-id. This can be useful in Prometheus rule evaluations, since it lets you generate a new metric for a series by appending labels from another info metric. When using the metric on a dashboard you can sum the values effectively "ignoring" the label(s) but generally it doesn't make sense to use labels like client id as the cardinality of the metric will be high and that will affect the performance. 0 allows you to specify multiple SRV records to be queried in a single scrape configuration, and also provides service-discovery-specific meta information that is helpful during the relabeling phase. Instrumentation, both for internal Prometheus monitoring and GHC statistics. 6 and install the latest Grafana Dashboards. /prometheus --config. e. yml so that it takes path of a target along with the host name? scrape_configs: - job_name: 'example-random' # Override the global default and scrape targets from this job every 5 seconds. 0. 2 weeks ago. In this case, can you also share some code that you Metric and label naming. Grafana: using instance label in title field of stats panel. May 8, 2023 · I am using Grafana 9. There is a label in common between the two metrics “node_meta” and “node_disk_bytes_read”: instance. *$. Aug 14, 2021 · Aug 14, 2021 • Tiago Ilieve. Prometheus label timeout use default value 0 if not exists. Jun 25, 2022 · on Jun 25, 2022. So I have quite a lot of labels for any query of the type: envoy_cluster_upstream_rq_total - a query which will return the number of requests for all clusters on all stands. - server3. However, since the summary is an annotation, it cannot be directly used in Alertmanagers groupBy configuration as it can only use labels. Going with gauge data type as for this use case we are "setting" the value as a timestamp. Each NameSpace (NS) has a label CLIENT_DEPLOYMENT_NAME with a different value. Labels in Loki perform a very important task: They define a stream. add more target option to separate instance and add labels. then you will only have label_foo available. To measure an application's stability: build_stability{application="my-billion-dollar-app"} To measure an cross-application build's stability: Oct 27, 2018 · How to display prometheus label, as value? 0. May 16, 2020 · Instead, we can use a static Recording Rule so availability target can be just set in one place. More specifically, the combination of every label key and value defines the stream. The first step is to understand that Prometheus is not just a monitoring system, but also a time series database. The library is intended to be easy to use, because instrumentation is already boring enough - the last thing you want is to be 5 pages deep in obscure GHC extensions just to bump a counter! Mar 7, 2023 · I'm adding serves to my prometheus config like this: - job_name: cluster1. counter call. i. Prometheus is configured via command-line flags and a configuration file. Step 2: Prometheus control plane configuration. # The PromQL expression to evaluate. route: Metrics with labels are not initialized when declared, because the client can’t know what values the label can have. The default prometheus registry class will run the collect once to store the metric definitions, then run collect to obtain updated metric values on each scrape. Is there any similar approach that is possible to add a dynamic job? Thank you in advance! Jul 17, 2021 · For example, let's imagine that for each incoming HTTP request, I want to add and fill a `user_group` label with value `admin` or `guest` depending on a few validations that happen at run time while processing the request. Contact points can access labels to dynamically Jobs and instances. Step 3: Registering the collector Jun 28, 2018 · The first dropdown is defined by variable datasource and is type datasource and type for datasource options is prometheus. Apr 14, 2022 · Grafana - create table with column values with Prometheus (dynamic) property/label data has each instance as a single metric with the right labels, instead of spread out over a number of metrics. Every evaluation cycle this is. Grafana lists these variables in dropdown select boxes at the top of the dashboard to help you change the data displayed in your dashboard. g. It takes four arguments: the input vector, the target label name, a regular expression to match the source label value, and a replacement string. After the metric name in {} are a set of labels which add dimensions to the histogram’s data (you can filter the values based on label criteria). Then you can route each label to different receivers. This multiple may also be a fraction. Mar 8, 2018 · Micrometer's tags are the same as Prometheus' keys/labels. Sep 24, 2019 · are created, and the labels added to the component are added as metric labels to the metrics. +)" Oct 2, 2021 · You can add external_labels to your prometheus. Exporting Prometheus metrics is quite straightforward, specially from a Go application - it is a Go project after all, as long as you know the basics of the process. Jun 28, 2017 · use prometheus feature flages --enable-feature=expand-external-labels. This documentation is open-source. register(); answered Jan 16, 2018 at 19:08. Base units. For example, we wanted to add a label "tenant" to the histogram labels so that we could show our users performance graphs of their own requests against our service. Still doesn't work. label_join: With this function, you can concatenate label Nov 5, 2020 · Sometimes it is needed to remove certain labels before storing metrics in Prometheus or VictoriaMetrics. // Prometheus will not allow you to ever export metrics with // inconsistent help strings or label names. May 10, 2020 · How can I configure prometheus. # By default, Prometheus stores its database in . test_value{asset_id="123"} 0. Grafana refers to such variables as template variables. The query works well for intervals less than 24 hours, but when I use a 24-hour or longer interval, the gauge keeps increasing. Jan 21, 2022 · 10. I am trying to achieve something like this: My current configuration looks like the following, but the pod URL is missing: metrics_path: /probe. Variables let you reuse a single dashboard for all your services. the helm chart's value. Hot Network Questions Why is the retention of ability to synthesize Jun 8, 2021 · IIUC, a solution would be to use one label to describe the application and another to represent the build number. The config file is reloaded on SIGHUP. Nov 6, 2023 · In this guide, we will explore the powerful feature of Prometheus relabeling, a technique central to refining data collection and enhancing monitoring efforts. Prometheus metrics are also to be created based on how many ever variables are there. tsdb. It provides metrics primitives to instrument code for monitoring. Metric names. c := &customMetricsRegistry{. For example, an API server job with four replicated instances: round(v instant-vector, to_nearest=1 scalar) rounds the sample values of all elements in v to the nearest integer. Then you can always query by the application value to see the stability value. 2. In this case you need to add the following relabeling rules into metric_relabel_configs section at the corresponding scrape_config: metric_relabel_configs: - source_labels: [__name__, name] regex: "node_systemd_unit_state;(. Then, we'll use the cluster label to group the jobs together. add_sample('prometheus_metric_name', value=float(0), labels May 6, 2019 · I have several metrics with the label "service". com. The optional to_nearest argument allows specifying the nearest multiple to which the sample values should be rounded. These will be the values of a grafana template variable. Try to keep values bounded to as small a set as possible. The Alertmanager uses labels to match alerts for silences and alert groups in notification policies. How can I use summary, or another dynamic way, for grouping alerts? Labels can cause some issues for Prometheus, such as high cardinality, label conflicts, and label misuse. How to concatenate labels in Prometheus relabel When using Prometheus' file-based service discovery mechanism, the Prometheus instance will listen for changes to the file and automatically update the scrape target list, without requiring an instance restart. record: <string>. target_label: customer. Object labels: __address__: "https://prometheus. However, are you using Grafana to watch your metrics ? If so, you can change the legend and print TOTAL instead of the metric name. Jul 1, 2022 · I'm using Prometheus with file_sd_configs sources. Just execute your alerting query directly in prometheus Jan 22, 2020 · There is a slightly more elegant way of doing this, that doesn't involve manipulating the address. What happened? Description. file=prometheus. The solution to this problem is to use an actual service discovery instead of static targets. Now let us expose our custom metric which is the last request time stamp. c = Counter('my_requests_total', 'HTTP Failures', ['method', 'endpoint']) To start Prometheus with your newly created configuration file, change to the directory containing the Prometheus binary and run: # Start Prometheus. Create YAML or JSON file : Folder Structure: # Prometheus Installed location. So now when you hit the metrics endpoint you will observe the newly created last_request_received_time metric. labels() method alone: from prometheus_client import Counter. May 29, 2019 · This stackoverflow question: Dynamic label values in Promethues alerting rules suggests that it might be possible to achieve what I want using recording rules, however following the pattern suggested in the only answer to this question results in recording rules that Prometheus doesn't seem to be able to parse: Feb 21, 2020 · You can notice that here we have labels allowing us to have a match between an instance IP address (10. yml config is below. 9. naming conventions, differently. Labels, along with dynamic labels. Instead you could add an additional label: labels: severity: critical. 4 for dashboarding and alerting. Before we start with our deep dive into relabeling metrics, let's first recap how labels work in Prometheus. # time series with the metric name as given by 'record'. In your case all those "labels" should be their own metrics/time series. Between two scalars, the behavior is obvious: they evaluate to another scalar Jun 8, 2021 · Dynamic label values in Promethues alerting rules. After checking the network response, I realized that For example, this expression returns the unused memory in MiB for every instance (on a fictional cluster scheduler exposing these metrics about the instances it runs): (instance_memory_limit_bytes - instance_memory_usage_bytes) / 1024 / 1024. You may provide the targets in a yaml or json format. So I have labels: envoy_cluster Jun 1, 2015 · Prometheus 0. But I cannot figure out how to access these values, or if it's The statsd_exporter can be configured to translate specific dot-separated StatsD metrics into labeled Prometheus metrics via a simple mapping language. If it is just docker-compose (I mean not Swarm), you can use DNS service discovery ( dns_sd_config) to obtain all IPs belonging to a service: Jan 6, 2023 · That's how Prometheus metrics work, you'll get a value per unique combination of labels. In the query options, I define the query as label_values (rt) but that gives the values of all labels irrespective of the cluster I chose in the first dropdown. There are several ways of providing dynamic targets to your Prometheus. OPEN QUESTION : Is it possible to change the value of a tag dynamically? So the 3rd and 4th label (or property) values change over time. If you have. For example, if you have 1,000 hosts in your environment it’s going to be just fine to have a host label with Aug 15, 2020 · Query = label_values(prometheus), here it means I want all the label values names prometheus(bad choice for federation though) Multi-value= Enabled, because it let the users select more than one value of the vars like “dev” + “qa” + “stg” + “prod” Custom all value= . relabel_configs: - action: labelmap. Labels that are available are the labels that are returned by the alerting query. Using this approach labels can be extracted from the filename. Between two scalars, the behavior is obvious: they evaluate to another scalar Oct 2, 2020 · Solution 3: Add functionality to Prometheus Remote Write Exporter. Jul 22, 2016 · Go to the "Queries" tab: Enter the name (here db2_prometheus_adapter_info) of the metric. 168. Step 3: Data plane pods to set up scraping. Unanswered. High cardinality occurs when there are too many unique combinations of labels and values Proposal Use case. but now I would like to label the instances with additional static labels, like the server1 and server2 with "database" and server3 with "mail" and maybe server1 additionally Prometheus collects and stores its metrics as time series data, i. For more elaborate overviews of Prometheus, see the resources linked from the media section. Aug 4, 2021 · Joins in PromQL. path). Aug 22, 2018 · Dynamic label values in Promethues alerting rules. A mapping definition starts with a line matching the StatsD metric in question, with * s acting as wildcards for each dot-separated metric component. You can have a label with your target that can be the source of your "relabel" action, something like this: The complete set of labels for an alert is what uniquely identifies an alert within Grafana alerts. For the second dropdown, I have variable service, type=Query. Please help improve it by filing issues or pull requests. Hi Team, I hope you are having a good weekend, I would like to have an information about CounterVec, is there a way to dynamically add Labels inside an already registered metric? For example: Creating a metric using two labels: status and environment. Jul 14, 2021 · Remember to check the names of your jobs, since each have their own labels that you’ll be able to use in Prometheus Query Language (PromQL) queries: For our examples, we'll use the instance as a unique label that will indicate the source of the metrics. The Prometheus model has four kinds of metrics: Counter: A cumulative metric that represents a single monotonically increasing counter whose value can only increase or be reset to zero on restart. A single query for metadata{} in table format has a column for the key and a column for the value, rather than Count and sum of observations. Above will create a Recording Rule job:availability:999 which will have no labels and will always yield 0. If you are familiar with Prometheus, the term used there is series; however, Prometheus has an additional dimension: metric name. But it is possible to add missing labels during queries with label_replace() function: label_replace( mongodb_exporter_last_scrape_duration_seconds, "cluster", "staging", "cluster", "" ) Add an extra dynamic label in the firing alerts #6496. example. The <type>MetricFamily class does the rest of the work. Teams are expected to operate their systems with I want to assign the values to variables and those to be assigned to prometheus metrics. This way Prometheus will scrape each replica during each iteration. Side note 2: Values are always floating-point numbers; timestamps are integers storing the number of milliseconds since the Unix epoch. QUESTION? How to query prometheus to have sum of “disk bytes read” by instance/node/server Aug 6, 2020 · Let’s imagine I have a metric called calories and two labels called type and name. First navigate to the directory containing the Node Jan 16, 2018 · 8. So if your query looks like: you will have no labels available. For you the code may like this: - job_name: 'development'. If you do: Then you'll get all the labels that are available on metric_name metric. Nov 15, 2017 · 13. And a few things that don't work. This is less critical for static labels. Prometheus should start up. Activate the "instant" option. update label in opentelemetry prometheus exporter. Feb 26, 2020 · If I'll simply define them via queries like label_values(controller) and label_values(action), they'll be independent, meaning that if controller"cont_a" is selected, all 4 actions will be available. Querying Prometheus. My desired output: metric = Metric('prometheus_metric_name', 'prometheus metric name', 'gauge') metric. expr: <string>. Use label as metric in Grafana Prometheus source. yml: serverFiles: prometheus. All other labels or annotations are user-defined and completely optional as they have sensible defaults. Here is my solution: use job_name as the group label. Jul 24, 2021 · I. I am a bit of a novice with the monitoring world. static_configs: - targets: - server1. For example, you might want to set the severity label for an alert based on the value of the query, or use the instance label from the query in a summary annotation so you know which server is experiencing high CPU Aug 28, 2020 · 5. Sub-packages allow to expose the registered metrics via HTTP (package promhttp) or push them to a Pushgateway (package push). Dec 4, 2017 · Prometheus doesn't provide the ability to add new labels to historical time series. metrics information is stored with the timestamp at which it was recorded, alongside optional key-value pairs called labels. Based on your description I'd suggest changing your metrics to Jan 17, 2019 · Prometheus allows renaming labels in the following places: During metric scrape. For example, you can use a counter to represent the number of requests served, tasks completed, or errors. So in order to collect metrics with Oct 24, 2023 · I’m using Prometheus as a data source, and I have a query to display the maximum latency within a given time interval. Note that the number of observations (showing up in Prometheus as a time series Jan 28, 2021 · But in the case where I want to add a dynamic list of jobs, like dummy2, dummy3, etc. file_sd_config is a more generic way to configure static targets. In this guide, you learned how the multi-target exporter pattern works, how to run a blackbox exporter with a customised module, and to configure Prometheus using relabeling to scrape metrics with prober labels. , a time series in Prometheus is defined by a combination of a metric name and a particular set of key-value labels. While the command-line flags configure immutable system parameters (such as storage locations, amount of data to keep on disk and in memory, etc. In Prometheus terms, an endpoint you can scrape is called an instance , usually corresponding to a single process. Could you please clarify your need, your programming language, and whether you are looking for a solution in the Prometheus server (that would be relabeling) or in client libraries (that would be declaring variables with labeled gauges/counters I think). Note on the "Instant" setting: This setting switches from a Mar 9, 2021 · The following promql queries with one on group filter (instance) and work as expected to produce a dynamic filter. Provide details and share your research! But avoid …. Prometheus provides a functional query language called PromQL (Prometheus Query Language) that lets the user select and aggregate time series data in real time. If you don't have access to promteheus config, or if you prefer doing it at query time for whatever reason, you can use label_replace to create the label in the Jul 21, 2022 · I have a Prometheus that collects statistics from envoy. Prometheus's main features are: May 2, 2020 · 2. static_configs: - targets: [ '192. . Note: value has changed from a label to an annotation. So you could avoid the first 'empty' metric declaration if you like and just keep the second Metrics. 1 What is list of Prometheus labels and variables that can be used in alert template. scrape_interval: 5s static_configs: - targets: ['localhost:8090','localhost:8080'] labels: group: 'dummy' . 1. io" Here are the main label functions in Prometheus: 1. The metric and label conventions presented in this document are not required for using Prometheus, but can serve as both a style-guide and a collection of best practices. Update the file whenever you want to add or replace the targets. You can find an example here. Prometheus labels marked with a star (*) are built-in and assignment to Alerta attributes happens automatically. Asking for help, clarification, or responding to other answers. Here is my question. Histograms and summaries both sample observations, typically request durations or response sizes. You cannot have two labels with the same key. help("All latencies. Oct 18, 2021 · roidelapluie commented on Nov 5, 2021. +" The following binary arithmetic operators exist in Prometheus: + (addition) - (subtraction) * (multiplication) / (division) % (modulo) ^ (power/exponentiation) Binary arithmetic operators are defined between scalar/scalar, vector/scalar, and vector/vector value pairs. Select the service you want to inspect from a drop-down menu, and watch panels update to only show you metrics from that service. type says whether we have fruit or vegetable, name is a name of the fruit or vegetable e. yml: global: # The labels to add to any time series or alerts when communicating with # external systems (federation, remote storage, Alertmanager). brian-brazil. Jul 24, 2020 · Each time Prometheus scrapes a histogram, it collects an instant vector like this one. carrot, onion, banana, orange, apple. They track the number of observations and the sum of the observed values, allowing you to calculate the average of the observed values. Package prometheus is the core instrumentation package. I have several booths (for testing, development and production) all running envoy. Labels. The syntax for recording rules is: # The name of the time series to output to. We don’t have perfect guidance as to what Loki can handle, but think single digits, or maybe 10’s of values for a dynamic label. Why is this important? we want to use the same metric, but use different tag in some case, for user or developer, because we have a lot of metric in our application, we cannot list them , after all, the metric is still The following binary arithmetic operators exist in Prometheus: + (addition) - (subtraction) * (multiplication) / (division) % (modulo) ^ (power/exponentiation) Binary arithmetic operators are defined between scalar/scalar, vector/scalar, and vector/vector value pairs. 1:9115 # Blackbox exporter address. Asked 5 years, 8 months ago. To demonstrate this, start up a second Node Exporter instance on port 9200. 1:9100' ] labels: Jun 23, 2019 · Prometheus metric with dynamic label value. - server2. Apr 14, 2018 · I have the same question before. params: module: [ping] kubernetes_sd_configs: - role: node. Golang prometheus show custom metrics. Remember to use continue: true for the pipeline to continue matching if you want the alert to reach multiple receivers. /data (flag --storage. Enter the label name as the legend using the {{[LABEL]}} notation (here {{app_state}} ). ") . Jul 20, 2020 · This way the set of labels to group by can be dynamic per alert since they are part of the summary. Configure the Prometheus to scrape the targets from a file. 001. Please refer the link here. Individual organizations may want to approach some of these practices, e. Must be a valid metric name. dwh: critical. Registry. Viewed 5k times. You can't change the label value of a PromQL query result. When querying the DNS-SRV records, a label named __meta_dns_name is attached to each target. externalLabels: prometheus_replica: $(POD_NAME) enableFeatures: - expand-external-labels. But I want to make the action variable dependent from the controller variable, so, that if controller "cont_a" is selected, then only values "act Summary. replacement: $1. I want to get a list of all the "service" levels that begin with "abc" and end with "xyz". This is that I have tried: label_values(service) =~ "abc. Prometheus template variables. Nov 26, 2019 · Grafana - create table with column values with Prometheus (dynamic) property/label data. 0. A collection of instances with the same purpose, a process replicated for scalability or reliability for example, is called a job. This can be done with action: labeldrop. vi hu tz pl ny py hb so gf or
Prometheus dynamic labels. Aug 4, 2021 · Joins in PromQL.