Prometheus monitoring
1. About
The following components deploy a Grafana, Prometheus server, node exporters, and alert managers on the VMs:
- Grafana automatically adds Prometheus server as its data source.
- Prometheus server automatically scrapes metrics from new node exporters, sends alerts to alert manager endpoints, as well as monitors Grafana and alertmanager endpoints.
- Security groups for TCP connections between components are auto-created.
- All connections between Grafana, Prometheus server, and node exporters are TLS protected with basic authentication (self-signed certificate and auto-generated password).
tip
You can use the exiting template Prometheus Monitoring
to create the above topology.
2. How to use
2.1. How to scrape metrics from a node exporter
- Put the component PrometheusServer on a compute node, where you want to deploy the Prometheus server.
- Put the component NodeExporter on any compute nodes, where you want to scrape the metrics. The PrometheusServer and NodeExporter can be on different compute nodes.
- Connect the scrape_metrics_from_node_exporters (on the right of the PrometheusServer) to the scrape_endpoint (on the left of the NodeExporter).
Set a version (optional)
- To customize which Prometheus version to deploy, click on the PrometheusServer / Set the component_version property (e.g.,
2.27.0
)
Set the metrics (optional)
- To customize the exported metrics, click on the NodeExporter / Set enabled_collectors properties.
info
- By default, node exporters enable the following collectors.
- Set the disabled_collectors properties to disable the default ones.
2.2. How to add an Alertmanager to Prometheus
- Put the component AlertManager on a compute node, where you want to deploy the Alertmanager.
- Connect the add_alert_managers (on the right of PrometheusServer) to the alertmanager_endpoint (on the left of the AlertManager).
Set a root route
- The Alert Manager requires a root route set with a default receiver.
- To set the root route, click on the AlertManager / Set the Route properties (e.g., Set
slack
in theReceiver
field).
Set receivers
- To add a receiver, click the Receivers properties (e.g., Set
slack
as the receiverName
). - To add a slack receiver, click slack_configs and set the required fields
api_url
andchannel
.
- Alternatively, to add an email receiver (e.g., gmail) click the email_configs (and do not use the slack_configs). Here is an example with gmail:
info
- The fields
route
,slack_configs
, andemail_configs
are the same configs as in the Alert manager official documentation.
2.3. How to add the Grafana dashboard
- Put the component Grafana on a compute node, where you want to deploy the dashboard (e.g., we put it on a public compute so that we can access it via floating IP).
- Connect the add_datasource_prometheus (on the right of Grafana) to the prometheus_endpoint (on the left of the PrometheusServer).
Set the admin user (optional)
- To set the admin user (on first login), click on the Security properties / Set the
admin_user
andadmin_password
fields. By default, it is set toadmin
/admin
.
Set the TLS certificates (optional)
- By default, we protect Grafana endpoint with TLS using an auto-generated self-signed certificate.
- To provide your own certificate, set the Server properties / Set the fields
cert_key
andcert_file
to the corresponding paths on the VM.
2.4. Set output attributes
- (Optional) Tick the attributes
public_url
of the Grafana component.
3. Expected result
3.1. Access Grafana
- After the deployment completes, click on the output
public_url
to access Grafana via a browser.
- Use the Grafana admin credentials set above to access the dashboard (e.g.,
admin/admin
).
3.2. Show the Grafana datasource
- Under Data Sources / Prometheus, you can see that the Prometheus endpoint is added.
tip
Click the Test
button to check the connection between Grafana and Prometheus server.
3.3. Show the metrics in the dashboard
- You can add a new Dashboard and query metrics (e.g., show the metric
up
from a node exporter)
3.4. Show node exporter configs
All node exporters are auto-protected with TLS (using a self-signed certificate) and basic authentication:
3.5. Show Prometheus configs
Prometheus is auto-protected with TLS (using a self-signed certificate) and basic authentication:
Prometheus scrapes metrics from the node exporter:
It also scrapes metrics from Prometheus itself:
It monitors Grafana and Alertmanager endpoint as well:
Prometheus sends alerts to the Alertmanager endpoint:
3.6. Show Alert manager configs
Alertmanager is configured with the receiver and the root route slack
:
3.7. Security group notes
- The orchestration engine auto-generates the following security groups:
- Public access (
0.0.0.0/0
) to Grafana on port3000
. - Internal access from Grafana to Prometheus on port
9090
. - Internal access from Prometheus to node exporters on port
9100
. - Internal access from Prometheus to alert manager on port
9093
.
- Public access (