Monitor a RisingWave cluster

If you run RisingWave in a Kubernetes cluster, you can monitor its performance, including the usage of resources like CPU, memory, and network, and the status of different nodes.

Install the monitoring stack

To monitor a RisingWave cluster, you can use the monitoring stack in the RisingWave-Operator repository. The stack includes Prometheus for collecting data, Grafana for visualization and alerting, and other necessary resources.

Clone or download the RisingWave-Operator repository to your environment.

git clone https://github.com/risingwavelabs/risingwave-operator.git

Run the script to install the monitoring stack. It will create the monitoring namespace and deploy all monitoring services in the namespace.

./monitoring/install.sh

Start monitoring

The RisingWave operator is now integrated with the Prometheus operator, which creates a service monitor for RisingWave and ensures that data is collected automatically.

1. Check if Prometheus is collecting data from RisingWave:

kubectl get servicemonitors -A -l risingwave/name

The expected output is like this:

NAMESPACE    NAME                         AGE
monitoring   risingwave-service-monitor   119m

2. Forward the web port of Grafana to localhost.

kubectl port-forward -n monitoring svc/prometheus-grafana 3000:http-web

The command above allows you to access Grafana from the local machine only. If you need to access Grafana from a different host, append --address 0.0.0.0 to the command to allow Grafana to accept requests from external hosts. The complete command is:

kubectl port-forward -n monitoring svc/prometheus-grafana 3000:http-web --address 0.0.0.0

3. Go to http://localhost:3000 if you access Grafana from a local machine, or http://<client_address>:3000 if you access Grafana from a different host, where <client_address> is the IP address of the machine running the Grafana service. When prompted, enter the default credentials (username: admin; password: prom-operator).
4. From Dashboards, select RisingWave -> RisingWave Dashboard. You can edit metrics in the dashboard and add new metrics.

Grafana dashboard for multi-cluster

If you have multiple RisingWave clusters deployed in a Kubernetes cluster, you need to generate a Grafana dashboard for the multi-cluster scenario.

The following steps can be run in any machine, it is only for generating the Grafana dashboard JSON model.

1. Clone the risingwave repository.

git clone https://github.com/risingwavelabs/risingwave.git

2. Follow the instruction to set up the toolchain.

3. Run the following command to generate the Grafana dashboard JSON model.

DASHBOARD_NAMESPACE_FILTER_ENABLED=true \
DASHBOARD_RISINGWAVE_NAME_FILTER_ENABLED=true \
DASHBOARD_DYNAMIC_SOURCE=true \
DASHBOARD_UID=risingwave_dashboard \
./generate.sh

4. Import the generated JSON models (*.gen.json) into Grafana.

Troubleshooting

If the namespace filter is not working, it is probably because the metrics collected does not contain the label risingwave_name. In that case, you need to use metricRelabelings and relabelings fields in the endpoint spec to add this label to the metrics.