In my first post, I installed the ESXi Arm Edition on a Raspberry Pi. This one is dedicated to a conceptual prototype enabling an end to end test. I focus on application components with minimal programming needs and adapted to my home lab capabilities. Hence, I have to leave NSX-T and Tanzu/K8s as well as the automation for following blog posts.
The Big Picture
This diagram will help you to keep the oversight. I will explain the different components from left to right. In this first version the first 3 tiers are all on the same physical network and only between my homelab and the public cloud I implemented a secure access by Inlets Pro.
The IoT Device (raspberrypi)
I got a Raspberry Pi 3B last year for my engagement as vExpert. I already did a lot of experiments running it with Raspbian. Luckily my 14 year old son had to spend a day with me for his search for a suitable career. I let him chose one sample from the tutorial of my sensor kit, he took the ColorfulSoftlight:
3 rotary potentiometers enable the user to set the RGB value that is sent to a LED (right top corner). It is controlled by a python loop that continuously reads the potentiometers and writes the values down to the interfaces of the LED. This raspberry Pi device is powered by a USB connection to the laptop and accessed over a VNC viewer over WLAN.
Having successfully assembled and tested everything, I gave him some commands to send the values to a REST endpoint running on the next tier. Later on, I integrated a very basic upgrade mechanism to replace the python loop with a newer version.
Please find the code for the IoT device here.
curl -SL https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-armv7.tar.gz > node_exporter.tar.gz && sudo tar -xvf node_exporter.tar.gz -C /usr/local/bin/ --strip-components=1
I added the service unit files to the repository as well.
The Far Edge Device (ubuntu-far-edge)
For this PoC I created an Ubuntu 20.10 (marked in yellow), installed java and the node exporter for monitoring. I created a Spring Boot REST Application with this controller. The first method, postValues accepts the RGB values from the IoT device, /metrics is delivering them in a prometheus metrics format and the last two are for uploading / upgrading the python file on the IoT device. It is polling every 5 minutes for a new version with the “If-Modified-Since” HTTP mechanism.
sudo apt install openjdk-14-jre-headless curl -SL https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-amd64.tar.gz > node_exporter.tar.gz && sudo tar -xvf node_exporter.tar.gz -C /usr/local/bin/ --strip-components=1 sudo adduser softlight
Now just build the maven project and copy the
on the target machine. The service unit files are contained in the git repo. If you do not want to use this user you might have to adapt the application.properties in the maven project.
On Prem Component (ubuntu-srv-1)
In my home lab installation ( 4 x NUCs, vCenter 7 ) I have another Ubuntu 20.10 (marked in green) server running. I installed Prometheus, the node exporter and Inlets Pro to create a safe websockets tunnel from Azure onto my on prem server component.
Node exporter installation is more of the same. Inlets will be covered in a separate paragraph. So here how to install Prometheus:
wget https://github.com/prometheus/prometheus/releases/download/v2.22.0/prometheus-2.22.0.linux-amd64.tar.gz tar xvf prometheus-2.22.0.linux-amd64.tar.gz mv prometheus-2.22.0.linux-amd64 /usr/lib chown prometheus:prometheus -R /usr/lib/prometheus-2.22.0.linux-amd64/ sudo nano /usr/lib/prometheus-2.22.0.linux-amd64/prometheus.yml #add scraping targets: static_configs: - targets: - ubuntu-srv-1.ne.local:9090 - ubuntu-far-edge.ne.local:8080 - raspberrypi.ne.local:9100 - ubuntu-far-edge.ne.local:9100 - ubuntu-srv-1.ne.local:9100
9100 -> these are the node exporters for each of the 3 components, 9090 is monitoring Prometheus itself and 8080 is the Spring Boot /metrics endpoint. You can test the installation by calling <yourserver>:9090:
Just use scrape_samples_scraped and you can check if the configured endpoints are running, if you have 0 scrapes, something is not working, yet. In this case I have stopped the IoT device.
Public Cloud Component (grafana)
On an Ubuntu VM on Azure I manually installed a grafana instance and set up the Inlets server component. Here the script that installs grafana:
sudo apt-get install -y adduser libfontconfig1 wget https://dl.grafana.com/oss/release/grafana_7.2.2_amd64.deb sudo dpkg -i grafana_7.2.2_amd64.deb sudo systemctl enable grafana-server sudo systemctl start grafana-server
After setting up the Inlets Pro tunnel I finally can see the metrics published in the internet:
That’s a dashboard that I set up myself, it is showing the values of the 3 rotary potentiometers with a minimal refresh rate of 5 seconds.
Question to the grafana cracks out there: how is it possible to create a pane with the chosen RGB color?
This is the Node Exporter Dashboard Full that can be installed simply by importing it by ID. The most important metrics of a VM or physical machine will be collected by the node exporter.
Grafana in the public cloud is consuming the data of the Prometheus instance in my home lab. To allow this connection it is possible to create an ssh tunnel or take the Inlets framework.
Inlets Pro Tunnel
intletsctl makes it very easy to install everything that you need:
#install inletsctl curl -sLSf https://inletsctl.inlets.dev | sudo sh #use inletsctl to download and install Inlets Pro sudo inletsctl download --pro #create a inlets server on the cloud (port sudo inlets-pro server --auto-tls --common-name grafana.niceneasy.ch --token <your token>
The inlets server (public cloud component) is listening on port 8123 hence you have to open the port to allow connections:
If you have a running server and the trial license from Alex Ellis, you can easily setup the tunnel on the other side, in this case on the ubuntu-srv-1 in my home lab:
inlets-pro client --url "wss://grafana.niceneasy.ch:8123/connect" --token "<your-token>" --license "<your-license>" --upstream "localhost" --ports "9090"
Done. Now you can configure the Prometheus datasource for Grafana on http://localhost:9090 and this will be routed to your on prem machine. For more please see Inlets documentation.
For the next iteration I’d like to do a similar setup on Kubernetes on every layer and to work on the automation of all the components. I have neglected networking and security and need to find good solutions for it.
Kudos to Alex Ellis, he is always an inspiration!