Achieving Observability on Fog Computing with the use of open-source tools

A smart city application is software developed to help manage and improve the quality of life in a city. These applications take advantage of advanced technologies such as sensors, IoT devices, 5G networks, and artificial intelligence to collect and analyse real-time data on different aspects of the city, including public safety, traffic, energy consumption, air quality, and more [21]. Mobile IoT-RoadBot [18] is a Smart City application that monitors and detects roadside asset maintenance issues. Furthermore, it is used as a real-world testbed to assess 5G performance. It has been deployed on 11 waste collection service trucks (WCST) in Brimbank City Council, located within the metropolitan area of Melbourne, Australia. This testbed provided an ideal real-world deployment scenario to assess the performance of the 5G network due to the natural mobility of the trucks. It allowed the performance of a 5G network to be tested using an application that streams a great amount of real-time video data across a large geographical area.

The 11 WCSTs have been equipped with IoT devices, including depth-sensing stereovision cameras, 5G routers, 5G dome antennas, edge computers, and Global Navigation Satellite Systems (GNSS). Mobile IoT-RoadBot collects footage from the roadside and transmits data continuously while being in service for garbage collection around Brimbank. The application must process the video on the edge to shrink its size and aggregate context data. After this, the data are sent to the cloud to allow server-side functionalities, such as recognition of roadside assets and integration into the maintenance dashboard. WCSTs use 5G technology, but the availability of 5G networks is not ubiquitous in the region. Therefore, sometimes there will be no available connection to send data to the server, and the videos must be stored for further transmission as soon as the network becomes available. Additionally, there are other issues that can cause data flow interruption between WCSTs and servers, such as lack of space to store videos, camera malfunction, and application crash. In such cases, the delay in identifying such problems may cause data loss, money waste, and low-quality service to the citizen.

It would be beneficial for the Smart City IoT application if such issues were identified quickly and automatically. A solution could be delivered faster and the system could return to its normal state, potentially actuating autonomically [17]. For example, after identifying that the application is stuck, i.e. hardware resources and network are available and healthy, but the application is not collecting roadside footage nor transmitting it, it could send a remote command to restart the application and verify after a short period of time whether a different actuation is still necessary. Thus, to provide a comprehensive and updated view of the whole system, it is necessary to increase its observability by promptly collecting the status of each component (sensors, hardware, network, application modules). These data can be processed to provide triggers for actuation, according to pre-defined thresholds and rules. Additionally, the data will be available for historical analysis.

In our previous work [12], we conducted a systematic review of the literature on fog monitoring solutions. We have found that no available proposal was capable of properly managing the collection of metrics, logs, and traces simultaneously in a Fog Computing environment. This work provides a clarification of the challenges of increasing the observability of the application in this scenario. The contributions of this work are: 1. The definition of observability in a Fog Computing environment; 2. Definition of a Fog Observability Data Life Cycle and 3. An empirical evaluation of delivering observability in a real world Fog Computing environment using a smart city use case.

This work is organised as follows. Section 2 summarises the characteristics of Fog Computing and presents a definition of observability for distributed systems. The most relevant related work is provided in Section 3. Section 4 presents a definition of Fog Observability based on a specific data life cycle, and modelling its overhead. A real-world Fog Computing testbed is detailed in Section 5. Section 6 describes the experiments and offers a discussion of the evaluation results. Finally, Section 7 concludes this work.

To obtain valuable information from each instrumentation domain and to increase the observability of an application running in a Fog environment, it is necessary to be aware of the following six-step Observability Data Life Cycle, depicted in Figure 2: 1. Collection; 2. IoT storage; 3. Transmission of data to the Fog; 4. Fog storage; 5. Data analysis and visualisation; 6.Cloud storage and analysis. The first three Steps make up the Data Collection phase of the life cycle. The last three Steps form the Data Analysis phase.

1. Collection - In the initial Step of the fog observability data life cycle, the data are collected. This can happen in a multitude of ways depending on the instrumentation domain in place. Metrics can be acquired from the operating system by means of system calls. Logs are written according to the specific event flow that was instrumented to be recorded in text. When previously instrumented, traces can be created by specific API calls that record the sequence and delay of each service call. 2. IoT Storage - data staging in the device awaiting transmission - Observability data are usually immutable and append-heavy [31]. In order to avoid running out of storage resources, a data removal policy should be in place. The period of time that a device can handle stored observability data will depend on several factors, such as data footprint by period of time, the frequency of generation, and the available storage space reserved for the system. Although metrics can be stable in terms of data volume, logs and traces have greater variability [32]. 3. Data transmission to Fog - Observability may allow timely and proper decision making. Although it is possible to make some minor decisions locally using a single device, critical decisions are expected to be made using a process that can assess a higher volume of data that came from different subcomponents of the system, granting a more comprehensive view of the system. Therefore, the data collected from the IoT layer should be transmitted to the Fog Layer, where a resource-richer node will store them and allow for a more comprehensive data analysis. The network connections used by the application to receive and respond to user requests may be the same as those used by the observability data flow. An adaptive process may be in place to define the amount of data that can be transferred from the devices, selecting which instrumentation domains will be included in each transmission, and the period of time to which the collected data will refer. 4. Fog Storage - Specialised pre-processing and storage according to the type of data and usage - Fog nodes are expected to be resource richer compared to IoT devices [4]. Due to this, it is on the Fog Layer where observability data from several IoT devices are stored with the aim of rapid actuation and decision-making. The metrics should be stored in a time series database (TSDB). However, logs and traces are structured differently and will benefit from other storage solutions, such as inverted index-based storage, due to the type of queries that are usually made to retrieve meaningful information from them [32]. Therefore, an observability data ingestion service on the fog should consider the data requirements that each instrumentation domain needs (see Table 1), while allowing cross-analysis to be performed. 5. Data analysis and visualisation for decision making - Once the observability data are available on the Fog, it is possible to query them and make decisions and actuations accordingly. Observability data tend to give more relevant answers when they are queried as soon as they arrive, which means that most queries and analysis use more recent data (less than 24 hours) [31]. Thus, it is important to guarantee fast access to this time window data. In addition to that, to save resources to continue receiving IoT data, it is important to provide automated mechanisms to send the data out of this range to long-term storage in the Cloud. 6. Cloud Storage - Long-term storage and historical analysis - Cloud is the appropriate environment to store large data volumes and run heavy data processing models, such as historical analysis of observability data [23].

Observability is not an end in itself. The positive outcome that an increased observability level can deliver must be balanced with the amount of computational resource needed (the overhead) to reach that level. Although obvious in a generic sense, this statement gains more relevance in a Fog Computing environment due to the characteristics of resource restriction and network uncertainty. This outcome can be modelled as in Equation 1:

where Overhead is a non-zero number between 0 and 100, representing the percentage of resources (i.e. CPU, memory, network bandwidth) consumed when running the life cycle described in Section 4.1. The higher the overhead, the lower could be the Outcome, considering that resource restriction is a fog characteristic.

The observability of a system is directly related to the cardinality of ID ($|ID|$) in addition to the synergistic interactions $X(ID)$, as seen in Section 2. Having $ID=\{ID_{Met},ID_{Log},ID_{Tra}\}$ means that the ODLC is capable of managing metrics, logs, and traces throughout the system. Due to connectivity uncertainty and resource restriction of Fog environments, part of the observability data set might be lost when there are no available resources such as storage, network bandwidth, device battery, etc. To cope with this, it is useful to define weights ($W_{i}$) for each instrumentation domain, representing the importance of that domain compared to the others in a given moment. As a data property, it is important to ensure that the overall sum of required weights is equal to 100%, so $\sum{W_{i}}=1$. These weights could be used to make adaptive decisions on which domain data will be collected or transmitted in the event that resources run out or in other specific situations. For example, a weight of 0% can mean that data from that domain should not be managed. Values above 0% will determine the rank of priorities if necessary. The weights may vary according to the needs, such as being in a regular system operation or after a massive error has occurred.

Using the observability definition of Section 2 in Equation 1, but replacing $ID$ by the weight set $W=\{W_{Met},W_{Log},W_{Tra}\}$. Splitting the resultant equation to isolate each instrumentation domain with its intrinsic overhead, and also isolating cross-analysis among the domains, the result is as follows:

When running the observability data life cycle, the overhead of each instrumentation domain, namely, $Over_{Met}$, $Over_{Log}$ and $Over_{Tra}$ for metrics, logs, and traces, respectively, can be modelled as a function of the amount of resources consumed to generate, store, and transmit the information for that domain. A suggested model is to assign the maximum percentage of resource consumption among key hardware resources, such as CPU, memory, and network bandwidth, as in $Over_{Met}=\max\{\%CPU_{Met},\%Mem_{Met},\%Net_{Met}\}$ * 100. For simplicity, we consider the first 3 factors of Equation 2 together as representing the data collection phase. The overhead of the synergistic interactions $Over_{X(ID)}$ can be modelled in the same way, but with respect to resources consumed to store and analyse the data in the fog, and transmit them to the Cloud (Steps 4 to 6 of ODLC).

In a dynamic Fog Computing environment, with resource restriction and network uncertainty, decision making can be more effective using a formula such as Equation 2 that balances the relative importance of each instrumentation domain with the effort required to collect and analyse it.

To assess the overhead of achieving a higher level of observability in a Fog environment, the testbed was used to measure: 1. The overhead of each tool in the IoT devices; 2. The overhead of the server component of each tool running on a fog node; 3. Volume of observability data collected, stored, and transmitted through the data life cycle. While Mobile IoT-RoadBot was running on a Raspberry Pi, we deployed each agent on the same device, and measured the overhead in terms of CPU and memory usage for 4 hours. We use the System Activity Reporter (SAR) [46] to get the average CPU and memory usage each 5 minutes. After recording this information, we uninstalled the tool and repeated the process with the other tools (Node Exporter, Filebeat, and OpenTelemetry SDK, described in Section 3). We performed a similar procedure related to the server side of the tools (Prometheus, ElasticSearch, and Jaeger, described in Section 3). After measuring the overhead, we run Mobile IoT-RoadBot for 9 hours (a regular whole-day journey for each truck) to assess the volume of data collected by each instrumentation domain and transmitted through the layers of the testbed architecture. Finally, to assess the benefits of achieving a higher level of observability on Fog environments, we replayed by 9 hours on the testbed the real-world output of 4 WCSTs. Each real-world WCST output is replayed by a Raspberry Pi that also runs observability agents sending data through ODLC from the IoT Devices to the Cloud server.

Figure 4 shows the CPU and memory usage of each component of the open source observability tool set being evaluated in this work. Figures 4(a) and 4(b) show the overhead of the IoT devices. We observe negligible CPU and memory overhead. The aggregated amount of CPU when all three components, i.e. NodeExporter, Filebeat, and OpenTelemetry SDK, run simultaneously is under 12% on average. In the case of memory usage, an aggregated footprint of less than 150MiB of RAM is required.

Figures 4(c) and 4(d) show the overhead on the Fog node. Unlike the IoT devices, CPU usage is higher. This is expected since the Fog node deals with 4 times the volume of data (sent by the 4 IoT devices), with the purpose of receiving, processing and storage of data. A CPU usage average of less than 25% for this kind of processing seems to be worth it. However, it prevents the server side of the observability tool set from being on the IoT layer, where devices have less resources. To guarantee steady performance, the volume of stored data was limited to a week. In terms of memory usage on fog nodes, Prometheus allocated 400 MiB of RAM on average, while Jaeger allocated around 200MiB, a very low memory footprint for the load of Mobile IoT-RoadBot observability data. However, ElasticSearch allocated almost 4.5GB of RAM. Mobile IoT-RoadBot has a steady data collection flow and usually does not generate peaks of transmitted data. However, when dealing with a more data-intensive application or an application that has a bursty behaviour, the server side overhead should be monitored to guarantee that it copes with the needed load.

In the last subsection, the overhead added to the Fog infrastructure after deploying open source observability tools that implement an ODLC (Section 4.1) was detailed. This subsection will show the benefits that Mobile IoT-RoadBot could have if it was using such a set of observability tools.

Figure 5 shows how the observability data can be analysed on the Fog using the open source framework deployed on the testbed. Figure 5(a) shows the value of some metrics (CPU, memory and bandwidth usage) collected by the experiments in the last hour from IoT devices. If any metric is outside the range considered safe, an alert message could be sent to the maintenance crew, allowing quick action. Figure 5(b) shows the visualisation of logs collected from IoT devices each 1 second. The logs can be easily preprocessed while being ingested to identify different fields of information, providing fast queries and allowing for alert management. Figure 5(b) shows the result of a query made on log data, detailing the latency between the IoT device and some servers of interest on the Internet. Figure 5(c) shows a dependency graph, created from the collected traces, where one can see the average delay of each dependent component after hundreds of requests and identify which cause the majority of the response time. This information is relevant for planning future performance improvements. Finally, Figure 5(d) shows the details of a specific trace, where it is possible to identify the components that cause the longest response time. When looking for the root cause of an identified issue, this information is very useful.

Table 4 presents the volume of observability data that were managed by the data life cycle during the experiments. NodeExporter was deployed with the default configuration. Although it is a tool with a small footprint in terms of CPU and memory usage [22], it may have a not negligible impact in terms of the volume of data it collects. The default set of metrics that it exposes accounts for 65KB of information. These data are presented only when Prometheus pulls them using an HTTP call. This means that there is no IoT storage for these data. Prometheus is configured by default to scrape the NodeExporter page every 5s, getting all the metrics exposed and storing them in its TSDB on the Fog node. Considering that there are four IoT devices exposing metrics, the data volume transmitted to and stored on the Fog node is about 8.75GB in the span of a week, the period when these data will be available for decision-making and other analysis on the Fog Layer. After reaching a week of age, the information is removed from the fog node and sent to the Cloud for long-term storage and historical analysis. As a matter of estimation, the volume on the Cloud will reach 75GB after 2 months of operation.

The default output from NodeExporter provides help text for each metric, as shown in Figure 6. This information accounts for at least 20% of the total output footprint and should be removed prior to exposing the metrics. The default set of metrics is very extensive and probably not all metrics are useful for every use case. For example, the node exporter exposes dozens of Go environment metrics (Figure 6) that are not of interest for the monitoring of Mobile IoT- RoadBot and should be removed. In addition to cutting off metrics that are not of interest, machine learning over historical data can be used to figure out metric correlations and keep the target metric set at minimum [5]. Furthermore, the frequency of scraping can be decreased in the Prometheus configuration without a relevant loss of opportunities for actuation. Increasing the scrap delay to 10 seconds will reduce the data volume transmitted to and stored on the Fog Layer by half. Using the strategies of removing the help text and changing the configuration of Node Exporter to expose only metrics about CPU, memory, disk, network, and power supply, and increasing the scrap delay to 10 seconds on Prometheus, we could reduce the volume of metric data on the Fog node by 87%, which also positively affected CPU and memory usage by Prometheus.

Regarding the logs generated by Mobile IoT-Roadbot while the trucks were moving around the city, they record information about 5G network analysis, such as latency and throughput, and contextual information (GNSS coordinates, truck speed, etc.). Although the application writes information in the logs every second, the volume of data written is low (0.67 GB, in the span of a week, as seen in Table 4), being smaller than the volume of data generated by Node Exporter after applying volume reduction strategies. Filebeat was configured to harvest only the logs written by Mobile IoT-RoadBot and transmit them to the Fog Layer. It was necessary to change the default configuration of Filebeat to turn off the auto-discovery feature. When this feature is active, Filebeat receives from the Docker manager every status change of any container on the device, consuming more memory than necessary. As the application was not originally instrumented to record trace calls, we instrumented it in a reporting feature, utilised to aggregate 5G data by the suburbs of Brimbank. To make this aggregation, geographic coordinates were used to find the full address of Australia using a service called MapBox [35]. Using the data footprint of these traces, we estimate the data volume to generate the traces of regular operation of the Mobile IoT-RoadBot. This use case does not have bursty behaviour in terms of request processing because it performs the same volume of operations while in service. Therefore, the volume of trace data is steady.

The aggregated data volume, collected by the four IoT devices, transmitted and stored on the fog node for a period of one week was approximately 10GB, considering the default configuration of the open source tools used as shown in Table 4. Using the strategies described above, the volume of aggregated data was reduced to 2GB, a reduction of 80%. The four trucks whose observability data are replayed by the IoT devices in this experiment transmitted 291 GB of video data using the 5G network in a week of real world operation. Therefore, the observability data (2GB) would represent an overhead of less than 1% in this use case. The experiments show that it is possible to collect the benefits of achieving a higher level of observability for a system in a Fog computing environment. In addition, the overhead of deploying an observability data life cycle can be low, if properly managed. The utilisation of Docker containers as the runtime environment for the observability tools help to address the Fog challenge of device heterogeneity. Due to the resource restriction of IoT devices, observability data collection should be done by lightweight agents. In addition, the data footprint should be minimised to reduce the risk of network congestion and increased overhead to collect and transmit the data to the Fog Layer. Each instrumentation domain has specific data requirements (Table 1) that must be met to optimise storage and minimise the average delay in analysing the observability data in the Fog Layer for decision-making and actuation. Leaving in the Fog Layer only a window of most recent observability data is another strategy to cope with the resource-restriction of fog nodes. Data that are outside the age range are sent to the Cloud for long-term storage and historical analysis.

The open source tools selected to make up the experimental setup are managed independently. This scenario makes more complex actuation difficult to implement. For instance, in the dynamic environment of Fog Computing, a system may present errors running on specific devices while it is functioning properly on others. In such cases, if there are not enough resources to transmit all observability data to the Fog Layer, a proper decision should be prioritising data from those specific devices and returning to regular operation when the issue is solved. To implement such adaptive and autonomous behaviour, it might be necessary to orchestrate the observability data life cycle and its agents. To our knowledge, there is no Fog solution in the literature that provides this functionality [12].