-
KISS methodologies for network management and anomaly detection
Authors:
Carlos Vega,
Javier Aracil,
Eduardo Magaña
Abstract:
Current networks are increasingly growing in size and complexity and so is the amount of monitoring data that they produce. As a result, network monitoring systems have to scale accordingly. As a possible approach, horizontal scalability can be achieved by large data centralization systems based on clusters, which are expensive and difficult to deploy in a real production scenario. In this paper w…
▽ More
Current networks are increasingly growing in size and complexity and so is the amount of monitoring data that they produce. As a result, network monitoring systems have to scale accordingly. As a possible approach, horizontal scalability can be achieved by large data centralization systems based on clusters, which are expensive and difficult to deploy in a real production scenario. In this paper we propose and evaluate a series of methodologies, deployed in real industrial production environments, for network monitoring and management, from the architecture design to the visualization system as well as for the anomaly detection methodologies, that intend to squeeze the vertical resources and overcome the difficulties of data collection and centralization.
△ Less
Submitted 6 February, 2019;
originally announced February 2019.
-
Evaluation of the performance challenges in automatic traffic report generation with huge data volumes
Authors:
Carlos Vega Moreno,
Eduardo Miravalls Sierra,
Guillermo Julián Moreno,
Jorge E. López de Vergara,
Eduardo Magaña,
Javier Aracil
Abstract:
In this paper we analyze the performance issues involved in the generation of auto- mated traffic reports for large IT infrastructures. Such reports allows the IT manager to proactively detect possible abnormal situations and roll out the corresponding cor- rective actions. With the ever-increasing bandwidth of current networks, the design of automated traffic report generation systems is very cha…
▽ More
In this paper we analyze the performance issues involved in the generation of auto- mated traffic reports for large IT infrastructures. Such reports allows the IT manager to proactively detect possible abnormal situations and roll out the corresponding cor- rective actions. With the ever-increasing bandwidth of current networks, the design of automated traffic report generation systems is very challenging. In a first step, the huge volumes of collected traffic are transformed into enriched flow records obtained from diverse collectors and dissectors. Then, such flow records, along with time series obtained from the raw traffic, are further processed to produce a usable report. As will be shown, the data volume in flow records is very large as well and requires careful selection of the Key Performance Indicators (KPIs) to be included in the report. In this regard, we discuss the use of high-level languages versus low- level approaches, in terms of speed and versatility. Furthermore, our design approach is targeted for rapid development in commodity hardware, which is essential to cost-effectively tackle demanding traffic analysis scenarios.
△ Less
Submitted 1 September, 2018;
originally announced September 2018.
-
Diluting the Scalability Boundaries: Exploring the Use of Disaggregated Architectures for High-Level Network Data Analysis
Authors:
Carlos Vega,
Jose Fernando Zazo,
Hugo Meyer,
Ferad Zyulkyarov,
Sergio Lopez Buedo,
Javier Aracil
Abstract:
Traditional data centers are designed with a rigid architecture of fit-for-purpose servers that provision resources beyond the average workload in order to deal with occasional peaks of data. Heterogeneous data centers are pushing towards more cost-efficient architectures with better resource provisioning. In this paper we study the feasibility of using disaggregated architectures for intensive da…
▽ More
Traditional data centers are designed with a rigid architecture of fit-for-purpose servers that provision resources beyond the average workload in order to deal with occasional peaks of data. Heterogeneous data centers are pushing towards more cost-efficient architectures with better resource provisioning. In this paper we study the feasibility of using disaggregated architectures for intensive data applications, in contrast to the monolithic approach of server-oriented architectures. Particularly, we have tested a proactive network analysis system in which the workload demands are highly variable. In the context of the dReDBox disaggregated architecture, the results show that the overhead caused by using remote memory resources is significant, between 66\% and 80\%, but we have also observed that the memory usage is one order of magnitude higher for the stress case with respect to average workloads. Therefore, dimensioning memory for the worst case in conventional systems will result in a notable waste of resources. Finally, we found that, for the selected use case, parallelism is limited by memory. Therefore, using a disaggregated architecture will allow for increased parallelism, which, at the same time, will mitigate the overhead caused by remote memory.
△ Less
Submitted 18 September, 2017;
originally announced September 2017.
-
Loginson: a transform and load system for very large scale log analysis in large IT infrastructures
Authors:
Carlos Vega,
Paula Roquero,
Rafael Leira,
Ivan Gonzalez,
Javier Aracil
Abstract:
Nowadays, most systems and applications produce log records that are useful for security and monitoring purposes such as debugging programming errors, checking system status, and detecting configuration problems or even attacks. To this end, a log repository becomes necessary whereby logs can be accessed and visualized in a timely manner. This paper presents Loginson, a high-performance log centra…
▽ More
Nowadays, most systems and applications produce log records that are useful for security and monitoring purposes such as debugging programming errors, checking system status, and detecting configuration problems or even attacks. To this end, a log repository becomes necessary whereby logs can be accessed and visualized in a timely manner. This paper presents Loginson, a high-performance log centralization system for large-scale log collection and processing in large IT infrastructures. Besides log collection, Loginson provides high-level analytics through a visual interface for the purpose of troubleshooting critical incidents. We note that Loginson outperforms all of the other log centralization solutions by taking full advantage of the vertical scalability, and therefore decreasing Capital Expenditure (CAPEX) and Operating Expense (OPEX) costs for deployment scenarios with a huge volume of log data.
△ Less
Submitted 7 March, 2017;
originally announced March 2017.
-
Multi-Gbps HTTP Traffic Analysis in Commodity Hardware Based on Local Knowledge of TCP Streams
Authors:
Carlos Vega,
Paula Roquero,
Javier Aracil
Abstract:
In this paper we propose and implement novel techniques for performance evaluation of web traffic (response time, response code, etc.), with no reassembly of the underlying TCP connection, which severely restricts the traffic analysis throughput. Furthermore, our proposed software for HTTP traffic analysis runs in standard hardware, which is very cost-effective. Besides, we present sub-TCP connect…
▽ More
In this paper we propose and implement novel techniques for performance evaluation of web traffic (response time, response code, etc.), with no reassembly of the underlying TCP connection, which severely restricts the traffic analysis throughput. Furthermore, our proposed software for HTTP traffic analysis runs in standard hardware, which is very cost-effective. Besides, we present sub-TCP connection load balancing techniques that significantly increase throughput at the expense of losing very few HTTP transactions. Such techniques provide performance evaluation statistics which are indistinguishable from the single-threaded alternative with full TCP connection reassembly.
△ Less
Submitted 17 January, 2017;
originally announced January 2017.
