approach.md

Bench-Routes

19th August 2019

Language - Go

• Why Go ?

To do:

1. A mechanism to auto-search the routes supported in a particular port (or web app): Currently, the input of routes in an application is decided to be manual. However, in real-world scenarios, organisations have routes in numbers of hundreds, sometimes thousands. In such scenarios, manual input is not a feasible approach. Hence, an algorithm to automatize this process is highly in need.
2. Running netstat on host computers.

Outline

• Resources

• Terms

• Overview

• Goals

• Learning Objectives

• High-level Model

• Basic Idea Explanation

• Methods

• Detailed Proposal

  • Ping
  • Monitoring
  • Report Changes
  • Request-Response delay
  • Network Congestion / Bottlenecking

• Data Flow

• Implementation overview

• Queries

Resources

1. Golang

2. Beginner's friendly level (excluding goroutines)

3. Standard level (covers nearly everything)

4. Benchmarks of golang vs other languages

5. Native ping support Linux kernel

6. Jitter

7. Run go scripts on system Ctrl+C call (or SIGINT)

8. Socket package used in BR

Terms

1. Route

Routes are paths to which a server listens or responds to, whenever the client pings (or requests) on that address. It is easy to confuse the route with the address (or URL).

Example :

• http://168.22.25:8080/ add_events
• http://168.22.25:8080/ remove_events

Here,

IP : 192.168.22.252

Port : 8080

Routes : ['add_events' , 'remove_events']

2. Monitoring

Monitoring refers to the act of constantly checking and observing all the routes at particular intervals, in order to check the status of the routes. The status of the routes can be running, stop, error or warning. In case of error or warning, the admin will be notified to check for the error and the log details.

3. tsdb

A time-series database (TSDB) is a database optimized for time-stamped or time-series data. Time series data are simply measurements or events that are tracked time-series and monitored. In these types of models, the data points are saved in stamps of time rather than any ID which is generally the case of SQL and NoSQL databases.

Why do we need a time-series database?

In traditional databases, values are stored simply as rows in tables or as an object, in a document. However, the data values in this project need to be stored at every fraction of a second. The use of traditional data-extraction mechanisms aren't convenient when it comes to storing and retrieving large amounts of data since the use of traditional data models increases space and time complexity during the course of data extraction in sorted order.

In a tsdb, the data is stored at a particular timestamp(at a particular instant of time) in which each record can be represented as a hash block. This block can be stored in the form of a chain of events, thereby forming a blockchain. This chain will be highly useful in making graphs on the UI end, as the time complexity of the extraction of data points would be highly optimized (since the extraction of data would follow chain propagation).

4. Benchmark

In computing, a benchmark is an act of running a computer program, a set of programs, or other operations, in order to assess the relative performance of an object, system or entity, by running a series of standard tests which are generally based on computationally expensive algorithms. Benchmarking determines the performance of the software or device.

Overview

Monitoring has been tough and with the increase in the routes used in any sophisticated project, the performance and metrics of an application are seriously affected. With an increase in server computational models, the probability of a complete request-response cycle without any throws is nowhere close to 1.

bench-routes acts as a routes-benchmarking, monitoring, and route-network analysis tool. It monitors the routes of the application and analyses the network pipe between the server-client.

Goals

1. Benchmark route-

2. Load-handling of application on the individual route.

3. Test various possibilities of data in params (Permutate params), like sending an empty param to see how the server response behaves.

4. Analyse network performance of the hosted application irrespective of containerization- 2.a. Network ping 2.b. Jitter analysis
   2.c. Packet loss

5. Log error handling capability of the application

6. Maintain a check on server-route output and alert on changes above the threshold

7. Graphical view using ElectronJS

Learning Objectives

1. Analysis of network-based calculations
2. Intensive use of channels (golang)
3. Familiarity with monitoring concepts
4. Time-series database(tsdb) [basic as of now]
5. Strong hold on Golang
6. Extensive use of go-subroutines
7. Simple blockchain concepts

High-Level Model

Please see the Data Flow for a better understanding of the propagation of data from the above modules

Brief description:

docs: This directory would contain all the related documents to the project. It will also support the project documentation for general users.

examples: This would contain use-case examples of benchroutes. This may also include native CLI examples if we plan to have a CLI version as well.

scripts: All shell and bash scripts related to setting up, building, execution or testing would be present in this directory. This would ease the development process.

src: This is the main directory that contains the development code. Majority of the development process would take place here.

lib: This directory will be responsible for the core implementation of ideas in golang.

service: Or in other words, server. This directory would contain all server related stuff like routes, controllers etc. which would run as a daemon.

tsdb: This directory contains the implemention of the time-series database specific to bench-routes.

Basic Idea Explanation

bench-routes is a GUI-powered highly scalable testing tool which monitors the performance of the routes in any web/routing - application. Bench-routes would perform a series of networking algorithms and calculations involved to find the real-time state of routes in an application. It monitors the routes at regular intervals and analyzes the response and the time involved in the same. The moment, the delta in response rises above a threshold limit, an alert would be sent to the admin of the respective manager.

Bench-routes (B) follows a basic idea of pinging the server at a respective route.

Consider a server S that contains routes R respectively.

Let,

r belongs to any route in R

t be any time constant

tres be a threshold above which the delta in response would lead to an alert to the admin

rmean be the mean value of responses from r (where r < tres , since if r exceeds tres , an alert would be sent as an exceptional behaviour of that particular route)

B sends a request to S at r route in every t instant. B logs the response to any local storage in t intervals after each response is received.

Consider tres be 0.7 or 70% respectively. This means that for every consecutive response if the difference of the response length from the mean response length is >= tres , it would lead to alerting. rmean would update itself for every response < tres so that the rmean behaves ideally and updates itself with time, dynamic to changes.

Methodology

1. Linux ping command
2. min/avg/max/mdev ping values
3. Flood ping
4. Jitter
5. Monitoring
6. Response monitoring
7. Report response changes
8. Req res delay
9. Network congestion

Detailed Proposal

1. Linux ping command: All Unix based system supports basic ping command right from their shell. This can be made into use by running it through a sub-process and pipe out the necessary details of our needs and scrap them accordingly. However, it is important to note that, ping works on the raw IP as a ping from the hosted service which runs as default in any network card. Hence, this method is just to check the server response time, requests performance and could be helpful in checking bottlenecking states. Following ways can make this process more helpful:

   1. min/avg/max/mdev ping values : These are the general ping values after the ping subprocess would end its successful execution. This can be made to run in count mode so that only a particular number of pings are made to the mentioned IP.

   2. Flood ping : These would send a high number of requests (often in thousands or lakhs) to the IP route specified. This leads to more accurate results but requires sudo if the minimum interval is set to < 0.2 (default). Consider the example below, the count is set to 10,000 requests with the interval set as 0 (since not mentioned) and hence requires sudo.

   3. Jitter : Jitter is the amount of variation in latency/response time, in milliseconds. Reliable connections consistently report back the same latency over and over again. Lots of variation (or 'jitter') is an indication of problems. Example: consider 5 ping samples: 136, 184, 115, 148, 125. Then, jitter found in the sample can be calculated by averaging consecutive differences in the ping response time as follows: 136 to 184, diff = 48; 184 to 115, diff = 69; 115 to 148, diff = 33; 148 to 125, diff = 23; The total difference is 173 - so the jitter is 173 / 4, or 43.25.

2. Monitoring: In a world of complex and sophisticated servers with several concurrent routes, monitoring remains an issue especially with the multi-deployment instance, handled by Kubernetes. This part aims to monitor the specific route and to alert the user in case of a downstate.

3. Response Monitoring:

   1. Report changes : Extension of the above point, in which the response of the particular route would be saved as a time-series and a mean of the response length would be saved on each incoming request. On each response, the per-cent change in the response length would be compared to the mean length of the response of that route. If the change is beyond a threshold, an alert will be sent to the user, since a change beyond the threshold would indicate some error in the VM.
   2. Req-Res delay : The delay or increase on the req-res time would lead to more likeliness of a request being a timeout and hence a bad performance functionally and non-functionally as well. The report changes monitoring method can be integrated in this module since we are dealing with routes and requests, so there is no sense to create another module for the cause.

Note: We have to create a separate database to store the values of response length to reuse it next time the user starts bench-routes.

4. Network Congestion: Network Congestion is the situation that occurs when the entire bandwidth of the network is occupied. For better understanding, consider a water flow pipe. When the water in the pipe has the highest Reynold's number (in the situation of highest pressure output), the pipe can be said to be congested, i.e., any other stream of water cannot be passed through it. A network can be related in the same way. Hence, any request in a congested network will be bound to get slow and sometimes may lead to time-out, resulting in termination of operations.

Checking congestion in a network will be carried out by using netstat. The following is sample output from the netstat command.

Note the columns Recv-Q and Send-Q. These two are the most important. The numbers in the two columns represent the leftover packets. Let's understand what leftover packets are.

In any system, Recv-Q and Send-Q are sockets which are responsible for inbound and outbound connections. Consider them as buffers. Whenever any program has to send information to the external network (or internet), the information is broken down into small transferable packets that are atomic in nature. These packets are then sent to Send-Q buffer. In Send-Q buffer, the packets are queued and the initial link is established to the corresponding foreign address. Once the link is established, these packets are then transferred one after another (serially) and removed from the buffer. Naturally, those packets which could not be transferred to the foreign address (or external network) would be leftover in the buffer. This could be due to various reasons, the most common and important being the congestion of outgoing network bandwidth.

Similarly, when a packet is received by the system, it gets stored in the Recv-Q buffer. Once the storing of the batch of incoming packets is done, the requested listening programs have signalled the arrival of the requested packets. These programs then collect the required packets. Ideally, the programs should collect all the packets from the buffer which they have requested while sending from the Send-Q buffer unless some internal error has occurred in the program. This leads to leftover packets in the Recv-Q buffer and hence indicates some issue in the internal network (or local network or localhost).

Hence, it must be clear by now, that for the successful execution of a request-response cycle, the Recv-Q and Send-Q must show values of 0 (i.e., 0 leftover packets in the respective buffer).

[ANNOTATION]:

BY 'Harkishen Singh' ON '2019-09-19T15:41:44' NOTE: 'That might not be possible since alerting firewall rules from the third party is next to impossible. Think about windows systems. They have firewalls from companies like Kaspersky, which have very tight inbound and outbound rules. So, dynamic manipulation should not be the preferred way.' NOTE: '' NOTE: 'However, we can use one way. If we keep the values of Send-Q and recv-Q stored in the VM of the host machine and that could be retrieved from the current machine on a simple HTTP request, then the problem of firewall blockage could be solved.' NOTE: '' NOTE: '*@all* what are the views on this approach?']

[ANNOTATION]:

BY 'Ganesh Patro' ON '2019-09-16T18:33:41' NOTE: 'Guys, the firewall thing, we discussed a way around right, Harkishen can you brief that here, and yes might be a lame question but, can't we dynamically add the IP of the host (for making it pass the firewall) from which we are expecting the netstat results?'] Now about the implementation. This approach can be done on the host system only. Hence, we need to make an independent script that would be deployed on the host server and that would be listening on some port X. Bench Routes would ping X in regular intervals to collect data and analyze whether the server is in congested state of network or not. It is obvious to have doubt that if in case the network is already congested, the ping to X might time out. Yes, that's true. To avoid this situation, the request needs to be highly optimized. Moreover, this timeout will itself indicate the congestion and once the congestion improves, the stats or reason could be studied from the PID or PName (as in the event of failure, the benchroute would still continue to ping until a response is received. Once the response is received, the entire picture will become clear).

Data Flow

Please read this as a continuation of the High-Level Model_._

The flow of data right from the requesting the bench-routes service by the frontend UI until the final response to the frontend UI is as follows. (All lines indicate bi-directional propagation, and / indicates a directory as related with the High-Level Model)

Implementation overview

The implementation of the above-proposed idea would be in the Go programming language. This project will launch itself on boot and work as a daemon service. The benchroutes-UI, an electron app(would be later discussed), will be responsible as the graphical aspect of this project.

Queries

1. How will a user be able to provide the routes in the software to bench-routes?

• The user can provide the required details about the software i.e. the IP address and the routes in the GUI of bench-routes.

2. Does the user have to import the package in the software?

• No, as mentioned in the explanation, bench-routes is a GUI tool, the user has to provide the IP address and the routes of the server in the GUI interface.

3. Is there any automation for finding out routes, as many sophisticated applications contain millions of routes in a single web-application?

• We plan a native mini-package of every possible language which would contain a fixed hidden route. The programmer will have to pass an array of strings to the imported function. When bench-routes is initiated, as the IP address is provided, the service will automatically ping the fixed hidden route and get the list of routes available and initiate the testing and monitoring process on it.