Monitoring Apache webserver and MySql

By • On • In apache, php, proyectos

Groovy scrips

http://www.mkyong.com/java/how-to-run-a-task-periodically-in-java/

http://mrhaki.blogspot.com.es/2009/11/groovy-goodness-run-code-at-specified.html

https://gist.github.com/kdabir/3176945

http://groovy-almanac.org/example-timer-and-timertask-in-groovy/

http://crazy4groovy.blogspot.com.es/2013/01/simple-timer-timertask.html

 

Data storage

http://blog.mongodb.org/post/65517193370/schema-design-for-time-series-data-in-mongodb

  • https://github.com/Glavin001/Cobra
  • https://www.quora.com/What-is-the-best-way-to-store-time-series-data-in-MongoDB

https://www.mongodb.com/customers/server-density?_ga=1.111702214.317250607.1443618784

http://www.hindawi.com/journals/ijdsn/2013/867693/

https://rubayeet.wordpress.com/2013/12/29/web-analytics-using-mongodb-aggregation-framework/

MongoDB

http://snmaynard.com/2012/10/17/things-i-wish-i-knew-about-mongodb-a-year-ago/

https://blog.codecentric.de/en/2014/01/realtime-analytics-mongodb-nodejs-smoothiecharts/

http://s3.amazonaws.com/info-mongodb-com/10gen-MongoDB_Operations_Best_Practices.pdf

Cube

http://square.github.io/cube/

Groovy

https://blog.sagaoftherealms.net/?p=176

http://blog.mongodb.org/post/18510469058/grails-in-the-land-of-mongodb

Node.js

http://bigspaceship.github.io/blog/2014/05/14/how-to-create-a-rest-api-with-node-dot-js/

http://www.robinwieruch.de/node-js-scheduling-periodic-tasks-with-agenda/

https://github.com/mikejihbe/metrics

http://stackoverflow.com/questions/9565912/convert-the-output-of-os-cpus-in-node-js-to-percentage

https://www.npmjs.com/package/nsutil

Python:

https://github.com/hcooper/check_mk-plugins

https://github.com/monitisexchange

Apache

http://httpd.apache.org/docs/2.4/mod/mod_status.html

 

local.conf

<VirtualHost *:80>
	DocumentRoot "/var/www/html"
	ServerName local.tld
	ServerAlias *.local.tld
	<Directory "/var/www/html">
		Options All
		AllowOverride All
		#Order allow,deny
		#Allow from all
		# Apache 2.4
		Require all granted
	</Directory>
	<Location /server-status>
		SetHandler server-status
		Require ip 127.0.0.1
		Require ip ::1
		Require ip 192.168.0.114
		Require ip 192.168.0.135
	</Location>
</VirtualHost>

 

https://github.com/fr3nd/apache-top

https://exchange.nagios.org/directory/Plugins/Web-Servers/Apache/check_apache2-2Epy/details

https://github.com/hcooper/check_mk-plugins/blob/master/mod_status/mod_status-dump.py

http://www.wellho.net/mouth/3019_Apache-httpd-Server-Status-monitoring-your-server.html

https://github.com/dioubernardo/apacheServerStatusCharts

http://pastebin.com/9iGkjSTv

 

How it works

There is an Apache module (mod_status) responsible for presenting the current view of Apache key parameters and a snapshot of all its request handlers with their statuses. The following lines in Apache’s configuration file present an example of mod_status switched on:

<Location /server-status>
	SetHandler server-status
	Order deny,allow
	Deny from all
	Allow from 127.0.0.1
</Location>

The statistics are provided in the form of a webpage – you can have a look at such a page of the Apache Project itself here:http://www.apache.org/server-status. Anturis Apache Monitor is able to read and understand this web page (for example: http://127.0.0.1/server-status), and present it on the Anturis Console.

Currently Anturis Apache monitor collects the following metrics:

  • aggregate CPU and memory consumption of all Apache processes
  • number of busy workers
  • number of idle workers
  • average number of requests per second*
  • average number of Kbytes served per second*
  • average number of Kbytes per request*
  • CPU usage by all Apache workers*

*provided ExtendedStatus is switched on in Apache configuration.

 

Monitoring web traffic

A good starting point for troubleshooting website problems is to check whether traffic was or is unusual. An unexpected surge in traffic may be caused by many reasons varying from a DOS attack to a marketing campaign that a sys administrator was unaware of. Anturis Apache Monitor (when mod_status and ExtendedStatus is enabled in your Apache server configuration) provides the following information about the overall traffic:

  • number of requests per second
  • number of bytes served per second
  • average number of bytes per request

Comparing these values with historical data may reveal any abnormalities, such as unusually high or low traffic being processed.

ver: http://piwik.org/

Monitoring Apache processes and threads

Latest Apache versions implement hybrid multi-threaded and multi-process models to serve requests. It means that you will see multiple Apache processes running. Each process may contain multiple threads. The number of processes and the number of threads per process, as well as a forking policy and other parameters, are governed by Apache configuration directives. Each request processing thread is called aworker.

Apache self-regulates the number of workers and maintains a pool of idle workers so as to have some ready to process traffic at any time. Anturis Apache Monitor allows you to keep an eye on the number of busy and idle workers.

Watching the number of busy and idle workers is a proactive way to catch Apache configuration problems early enough. Anturis Apache Monitoring tool will let you discover, for example, these problems:

  • insufficient workers for the given traffic
  • too many workers – some processes get swapped out, with a heavy overall performance penalty
  • non-optimal forking policy or MPM model
  • stalling application code (e.g. due to external calls)
  • server resources insufficient for the given traffic

Monitoring Apache processes CPU and RAM consumption

The most important server characteristics in relation to a web server is the amount of RAM. A web server should never swap because swapping out a process will make request processing latency unacceptable. So it is important to understand the maximum number of workers that fit the RAM of your server and to adjust the MaxClients (or MaxRequestWorkers) Apache configuration directive accordingly. In order to do this you should observe how much RAM is consumed by how many Apache processes under normal conditions. Divide the first by the second, and compare it to your total available physical memory to understand how many Apache processes you can have on this server. The same procedure should be applied to the CPU, as your web application may be CPU-intensive.

Knowing the maximum number of Apache workers per server will in turn give you some insight into the traffic you can serve by the server. You can use this information to anticipate future upgrades of your infrastructure.

Anturis Apache Monitor collects data about Apache processes’ aggregate CPU and memory consumption. It will also alert you if the values exceed the desired limits.

Best practices: http://cdn.swcdn.net/creative/v14.9/pdf/techtips/1202_SAM_BestPracticesApache_TT.pdf

Mysql

http://jonathanhui.com/mysql-monitoring-performance

https://github.com/shoma/mysqlmonitor/blob/master/src/mysqlstatus.py

Get to grips with the Basics

Fundamental Server Resources

Monitoring of the underlying server’s resources is the basis of MySQL database monitoring. Server CPU and memory usage, free disk space, free swap space, and network usage are all important metrics and the perfect starting point for troubleshooting in case of problems. Anturis will let you know in advance about any overloads of these resources, giving you the chance to prevent negative impact on database operations. You will also be able to spot usage trends over time so as to get insights on when you need to replace/upgrade the hardware serving the MySQL database.

MySQL Availability

The next obvious thing to know is MySQL availability. Failure of MySQL usually makes reliant applications completely inoperable. Anturis alerts you immediately in case MySQL itself or the underlying server goes down.

MySQL Error Log

MySQL error log contains all the errors occurring while MySQL is running as well as some context information, like stack traces. Anturis will watch the log file for error entries and keep you informed when it finds them.

Key MySQL Server Metrics

There are two ways in which MySQL metrics are useful to you: error detection and trending. In the first case you are notified of hazardous conditions that require your attention. Such conditions are defined by the thresholds on the selected metrics. In the second case, you have access to the metric graphs and you are able to troubleshoot by looking for unexpected changes and unusual patterns, and analyzing their reasons.

Here are a few examples of MySQL key performance metrics available with Anturis MySQL monitor.

  • Slow query rate. Slow queries are eating the precious resources of the server, leading to degraded performance and application failures. Having a slow query rate that is too high indicates that it is a good time to look into the slow query log and check what’s going on.
  • Connections usage. If your MySQL server reaches the maximum number of allowed connections the new client connections will simply be refused. Running out of connections signals an abnormal situation, such as when configuration changes may be needed.
  • InnoDB statistics. This includes InnoDB buffer pool usage, its’ miss and wait rates, and the InnoDB log cache wait rate. Using this data you are able to understand whether or not InnoDB performance is optimal.
  • MySQL query and key cache statistics. A low cache hit rate signifies that cache configuration is misaligned with application’s needs.

 

Tomcat

https://github.com/msacks/jythonJMXClient/blob/master/jythonJmxClient.py

 

Visualization

http://www.citronlab.com/programacion/javascript-graficos-con-morris-js/

https://anmolkoul.wordpress.com/2015/06/05/interactive-data-visualization-using-d3-js-dc-js-nodejs-and-mongodb/

https://github.com/worldline/beekeeper

http://www.grepstar.org/iot-recipe-1-yun-mongodb-d3js/

http://adilmoujahid.com/posts/2015/01/interactive-data-visualization-d3-dc-python-mongodb/

https://github.com/StackStorm/collectd3

https://www.digitalocean.com/community/tutorials/an-introduction-to-tracking-statistics-with-graphite-statsd-and-collectd

Real Time

https://github.com/leggetter/socketalytics

Comments are closed.