Groovy scrips
http://www.mkyong.com/java/how-to-run-a-task-periodically-in-java/
http://mrhaki.blogspot.com.es/2009/11/groovy-goodness-run-code-at-specified.html
https://gist.github.com/kdabir/3176945
http://groovy-almanac.org/example-timer-and-timertask-in-groovy/
http://crazy4groovy.blogspot.com.es/2013/01/simple-timer-timertask.html
Data storage
http://blog.mongodb.org/post/65517193370/schema-design-for-time-series-data-in-mongodb
- https://github.com/Glavin001/Cobra
- https://www.quora.com/What-is-the-best-way-to-store-time-series-data-in-MongoDB
https://www.mongodb.com/customers/server-density?_ga=1.111702214.317250607.1443618784
http://www.hindawi.com/journals/ijdsn/2013/867693/
MongoDB
http://snmaynard.com/2012/10/17/things-i-wish-i-knew-about-mongodb-a-year-ago/
https://blog.codecentric.de/en/2014/01/realtime-analytics-mongodb-nodejs-smoothiecharts/
http://s3.amazonaws.com/info-mongodb-com/10gen-MongoDB_Operations_Best_Practices.pdf
Cube
http://square.github.io/cube/
Groovy
https://blog.sagaoftherealms.net/?p=176
http://blog.mongodb.org/post/18510469058/grails-in-the-land-of-mongodb
Node.js
http://bigspaceship.github.io/blog/2014/05/14/how-to-create-a-rest-api-with-node-dot-js/
http://www.robinwieruch.de/node-js-scheduling-periodic-tasks-with-agenda/
https://github.com/mikejihbe/metrics
http://stackoverflow.com/questions/9565912/convert-the-output-of-os-cpus-in-node-js-to-percentage
https://www.npmjs.com/package/nsutil
Python:
https://github.com/hcooper/check_mk-plugins
https://github.com/monitisexchange
Apache
http://httpd.apache.org/docs/2.4/mod/mod_status.html
local.conf
<VirtualHost *:80> DocumentRoot "/var/www/html" ServerName local.tld ServerAlias *.local.tld <Directory "/var/www/html"> Options All AllowOverride All #Order allow,deny #Allow from all # Apache 2.4 Require all granted </Directory> <Location /server-status> SetHandler server-status Require ip 127.0.0.1 Require ip ::1 Require ip 192.168.0.114 Require ip 192.168.0.135 </Location> </VirtualHost>
https://github.com/fr3nd/apache-top
https://exchange.nagios.org/directory/Plugins/Web-Servers/Apache/check_apache2-2Epy/details
https://github.com/hcooper/check_mk-plugins/blob/master/mod_status/mod_status-dump.py
http://www.wellho.net/mouth/3019_Apache-httpd-Server-Status-monitoring-your-server.html
https://github.com/dioubernardo/apacheServerStatusCharts
http://pastebin.com/9iGkjSTv
How it works
There is an Apache module (mod_status) responsible for presenting the current view of Apache key parameters and a snapshot of all its request handlers with their statuses. The following lines in Apache’s configuration file present an example of mod_status switched on:
<Location /server-status> SetHandler server-status Order deny,allow Deny from all Allow from 127.0.0.1 </Location>
The statistics are provided in the form of a webpage – you can have a look at such a page of the Apache Project itself here:http://www.apache.org/server-status. Anturis Apache Monitor is able to read and understand this web page (for example: http://127.0.0.1/server-status), and present it on the Anturis Console.
Currently Anturis Apache monitor collects the following metrics:
- aggregate CPU and memory consumption of all Apache processes
- number of busy workers
- number of idle workers
- average number of requests per second*
- average number of Kbytes served per second*
- average number of Kbytes per request*
- CPU usage by all Apache workers*
*provided ExtendedStatus is switched on in Apache configuration.
Monitoring web traffic
A good starting point for troubleshooting website problems is to check whether traffic was or is unusual. An unexpected surge in traffic may be caused by many reasons varying from a DOS attack to a marketing campaign that a sys administrator was unaware of. Anturis Apache Monitor (when mod_status and ExtendedStatus is enabled in your Apache server configuration) provides the following information about the overall traffic:
- number of requests per second
- number of bytes served per second
- average number of bytes per request
Comparing these values with historical data may reveal any abnormalities, such as unusually high or low traffic being processed.
ver: http://piwik.org/
Monitoring Apache processes and threads
Latest Apache versions implement hybrid multi-threaded and multi-process models to serve requests. It means that you will see multiple Apache processes running. Each process may contain multiple threads. The number of processes and the number of threads per process, as well as a forking policy and other parameters, are governed by Apache configuration directives. Each request processing thread is called aworker.
Apache self-regulates the number of workers and maintains a pool of idle workers so as to have some ready to process traffic at any time. Anturis Apache Monitor allows you to keep an eye on the number of busy and idle workers.
Watching the number of busy and idle workers is a proactive way to catch Apache configuration problems early enough. Anturis Apache Monitoring tool will let you discover, for example, these problems:
- insufficient workers for the given traffic
- too many workers – some processes get swapped out, with a heavy overall performance penalty
- non-optimal forking policy or MPM model
- stalling application code (e.g. due to external calls)
- server resources insufficient for the given traffic
Monitoring Apache processes CPU and RAM consumption
The most important server characteristics in relation to a web server is the amount of RAM. A web server should never swap because swapping out a process will make request processing latency unacceptable. So it is important to understand the maximum number of workers that fit the RAM of your server and to adjust the MaxClients (or MaxRequestWorkers) Apache configuration directive accordingly. In order to do this you should observe how much RAM is consumed by how many Apache processes under normal conditions. Divide the first by the second, and compare it to your total available physical memory to understand how many Apache processes you can have on this server. The same procedure should be applied to the CPU, as your web application may be CPU-intensive.
Knowing the maximum number of Apache workers per server will in turn give you some insight into the traffic you can serve by the server. You can use this information to anticipate future upgrades of your infrastructure.
Anturis Apache Monitor collects data about Apache processes’ aggregate CPU and memory consumption. It will also alert you if the values exceed the desired limits.
Best practices: http://cdn.swcdn.net/creative/v14.9/pdf/techtips/1202_SAM_BestPracticesApache_TT.pdf
Mysql
http://jonathanhui.com/mysql-monitoring-performance
https://github.com/shoma/mysqlmonitor/blob/master/src/mysqlstatus.py
Get to grips with the Basics
Fundamental Server Resources
Monitoring of the underlying server’s resources is the basis of MySQL database monitoring. Server CPU and memory usage, free disk space, free swap space, and network usage are all important metrics and the perfect starting point for troubleshooting in case of problems. Anturis will let you know in advance about any overloads of these resources, giving you the chance to prevent negative impact on database operations. You will also be able to spot usage trends over time so as to get insights on when you need to replace/upgrade the hardware serving the MySQL database.
MySQL Availability
The next obvious thing to know is MySQL availability. Failure of MySQL usually makes reliant applications completely inoperable. Anturis alerts you immediately in case MySQL itself or the underlying server goes down.
MySQL Error Log
MySQL error log contains all the errors occurring while MySQL is running as well as some context information, like stack traces. Anturis will watch the log file for error entries and keep you informed when it finds them.
Key MySQL Server Metrics
There are two ways in which MySQL metrics are useful to you: error detection and trending. In the first case you are notified of hazardous conditions that require your attention. Such conditions are defined by the thresholds on the selected metrics. In the second case, you have access to the metric graphs and you are able to troubleshoot by looking for unexpected changes and unusual patterns, and analyzing their reasons.
Here are a few examples of MySQL key performance metrics available with Anturis MySQL monitor.
- Slow query rate. Slow queries are eating the precious resources of the server, leading to degraded performance and application failures. Having a slow query rate that is too high indicates that it is a good time to look into the slow query log and check what’s going on.
- Connections usage. If your MySQL server reaches the maximum number of allowed connections the new client connections will simply be refused. Running out of connections signals an abnormal situation, such as when configuration changes may be needed.
- InnoDB statistics. This includes InnoDB buffer pool usage, its’ miss and wait rates, and the InnoDB log cache wait rate. Using this data you are able to understand whether or not InnoDB performance is optimal.
- MySQL query and key cache statistics. A low cache hit rate signifies that cache configuration is misaligned with application’s needs.
Tomcat
https://github.com/msacks/jythonJMXClient/blob/master/jythonJmxClient.py
Visualization
http://www.citronlab.com/programacion/javascript-graficos-con-morris-js/
https://github.com/worldline/beekeeper
http://www.grepstar.org/iot-recipe-1-yun-mongodb-d3js/
http://adilmoujahid.com/posts/2015/01/interactive-data-visualization-d3-dc-python-mongodb/
https://github.com/StackStorm/collectd3
https://www.digitalocean.com/community/tutorials/an-introduction-to-tracking-statistics-with-graphite-statsd-and-collectd
Real Time
https://github.com/leggetter/socketalytics
