Estándar

Monitoring Apache webserver and MySql

Groovy scrips

http://www.mkyong.com/java/how-to-run-a-task-periodically-in-java/

http://mrhaki.blogspot.com.es/2009/11/groovy-goodness-run-code-at-specified.html

https://gist.github.com/kdabir/3176945

http://groovy-almanac.org/example-timer-and-timertask-in-groovy/

http://crazy4groovy.blogspot.com.es/2013/01/simple-timer-timertask.html

 

Data storage

http://blog.mongodb.org/post/65517193370/schema-design-for-time-series-data-in-mongodb

  • https://github.com/Glavin001/Cobra
  • https://www.quora.com/What-is-the-best-way-to-store-time-series-data-in-MongoDB

https://www.mongodb.com/customers/server-density?_ga=1.111702214.317250607.1443618784

http://www.hindawi.com/journals/ijdsn/2013/867693/

https://rubayeet.wordpress.com/2013/12/29/web-analytics-using-mongodb-aggregation-framework/

MongoDB

http://snmaynard.com/2012/10/17/things-i-wish-i-knew-about-mongodb-a-year-ago/

https://blog.codecentric.de/en/2014/01/realtime-analytics-mongodb-nodejs-smoothiecharts/

http://s3.amazonaws.com/info-mongodb-com/10gen-MongoDB_Operations_Best_Practices.pdf

Cube

http://square.github.io/cube/

Groovy

https://blog.sagaoftherealms.net/?p=176

http://blog.mongodb.org/post/18510469058/grails-in-the-land-of-mongodb

Node.js

http://bigspaceship.github.io/blog/2014/05/14/how-to-create-a-rest-api-with-node-dot-js/

http://www.robinwieruch.de/node-js-scheduling-periodic-tasks-with-agenda/

https://github.com/mikejihbe/metrics

http://stackoverflow.com/questions/9565912/convert-the-output-of-os-cpus-in-node-js-to-percentage

https://www.npmjs.com/package/nsutil

Python:

https://github.com/hcooper/check_mk-plugins

https://github.com/monitisexchange

Apache

http://httpd.apache.org/docs/2.4/mod/mod_status.html

 

local.conf

<VirtualHost *:80>
	DocumentRoot "/var/www/html"
	ServerName local.tld
	ServerAlias *.local.tld
	<Directory "/var/www/html">
		Options All
		AllowOverride All
		#Order allow,deny
		#Allow from all
		# Apache 2.4
		Require all granted
	</Directory>
	<Location /server-status>
		SetHandler server-status
		Require ip 127.0.0.1
		Require ip ::1
		Require ip 192.168.0.114
		Require ip 192.168.0.135
	</Location>
</VirtualHost>

 

https://github.com/fr3nd/apache-top

https://exchange.nagios.org/directory/Plugins/Web-Servers/Apache/check_apache2-2Epy/details

https://github.com/hcooper/check_mk-plugins/blob/master/mod_status/mod_status-dump.py

http://www.wellho.net/mouth/3019_Apache-httpd-Server-Status-monitoring-your-server.html

https://github.com/dioubernardo/apacheServerStatusCharts

http://pastebin.com/9iGkjSTv

 

How it works

There is an Apache module (mod_status) responsible for presenting the current view of Apache key parameters and a snapshot of all its request handlers with their statuses. The following lines in Apache’s configuration file present an example of mod_status switched on:

<Location /server-status>
	SetHandler server-status
	Order deny,allow
	Deny from all
	Allow from 127.0.0.1
</Location>

The statistics are provided in the form of a webpage – you can have a look at such a page of the Apache Project itself here:http://www.apache.org/server-status. Anturis Apache Monitor is able to read and understand this web page (for example: http://127.0.0.1/server-status), and present it on the Anturis Console.

Currently Anturis Apache monitor collects the following metrics:

  • aggregate CPU and memory consumption of all Apache processes
  • number of busy workers
  • number of idle workers
  • average number of requests per second*
  • average number of Kbytes served per second*
  • average number of Kbytes per request*
  • CPU usage by all Apache workers*

*provided ExtendedStatus is switched on in Apache configuration.

 

Monitoring web traffic

A good starting point for troubleshooting website problems is to check whether traffic was or is unusual. An unexpected surge in traffic may be caused by many reasons varying from a DOS attack to a marketing campaign that a sys administrator was unaware of. Anturis Apache Monitor (when mod_status and ExtendedStatus is enabled in your Apache server configuration) provides the following information about the overall traffic:

  • number of requests per second
  • number of bytes served per second
  • average number of bytes per request

Comparing these values with historical data may reveal any abnormalities, such as unusually high or low traffic being processed.

ver: http://piwik.org/

Monitoring Apache processes and threads

Latest Apache versions implement hybrid multi-threaded and multi-process models to serve requests. It means that you will see multiple Apache processes running. Each process may contain multiple threads. The number of processes and the number of threads per process, as well as a forking policy and other parameters, are governed by Apache configuration directives. Each request processing thread is called aworker.

Apache self-regulates the number of workers and maintains a pool of idle workers so as to have some ready to process traffic at any time. Anturis Apache Monitor allows you to keep an eye on the number of busy and idle workers.

Watching the number of busy and idle workers is a proactive way to catch Apache configuration problems early enough. Anturis Apache Monitoring tool will let you discover, for example, these problems:

  • insufficient workers for the given traffic
  • too many workers – some processes get swapped out, with a heavy overall performance penalty
  • non-optimal forking policy or MPM model
  • stalling application code (e.g. due to external calls)
  • server resources insufficient for the given traffic

Monitoring Apache processes CPU and RAM consumption

The most important server characteristics in relation to a web server is the amount of RAM. A web server should never swap because swapping out a process will make request processing latency unacceptable. So it is important to understand the maximum number of workers that fit the RAM of your server and to adjust the MaxClients (or MaxRequestWorkers) Apache configuration directive accordingly. In order to do this you should observe how much RAM is consumed by how many Apache processes under normal conditions. Divide the first by the second, and compare it to your total available physical memory to understand how many Apache processes you can have on this server. The same procedure should be applied to the CPU, as your web application may be CPU-intensive.

Knowing the maximum number of Apache workers per server will in turn give you some insight into the traffic you can serve by the server. You can use this information to anticipate future upgrades of your infrastructure.

Anturis Apache Monitor collects data about Apache processes’ aggregate CPU and memory consumption. It will also alert you if the values exceed the desired limits.

Best practices: http://cdn.swcdn.net/creative/v14.9/pdf/techtips/1202_SAM_BestPracticesApache_TT.pdf

Mysql

http://jonathanhui.com/mysql-monitoring-performance

https://github.com/shoma/mysqlmonitor/blob/master/src/mysqlstatus.py

Get to grips with the Basics

Fundamental Server Resources

Monitoring of the underlying server’s resources is the basis of MySQL database monitoring. Server CPU and memory usage, free disk space, free swap space, and network usage are all important metrics and the perfect starting point for troubleshooting in case of problems. Anturis will let you know in advance about any overloads of these resources, giving you the chance to prevent negative impact on database operations. You will also be able to spot usage trends over time so as to get insights on when you need to replace/upgrade the hardware serving the MySQL database.

MySQL Availability

The next obvious thing to know is MySQL availability. Failure of MySQL usually makes reliant applications completely inoperable. Anturis alerts you immediately in case MySQL itself or the underlying server goes down.

MySQL Error Log

MySQL error log contains all the errors occurring while MySQL is running as well as some context information, like stack traces. Anturis will watch the log file for error entries and keep you informed when it finds them.

Key MySQL Server Metrics

There are two ways in which MySQL metrics are useful to you: error detection and trending. In the first case you are notified of hazardous conditions that require your attention. Such conditions are defined by the thresholds on the selected metrics. In the second case, you have access to the metric graphs and you are able to troubleshoot by looking for unexpected changes and unusual patterns, and analyzing their reasons.

Here are a few examples of MySQL key performance metrics available with Anturis MySQL monitor.

  • Slow query rate. Slow queries are eating the precious resources of the server, leading to degraded performance and application failures. Having a slow query rate that is too high indicates that it is a good time to look into the slow query log and check what’s going on.
  • Connections usage. If your MySQL server reaches the maximum number of allowed connections the new client connections will simply be refused. Running out of connections signals an abnormal situation, such as when configuration changes may be needed.
  • InnoDB statistics. This includes InnoDB buffer pool usage, its’ miss and wait rates, and the InnoDB log cache wait rate. Using this data you are able to understand whether or not InnoDB performance is optimal.
  • MySQL query and key cache statistics. A low cache hit rate signifies that cache configuration is misaligned with application’s needs.

 

Tomcat

https://github.com/msacks/jythonJMXClient/blob/master/jythonJmxClient.py

 

Visualization

http://www.citronlab.com/programacion/javascript-graficos-con-morris-js/

https://anmolkoul.wordpress.com/2015/06/05/interactive-data-visualization-using-d3-js-dc-js-nodejs-and-mongodb/

https://github.com/worldline/beekeeper

http://www.grepstar.org/iot-recipe-1-yun-mongodb-d3js/

http://adilmoujahid.com/posts/2015/01/interactive-data-visualization-d3-dc-python-mongodb/

https://github.com/StackStorm/collectd3

https://www.digitalocean.com/community/tutorials/an-introduction-to-tracking-statistics-with-graphite-statsd-and-collectd

Real Time

https://github.com/leggetter/socketalytics

proyecto 52 52
Estándar

Proyecto 52 para el 2012

proyecto 52 52
proyecto 52 52

El momento se acerca, se acaba el año y toca revisar como ha ido el 2011 y hacer la lista de los buenos propósitos para el año nuevo.

Yo éste año me voy embarcar en un reto personal al que llevo dando vueltas desde hace tiempo, 52 semanas 52 fotos para el 2012.

El objetivo es aprender y perfeccionar mi hobbie, la fotografía,  y como a andar se aprende andando,  a hacer fotos se aprenderá igual (haciendo fotos no andando 😀 ).

La planificación inicial de temas o ideas para el proyecto, aunque sólo como guía, creo que ha sido un buen ejercicio pensarla y escribirla. Supogo que en un proyecto a tan largo plazo pasaré por diferentes fases y estados de motivación e inspiración así que será de ayuda. No me voy a ceñir exactamente a la lista, es posible que la cambie sobre la marcha.

Por cada semana, un tema y la fecha de inicio de semana.

Comienzo del proyecyo 1 – Enero – 2012
Semana 1  – Juguetes             – Ene 2
Semana 2  – Malos hábitos   – Ene 9
Semana 3  – Diminuto        – Ene 16
Semana 4  – Lineas          – Ene 23
Semana 5  – Acero           – Ene 30
Semana 6  – Cables          – Feb 6
Semana 7  – Enchufes        – Feb 13
Semana 8  – Velocidad       – Feb 20
Semana 9  – Mascotas        – Feb 27
Semana 10 – En el espejo    – Mar 5
Semana 11 – Suave           – Mar 12
Semana 12 – Puertas         – Mar 19
Semana 13 – Verde           – Mar 26
Semana 14 – Musica          – Abr 2
Semana 15 – Manos           – Abr 9
Semana 16 – Lluvia          – Abr 16
Semana 17 – Fuego           – Abr 23
Semana 18 – Hielo           – Abr 30
Semana 19 – Blanco y negro  – May 7
Semana 20 – Rojo            – May 14
Semana 21 – Ropa            – May 21
Semana 22 – Arquitectura    – May 28
Semana 23 – Enrollado       – Jun 4
Semana 24 – Parar           – Jun 11
Semana 25 – Nuevo           – Jun 21
Semana 26 – Comer           – Jun 28
Semana 27 – Ojos            – Jul 2
Semana 28 – Arma            – Jul 9
Semana 29 – Límites         – Jul 16
Semana 30 – Yo          – Jul 23
Semana 31 – No              – Jul 30
Semana 32 – Descanso        – Ago 6
Semana 33 – Naranja         – Ago 13
Semana 34 – Suciedad        – Ago 20
Semana 35 – Delicioso       – Ago 27
Semana 36 – Abajo           – Sep 3
Semana 37 – Arriba          – Sep 10
Semana 38 – Risa            – Sep 27
Semana 39 – Herramientas    – Sep 24
Semana 40 – Doblado         – Oct 1
Semana 41 – Azul            – Oct 8
Semana 42 – Ventanas        – Oct 15
Semana 43 – Altura          – Oct 22
Semana 44 – De miedo        – Oct 29
Semana 45 – Metal           – Nov 5
Semana 46 – Centro          – Nov 12
Semana 47 – Textura         – Nov 19
Semana 48 – Negro           – Nov 26
Semana 49 – Luz             – Dic 3
Semana 50 – Libros          – Dic 10
Semana 51 – Viejo           – Dic 17
Semana 52 – Navidad         – Dic 24
Resumen final – 31 – Dicembre – 2012

Pues nada, ésta es la lista de temas.

Por cierto, si alguien se anima podríamos intentar hacer el proyecto a la vez, deja un comentario o mandame un correo y lo vemos.

Estándar

Grails LDAP authentication and authorization

Today I’ve been investigating how to integrate a Grails aplication with an LDAP server to perform authentication and authorization.

There are several plugins available to do the authentication in a grails way.

My choice is Spring Security Core Plugin because it is mature and has several extension plugins to extend the functionality and provide integrations with external systems like Facebook, Twitter, OpenId and LDAP .

Installation and configuration are very well documented at : http://grails-plugins.github.com/grails-spring-security-core/docs/manual/

Peter Ledbrook wrote a great introductory article Simplified Spring Security with Grails .

I’ve updated the example application in the post to do the authentication using an LDAP Server.

My server has the following structure:

Sample LDAP structure

Yo can download the sample.ldif

Once you have all the LDAP structure, install the extension LDAP plugin for Spring Security Core

1
grails install-plugin spring-security-ldap

Now modify Config.groovy adding the specific configuration for LDAP spring-security-ldap

1
2
3
4
5
6
7
8
9
10
11
grails.plugins.springsecurity.providerNames = ['ldapAuthProvider','anonymousAuthenticationProvider','rememberMeAuthenticationProvider']
grails.plugins.springsecurity.ldap.context.managerDn = 'uid=admin,ou=system'
grails.plugins.springsecurity.ldap.context.managerPassword = 'YOUR_PASSWORD'
grails.plugins.springsecurity.ldap.context.server = 'ldap://10.99.8.135:10389'
grails.plugins.springsecurity.ldap.authorities.groupSearchBase = 'ou=Groups,dc=example,dc=com'
grails.plugins.springsecurity.ldap.authorities.retrieveGroupRoles = true
grails.plugins.springsecurity.ldap.authorities.retrieveDatabaseRoles = false
grails.plugins.springsecurity.ldap.authorities.groupSearchFilter = 'member={0}'
grails.plugins.springsecurity.ldap.search.base = 'dc=example,dc=com'
grails.plugins.springsecurity.ldap.search.attributesToReturn = ['mail', 'cn', 'sn', 'givenName', 'jpegPhoto' , 'telephoneNumber']
grails.plugins.springsecurity.ldap.authenticator.attributesToReturn = ['mail', 'cn', 'sn', 'givenName', 'jpegPhoto' , 'telephoneNumber']

 

Now you are ready to authenticate against your LDAP, also the group membership is readed from LDAP.

In the expample application the sec tag lib is used to show a link to the create post action based on the role of the logged user.

1
2
3
<sec:ifAllGranted roles="ROLE_USER">
  <g:link controller="post" action="timeline">My Timeline</g:link>
</sec:ifAllGranted>

How is this managed if you are using an LDAP? the answer is simple as everything in grails. Create a group in your LDAP named USER and add the users to it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
dn: cn=USER,ou=Groups,dc=example,dc=com
objectClass: groupOfNames
objectClass: top
cn: USER
description: USER_ROLE
member: uid=wpauli,ou=Users,dc=example,dc=com
member: uid=aeinstein,ou=Users,dc=example,dc=com
member: uid=mborn,ou=Users,dc=example,dc=com
member: uid=mcurie,ou=Users,dc=example,dc=com
member: uid=sito,ou=Users,dc=example,dc=com
createTimestamp: 20111121102018Z
creatorsName: 0.9.2342.19200300.100.1.1=admin,2.5.4.11=system
modifiersName: 0.9.2342.19200300.100.1.1=admin,2.5.4.11=system
modifyTimestamp: 20111121110901Z

I’ve added to the application some other properties that came from LDAP ( Photo, Telephone number, Full Name ).

This is done extending the org.springframework.security.core.userdetails.User to add all new attributes:

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
import org.springframework.security.core.GrantedAuthority
import org.springframework.security.core.userdetails.User
 
class MyUserDetails extends User {
 
    // extra instance variables final String fullname final String email final String title
    String fullname
    String email
    String title
    String phone
 
    byte[] photo
 
    MyUserDetails(String username, String password, boolean enabled, boolean accountNonExpired, boolean credentialsNonExpired, boolean accountNonLocked, Collection authorities, String fullname, String email, String title, byte[] photo, String phone) {
 
        super(username, password, enabled, accountNonExpired, credentialsNonExpired, accountNonLocked, authorities)
 
        this.fullname = fullname
        this.email = email
        this.title = title
        this.photo = photo
        this.phone = phone
    }
 
}

And providing your own implementation for the org.springframework.security.ldap.userdetails.UserDetailsContextMapper interface.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
import org.springframework.ldap.core.DirContextAdapter
import org.springframework.ldap.core.DirContextOperations
import org.springframework.security.core.userdetails.UserDetails
import org.springframework.security.ldap.userdetails.UserDetailsContextMapper
/**
 *
 * @author SiTo
 */
 
class MyUserDetailsContextMapper implements UserDetailsContextMapper {
 
    UserDetails mapUserFromContext(DirContextOperations ctx, String username, Collection authorities) {
 
        String fullname = ctx.originalAttrs.attrs['cn'].values[0]
        String email = ctx.originalAttrs.attrs['mail'].values[0].toString().toLowerCase()
 
        def title = ctx.originalAttrs.attrs['sn']
 
        def phone = ctx.getStringAttribute('telephoneNumber')
 
        byte[] photo = (byte[])ctx.getObjectAttribute('jpegPhoto')
 
        def userDetails = new MyUserDetails(username, '', true, true, true, true,
            authorities, fullname, email, title == null ? '' : title.values[0], photo, phone)
 
        return userDetails
    }
 
    void mapUserToContext(UserDetails user, DirContextAdapter ctx) {
        throw new IllegalStateException("Only retrieving data from LDAP is currently supported")
    }
 
}

The final step is register the custom implementation using the Spring DSL in the resources.groovy file

1
2
3
4
5
beans = {
   ldapUserDetailsMapper(MyUserDetailsContextMapper) {
      // bean attributes
   }
}

Now we are ready to use all new properties that are maintained by the LDAP into our app.

For example, show the Photo of the logged user:

Add the following method to your PersonController.groovy

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
class PersonController {
 
    def springSecurityService
 
    // PersonController CODE
 
    def photo = {
        def userDetails = springSecurityService.principal
        def photo = new File(GrailsResourceUtils.WEB_APP_DIR  + "/images/person.jpg").readBytes()
 
        if( userDetails.photo != null ){
            photo = userDetails.photo
        }
 
        response.outputStream << photo
        response.setHeader("Content-disposition", "attachment; filename=avatar.jpg")
        response.contentType = 'image/jpeg'
        response.outputStream << photo
        response.outputStream.flush()
        return;
 
    }
}

And modify the main.gsp to show the photo:

 

1
2
3
4
         <sec:ifLoggedIn>
          Hola <sec:loggedInUserInfo field="fullname"/> [<sec:loggedInUserInfo field="phone"/>] (<g:link controller="logout">Salir</g:link>)
          <img src="${createLink(controller:'person', action:'photo')}" width="40px" />
         </sec:ifLoggedIn>

 

Done!

 

This is how it looks:

That’s all…

Grails Rocks!

See you.

Estándar

Proyecto blanco y negro

Decidido, voy a comenzar a realizar más fotografías en blanco y negro.

Éste verano intentaré olvidarme del color para intentar mejorar mi técnica , composición, gestión de la luz y post-proceso.

Como inspiración os dejo unos enlaces y fotos:

 

Iré poniendo fotos en el blog a ver que tal.