en faite je me dois d'etres plus precis j'aimerais pouvoir utiliser votre outil pour pouvoir analyser des logs au format XML a partir d'une base de données
qui regroupes les logs generés par une aplications a partir de plusieurs serveur !! donc si eventuellement vous avez une solution assez rapide elle serais la bienvenue !!
Joined: 01 Jan 1970 Posts: 24 Location: Geneva, Switzerland
Posted: Thu Nov 02, 2006 8:07 pm Post subject:
Bonjour,
Dès lors que les logs XML contiennent les champs habituels des requêtes web (useragent, referer, url, ip, etc.), il devrait être relativement simple de créer une nouvelle source de donnée de log.
Cependant, vous ne parlez pas spécifiquement d'un serveur web, mais d'une application. Pouvez-vous donner un extrait de log XML?
Les fichiers de log XML peuvent être lus par Chainsaw, qui permet de manipuler les logs (p.ex. le niveau d'alerte), mais qui ne permet pas d'obtenir des statistiques détaillées. Peut-être que vous devriez essayer cet outil. _________________ Julien
Log(psycho)Analyst developer.
Joined: 01 Jan 1970 Posts: 24 Location: Geneva, Switzerland
Posted: Sat Nov 04, 2006 9:38 am Post subject:
Le problème, c'est que ce format de log ne provient pas d'un serveur web, donc la majorité des fonctions de Log(psycho)Analyst (LPA) ne peuvent pas être exploitées.
Néanmoins, il est possible de traiter ce fichier de log, mais cela implique l'ajout de plusieurs classes à LPA, ce qui est facilité par une architecture de plugins (pour le moment non documentée, mais ça viendra: je vous la décrirai si vous voulez poursuivre dans cette voie).
Sont a créer:
un parser de log XML héritant de la classe LogFormat de LPA (et exploitant les parsers XML java). Ce serait par exemple une classe BxssLogFormat qui retourne une ligne (ou message) à chaque fois que l'on fait appel à lui. Cette classe devra possèder des getters qui permetrons d'accéder aux champs du message BXSS.
un plugin d'entrée qui stocke les données de chaque ligne (ou message) du log (dans votre cas, l'error level, la classe ou le nom du thread). Ce plugin sera appelé automatiquement par LPA pour chaque ligne/message du fichier de log. Le plugin mémorisera les informations dans une structure de donnée fournie par LPA.
un plugin de sortie qui génère des tables et des graphiques qui sont intégrés aux rapports LPA. Le plugin de sortie va rechercher les informations stockées dans le plugin d'entrée, le rendu des tables et graphiques étant facilités par des fonctions de LPA.
Je n'ai malheureusement pas le temps de réaliser cela moi-même, mais je pense qu'il y a environ une semaine de travail (le temps de comprendre l'API de LPA, de coder le parser de log, le plugin d'entrée et le plugin de sortie).
Si vous désirez poursuivre dans cette voie, je peux vous fournir l'API et un exemple de parser et de plugins d'entrée et de sortie. De ma part, ça demandera un peu de travail (je n'ai pas encore généré l'API et les plugins ne sont pas encore chargés automatiquement).
Je peux vous expliquer plus en détail _________________ Julien
Log(psycho)Analyst developer.
Joined: 01 Jan 1970 Posts: 24 Location: Geneva, Switzerland
Posted: Mon Nov 06, 2006 9:11 pm Post subject: How to create your own log format
Note: customized log format is an experimental feature and is subject to changes.
In order to create your own log format, you must create a class which parses the log file:
Code:
package ch.kronospace.logpsychoanalyst.logformat;
public class MyLogFormat extends LogFormat {
// register the log format (it will be available in the GUI)
static {
LogFormatManager.register(new MyLogFormat ());
}//end static block
public void parse(String s) {
...
}//end parse
}//end class
You must uncompress the $INSTALLATION_DIR\lib\logpsychoanalyst.jar (before doing that, be sure you are using the last version).
Your MyLogFormat class must be placed in the $INSTALLATION_DIR\lib\ch\kronospace\logpsychoanalyst\logformat\ directory.
You will need to configure a FileDataSource in the GUI for a file or directory associated with your ch.kronospace.logpsychoanalyst.logformat.MyLogFormat class.
For each file of this datasource, the following sequence will be called:
Code:
myLogFormatInstance.setBufferedReader(aBufferedReader);
String current_log_entry = myLogFormatInstance.getNextEntry();
while (current_log_entry!=null) {
myLogFormatInstance.parse(current_log_entry);
... // some internal processing
current_log_entry=myLogFormatInstance.getNextEntry();
}//end while
Note: I'm aware this is maybe not the simplest way to process, e.g. if you have a parser which does not return log entry as String. This behavior is currently required in order to compute the progression in percents. I will change it probably later by introducing a parseNextEntry().
The LogFormat implements a default getNextEntry() method for log files which have one log entry per file line. For reference, the source code shown at the end of this post.
You should at least override the parse(String) method. You can also add your custom getters which will be used in your custom report plugins (more information on that in some days).
I hope these tips are sufficient to help you designing a new log format. I will explain in a few days how you can design custom reports, but you can start by designing your custom log format.
public class LogFormat {
/**
The date format used to render the log entry.
*/
public static SimpleDateFormat df = new SimpleDateFormat("dd/MMM/yyyy:HH:mm:ss", Locale.ENGLISH);
/**
IP Address or host name of the machine that requested the document. Must be set in the "parse" method of the subclass. Default value is null;
*/
protected String host=null;
/**
IP Address or host name of the machine that requested the document. Must be set in the "parse" method of the subclass. Default value is null;
*/
protected long host_ip=0;
protected InetAddress host_inet_address=null;
/**
Login name of the user that requested the document (in most cases not specified for security reasons). Must be set in the "parse" method of the subclass. Default value is null.
*/
protected String logname=null;
/**
Date/time of the request (day, month, year, hour, minute, second). Must be set in the "parse" method of the subclass. The default value is the current date/time.
*/
protected Calendar date=Calendar.getInstance();
/**
Method used to request the document (GET, POST, OPTIONS, ...). Must be set in the "parse" method of the subclass. Default value is null.
*/
protected String method=null;
/**
URL of the requested document. Must be set in the "parse" method of the subclass. Default value is null.
*/
protected String url=null;
/**
Return code of the document. Must be set in the "parse" method of the subclass. Default value is 0.
*/
protected int code=0;
/**
Length of the requested document in bytes. Must be set in the "parse" method of the subclass. Default value is 0.
*/
protected long size=0;
/**
Referer of the document (=URL of the previous document). Must be set in the "parse" method of the subclass. Default value is null.
*/
protected String referer=null;
/**
User-agent string of the application that requested the document. Must be set in the "parse" method of the subclass. Default value is null.
*/
protected String useragent=null;
/**
Tell if the record is corrupted.
*/
protected boolean corrupted = false;
/**
Host name of the referer (for example www.foo-bar.com).
*/
protected String ref_host_name=null;
/**
Tell if the requested URL is on the same virtual host than the Referer.
*/
protected boolean is_referer_on_same_vhost = false;
/**
Referer requested page (e.g. "/index.html"). Does not contain the referer host.
*/
protected String ref_page=null;
/**
Virtual Host that use the current logformat. Used to determine if an URL (requested or referer) is on the same VirtualHost (=alias).
*/
protected DataSource vhost=null;
protected int hour=-1;
protected int minute=-1;
protected int day=-1;
protected int month=-1;
protected int year=-1;
protected int day_of_week=-1;
String last_entry =null;
protected BufferedReader in=null;
public void setDataSource(DataSource vhost) {
this.vhost = vhost;
}//end setDataSource
public DataSource getDataSource() {
return vhost;
}//end getDataSource
public void setBufferedReader(BufferedReader in) {
this.in = in;
}//end setBufferedReader
/**
Default entry reader. The entry is supposed to have only one line.
*/
public String getNextEntry() throws IOException {
last_entry = in.readLine();
return last_entry;
}//end getNextEntry
public String getLastEntry() {
return last_entry;
}//end getLastEntry
/**
Parses a log entry (= a line in the log file). Must be implemented by subclasses.
*/
public void parse(String log_entry) {}
public boolean isCorrupted() {
return corrupted;
}//end isCorrupted
/**
Returns the host name. The host must be set in the "parse" method of the subclass.
*/
public String getHostAsString() {
return host;
}//end getHostAsString
/**
Returns the host name. The host must be set in the "parse" method of the subclass.
*/
public long getHost() {
return host_ip;
}//end getHost
/**
Returns the host name. The host must be set in the "parse" method of the subclass.
*/
public InetAddress getHostAsInetAddress() {
return host_inet_address;
}//end getHost
/**
Returns the client user log name. The log name must be set in the "parse" method of the subclass.
*/
public String getLogName() {
return logname;
}//end getLogName
/**
Return the hour of the record.
*/
public int getHour() {
return hour;
}//end getHour
/**
Return the minute of the record.
*/
public int getMinute() {
return minute;
}//end getMinute
public int getDayOfWeek() {
return getDate().get(Calendar.DAY_OF_WEEK);
}//end getDayOfWeek
public int getMonth() {
return getDate().get(Calendar.MONTH);
}//end getMonth
/**
Returns the day of year (1..366)
*/
public int getDayOfYear() {
return getDate().get(Calendar.DAY_OF_YEAR);
}//end getDayOfYear
/**
Returns the date of the request. The date must be set in the "parse" method of the subclass.
*/
public Calendar getDate() {
return date;
}//end getDate
/**
Returns the method used (GET, POST, OPTIONS, ...). The method must be set in the "parse" method of the subclass.
*/
public String getMethod() {
return method;
}//end getMethod
/**
Returns the URL of the requested document. The URL must be set in the "parse" method of the subclass.
*/
public String getURL() {
return url;
}//end getURL
/**
Returns the return code of the request (200, 404, ...). The code must be set in the "parse" method of the subclass.
*/
public int getCode() {
return code;
}//end getCode
public void setCode(int code) {
this.code = code;
}//end setCode
/**
Returns the length of the requested document in bytes. The host must be set in the "parse" method of the subclass.
*/
public long getSize() {
return size;
}//end getSize
/**
Returns the referer of the requested document (=previous hypertext document). The referer must be set in the "parse" method of the subclass.
*/
public String getReferer() {
return referer;
}//end getReferer
/**
Returns the user agent string of the application that requested the document. The useragent must be set in the "parse" method of the subclass.
*/
public String getUserAgent() {
return useragent;
}//end getUserAgent
public void setUserAgent(String ua) {
this.useragent = ua;
}//end setUserAgent
public boolean isRefererOnSameVirtualHostThanURL() {
return is_referer_on_same_vhost;
}//end isRefererOnSameVirtualHost
public String getRefererHost() {
return ref_host_name;
}//end getRefererHost
public String getRefererPage() {
return ref_page;
}//end getRefererPage
public boolean hasReferer() {
return true;
}//end hasReferer
public boolean hasUserAgent() {
return true;
}//end hasUserAgent
/**
Returns the String representation of the log entry. Should be used for debug only.
*/
public String toString() {
return
"Host: *"+getHost()+"*\n"+
"Date: *"+df.format(getDate().getTime())+"*\n"+
"URL: *"+getURL()+"*\n"+
"Method: *"+getMethod()+"*\n"+
"Code: "+getCode()+"\n"+
"Size: "+getSize()+"\n"+
"Referer: "+getReferer()+"\n"+
"UserAgent: "+getUserAgent()+"\n";
}//end toString
}//end class
Joined: 01 Jan 1970 Posts: 24 Location: Geneva, Switzerland
Posted: Wed Nov 08, 2006 9:27 pm Post subject: How to build a custom plugin
In a previous post, we saw how to create a custom log file format parser. Now, we will see how to create a custom plugin. Each plugin will be called for each log entry. The plugins are automatically loaded on startup (if their .class file is in the plugin directory).
You will need to design a custom plugin such as this prototype:
// store your custom information here (e.g. a DailyActivityStorage)
/**
Return an object if the key is supported by the plugin. Not currently used.
*/
public Object getValue(String key) {
return null;
}//end getValue
/**
This function is called for every log entry of the log files.
*/
public void processRecord(LogFormat lf, Visit visit, HTTPStatistics stats) {
// save the information you want in the attributes
}//end processRecord
public boolean isAHit(LogFormat lf) {
return false;
}//end isAHit
}//end class
This class must be compiled to the $INSTALLATION_DIR/lib/ch/kronospace/logpsychoanalyst/plugins/
You will need to store the log entries information is the plugin you will write. LPA stores all informations in instances of DailyActivityStorage, a way to store daily activity. The simplest way is to use the same mechanism (which is explained below). If the day granularity does not fit your needs, you will need to implement your own storage structure (probably based on arrays and hashtables).
The information can be stored in objects of class ch.kronospace.logpsychoanalyst.DailyActivityStorage (this could seem a bit fastidious/complex, but this IMHO the best way to store generic information with good scalability):
Code:
// create a new DailyActivityStorage wich will store your own piece of information on a daily basis
DailyActivityStorage das = new DailyActivityStorage();
das.setEmptyActivity(new MyCustomInfo()); // MyCustomInfo implements StatisticElement
// create a way to update the DailyActivityStorage
MyUpdater updater = new MyUpdater(); // implements StatisticElementUpdater
updater.setXXX(..); // set the updater parameters (e.g. set the increment)
// update the statistics for the day of the request
das.update(lf.getDate().getTimeInMillis(), updater);
Implementation note: DailyActivityStorage stores the daily informations as a sparse array. The performance is optimized for reading/writing : O(1).
Your MyCustomInfo class could look like:
Code:
package ch.kronospace.logpsychoanalyst;
public class MyCustomInfo implements StatisticElement {
public long count;
public MyCustomInfo() {
reset();
}//end constructor
/**
Merge the statistic with the current one. In fact, only the stats are merged.
*/
public void merge(StatisticElement ua) {
MyCustomInfo el = new MyCustomInfo();
count += el.count;
}//end merge
public void reset() {
count=0;
}//end reset
/**
Makes a copy of the current object. Do not deep clone the user object (reference clone only).
*/
public Object clone() {
MyCustomInfo el = new MyCustomInfo();
el.count = count;
return el;
}//end clone
public String toString() {
return "foo-bar"
}//end toString
}//end class
Your MyUpdater class could look like:
Code:
package ch.kronospace.logpsychoanalyst;
public class MyUpdater implements StatisticElementUpdater {
public long increment = 0;
/**
from the StatisticElementUpdater interface.
*/
public void update(StatisticElement element) {
MyCustomInfo el = (MyCustomInfo)element;
el.count += increment;
}//end update
public void setXXX(...) {
// your custom setters
}//end setXXX
}//end class
I will explain later how to retrieve information from the DailyActivityStorage in the plugin report. _________________ Julien
Log(psycho)Analyst developer.
Joined: 01 Jan 1970 Posts: 24 Location: Geneva, Switzerland
Posted: Sun Nov 12, 2006 8:59 am Post subject: How to make a custom output report?
In previous posts, we saw how to make a custom log format and a custom input plugin. Now, we will see how to make an output plugin, whose main purpose is to generate HTML code which will be integrated in the full report.
/**
Implementing the ReportPlugin interface is required
*/
public class MyReport implements ReportPlugin {
// register the plugin is required so that LPA can know it.
static {
PluginManager.registerReportPlugin(new MyReport());
}
// you will need to implement one or more than one StatisticElementArrayGetter.
// The StatisticElementArrayGetter allows to extract information from a DailyActivityStorage, on a form of an array of double.
public static final StatisticElementArrayGetter getter_nbr_comments = new StatisticElementArrayGetter() {
// return the name of the value which is extracted by the getter.
// This will be used on the generated graphics.
public String getValueName() {
return "Nombre de commentaires";
}//end getValueName
// returns the values extracted from a DailyActivityStorage, between two dates (inclusive).
public double[] getValues(DailyActivityStorage das, long first_date, long last_date) {
StatisticElement[] elements = das.getRange(first_date, last_date);//
double[] values = new double[elements.length];
for (int i=0; i<elements.length; i++) {
RestorangStatus el = (RestorangStatus)elements[i];
values[i]=el.nbr_comments;
}//end
return values;
}//end getValues
};//end StatisticElementArrayGetter
// reference to the class which build the full report.
HTTPStatisticsHTMLReportBuilder report_builder = null;
// mandatory method which returns the HTML code which will be integrated in the report.
public String getReport(HTTPStatisticsHTMLReportBuilder report_builder, String visitor_type) {
// the visitor type can be "human" or "non-human", e.g. if this is a robot crawling your site. Depending on your log format this may not have a meaning for you.
if (visitor_type.equals("non-human")) return "";
this.report_builder = report_builder;
// get the associated input plugin
MyCustomPlugin input_plugin = null;
for (Enumeration e=PluginManager.getReportPlugins().elements(); e.hasMoreElements(); ) {
Object plugin = e.nextElement();
if (plugin instanceof MyCustomPlugin) {
input_plugin = (MyCustomPlugin)plugin;
break;
}//end if
}//end for
// if the input plugin is not used, no need to generate a report
if (!input_plugin.used) return "";
String result = "";
DailyActivityStorage das = input_plugin.myDAS;
// set the parameters for the graphic generator
String figure_title = "Nombre de vues/commentaires/votes";
String figure_xlabel = "Time";
String figure_ylabel = ""; // no label
String filename = "resto-rang."+key+".nbr.png";
boolean normalized = false;
int figure_width = 450;
int figure_height = 150;
int smoothing_over_x_days=14;
StatisticElementArrayGetter[] getters = new StatisticElementArrayGetter[] {getter_nbr_comments};
result += "<p><table class=\"reporttable\">"+
"<tr><td class=\"tabletitle\"> Nombre de commentaires</td><td align=right><a href=help/myCustomHelpPage.html><img border=0 src=images/help.png></a></td></tr>"+
"<tr><td colspan=2 nowrap>"+
"<img src=\""+filename+"\">"+
"</td></tr></table>";
return result;
}//end getReport
}//end class
Your class file will be stored to $INSTALLATION_DIR/lib/ch/kronospace/logpsychoanalyst/plugins/report/ and will be detected automatically at load time.
Your report will be integrated into the full report. _________________ Julien
Log(psycho)Analyst developer.
Last edited by jkronegg on Wed Dec 06, 2006 8:12 pm; edited 1 time in total
Joined: 01 Jan 1970 Posts: 24 Location: Geneva, Switzerland
Posted: Mon Nov 13, 2006 9:47 pm Post subject:
Hi MasterOfSword!
Well, currently the log format I advised in the specified page is not supported . This could seem a bit strange, but I wanted people to get the most complete log file format, get a backup of their log files and process them later. In fact, a parser for this log format is currently being developped, but I did not get enough traffic on my test web site to fully test my log parser.
If you already have some log files using this format, I will be happy to use them to test my parser. If you agree to share them, can you send me a zipped file by private message?
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
. This could seem a bit strange, but I wanted people to get the most complete log file format, get a backup of their log files and process them later. In fact, a parser for this log format is currently being developped, but I did not get enough traffic on my test web site to fully test my log parser.