May 25, 2005 11:23 AM
Logging Conversations
-
Like (0)
I have setup the server to log conversations between users in a corporate environment. But the log file has a lot of stuff that I really don''t need and it is difficult to navigate when looking for a specific item. Is there an app I can use to read the file easier or a plugin I can use to save them as a plain text conversation?
ItsNewToYou
I''d also be interested in a feature like this if it isn''t available. I''m doing some searches for an XML parser app that I can use in the meantime.
There''s been some discussion before about how to improve logging and auditing (here''s one url=http://www.jivesoftware.org/forums/thread.jspa?threadID=14432thread[/url] in particular) but I don''t believe anyone is actively working on anything.
Hope that helps,
Ryan
Hey ItsNewToYou,
There aren''t any tools for it yet, but we do hope to add some ways to view/search it in the admin console in a future release. Meanwhile, if you are logging stuff that you don''t need you can only check to audit Message packets and leave unchecked IQ and Presence packets in the Admin Console.
Regards,
-- Gato
I do have it set to only "Audit Message Packets" but there is still a ton of stuff you just don''t need. So what I found and am using is a proxy called JabLogger and it looks like it is doing just what I need it to so far.
http://www.michaylov.com/html/jablogger.html
ItsNewToYou
Well after trying this approach it worked fine for awhile. But after about 30 people were logged into the server the connection would be lost after they send an IM to someone. So I had to revert back to the old way of doing things.
ItsNewToYou
I''ve been doing a little work on this.
I''ve put together a small Perl script that parses the jive.audit log files and creates dated log files from them (default is the current date but an offset in days can be passed in). This could be run as a cron job (or scheduled task) early in the morning, specifying an offset of -1 to grab all the previous day''s traffic. All log files are parsed as there is no accounting for preferred rollover file sizes and also depending on heavy usage, a single day''s traffic could span several files. The original log files remain intact.
I have submitted a feature request to have the facility to specify log rollover by time rather than by file size and Matt has kindly filed it as
JM-298, so the Perl script would eventually become redundant.
I then have a program in C# - .NET but I''m sure it can be ported to Mono, (I''m all over the place with languages
, except Java
) that parses the dated log file, although it can quite happily parse the original audit file as the only difference being the dated log files only contain message packets for a specific date. I''ve not tested very large files.
Messages are presented in the form of a list, giving date, sender, message type (chat or message), subject and first 20 words of message body. Double clicking will provide the whole message. This list can then be filtered by senders and message type, i.e. being able to focus in on a chat, or just view all messages sent by a particular user.
As message packets are logged twice, ie. from the sender''s stream and also to the recipient''s stream, these are ignored based on sender, subject (if one exists), and message body. Recipient is ignored to allow for broadcast messages.
I could post the code if anyone is interested, but bear in mind that I tend to take extreme programming to extremes and just omit design completely
. Currently there are no configuration options so lots of stuff is hard coded, but I could fix that before posting. Also none this is internationalised in terms of date/time formats, but again, date formatting could be set as an option.
Things to do:
Stuff this all into a database to allow for easy searching, and add a liberal sprinkling of nice icons.
Cheers,
Jason.
Jason,
Sounds cool.
I''ve given you permission to post attachments in case you want to attach your app or screenshots.
Regards,
Matt
Hi,
I''ve attached a png image of the log viewer itself to whet your appetites. It shows the main points of the application. Appearance is a little bland at the moment. No screen shot for the Perl script - no output on stdout (hopefully none on stderr either!).
The context menu - Focus this type lets you easily focus in on a chat or a series of messages between two users, without having to set the filter via the filter window. Focus all types selects all types of message between the two users. The last one is handy here as lots of users still haven''t got to grips with the difference between chat and message. The Filter window allows you to select from all available users (that are listed in the log file) and view various types of message between them.
For the list of users, the application strips off the domain section of the JID and assumes that if the first portion has a . in it then it is Firstname.Surname and uses proper case on the username (only for display - the original message is stored intact).
Currently, the application only reads the dated log file, however, I will get around to enabling this to be entered into a database.
I''ll do a little work on putting some of the hard coded settings in the log viewer into some sort of config file to make things easier. Hopefully being able to post the code in the next few days.
Sorry, no smilies in the message window
, not yet at least. I''m using a RichText control and displaying images is a little involved if you want to avoid using the clipboard.
I have also attached the Perl script that splits the audit logs into dated files. This should work on both Windows and Linux, the path separator really being the only difference.
The Perl script can take 4 options:
-o specify an offset in days previous to the current day. I.e -o "-1" for yesterday
Default is 0 i.e. today. I suppose that means you can supply and offset for a future date, but that won''t get you anywhere ![]()
-O output location for the dated log file
Default is ./logs (I think this works on Windows regardless of the / separator)
-l location of the jive audit files
Default is /opt/jive_messenger/logs (no idea where the default is on Windows)
-f a template for the audit file
Default is jive.audit-*.log. The script then ''globs'' all matching files.
The output log file is named yyyy-mm-dd.log and is in the same format as the original, just containing the message packets from the specified date.
The current BIG assumption in the Perl script is that the message packet dates are in the form: Fri Jun 02 09:35:00 BST 2005, with BST and GMT being used depending on daylight savings being in effect. I''m sure someone can point out the error of my ways here.
Actually, the Log viewer makes this assumption as well (as that is the format our logs take).
Hope this helps a few people.
Cheers,
Jason.
Message was edited by:
jasonmcclean
Ok, turned out there weren''t that many hard coded settings ![]()
I''ve attached the C# source code. You will need, at least, the .NET v1.1 SDK to compile. Much easier if you have something like Visual Studio .Net, or what I use, a GPL .NET IDE called SharpDevelop (http://www.icsharpcode.net/OpenSource/SD/Default.aspx).
Create a new project, add in all the source files. The file called Log Viewer.exe.config should be placed in the output folder and renamed to match the executable that you produce, i.e. yourApp.exe.config. This file contains the broadcast address, your domain (not used at moment), date/time format for display and the window caption.
I can post the executable if you wish, but you will need the .NET v1.1 runtime to run it.
The whole thing needs a good polish, but it works. Let me know when you find all the bugs.
EDIT
Just checking and noticed something. If you are going to use it on the actual log files produced by Jive messenger, use a copy. This is because in the current log file, the closing </jive> tag as not yet been added. The log viewer checks for this and adds it if it can''t be found. This avoids errors being thrown from the Xml classes being used. Not sure what impact that would have if Jive Messenger found the closing tag when it wasn''t supposed to be there, unless it just blindly appends to the file. However, you would then still end up with an invalid XML file at the end due to two closing tags being present.
Sorry ![]()
End of edit
Cheers,
Jason.
Message was edited by:
jasonmcclean
edit
source added to message towards end
Any feedback?
I''ve added support for emoticons
and you can easily recreate psuedo chats from either chats or related messages.
I''ll update the previous post with new code if anyone interested. If not, nevermind, it''s doing what I need ![]()
Jason.
Added image of chat window created from the logs. (couldn''t add it to previous post as it didn''t originally have an attachment).
I''ll stop now. ![]()
Jason.
Jason,
It''s starting to look very cool! Would you be interested in doing binary releases so that people can more easily use the software?
Regards,
Matt
Would you be
interested in doing binary releases so that people
can more easily use the software?
Not a problem. As long as .NET v1.1 runtime is installed, there shouldn''t be a problem.
Have attached binary.
Emoticon support was added by using the RichEdit control by Khendys Gordon (http://www.codeproject.com/cs/miscctrl/csexrichtextbox.asp)
The emoticons are taken from the Psi client (http://psi.affinix.com) as that is the client we use - using the same icondef.xml as Psi uses. That means any icon set with a corresponding icondef.xml can be used.
Just extract and run.
edit
Oh, edit the Log Viewer.exe.config file first with your broadcast address to pick up broadcast messages. Will work without, but you won''t see broadcasts marked as such, just the first recipient message.
Jason.
Message was edited by:
jasonmcclean
edit
Seem to have lost the binary. Reattached to a message towards the end.
Just wanted to thank you all for your efforts in doing this work... especially the spiffy log reader!
Hi!
I am trying to use the LogViewer, but in both scenarios (source and bin) I get the same error when i open the log file:
==========================================
See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.
Exception Text **************
System.NullReferenceException: Object reference not set to an instance of an object.
at Listawood.Xmpp.LogViewer.XmppLog.Add(XmlNode value) in c:\Documents and Settings\nromero\My Documents\SharpDevelop Projects\LogViewer\XmppLog.cs:line 55
at Listawood.Xmpp.LogViewer.MainForm.LoadFile(String fileName) in c:\Documents and Settings\nromero\My Documents\SharpDevelop Projects\LogViewer\MainForm.cs:line 413
at Listawood.Xmpp.LogViewer.MainForm.OpenLogFile() in c:\Documents and Settings\nromero\My Documents\SharpDevelop Projects\LogViewer\MainForm.cs:line 281
at Listawood.Xmpp.LogViewer.MainForm.MainToolBarButtonClick(Object sender, ToolBarButtonClickEventArgs e) in c:\Documents and Settings\nromero\My Documents\SharpDevelop Projects\LogViewer\MainForm.cs:line 235
at System.Windows.Forms.ToolBar.OnButtonClick(ToolBarButtonClickEventArgs e)
at System.Windows.Forms.ToolBar.WmReflectCommand(Message& m)
at System.Windows.Forms.ToolBar.WndProc(Message& m)
at System.Windows.Forms.ControlNativeWindow.OnMessage(Message& m)
at System.Windows.Forms.ControlNativeWindow.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
Loaded Assemblies **************
mscorlib
Assembly Version: 1.0.5000.0
Win32 Version: 1.1.4322.573
CodeBase: file:///c:/windows/microsoft.net/framework/v1.1.4322/mscorlib.dll
-
LogViewer
Assembly Version: 1.0.2020.24043
Win32 Version: 1.0.2020.24043
CodeBase: file:///C:/Documents%20and%20Settings/nromero/My%20Documents/SharpDevelop%20Pro jects/LogViewer/bin/Debug/LogViewer.exe
-
System.Windows.Forms
Assembly Version: 1.0.5000.0
Win32 Version: 1.1.4322.573
CodeBase: file:///c:/windows/assembly/gac/system.windows.forms/1.0.5000.0__b77a5c561934e0 89/system.windows.forms.dll
-
System
Assembly Version: 1.0.5000.0
Win32 Version: 1.1.4322.573
CodeBase: file:///c:/windows/assembly/gac/system/1.0.5000.0__b77a5c561934e089/system.dll
-
System.Drawing
Assembly Version: 1.0.5000.0
Win32 Version: 1.1.4322.573
CodeBase: file:///c:/windows/assembly/gac/system.drawing/1.0.5000.0__b03f5f7f11d50a3a/sys tem.drawing.dll
-
System.Xml
Assembly Version: 1.0.5000.0
Win32 Version: 1.1.4322.573
CodeBase: file:///c:/windows/assembly/gac/system.xml/1.0.5000.0__b77a5c561934e089/system. xml.dll
-
JIT Debugging **************
To enable just in time (JIT) debugging, the config file for this
application or machine (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.
For example:
When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the machine
rather than being handled by this dialog.
=================================================
What am I missing?
thanks!
I''m guessing that there is a problem with the XML, so the XML classes are not happy. Are you accessing the raw log file or using the Perl script to extract specific dates from the raw log file, then accessing the file that the Perl script creates?
If it is the raw log file, then maybe the bit of code I shoved in to add the missing </jive> tag isn''t working. I''ll have a play about tomorrow and get back to you. Unfortunately no time at the moment. ![]()
Jason.
Nope,
it''s not that. Just taken a file and removed the </jive> tag from it and everything still worked. opened the file up in notepad after and the </jive> was back again, so that is definitely working.
Hmm, if you are using the raw log file, do you have write access to it?
edit
Unfortunately, as I just threw this together for internal use I didn''t put in a lot of error checking code - darn, remember that next time I release code on the unsuspecting world.
Message was edited by:
jasonmcclean
Thanks for your fast response. In fact I''m using the raw data.... first I thought I had to add </jive> to the copy I made...but since i got errors i tried without it as well and same result. Permissions are okey... though now that i thing I might have been trying when the audit-file was still opened (editor)... Now I can''t check that .. but I will do first thing 2morrow morning...
Any other suggestion please tell me!
I will also try to use the perl script to see if that makes a different.
I am pulling up the same exact error. Any clue on how to fix this? I''m looking to implement this fantastic Jive Messenger in a corp. environment where everything can be controlled.. unlike using AIM or MSN or the other crap.
Thanks for your help you guys are very bright.
Thanks,
Steve
Just an FYI - I''m looking to log ALL conversations to reference at any time - without having to install anything locally on each machine.
This is what I am looking for right?
Thanks again,
Steve
Just an FYI - I''m looking to log ALL conversations to
reference at any time - without having to install
anything locally on each machine.
This is what I am looking for right?
All this program does is organise the logs that Jive messenger outputs into a more useable form. The only plave you need to use this program in on the machine that your want to view the logs from.
As for the error, still trying to reproduce it. ![]()
edit
If any use, I''ve updated the source code on the earlier post, however this should give the binary that I posted. You get the error with that as well.
From the debug output, the problem seems to be happening on the initial parse of the logfile. I''ve just tried a raw log and still didn''t get any errors.
Question to devs: Are the xml tags in the logs the same regardless of localisation? What about time stamps?
I''ll update the sources again once I''ve added some error checking around the log file loading code.
Another edit
Source and binary attachments updated. Error checking just reports whether the log file can be loaded into an XmlDocument. If it can''t, that points to invalid XML. It''s a start.
Jason.
Message was edited by:
jasonmcclean
What settings do I have to enable under administration for it to log all conversations had?
Including private messages
Just an FYI - I downloaded the new binary and am getting these errors when trying to view the log file:
First message box: Error parsing child nodes of root node jive
Second message box: Error loading log file
Thanks,
Steve
Nothing was "fixed" with the new binary, just some error tracking. I need to find out where it is going wrong first. ![]()
Ok, we''re zooming in on the problem.
btw, to log all conversations, go to the Server tab (the default one after entering the admin console), then click on Message Audit policy on the left. Click on ''Enable Message Auditing'', then if it is only messages you are after, just click on ''Audit Message Packets''.
This will produce files like jive.audit-0.log etc. These files will contain all the message packets, and it is these files that the log viewer (and associated Perl script) process.
If you don''t have message auditing switched on, what are you trying to view with the log viewer??
I get the same errors as Steve.
I did also check the audit file after trying to open it in LogViewer and now it does have a </jive> at the end.
Btw, I used the binary.
oh, I was just double checking that I had the settings correct - which I did.
This utility would be AWESOME if you programmer guru''s can zoom in on this error ![]()
I''ve sent you a private message with my email address. Could you send me a copy of your log so that I can try and reproduce the error?
Jason.
Ok,
I''ve received a copy of Natata''s log file and the problem is now blindingly obvious. I wrote the log viewer only to be expecting message packets, not everything! This is a bit of major boob on my part
It is the status packets, log on/off packets etc. etc. that is completely throwing the viewer.
Hopefully I''ll have some time today to modify the code to filter out the rest of the other stuff and concentrate only on message packets.
Once again, oops!
Jason.
Hi Natata & Steve,
I think I''ve almost sorted the problem. If you are logging all packets, only message packets are used, everything else is ignored. Also fixed a time stamp problem, but as you didn''t get that far, you wouldn''t have noticed it ![]()
Another thing that I didn''t consider was groupchats, doh! As we don''t currently use them, support for them is limited in the viewer at the moment. You can view individiual messages by double clicking the entry in the list, but you cannot recreate a chat window ... yet. I''ve added the type so that the viewer correctly reports the message type (groupchat), but currently that is as far as it goes at the moment.
Hopefully I''ll get a little more time again later today to try and address the remaining problems, then I will post updated source and binary.
so the Viewer will work if I stop logging everything and simply only log message packets?
so the Viewer will work if I stop logging everything
and simply only log message packets?
Most likely, however, I have removed this restriction. You shouldn''t have to stop logging everything just to use this utility. That''s not the way it should go.
Natata''s log file has highlighted a few other little problems like extracting the date from the message, I was assuming only a 3 character time zone code (like BST), but Natata''s was CEST, which threw it a bit. Fixed that.
Group chat. We don''t use that here yet, so the logger doesn''t cope well with that. When viewing a chat, the log viewer assumes just two participants, however when it can be many (including the room itself), it becomes a little more involved to reconstruct the conversations. Working on that now.
Hopefully I''ll get something posted today.
Jason.
I seem to have fluffed updating the attachments further back in the thread and managed to lose them. So, here they are again.
If logging everything is turned on, only message, chat and groupchat packets are processed. Everything else is ignored.
Reconstruction of a group chat sort of works, but I have by no means thoroughly tested it
Just right click and choose "View as chat".
In order to recognise when the sender or recipient is actually the chat room, the key "Conference" in the config file must be set to the subdomain of your conference, i.e. conference.yourdomain.com. There will be repetitions of messages to and from chat rooms in the main window, but when viewing as a chat, the messages sent from the room are ignored so all you get are the messages sent by the participants of the group chat.
I suppose that I could combine all of these so that you can just specify your domain then just the subdomain part for broadcasting and groupchats. Maybe in the next update.
BTW I''ve not checked how the Perl script copes with everything. I think it just picks out packet nodes so it shouldn''t be affected by what is being logged.
Let me know how you get on.
Jason.
edit
Attachments have been updated - 01-09-2005
Message was edited by:
jasonmcclean
Many many thanks Jason!
I am able now to load a log file. Somehow when I use focus or filter in GROUPCHAT sender and receiver get a bit messed up. I can perfectly "view as chat" normal chats but with groupchat I got the following undhandled exception:
- 71 is not a valid message item
Parameter name: index
and in details:
See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.
Exception Text **************
System.ArgumentOutOfRangeException: 71 is not a valid message item
Parameter name: index
at Listawood.Xmpp.LogViewer.Sender.get_Item(Int32 index)
at Listawood.Xmpp.LogViewer.ChatWindow.PopulateGroupChatWindow(Boolean useEmoticons)
at Listawood.Xmpp.LogViewer.ChatWindow.Setup(XmppLog log, XmppMsgType msgType, String room)
at Listawood.Xmpp.LogViewer.ChatWindow..ctor(XmppLog log, XmppMsgType msgType, String room)
at Listawood.Xmpp.LogViewer.MainForm.GroupChatWindow(XmppMsg msg)
at Listawood.Xmpp.LogViewer.MainForm.MenuItem3Click(Object sender, EventArgs e)
at System.Windows.Forms.MenuItem.OnClick(EventArgs e)
at System.Windows.Forms.MenuItemData.Execute()
at System.Windows.Forms.Command.Invoke()
at System.Windows.Forms.Control.WmCommand(Message& m)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ListView.WndProc(Message& m)
at System.Windows.Forms.ControlNativeWindow.OnMessage(Message& m)
at System.Windows.Forms.ControlNativeWindow.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
Loaded Assemblies **************
mscorlib
Assembly Version: 1.0.5000.0
Win32 Version: 1.1.4322.573
CodeBase: file:///c:/windows/microsoft.net/framework/v1.1.4322/mscorlib.dll
-
Log Viewer
Assembly Version: 1.0.2026.15186
Win32 Version: 1.0.2026.15186
CodeBase: file:///C:/Documents%20and%20Settings/nromero/My%20Documents/TUe/Development/ji ve_messenger_src/LogViewer-bin/Log%20Viewer.exe
-
System.Windows.Forms
Assembly Version: 1.0.5000.0
Win32 Version: 1.1.4322.573
CodeBase: file:///c:/windows/assembly/gac/system.windows.forms/1.0.5000.0__b77a5c561934e0 89/system.windows.forms.dll
-
System
Assembly Version: 1.0.5000.0
Win32 Version: 1.1.4322.573
CodeBase: file:///c:/windows/assembly/gac/system/1.0.5000.0__b77a5c561934e089/system.dll
-
System.Drawing
Assembly Version: 1.0.5000.0
Win32 Version: 1.1.4322.573
CodeBase: file:///c:/windows/assembly/gac/system.drawing/1.0.5000.0__b03f5f7f11d50a3a/sys tem.drawing.dll
-
System.Xml
Assembly Version: 1.0.5000.0
Win32 Version: 1.1.4322.573
CodeBase: file:///c:/windows/assembly/gac/system.xml/1.0.5000.0__b77a5c561934e089/system. xml.dll
-
Khendys.Controls.ExRichTextBox
Assembly Version: 1.0.1985.28025
Win32 Version: 1.0.1985.28025
CodeBase: file:///C:/Documents%20and%20Settings/nromero/My%20Documents/TUe/Development/ji ve_messenger_src/LogViewer-bin/Khendys.Controls.ExRichTextBox.DLL
-
JIT Debugging **************
To enable just in time (JIT) debugging, the config file for this
application or machine (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.
For example:
When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the machine
rather than being handled by this dialog.
==========
The tool looks very promising!! thanks once more for all your effort Jason... I''m trying to go through the code now to understand how it works so I can help with the errors ![]()
Hi Natata,
Glad to hear that you can load a log file. Group chat, hmm, that was only added yesterday and is still up in the air in terms of working. I need to test that further. The problem is that we don''t really use groupchat here yet, so I''ve not really had anything to test the code with. I''ll probably be setting up a testing group chat later today with a few colleagues so that I can get some log entries to test with.
BTW it''s just the first few lines of the error that are most useful as that is closest to where the error actually is. I.e. the exception text.
Thanks,
Jason.
I had the error with the root node jive also.
I was compiling the source. it turned out that if my exe was named anything other than log viewer.exe it errored.
Looks really nice, yes!
Just tried to compile it on Mono, but in spite of a (citing the website) 98% complete implementation of System.Windows.Forms it''s still missing the RichTextBox.. sigh
"Major areas still needing work are DataGrid, RichText, MDI and Printing."
Would be very cool to have your tool on a cross-platform framework..
Regards,
Ben
Looks really nice, yes!
Just tried to compile it on Mono, but in spite of a
(citing the website) 98% complete implementation of
System.Windows.Forms it''s still missing the
RichTextBox.. sigh
"Major areas still needing work are DataGrid,
RichText, MDI and Printing."
Would be very cool to have your tool on a
cross-platform framework..
Haven''t played with Mono yet (forgot about Windows.Forms Doh!). Maybe soon ![]()
edit
don''t see a reason why it can''t be re-implemented using something other than C#. That just happens to be my language of the moment.
Jason
Message was edited by:
jasonmcclean
what about *nix?
i mean bin
what about *nix?
i mean bin
Do you mean a *nix binary? Sorry, not until this could be compiled with Mono... ![]()
I''ve only got as far a scripting (Perl & PHP) on *nix.
Well - a mono assembly is not exactly what I associate with a unix binary (since it runs on MS .NET as well and is nothing native).
I''ve some experiences on *nix, so I''m not whining here.. It''s just that I tried (curiosity..) to compile the app with mono.
Well - a mono assembly is not exactly what I
associate with a unix binary (since it runs on MS
.NET as well and is nothing native).
What I meant was, Mono is the only way this particular app will most likely ever end up running on a *nix system. It''s a long time since I played with c/c++. Pascal (whatever version), not sure.
Jason,
Any interest in playing with desktop Java?
It''s actually pretty great these days. It''s fast and looks native. It would certainly fit well with JM itself...
-Matt
Jason,
Any interest in playing with desktop Java?
It''s
actually pretty great these days. It''s fast and looks
native. It would certainly fit well with JM
itself...
Definitely an idea. I''ve not really tried Java before, but I guess this could be the perfect opportunity to start ![]()
edit s/not really/never/ ![]()
Message was edited by:
jasonmcclean
Work has started on porting the log viewer to Java. Not as bad as I thought as C# is quite close syntactically (since java was a big influence). GUI will come last.
Any views as to whether this should still be a stand alone application, an applet or a messenger plugin (no idea how to tackle that one
)?
Jason.
That''s good news to hear.
One thing that would be very interesting is indexing the logs using Lucene (http://lucene.apache.org/). That would let people do full-text searches to locate conversations.
As for the medium (app, applet, plugin). Eventually, we''ll definitely need a robust tool inside the Jive Messenger admin console for viewing log information. However, there may definitely be a place for a desktop application as well. So, I would start where you want to and see where it goes.
-Matt
and i vote for a plugin first:)
I have Jive 2.1.5 running on a fresh install of Solaris 9. I was able to take the audit log from the Solaris box, ftp the log over to a Windows 2000 machine, run the binary that was posted and viewed the chats perfectly.
Works very well ... unless you have 0 Windows machines at your location. ![]()
windows binary?? Do i''ve missed something?
LogViewer-bin.zip (55.1 K)
Would you be
interested in doing binary releases so that people
can more easily use the software?
Not a problem. As long as .NET v1.1 runtime is installed, there shouldn''t be a problem.
Check the first page of posts ... the one from Jason with the ZIP attached has an EXE file to view logs in a very easy to use format ... I''ve already busted a few people with it. ![]()
I''ve got Jive running on Linux, so I''m just using a restricted Samba share to access the dated log files produced by the Perl script. No copying required. ![]()
Haven''t played with Mono yet (forgot about
Windows.Forms Doh!). Maybe soon
It''s currently "in the pipeline" (Talking about RichTextBox here). But the next problem would be, that you are using the Codecentral approach of inserting icons, right? With P/Invoke on gdi/windows dlls..
edit
don''t see a reason why it can''t be re-implemented
using something other than C#. That just happens to
be my language of the moment.
I do like C# as well (currently even better than Java sometimes). It certainly could be implemented cross-platform with C# / gtk# or lots of other languages of course (Well - Java would be quite a good idea for a java IM server.. ponder)
wroot: No idea what you mean, sorry?
wroot: No idea what you mean, sorry?
nevermind. i guess i''ll go to bed right now, this is the best solution so far for me:)
It''s currently "in the pipeline" (Talking about
RichTextBox here). But the next problem would be,
that you are using the Codecentral approach of
inserting icons, right? With P/Invoke on gdi/windows
dlls..
The RichEdit control that I''m using has extended the standard one by adding in functions to wrap up an image in a Windows MetaFile (WMF) and inserting directly into the control wrapped in RTF codes. If you looked at one of the icons in this state you would see the RTF control codes and the image represented in Hex.
The way I''m using it is I''ve created a class to search for the emoticons using regexes then generate the RTF codes required and replacing in the text of the message. The message is then inserted into the RichText control as standard RTF text.
As RTF is a well known standard, as soon as Mono sort out the control, this should work
(I think)
It''s currently "in the pipeline" (Talking about
RichTextBox here). But the next problem would be,
that you are using the Codecentral approach of
inserting icons, right? With P/Invoke on
gdi/windows
dlls..
The RichEdit control that I''m using has extended the
standard one by adding in functions to wrap up an
image in a Windows MetaFile (WMF) and inserting
directly into the control wrapped in RTF codes. If
you looked at one of the icons in this state you
would see the RTF control codes and the image
represented in Hex.
The way I''m using it is I''ve created a class to
search for the emoticons using regexes then generate
the RTF codes required and replacing in the text of
the message. The message is then inserted into the
RichText control as standard RTF text.
If you are using the Codecentral version (http://www.codeproject.com/cs/miscctrl/csexrichtextbox.asp): That one uses Platform/Invoke and uses windows-only functions. Nothing wrong with that, but since it uses for example
[DllImportAttribute("gdiplus.dll")]
private static extern uint GdipEmfToWmfBits (IntPtr _hEmf,
uint _bufferSize, byte[] _buffer,
int _mappingMode, EmfToWmfBitsFlags _flags);
this won''t ever work on mono on a non-windows system.. Well - it would work, if you have the same function and can map the dll.. But - GDI is quite windows-specific, I guess. So is this function.. ![]()
As RTF is a well known standard, as soon as Mono sort
out the control, this should work
(I think)
Yes, what ends up in the box is RTF and fine. The way it gets there is what I was talking about..
But I may be wrong, of course. And it''s seems a great tool anyway, independent of that "cosmetic" thing.
If you are using the Codecentral version
(http://www.codeproject.com/cs/miscctrl/csexrichtextbo
x.asp): That one uses Platform/Invoke and uses
windows-only functions. Nothing wrong with that, but
since it uses for example
[DllImportAttribute("gdiplus.dll")]
private static extern uint GdipEmfToWmfBits (IntPtr
tr _hEmf,
uint _bufferSize, byte[] _buffer,
int _mappingMode, EmfToWmfBitsFlags _flags);
this won''t ever work on mono on a non-windows
system.. Well - it would work, if you have the same
function and can map the dll.. But - GDI is quite
windows-specific, I guess. So is this function..
Just goes to show I didn''t look closely at the code ![]()
edit
I wonder if a browser type component could be used instead?
Message was edited by:
jasonmcclean
Good idea.. Well - mono would suggest Gecko I guess. Not sure what MS .NET offers there.. I''ll have a look at it this morning.. ![]()
How do i use this?
I''m not used about Perl...
I''ve installed Jive messenger and it''s working ok!
How to run Log Viewer? Need to install Perl?
Thank''s in Advance
How do i use this?
I''m not used about Perl...
I''ve installed Jive messenger and it''s working ok!
How to run Log Viewer? Need to install Perl?
Thank''s in Advance
Hi there,
No you don''t necessarily need Perl. All the Perl script does is to enable splitting the audit log files into dated files, so it is easier to keep track. At some point I think the option to roll over the log files based on time will be included, just not yet ![]()
To run the Log viewer, however, you will need the Microsoft .NET 1.1 runtime installed (available from the Microsoft web site).
All the log viewer does is to present the Jive audit logs in a more readable form. It is still work in progress. You need to have auditing turned on (In the admin console - in the logs section).
Hope that helps.
Jason.
Jason,
Just wanted to say Excellent work on the chat logger!!
Makes searching a lot easier then using Hurrican Search or WinGrep to find stuff
Thanks again!
Edit
I see no option to export from filtered searchs, this would be a nice thing to see added
Message was edited by:
lcameron
I see no option to export from filtered searchs, this
would be a nice thing to see added
Good idea. I''d also like to add this stuff to a database aswell, unfortunately, I just don''t have much time at the moment. I''m also in the process of familiarising myself with Java so I can port this to Java and be a little more intune with Jive and a little more cross platform.
Things to do:
- Port to Java
- enable phrase/word searching withing the viewer
- export filtered results
- DB storage (so you don''t have to keep opening older files)
- Maybe turn the viewer into a plugin... That would be nice.
The source is posted in this thread so anyone is more than welcome to have a go too ![]()
Cheers,
Jason.
Log Viewer shows me all dates as 01/01/1980, does anybody know why?!
Thank''s!
the_lok,
The log viewer grab''s the date from the log files, so you may want to check your system date and make sure it is correct.
That is the default date given if for some reason the date can''t be obtained from the log. It may be a format thing. I thought that would be a better idea than throwing an error.
There are lots of ways this app can be improved, internationalising it and coping with different date and time formats is one of them.
How is the date formatted in your log?
Jason
My date format:
timestamp="Mon Aug 29 08:04:44 GMT-03:00 2005"
My date format:
timestamp="Mon Aug 29 08:04:44 GMT-03:00 2005"
That''s the kiddy
. The regex that pulls out the date and time is being thrown off by the -03:00 bit.
I''ll have a fiddle with that and upload new source and binary packages on the message a little further back on this thread.
Will try and get that sorted today
Jason
edit
Done. I don''t actually do anything with the -03:00 part, just that it doesn''t cause the regex to fail anymore.
Attachments further back in the thread have been updated.
Message was edited by:
jasonmcclean
Question to the Devs.
Do you think that I should move this off the forums, just leaving a pointer to where it has gone, or are you quite happy for this thread to keep getting bigger?
Cheers,
Jason.
Again wanted to thank Jason for his dedication to getting this thing up and running... I wish I knew something about programming so I could be of assistance to you on this. But I am sure that everyone who has found this has rejoiced at finding something other than opening the log in a text editor.
Brett
I started using this perl script on my corporate server recently and I noticed a bug in the code.
At the bottom of the script, the code reads as follows:
Re: Logging Conversations #short date
#add leading zero to month, if required
#first make it non-zero based
$mon++;
if ($mon<10) {
$month = "0$mon";
}
my $shortDate = "$year-$month-$day";[/i]
It needs to be modified in one of several ways (I have tested multiple which work, though I''ll leave it up to each person to execute it in their own way as I have zero perl experience and I know there is a much much more elegant solution, but I simply don''t know enough about the language syntax to code it myself.)
The problem is this: if the value variable "mon" is 10 or greater, the variable "month" (which is used to create "shortDate") has a value of null. This causes the resulting filename of the log file to be "2005--04.log" instead of "2005-10-04.log"
The fixes I have tried:
Re: Logging Conversations1)[/b] Add an else statement to the if statement that sets "month" to "mon"
Re: Logging Conversations #short date
#add leading zero to month, if required
#first make it non-zero based
$mon++;
if ($mon<10) {
$month = "0$mon";
}
Re: Logging Conversations else {
$month = "$mon";
}[/b]
my $shortDate = "$year-$month-$day";[/i]
Re: Logging Conversations2)[/b] Re: Logging ConversationsModify the line assigning a value to "shortDate" to use "mon" instead of "month" and modify the if statement to assign the resulting value back to "mon"
#short date
#add leading zero to month, if required
#first make it non-zero based
$mon++;
if ($mon<10) {
Re: Logging Conversations$mon = "0$mon";[/b]
}
Re: Logging Conversationsmy $shortDate = "$year-$mon-$day";[/b][/i]
Hope this helps others that are using this perl script who may or may not have noticed this bug.
Peace,
-Bartman007
Thanks for the fix.
I''m by no means an expert with Perl either
but that was just simply stupid of me! Just shows also that I''ve not actually checked my logs for the last week or so, otherwise I would have seen that too - Doh!
Undoubtedly there are many more ''issues'' to be found and I welcome people pointing them out.
Cheers,
Jason.
edit
p.s. I think I favour the shorter solution. Why introduce variables when you don''t have too. Good point! Why did I? ![]()
Hi Jason!
Just checked that software and liek it so far, bhut:
- i second the option to export or print results (export is probably better for formatting purposes -> Export to CSV)
then one thing i miss a lot:
The filter seems to only filter the ''from'' address, and not the too address'', so if i want to check a complete conversation between user1 and all other users it should also show replies user1 got from user 2, so everythign where ''to'' is user2 too.
also a selction by date (grouped) woudl be nice ![]()
I think the filter thing broke a little when I had to rejig some things a little while ago. Exporting would be good, I totally agree, however, I''m a really bogged down in other projects at the moment ![]()
I think the thing to do would be to move this onto another forum where anyone wishing to contribute can do so more easily.
Jason.
When pointing the viewer to the log file, I get an message "Error parsing child nodes of root node jive" then a "Error loading log file." I''m trying to run the binary and view Wildfire 2.4.0 logs. Did the logs change with the change to Wildfire by chance...they look the same to me. ![]()
Thanks,
Chad
Not sure about the log files. I''m not currently using the latest version :-o
I''ll look into it anf get back to you.
Jason.
ignore this, ironing out the kinks with it working with 2.4
Hello everyone - has this project been abandoned? if so could someone please point me to the right place to look?
I have an issue with the current binaries where when i pull my log into the viewer every chat has a date / time of 1/1/1980 00:00:00 - this really screws up attempting to make sense of a conversation.
When I open the .log file itself i can definately see that the dates are in there correctly. Please help!!
Thank you
Hello everyone - has this project been abandoned?
It''s not been abandoned per se, however I just don''t have any time to spend on it. Shucks, I didn''t even notice the name change to Wildfire until I got notification of this post. Is that bad? Do i really need to be upgrading my Jive Messenger installation (2.2.0 and working wonderfully - especially since I got my new server
). Maybe I need to look at that anyway.
>if so could someone please point me to the right place
to look?
I have an issue with the current binaries where when
i pull my log into the viewer every chat has a date /
time of 1/1/1980 00:00:00 - this really screws up
attempting to make sense of a conversation.
This will be due to the date handling and the huge assumptions that I made when creating the regex to extract them. what is happening is that it is not properly recognising the dates, therefore a default date is being used. Regular expressions are not my strongest subject, but I''m getting better at it.
When I open the .log file itself i can definately
see that the dates are in there correctly. Please
help!!
Oh yes, the dates will be fine in the log file. It is the log viewer that is having the problem.
If you can post an extract of the date format in your log, that will help me refine the date extraction regex. Really needs overhauling anyway - bit too specific.
Please bear in mind that this project started of as an internal project so that I, and a few others in management could easily review what was happening on our system. The viewer still works fine for us - this is where the big assumptions came in. I had no access to the formats of other installations. I threw the code into this forum so others could take advantage. Please, please feel free to review the source and see if anyone can help take this project forward.
Hopefully, soon I will have more time to spend on this and take it to the next level, fix all the broken bits and finally port it to Java (I have started learning Java, honest
).
Fingers crossed, I may soon have some splutter free time.
Jason.
Just had a thought. As I''m several versions behind, has any progress been made with the internal log viewer? There was talk of this some time ago.
Just wondering. Not had a chance to trawl through the forums and stuff.
Jason.
Just had a thought. As I''m several versions behind,
has any progress been made with the internal log
viewer? There was talk of this some time ago.
It seems to me that this would be the best way to handle it. IMHO, either the core or at least a plugin needs to be developed that will put the logs in a readable fomat.
cb
There was talk of it, but that appears to have fall by the wayside. There was also discussion of date-based log files, http://www.jivesoftware.org/issues/browse/
JM-298 , but the version it was to be implemented in has been bumped back several times and now sits at 2.5.0
khoehn, a forum member has created a admin console plugin (http://sourceforge.net/projects/iball-auditor) that creates logs - separate from the built-in Message Auditing - which are specifically for chat and MUC. This plugin integrates a viewer into the Admin console. A request has been made to integrate this plugin into Jive/Wildfire itself, http://www.jivesoftware.org/issues/browse/
JM-523 Based on posts khoehn has made on the forums, he plans to continue developing the plugin and has plans for the ability to search through the logs in various methods.
I plan to test this plugin but probably won''t be able to for several months as other projects take priority.
Message was edited by: Bartman007
I plan to test this plugin but probably won''t be able
to for several months as other projects take
priority.
Know the feeling ![]()
Jason.
I am almost done with a related project for my coursework here in graduate school. I have written an app that listens for message packets and writes them to an .xml file. I currently filter to only write MUC messages, but it could be modified. These files are then mirrored into a WEBDAV enabled native-xml database (using eXist). The files are then accessible for query via an xquery webapp.
This stuff has a lot of design tweaks that are specifically aimed at meeting the needs of my problem, but the basic principles of logging to an XML doc, importing into an XMLdb, and then web-enabled query access is working out pretty slick and isn''t too tough to implement either.
I am having the same problem as ChiefofTheClass is having with the 0 dates. The date format in the log shows up as:
Mar 17, 2006 6:03:19 PM
Mar 18, 2006 12:00:40 AM
Hopefully this can be corrected quickly, I know your pressed for time but we use the chat for internal communication and a small HR issue has arisen in which we need a manager who isn''t quite as ready to read raw logs.
Hi Traciatim,
well you know how it is. If one falls off a roof and can not fly it has only little time to learn flying *) **), if one falls in the water and can not swim it has only little time to learn swimming. So you have a manager which can not read raw logs, so it may teach itself very fast to read them
unless one can provide a solution.
( Someone mentioned: "If you disappeared from the roof falling brake before you down land." *)
( some lyrics: "The important thing is not the fall but the landing" **)
LG
Traciatim
May be, it will get the fix you want in WildFire 2.6.0 ?
JM-298 Try voting there and please comment the date format you want there so that the developer can implement that too.
Regards,
wmhtet
As far as I understand it,
JM-298 just allows raw log files to be rolled over based on date rather than size. This is a feature that I requested and removes the need to run the Perl script (see earlier in the thread) to split the log files into dated files.
This doesn''t get around the date parsing problem.
What we have is a multitude of date formats. If the log viewer ran on the same platform that generates the logs, we would probably not have this problem as it would probably utilise the same date format as Wildfire.
However, the log viewer is currently restricted to Windows and .NET (Windows Forms and the Rich Edit control prevent compilation under Mono at this time).
There are several possible solutions to this problem:
1. Having some sort of option to define the timestamp format when the logs are actually generated - This really isn''t a fair solution as that just offloads the problem onto the Wildfire devs and they have got better things to do.
2. Port the log viewer to Java so it can run on the same platform and automatically utilise the same sort of timestamp format - This was mooted some time ago, but I haven''t got time at the moment to do a port (whilst learning Java in the process), however this is something I want to do.
2.5 Related to above - transform the viewer into a console plugin. The only problem I can think about that one is providing access to other people to the admin console to view the logs. It may not always be the tech guys that are reviewing these.
3. Have an option in the current implementation of the Log Viewer to define the log timestamp format. Obviously this is the quickest solution, but how to implement it?
The Log Viewer user could probably define a regex in the config file that would be used during parsing, but that means the user needs to have some understanding of regular expressions. If some people balk at just looking at XML, maybe that isn''t a good way to go.
Use standard dd/mm/yyyy type definitions in the config file - probably the best way and is probably what I will try and do.
I know that there are other things that need doing with this application, like fixing the filtering options, making sure group chats are rendered correctly and introducing indexing and exporting, but the date thing seems to be the most critical.
I know that I have left the project languishing for a while, but I will try and get new source and binary files posted as soon as I can.
Jason.
edit added clarification of restriction to .NET
another edit Has anyone had a go at hacking the source?
Message was edited by: jasonmcclean
I took a crack at the source code since my computer is logging the dates in the same format as described above..
<jive xmlns="http://www.jivesoftware.org"><packet xmlns="http://www.jivesoftware.org" streamID="7915039" status="auth" timestamp="Aug 1, 2006 2:13:17 PM">
I modified the code in XmppMsg.cs
I also configured my date time format in the config file to be yyyy/MM/dd HH:mm:ss because i like my dates and times to be chronologically sorted. i at first used a smal mm for the months, but that caused it to put the minutes in the month''s spot. The capital MM works for months properly, my bad
The XmppMsg.cs section modified i will paste here. The RED code is what i put in/changed I have a binary that works for this. if anyone is interested email me and I''ll send it to you.
public void SetTimeStamp(DateTime timeStamp)
{
_timeStamp = timeStamp;
}
public void SetTimeStamp(string dateTime)
{
//Msg time stamp in form of
//day month date time zone year
//e.g Mon Aug 29 08:04:44 GMT-03:00 2005
{color:#ff0000}dateTime = string.Format("",DateTime.Parse(dateTime));
Regex r = new Regex(@"(\w?)\s(\w?)\s(\d)\s(\d:\d:\d)\s\w{3,4}[0-9\-:|0-9/-:]*?\s(\d)");
Match m = r.Match(dateTime);
if(m.Success)
{
_timeStamp = GetDateTime(m);
}
else
{
//this is what you get if, for some reason, the regex fails to extract the date and time
{color:#ff0000}//_timeStamp = DateTime.Parse("01/Jan/1900 00:00:00");
_timeStamp = DateTime.Parse(dateTime); }
}
I am not good at per Regular expressions so i just formatted the string containing the date to match the format of the 01/Jan/1900 and forced it to pass my variable instead. If there is a better format that would be best. Please contact me if you have any questions or would like to help me with writing this section better.