Jahia DX > Dynamic module showing...

0 (0 Good)
0 (0 Bad)

Dynamic module showing IRC logfiles

by  rvt »  2012/11/16 02:25

Hello,

I am converting my website to Jahia and looking into showing my eggdrop IRC logfiles again, to show you what I mean, here is my current output site: http://www.rvantwisk.nl/librecad-irc-log/2012/november/1.html

A little background around eggdrop:

Eggdrop stores each days as a single file in the following file format <channelname>.log.<day><month><year> for example librecad.log.01Apr2012 having each entry of the log as follows: 

[19:56] <ries> I think shared_ptr will work, unfortunately it now crashes 'somewhere' and my debugger decided to bail out

Currently give or take I have around 100Mb of logfiles over a couple of channels.

What would be the correct/suggested way in to do this in Jahia in such a way that I can stil have a search box on my website that enabled searching in logs, and showing the search result entries on the right page. But at the same time have a sensible method of loading the logfiles into Jahia, I currently lack knowledge on the default search engine of Jahia...

I thought of the following methods:

  1. Load files into the repository in the files section. Then create a module that can read and parse the files and display them on a page as a dynamic module. However, in that case I am not sure if the search engine will pickup and indexes the dynamic pages correctly. I could install this (https://github.com/bpapez/Jahia-Crawler) apache nutch, but I try to avoid that as of now...
  2. Create a document parser (extend Tika?) and create a module that can display this content, however this will properly allow the search engine to index the documents, but will show them on the wrong location as I want to have the correct 'URL' show up in the search results, and nota indexed document's location in the files section.
  3. Create a cron job and load each line of each file as a seperate content element / nodes into Jahia on the right page. This will properly work for search indexging, and filtering of content. However, this somehow look bloated because each line in each file is a seperate node. This will add around 780000 additional nodes, one node per line entry as this the current number of lines I have in my logfiles.

I am currently thinking of method 2. because this seems to make most sence. I am pretty sure that 3. will work, but that just looks very evil to me.

Looking forward to your opinion!

 - Ries

  • Dynamic module showing IRC logfiles
    2012/11/16 02:25

    rvt <p> Hello,</p> <p> I am converting my website to Jahia and looking into showing my eggdrop IRC logfiles again, to show you what I mean, here is my current output site:&nbsp;http://www.rvantwisk.nl/librecad-irc-log/2012/november/1.html</p> <p> A little background around eggdrop:</p> <p> Eggdrop stores each days as a single file in the following file format &lt;channelname&gt;.log.&lt;day&gt;&lt;month&gt;&lt;year&gt; for example librecad.log.01Apr2012&nbsp;having each entry of the log as follows:&nbsp;</p> <p> <em>[19:56] &lt;ries&gt; I think shared_ptr will work, unfortunately it now crashes &#39;somewhere&#39; and my debugger decided to bail out</em></p> <p> Currently give or take I have around 100Mb of logfiles over a couple of channels.</p> <p> What would be the correct/suggested way in to do this in Jahia in such a way that I can stil have a search box on my website that enabled searching in logs, and showing the search result entries on the right page. But at the same time have a sensible method of loading the logfiles into Jahia, I currently lack knowledge on the default search engine of Jahia...</p> <p> <u>I thought of the following methods:</u></p> <ol style=""> <li> Load files into the repository in the files section. Then create a module that can read and parse the files and display them on a page as a dynamic module. However, in that case I am not sure if the search engine will pickup and indexes the dynamic pages correctly. I could install this (https://github.com/bpapez/Jahia-Crawler) apache nutch, but I try to avoid that as of now...</li> <li> Create a document parser (extend Tika?) and create a module that can display this content, however this will properly allow the search engine to index the documents, but will show them on the wrong location as I want to have the correct &#39;URL&#39; show up in the search results, and nota indexed document&#39;s location in the files section.</li> <li> Create a cron job and load each line of each file as a seperate content element / nodes into Jahia on the right page. This will properly work for search indexging, and filtering of content. However, this somehow look bloated because each line in each file is a seperate node. This will add around&nbsp;780000 additional nodes, one node per line entry as this the current number of lines I have in my logfiles.</li> </ol> <p> I am currently thinking of method 2. because this seems to make most sence. I am pretty sure that 3. will work, but that just looks very evil to me.</p> <p> Looking forward to your opinion!</p> <p> &nbsp;- Ries</p>

  • Number of messages  185
    Registration date Nov 16, 2012
    0 (0 Good)
    0 (0 Bad)

    Re: Dynamic module showing IRC logfiles

    by  shuber »  2012/11/16 07:29

    Hi Ries, 

    Nice to hear from you again ! How are things it's been a while ! 

    I think you should either use option 1 or 2. In option 1, the only thing you would have to check is if the file format is properly recognized by the indexer. You could then modify the search result page to build a custom link to display a file, and do whatever processing you need on it at display time.

    For option 2, I would think that you would probably want to customize the search field to split the operation between searching in Jahia's index and your own. Not the easiest thing to do but knowing you you might be able to pull it off :) 

    Option 3 I would forget, creating that many nodes in Jahia is not a good idea if it can be avoided.

    Regards,

      Serge Huber.

    ps : personnally I would probably try option 1 first, as it requires the less work :)

    Serge Huber (shuber)

    Number of messages  271
    Registration date
    0 (0 Good)
    0 (0 Bad)

    Re: Re: Dynamic module showing IRC logfiles

    by  rvt »  2012/11/19 13:45

    Hello Serge,

    thanks for the answer..

    I have been implementing Option 1 as per your suggestion and with Nutch the complete website was indexed. They only item I didn't do was importing the file's into the repository as I could just as well get them straight from the filesystem anyways.

    Next step is to refactor the module a bit so we can also load in other chatlog file types then I need to add some options to ensure I can allow/deny specific items so the system cannot be used for abuse (generate links to other websites). I will put the module on github later this week.

    Ries

  • Re: Re: Dynamic module showing IRC logfiles
    2012/11/19 13:45

    rvt <p> Hello<font color="#333333" face="Helvetica Neue, Helvetica, Arial, sans-serif"><span style="font-size: 14px; line-height: 20px;">&nbsp;Serge,</span></font></p> <p> thanks for the answer..</p> <p> I have been implementing Option 1 as per your suggestion and with Nutch the complete website was indexed. They only item I didn&#39;t do was importing the file&#39;s into the repository as I could just as well get them straight from the filesystem anyways.</p> <p> Next step is to refactor the module a bit so we can also load in other chatlog file types then I need to add some options to ensure I can allow/deny specific items so the system cannot be used for abuse (generate links to other websites). I will put the module on github later this week.</p> <p> Ries</p>

  • Number of messages  185
    Registration date Nov 19, 2012
    0 (0 Good)
    0 (0 Bad)

    Re: Re: Re: Dynamic module showing IRC logfiles

    by  rvt »  2012/11/21 19:58

    Serge, and others interesetd...

    Here you can find the beta version of the module: https://github.com/rvt/rvt_irclogs

    Ries

  • Re: Re: Re: Dynamic module showing IRC logfiles
    2012/11/21 19:58

    rvt <p> Serge, and others interesetd...</p> <p> Here you can find the beta version of the module:&nbsp;https://github.com/rvt/rvt_irclogs</p> <p> Ries</p>

  • Number of messages  185
    Registration date Nov 21, 2012
    0 (0 Good)
    0 (0 Bad)

    Re: Re: Re: Re: Dynamic module showing IRC logfiles

    by  faissah »  2012/12/04 17:28

    Hi Ries,

    Thanks for sharing. I just fork it in our Jahia Git repository.

    Regards,

    Fabrice

    Fabrice  (faissah)

    Number of messages  262
    Registration date
    Contact
    Share
    Feedback

    Get in touch

    Whether you are a current user or if you are just evaluating Jahia, we are here to help.

    Contact us

    Share this page