Jahia Dev Forum > Huge site export

0 (0 Good)
0 (0 Bad)

Huge site export

by  mxlab »  2012/11/19 09:09

Hi all!

I have a site with lots of contents like news, pages, articles, files, ... for a total size of 8GB.

I'd like to export it with the site export tool but when I try to do so the system hangs, the file downloaded is corrupted and its size is much smaller than the total size (i.e. 300MB).

How can I do? Are there any tools, maybe in jahia enterprise?

Thanks!

  (mxlab)

Number of messages  33
Registration date
0 (0 Good)
0 (0 Bad)

Re: Huge site export

by  rvt »  2012/11/19 13:31

mx,

our website is not as large as yours, but we do have PDF and video's online. Our exported repository exported as a zip is a little less then 6GB. I wrote a script a while ago that could export it using shell access, it's located here : http://www.jahia.com/cms/home/download/forum/jahia-user/jahia-65/script-to-backup-your-sites-with.html

I use this script with Jahia 6.6 to create a zipfile of our sites so the ycan be stored away.

Ries

 

  • Re: Huge site export
    2012/11/19 13:31

    rvt <p> mx,</p> <p> our website is not as large as yours, but we do have PDF and video&#39;s online. Our exported repository exported as a zip is a little less then 6GB.&nbsp;I wrote a script a while ago that could export it using shell access, it&#39;s located here :&nbsp;http://www.jahia.com/cms/home/download/forum/jahia-user/jahia-65/script-to-backup-your-sites-with.html</p> <p> I use this script with Jahia 6.6 to create a zipfile of our sites so the ycan be stored away.</p> <p> Ries</p> <p> &nbsp;</p>

  • Number of messages  185
    Registration date Nov 19, 2012
    0 (0 Good)
    0 (0 Bad)

    Re: Re: Huge site export

    by  mxlab »  2012/11/19 14:38

    mx,

    our website is not as large as yours, but we do have PDF and video's online. Our exported repository exported as a zip is a little less then 6GB. I wrote a script a while ago that could export it using shell access, it's located here : http://www.jahia.com/cms/home/download/forum/jahia-user/jahia-65/script-to-backup-your-sites-with.html

    I use this script with Jahia 6.6 to create a zipfile of our sites so the ycan be stored away.

    Ries

     

    Thank you very much!!!!

    But...is it normal that I can see only a white page?

      (mxlab)

    Number of messages  33
    Registration date
    0 (0 Good)
    0 (0 Bad)

    Re: Re: Re: Huge site export

    by  rvt »  2012/11/19 19:11

    mx,

    I noticed in our case our proxy server would give kill the connection after XX seconds of 'inactivity' so we needed a method to

    1. Automate it
    2. ensure we don't pull data through our proxy server we still wanted to have a short timeout towards clients

    The whitepage you are seeing is highly lickly because the connection get's killed, and I have seen it myself a bunch of times  with similar effects (not Jahia related apps even).

    Come to think of it, in GlassFish there was a setting we also had to change: 

                <http request-timeout-seconds="3600" default-virtual-server="_server">                                                                                                     
                  <file-cache></file-cache>                                                                                                                                               
                </http> 
     
    I remember I had to set the *request-timeout-seconds* also to ensure that glassfish wouldn't kill the process, if you are running Tomcat then I am sure it also has such setting.
     
    Both timeouts are rather easy to spot because you will see that your export always stops at around teh same datasite, but more import anround the same time after you initiate the process of exporting.
     
    Ries
     
  • Re: Re: Re: Huge site export
    2012/11/19 19:12

    rvt <p> mx,</p> <p> I noticed in our case our proxy server would give kill the connection after XX seconds of &#39;inactivity&#39; so we needed a method to</p> <ol> <li> Automate it</li> <li> ensure we don&#39;t pull data through our proxy server we still wanted to have a short timeout towards clients</li> </ol> <p> The whitepage you are seeing is highly lickly because the connection get&#39;s killed, and I have seen it myself a bunch of times &nbsp;with similar effects (not Jahia related apps even).</p> <p> Come to think of it, in GlassFish there was a setting we also had to change:&nbsp;</p> <div> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;http request-timeout-seconds=&quot;<strong>3600</strong>&quot; default-virtual-server=&quot;_server&quot;&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</div> <div> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;file-cache&gt;&lt;/file-cache&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</div> <div> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;/http&gt;&nbsp;</div> <div> &nbsp;</div> <div> I remember I had to set the *request-timeout-seconds* also to ensure that glassfish wouldn&#39;t kill the process, if you are running Tomcat then I am sure it also has such setting.</div> <div> &nbsp;</div> <div> Both timeouts are rather easy to spot because you will see that your export always stops at around teh same datasite, but more import anround the same time after you initiate the process of exporting.</div> <div> &nbsp;</div> <div> Ries</div> <div> &nbsp;</div>

  • Number of messages  185
    Registration date Nov 19, 2012
    0 (0 Good)
    0 (0 Bad)

    Re: Re: Re: Re: Huge site export

    by  mxlab »  2012/11/20 08:22

    Sorry, perhaps I did not explain well!

    When I click on this link http://www.jahia.com/cms/home/download/forum/jahia-user/jahia-65/script-to-backup-your-sites-with.html 
    I get only a white page and I can't download the script!

    Moreover, is there any method to automatically remove every versioning copy, so I can export only the last content's version?

    Thanks for the help!

      (mxlab)

    Number of messages  33
    Registration date
    0 (0 Good)
    0 (0 Bad)

    Re: Re: Re: Re: Re: Huge site export

    by  rvt »  2012/11/20 19:46

    mx,

    mail me at rvt@dds.nl and I will mail you the script.

    Ries

  • Re: Re: Re: Re: Re: Huge site export
    2012/11/20 19:46

    rvt <p> mx,</p> <p> mail me at rvt@dds.nl and I will mail you the script.</p> <p> Ries</p>

  • Number of messages  185
    Registration date Nov 20, 2012
    Contact
    Share
    Feedback

    Get in touch

    Whether you are a current user or if you are just evaluating Jahia, we are here to help.

    Contact us

    Share this page