Is your Sitecore publishing stucks?

Challenge:

On web and in person I’ve read and faced so many issues on publishing stuck scenario with Sitecore and most of them got resolved by doing Application Pool Recycling Or restarting IIS. But I’ve never read/found its root cause.

We were also facing lots of publishing stuck issues and we were also following the same approach what others were following [IISRestart] which was causing us a lot of man hours lost, clients unhappiness, Content Authors unhappiness etc. So, we decided to find the root cause of this issue and fix it permanently! [My learning — The best method to fix any issue/bug permanently is to first find the root cause [I know it’s not easy] and then find the fix for it, else you will be doing same thing for days and years :)]

So, if you are also facing Publishing Stuck issue this article is for you!

Solution:

Is your sitecore publishing stucks? Before I explain our problem. It would be great to check basic things first which are as below:

1. Check network speed\firewall\antivirus is everything looks fine?

2. If your Sitecore items are huge and If your CAs are using Publish Site [Shortcut F9]. This may also cause an issue for slow publishing.

3. Check Stager/Staging Module files [If your Sitecore version < 6.3].

Basically Stager or Staging module creates files on your CM environment which tells your CD environment to clear cache. Sometimes your Stager/Staging module does not clear this files and due to lots of Files the I/O operation takes time and your publishing stucks.

Below all steps applies to Sitecore Version >= 6.3

4 . Check EventQueue configuration.

5. Disable your WebDeploy, and check does your publishing works fine? If yes then you need to check your WebDeploy.

6. If you are using Sitecore CMS 6.3.0 rev.101029 (Update-3)  with publishing Instance and if your publishing is showing “Initializing” then you need to upgrade to Sitecore CMS 6.3.1 rev.110112 (Initial Release).

Publishing

• [Backported from 6.4.1] Publishing jobs would not be executed if the publish operation was initiated after a restart of the Sitecore Publishing instance or while it was down. The publish dialog would display “Initializing…” and never show any progress. (336232, 336668)

7. have you added any hook on publish:begin, publish:end, publish:end:remote? If yes check its code or to quickly verify that they are creating an  issue or not try disabling  them one by one. And if they seems to be creating an issue then check its code.

Okay, you’ve checked all above basic things, and still publishing stucks? [Which happened with us :)]. Then check things as mentioned below:

Do you have two publishing targets? if yes then Publishing stucks on second target? Does your Sitecore log file shows following pattern in log file? [We just selected Job Started entries for each item in following log snippet]

    Line 22533: ManagedPoolThread #10 13:27:46 INFO  Job started: Publish to ‘web1’
    Line 22721: ManagedPoolThread #16 13:32:34 INFO  Job started: Publish to ‘web2’
    Line 22725: ManagedPoolThread #16 13:32:35 INFO  Job started: Publish to ‘web1’
    Line 22729: ManagedPoolThread #16 13:32:35 INFO  Job started: Publish to ‘web2’
    Line 22734: ManagedPoolThread #16 13:32:36 INFO  Job started: Publish to ‘web1’
    Line 22992: ManagedPoolThread #16 13:38:12 INFO  Job started: Publish to ‘web2’
    Line 23002: ManagedPoolThread #16 13:38:13 INFO  Job started: Publish to ‘web1’
    Line 23447: ManagedPoolThread #16 13:48:32 INFO  Job started: Publish to ‘web2’
    Line 23705: ManagedPoolThread #16 13:56:38 INFO  Job started: Publish to ‘web1’

Here there is no entry for Job started: Publish to web2′ It means that system has completed “ Job started: Publish to ‘web1′” and after that before startingPublish to ‘web2’ it got stuck somewhere.

If answer is yes then below steps contains its solution, steps to reproduce it, and reason why it occurs! Wow! Sounds interesting! then let’s go..

We raised Sitecore Support ticket for the same and I should say big thanks to geniuses at Sitecore Support for providing us a hint on its problem and solution — Especially Sergey Kuts

Reason:

Our Sitecore instance was running out of thread, so it doesn’t have thread to run another publishing operation.
By default Sitecore has 20 threads in thread pool. According to our log file we have

ManagedPoolThread #XX where XX is from 0 to 19, so it is quite possible that all threads are in use. [Open your log file and check this out!]

Reproduce:

Above reason sounded logical to us. But it was theoretical and we wanted to check our theory, before we fix the issue. [Basically, we are engineers for whom E=MC² also need to be proved :)].

To reproduce the scenario we created one ASPX page in our Sitecore Instance which takes Number of Jobs to run as an input in text box and based on that input. It will Start that number of Jobs.

If you are new to Sitecore Jobs, then no worries check my earlier article here.

So, in our Job’s method we written some code which will run for long period of time [Technically iterating through /sitecore/ item recursively]. So, it will occupy the thread for long period of time.

Then we kept big items for publish with two publishing targets [Web1 and Web2]. And as it is a huge publishing  it takes sometime to publish in the mean time we used our ASPX page to create 19 Jobs. So, Now our threadpool is full.  [Currently 20 jobs are running – One Publish Job and  19 custom created jobs.]

Our publishing process completes publishing on first publish target [Web1], and the moment it starts publishing on another publishing target [web2] it get stucks [So, we conclude from this is that One Publish target creates one thread to execute itself].

And yes, we successfully reproduced the issue. [Have you!]

Okay, I am sure you are eager to see its solution. Let’s move on to it.

Solution:

Sitecore guys suggested us to try  to increase available threads to at least 100 and see if it helps.  It can be done by adding the following setting to the <settings> section in the web.config

<setting name="MaxWorkerThreads" value="100"/>

Just a note : as it is config change it will restart IIS

And after this change we tried to reproduce it again. And we can’t — That’s what we wanted to do! 🙂

Finally, sorry for such a long post. But I hope you found it useful!

UPDATE : 02/June/2014:

We were facing an error — Where publishing dialog will say — Initializing forever — And to resolve it we have to recycle app pool. After spending lot of hours we found that our SQL Server and Web Server’s timings were not in sync. Syncing them fixed our issue. Give it a try, it might work for you as well!

Good read:

http://sdn.sitecore.net/SDN5/Forum/ShowPost.aspx?PostID=59795

Happy Smooth Publishing! 🙂