Sitecore Troubleshooting Path

Challenge:

We have invested a lot of time in troubleshooting Sitecore challenges, e.g. CM/CD/PI is slow, CM/CD/PI is crashed, CM/CD/PI is slow etc.

When we started working on these challenges, we had either no path to look at OR lot of paths to look at. And trust me it has been always fun and learning to work on such challenges, It teaches you a lot! And when you fix it, you really feel great!

During our journey (“Performance is a Journey, not a Destination”) we learnt the lot of new things by doing few things that were right and doing few things that were wrong! Right/wrong they were need of the time!

And the sole purpose of writing this post is to share our learning with you! (“Knowledge shared is knowledge gained!”) So, you may not face the challenges which we faced while working on these challenges.

Solution:

Just a note : Before, we start one thing I would like to share with you, which is not technical. But it is as important as technical knowledge. When you troubleshoot any challenge, things might be burning. Because your live servers are down. People looking at you for solution. You are getting chased for solution. Whatsoever happens, keep your self cool and calm!. This is the most basic required thing to troubleshoot and solve any challenge (Virtual/Real world!). And always Stick to your basics! Wish you luck! I’m sure you will solve it!

There are plenty of things to share, and was wondering what will be the best way to write an article which will share knowledge with you in an interesting way! And as someone rightly said “A picture is worth a thousand words“. Finalized to create a poster which will describe Sitecore Troubleshooting Path. Eager to see it?!, here you go!:

You can have a quick look from below and if you find it interesting then only you can download from link given in left side of preview area 🙂

View this document on Scribd

Sounds simple, correct? Let me explain few basic things, how this poster has been designed:

Environments:

CM – CM stands for Content Management server, a server which is responsible for your content editing and content management.
PI – PI stands for Publishing Instance server, a server which is responsible for your publishing operation only!
CD – CD stands for Content Delivery server, a server which is responsible for serving your front-end/live content only!

Troubleshooting source detailed information:

Sitecore log – Sitecore log file from your data folder. Good to use Log Parser Lizard/SCLogExplorer/Sitecore’s log analysis tool for analyzing SC log files.
Sitecore Diagnostics Log – Sitecore Diagnostics log, for more information, please refer earlier post — https://sitecorebasics.wordpress.com/2011/07/30/basics-of-counterdumpagent/
Sitecore configurations – Your sitecore configurations.
Windows Event Log – Windows event log files
IIS Log — IIS logs, in some case if your server becomes slow/gets crashed, it may due to one of the page which is having a rendering which may cause slowness or crash. But you need to build URL pattern to conclude from this source. And good to use Log Parser Lizard for analyzing IIS log files.
Performance counters – Performance counters, you can use PAL to generate reports easily!
Server configurations – Your server configurations, If your CM/CD becomes slow at particular time. Do check that is there any windows service/scheduled task/Scheduled Antivirus scan runs on that particular time.
SQL Server (Performance counters, System usage) or Optimize Sitecore databases (DB cleanup, Rebuild, etc.) – Your SQL server performance. Good to check CMS Performance Tunning guide or http://sdn.sitecore.net/Scrapbook/Optimizing%20Sitecore%20Performance.aspx
Database Test Page results (/sitecore/admin/DatabaseTest.aspx) – DatabaseTest page from Sitecore, for more information, please refer earlier post – DatabaseTest
Is cache tuning required? /sitecore/admin/cache.aspx or Cachetuner.aspx- Cache tuning is required?, for more information, please refer earlier post on Cache Tuning
Your Custom code – If possible disable it and check it, else do logging for it, deploy it and verify – It might be due to custom code, if possible disable it and check it or do detailed logging and verify.
Sitecore Debugger/Profiling – for more information, please refer earlier post, Basics of Sitecore Debugger
Sitecore Admin Pages (Cache.aspx, Stats.aspx, Showconfig.aspx) – Nice post Sitecore admin pages explained
Monitoring tool if any e.g. New Relic — New Relic is APM
Your environment e.g. Network, Load Balancer, Local Switch, If your servers are on VM then VM performance and other Information from your Hosting provider.
Memory Dump – If you are new to Memory Dump and Crash Dump, then I would suggest you to read Ekaterina, Tess and Muktesh‘s blog. Basically, using DebugDiag you can generate dump automatically/manually and using WinDbg and SOS.dll you can analyze what was the root cause of your challenge. Initially you will find it bit complex. But once you got to know it you will be the champ! [Similar as Maths! :-)]
Crash Dump
MemoryThreshold related details from Sitecore log file – It is not good idea to use this if you are hosting multiple websites from single Sitecore – For more information, please refer earlier post – Threshold exceeds and MemoryMonitorHook
Cache Debug related information from Sitecore log file (Caching.DebugEnabled) — For more information, please refer earlier post – Caching Debug
Detailed event level details from Sitecore log file (events timingLevel=”high”) – For more information, please refer earlier post – EventQueue troubleshooting
IIS worker process configuration – Idle Timeout may shutdown your Worker process, if not configured properly.Also, it’s always good to run your SC worker Process on 64 bit — For more information, please refer earlier post – KeepAlive, CD server crash
GC mode – server or workstation? – For more information, please refer earlier post – Do you really need AccessResultCache on CD Server?
System usage – CPU, Memory etc.
Any custom/Sitecore agent causing issues?