Does your CM server performing slow?
Our CM Instance was responding very slow. When the day starts it works well. But as the day moves on it starts becoming slow. And sometime it becomes slow from start of the day itself.[It was totally random] Locking an item, navigating from one item to another item, saving an item, previewing an item etc. was taking lot of time.
After investigating lot of things starting from network till application, finally we solved this challenge! [Touchwood!]. I know you must be eager and impatient to know the solution. But while working on this challenge, we learnt to have some patience and passion to solve these kinds of issues!
Frankly, for performance issue there is no any master-key available which you put in to lock and your issue will get solved, you have to keep trying all the keys you have and trust me at one good day, your one key will surely help you to open the lock!
Basic: Whenever any application is performing slow, as per the basic concept. It can be issue at any place like Network, routing, OS, Hardware, Database server, Database configuration, Application Server, Application etc. and you should not miss any of these!
Before you start trying all options given below, have some mechanism and statistical data to measure the performance improvement.
Just a note: For few options I will follow the DRY [Don’t repeat yourself] rule. So, will provide you link to my earlier articles! So, please enjoy reading them as well!
So, let me share all the areas [keys] we tried to get to the bottom of this issue:
- Log files: In a Sitecore world, Application log files are our best friend! [Good to know that Sitecore uses Log4Net for log files]. So, whenever you face any issue. First place to check is Sitecore log files, is anything wrong there? Try looking at some pattern [TIP: Use Notepad++ or Log Parser Lizard - for querying your Sitecore log file]. Keep checking it always and if in a series nothing wrong found in log file. Then you should try next option. Also, enable Threshold logs.
- IIS Logs – Here you can check is there any page taking lot of time? Are there any crawler sending lots of requests on your CM Server? [Tip : Log Parser Lizard will help you to query your IIS log file]. Digging gold in IIS logfiles
- System resources: Check your CM Instance’s CPU/Memory usage trend. Troubleshooting guide.
- Performance counters: If you reached here and found no solution yet. My dear friend don’t worry, we will surely be able to solve your issue
Now, we need to check that whether Network, Application, Database server is an issue or not?
When application becomes slow, precisely at that time what is the stage of system resources? Obviously we can’t ask one person to constantly seat and keep checking it. To achieve this we can use perfmon. Configure following performance counters on Database server and Application server.
Category Counter Process Memory Available Mbytes All PhysicalDisk Avg. Disk Queue Length _Total Process Private Bytes Worker Process for Sitecore,_Total Process Virtual Bytes Worker Process for SITECORE,_Total Process Page File Bytes Worker Process for SITECORE,_Total Process IO Data Operations/sec Worker Process for SITECORE,_Total Process IO Other Operations/sec Worker Process for SITECORE,_Total Process % Processor Time Worker Process for SITECORE,_Total Network Interface Bytes Total/sec All Instances Network Interface Output Queue Length All Instances
[Tip: to analyze performance counters use PAL]
- Database Test : Sitecore has a tool known as “Database performance Test” which helps us to identify that whether there is any issue with Database [Fragmentation?], Database server, Connectivity between application and Database server, etc. [It does not pin point directly to any possibility. But if you get the statistics which are alarming then you can check all above suggested possibilities].
If you have not used this tool earlier, and would like to read more about it, Please do read my earlier blog on Database Test
Basically, If Average Time exceeds Boundary Average Time, this may imply SQL database performance problems.
- CounterDumpAgent : CounterDumpAgent is very helpful agent which comes with Sitecore – Basically it dumps all the useful Sitecore counters related information in one file. If you would like to read more about it, I would suggest you to read my earlier blog on CounterDumpAgent.
After configuring this Agent, we found that there were heavy cache clearings going on, which was causing us slowness. [Thanks to Maxim Savelyev from Sitecore support for helping us out to find this].
Majorly following two caches:
AccessResultCache: It was getting fully cleared in following scenarios:
- Using Sitecore API we call UserProfile.Save() method
- Whenever you click on accept button of Rich Text Editor
- Whenever you proceed on Publish wizard
Sitecore support team accepted this as a BUG and they provide us a HotFix for it with HotfixID – 349789-2
Master[Items]: It was getting fully cleared in following scenarios:
- Whenever you save any template [as per our architecture our CAs can add/update/delete templates, which was flushing this cache whole day. We discussed this issue with Sitecore support and they explained us that due to technical limitations they can’t provide any solution – and I agree with them! But they suggested us to fine tune the caches for layers which are below master cache – To read more about Caching read my earlier blog on Caching basics]
- Whenever you save any template without making any changes in to template (As per window practice few users keep pressing CTRL + S again and again) [Sitecore accepted it as a bug and they provided us a Hotfix for the same. HotfixID - 351159]
- Whenever you publish any template [Basically we added one handler on publish item processing which modifies an item which is being published and save it. It was doing this operation for all the items. Instead of that we added one check that if currently published item is Template then don’t do any modifications]
7. Cache Tuning: Sitecore heavily and nicely uses cache for giving better performance. So, check your cache
utilization and if needed fine tune it. [Tip : use Cache Tuner for fine tuning your cache]
8. TCP Chimney: Basically, TCP Chimney offloads all the network related operations from CPU to NIC. So, your CPU Usage stays normal. But couple of NICs (BroadCom NetXtreme II,Hewlett-Packard NC373i Multifunction Gigabit Server Adapter) implementation for TCP Chimney is buggy. [Good to read]
9. IIS 7.5: Earlier we were running our CM instance on IIS 6.0 and after migrating it to 7.5 It can handle 57% more requests and it can utilize CPU. And it gave us a good performance improvement [Good to read : IIS 7.5]
After doing step#6 and #7 we got around 30% improvements in response time!
And after doing step#9 we got around 40% improvements in response time!
Thanks to Maxim Savelyev – from Sitecore support team for helping us out to resolve this issue!
Sorry for writing such a long article, and thanks for reading it patiently!
Happy Performance Tuning!
Good to read:
CMS Performance Tuning guide