Skip to content
February 15, 2020 / kiranpatils

RedisTimeoutException with Sitecore 9.0.2 – 9.3

Challenge:

In recent time, was involved in Sitecore 9 Azure PaaS Implementation for few of our enterprise clients, who had humongous amount of data, traffic and multi-region CD servers to handle load.

We faced RedisTimeout challenges during 9.0.2 launch and recently we faced similar challenges for 9.3 upgrade. It was moderate for later than earlier. But in both case, we spent good amount of time and we have learnt something new. So, thought to share with you. So, hopefully it can reduce your resolution time.

So, If you are also haunted with RedisTimeout exceptions and your launch is due soon and looking for some solutions this post might help you.

Solution:

I won’t spend much time explaining basic details about it. But I would strongly recommend you to read following great posts:

If you’ve already been through above posts, Great! Please keep reading.

Before we talk about Solution, Let’s talk about Why this happens?

If you want to know more about Session and how it works, I would strongly recommend you to read this great post : https://mhwelander.net/2016/05/19/lets-talk-about-session-state/

By this time, your basic understanding should be clear. In XP Scaled and Multi-region most applications will prefer to use Out-Proc session for obvious reasons explained in the post earlier.

So, if you are expecting lot of traffic then your CD Servers should have enough power to handle session threads as well as your REDIS server should also be powerful enough to handle those session requests.

When you are troubleshooting REDISTimeout issues, find your bottleneck first. Here are few of our learnings to identify bottleneck:

  1. Check CPU Usage of CD Server – It is high? :Then your CD server needs more power (Just a note : If your CD’s CPU is high, then there might be another reasons for that. Which needs to be checked. But for simplicity of this post, We are assuming all other issues are already taken care). Give a tier upgrade and check again.
  2. If you are not noticing high CPU on CD and REDIS is overloaded then good to add more power to your REDIS app.
  3. If you’ve upgraded them to both of them to higher tier and if you are still noticing REDIS errors then you should apply configurations given in KB Article and mentioned below – Please keep reading this post.

Can you show me some more symptoms?

If you are not sure you are impacted by REDISTimeout exceptions or not following are few symptoms of it:

AI Logs


Sitecore Logs


Azure tools (Latest) query “- traces | where timestamp > now(-1d) | where message contains “Exception” or customDimensions.StackTrace contains “Exception” | where customDimnsions.Role == “CD” | sort by timestamp desc | limit 100″
Dump file analysis

I’ve faced this error on multiple versions and solution for each version had slightly different. So, will divide solution in two parts:

In either of the version, you must need to refer this KB Article : https://kb.sitecore.net/articles/464570 – Please don’t get overwhelmed with information it provides, take few deep breaths and read – It’s good info. I personally feel Solution steps are not as straight forward as it should have been. But we all are learning together 🙂

  • Sitecore 9.0.2 : We noticed lot of performance challenges and during troubleshooting we found that our applications were setup (~2 years ago) as per Sitecore’s old tiers recommendation, And it was not capable to handle load. also, we had to upgrade our Redis Tier to C2 (C1 might work, if your site is not having heavy load)
    • We did some load testing and performance tuning and we learnt that upgrading CD apps to P2V2 (Do some performance test to find out right tier P1V2 might be a good one to go with. Also, good to move away from Standard tiers to Premium tiers as they are available at same price with better hardware! – Please refer this : https://azure.microsoft.com/en-us/updates/announcing-pricing-decrease-for-azure-app-service-on-the-premium-plan/)
    • We also had to upgrade our REDIS Apps to C2 (If you got moderate load than C1 might work for you!)
    • Apart from this we applied steps mentioned in KB Article : https://kb.sitecore.net/articles/464570#Solution(XP8.0-9.0)
      • Apply Step#1-3
      • But please make sure you read Notes section first. Whatever values you applied as per this KB article is just for reference, and you need to do some load testing before finding optimal value for your needs.
      • I’ve created gist with our configurations : https://gist.github.com/klpatil/9c19fe6f0b8c3cfa8e9ed76629e16c79
      • Please play close attention to following values in gist, we updated them for better results:
        • pollingInterval : Specifies the time interval in seconds that the session state provider uses to check if any sessions have expired. The default value is: 2.
        • executionTimeout
        • timeoutBetweenLockAttempts
        • Sitecore.Support.210408.config – Especially values given in this file needs close attention.
  • Sitecore 9.3 : As we had learnings from our last project (9.0.2) project we were not expecting any Redis error. But life is not as easy it seems to be :-). We again got haunted by REDISTimeout exceptions during our Performance testing period. Here’s what we learnt.
    • We were facing one odd error during UAT period. But during load testing. We noticed that REDIS errors count went high. And that’s when we started to investigate further on this before go-live:
    • Tiers : Luckily, we haven’t faced much challenges with tiers. CD servers were good. We only had to upgrade REDIS from C1 to C2 to handle load.
    • Sitecore Patch as per KB Article: As you must have noticed on this article it says – Fixed in 9.2. And if you scroll further it has suggested some steps : https://kb.sitecore.net/articles/464570#Solution(XP9.2InitialReleaseOrLater) and steps are also not super clear. Feeling same? We also felt same, And thought to reach out after couple of back and forth. Sitecore support also accepted that those steps need some work for 9.3 and here are our learnings till Sitecore updates KB article:
      • Good part is – you no need to apply and DLL patch for this.
      • Just some configuration updates as listed in this gist: https://gist.github.com/klpatil/0b395c5cf46a8ff5f04118ea49f05546
      • Biggest learning was maxConcurrencyLevel value should not be changed at all. As per 9.3 logic, if this value is not provided then 9.3 handles it in conjunction with batch size, else size you provided in config. And as per our learning default logic is optimal.
    • Load testing : Above steps reduced REDIS Timeout exceptions drastically. But whenever client used to run JMeter load test, they noticed REDISTimeout exception. After some consultation with Sitecore support and analysis we learnt that load testing script was sending unrealistic traffic and it was also not handling cookies properly.
  • You are still facing error? : If you are facing error even after following steps given above, then it might be good idea to create memory dump using procdump and try to analyze this with help of Sitecore support.
D:\hone\site\wwwroot\App_Data\dumps>D:\devtools\sysinternals\procdump -accepteula -e 1 -f "StackExchange.Redis.RedisTimeoutExceptio 
" -ma 285441
D:\devtools\sysinternals\procdump -accepteula -e 1 -f “StackExchange.Redis.RedisTimeoutException ” -ma <PID>

Both our environments are serving live traffic and we are having stable systems with No REDIS Timeout exceptions!

Hope this post helps you to fix you REDIS timeout exceptions or avoid them before you see them. Special thanks to Sitecore support team and my colleagues Joseph and Khushboo for all their help during troubleshooting!

November 30, 2019 / kiranpatils

200th Sitecore Blog Post

I’ve been sharing my Sitecore basics via this blog since 2011. Today, I’m super happy and surprised that, I’m writing my 200th blog post. (Double century is really special – You can understand that I’m coming from Cricket playing nation 🙂 )

This would have not been possible without following people:

  • All those great souls who inspired me to write!
  • Great souls like you who kept reading my blog and appreciate what I did – Which motivated me to keep going. As starting Sitecore or any blog is easy. But to keep it going needs more efforts. Would have used this time doing something else. But I feel grateful that I invested my time on this blog which helped lot of Sitecore Souls across the globe!

Some Stats:

This post will make posts count to 200
All time views
Annual Site stats
Total followers
This is my favorite – Could contribute on larger part of the earth in some way!

Summary:

  • Total posts : 200
  • Total views : 519518
  • Total visitors : 254007
  • Total words : 107450

Perfect day [Thanksgiving] to write this blog. I’m Thankful for:

  • Those who taught me Programming
  • Those who taught me Sitecore
  • Those who inspired me to blog
  • Those who read my blog and kept me going

Hope this post inspires you to start your blog or restart your blog which you haven’t updated since quite some time.

Happy blogging! 🙂 – Start your blog, might not for others. But for your self! [At times, I find solution from my old post 😉 ]

November 30, 2019 / kiranpatils

SearchStax with Sitecore Azure PaaS

Challenge:

Sometime back, we were trying to integrate SearchStax with Sitecore 91 PaaS environment.

I got some good documentation to begin with. But it took me sometime to accomplish my goal. With the help of SearchStax team [They are great!] was able to achieve my goal.

Recently came across similar question by someone in Sitecore Slack group. So, thought to blog it here.

Solution:

This post assumes, that you are already aware about SearchStax – Basics and how to get started, Solr, SolrCloud basics.

As you do, I also did quick search and came across following great posts!

Initial posts clarified lot of my basic fundamentals, Last post : https://www.searchstax.com/docs/sitecore-9-1-solr/ is really good. But If you are trying to do it for your Sitecore Azure PaaS environment. You need to alter some steps and I also faced some errors. Let me outline that for you:

  • You no need to do Step#1.
  • Do Step#2.
  • Step#3 : SearchStax team has created nice plugin, which helps you create all initial required collections — Basically, Plugin abstracts zookeeper config upload steps:
    • Instead of using searchstax-sitecore-plugin-1.0 use 1.1. As we found bug in 1.0. Use 1.1 from here : https://github.com/searchstax/searchstax-sitecore-plugin/releases or direct link https://github.com/searchstax/searchstax-sitecore-plugin/releases/tag/v1.1
    • Script relies on PowerShell core. To start PowerShell Core – you need to type pwsh.exe in CMD/Powershell – See this video : https://www.youtube.com/watch?v=UGCMbYAufp0
    • Extract downloaded plugin and update YML file as per given instruction. [pathToWWWRoot : Is not important in Azure PaaS. But still you have to provide it and sitecorePrefix (Will be used to prefix your Solr collection/cores.) also should be valid site. Plugin will update config files. So, you might need to take backup of local file – ConnectionStrings.Config]
    • Run this from CMD/Powershell. Make sure you are in PowerShell Core. It will also ask your SearchStax credentials. So, please keep them handy!
  • Once above step is done, your Solr collections are ready. You can access Solr Admin screen to confirm that (You should see 11 collections). It also updates Solr Connection string in your local. Please revert those changes to your local Solr instance. [We took backup in earlier step]

Note : It doesn’t create XDB Core. https://doc.sitecore.com/developers/90/platform-administration-and-architecture/en/walkthrough–using-solrcloud-for-xconnect-search.html#UUID-00ee25a3-647f-aa18-6f98-805fa4f305de_section-5c3ef326406c3-idm45266705934112– As of now you have to create manually or update scripts to do that.

  • Till this point of time we have setup our Solr Cloud instance. Now. let’s connect it with our Sitecore Instance.
  • Once all above steps are done, Login to your Sitecore instance and from Control Panel – Do “Populate Schema from the Control Panel in Sitecore” step.

This post depicts steps with Sitecore 9.1. But I believe it should work with 9.2 https://www.searchstax.com/docs/sitecore-9-2-solr/ as well. I’m not sure Search Stax Plugin issue is fixed in 9.2 document or not. Also, I heard Search Stax is coming up with Sitecore LaunchPad Plugin (https://sitecorebasics.wordpress.com/2019/11/30/searchstax-with-sitecore-paas/) to simplify these steps. I’m sure you must be excited to try this. So, as I!

Hope you found this post helpful – Big Shout out to SearchStax team for coming up with such a great solution.

Image Credits : SearchStax

November 23, 2019 / kiranpatils

Decade’s worth of 11 Sitecore modules

Challenge:

Last week Robert Hock published new UI for my Cache tuner module. Which looks great! This module got lot of buzz in Sitecore community last week, and I was surprised to know that lot of folks were either not aware of this tool (Which is published since 27-April-2011 : https://sitecorebasics.wordpress.com/2011/04/27/cache-tuner-user-guide/) or used it in the past. But forgot about it.

As Rob mentioned in the post, At times we used some module. But we forget about it (At times, I also come across my modules from Google or talking to others and same happens for my blog posts as well 😉 – But that’s why I blog – So, as you?!)

In this blog post, Thought to list down all my modules and their usage and link to download for you! [And for me as well ;)]

Sounds great?! – Let’s delve into it

Solution:

List of modules are as below:

  1. Cache Tuner : Cache Tuner Module helps you to understand your Sitecore Caching usage and helps you for fine tuning cache sizes as per Sitecore’s Caching Guide Recommendation. It has two versions Pre Sitecore 8.X and after that [as Sitecore 8.2ish had major caching updates]
    1. https://marketplace.sitecore.net/Modules/C/Cache_Tuner.aspx?sc_lang=en
    2. https://sitecorebasics.wordpress.com/2011/04/27/cache-tuner-user-guide/
    3. V2 : https://sitecorebasics.wordpress.com/2019/06/10/cache-tuner-v2/
    4. https://github.com/klpatil/CacheTuner
  2. Hot Fix Checker : It will list all packages installed in your Sitecore Instance. It will also help you to answer questions related to packages installation like Who created package?, Who Installed it?, When it got installed?
    1. https://github.com/klpatil/HotFixChecker
    2. https://sitecorebasics.wordpress.com/2019/09/27/module-hotfix-checker-hfc/
  3. Advanced Stats Page : This tool allows users to visualize Sitecore rendering stats data in an intuitive and simplified way. Which will help you to identify slow renderings, fix them and have happy end users!
    1. http://klpatil.github.io/Advanced-Stats-Page/
    2. https://marketplace.sitecore.net/en/Modules/A/Advanced_Stats_Page.aspx
  4. Item Utility : Sitecore Item Utility to do bulk operations
    1. https://github.com/klpatil/SCItemUtil
    2. https://sitecorebasics.wordpress.com/2017/08/30/sitecore-item-utility/
  5. Quick Item Search : This module is upgraded version of http://trac.sitecore.net/QuickItemSearch module which works on Sitecore 5.1. But this module will work on all Sitecore 6.X versions [I’ve tested with Sitecore 6.4/6.5].
    1. https://sitecorebasics.wordpress.com/2011/06/01/quick-item-search-user-guide/
    2. https://marketplace.sitecore.net/en/Modules/Q/Quick_Item_Search.aspx
  6. Package History : It will list all packages installed in your Sitecore Instance. It will also help you to answer questions related to packages installation like Who created package?, Who Installed it?, When it got installed? V2 : Package History V2 has been launched. In which we provided a functionality using which you can uninstall Sitecore Package/Module just by few clicks!
    1. https://github.com/klpatil/PackageHistory
    2. https://sitecorebasics.wordpress.com/2014/02/10/package-history-v2-user-guide/
    3. https://marketplace.sitecore.net/en/Modules/P/PackageHistory.aspx
  7. Sitecore History Viewer : This Tool allows users to view,search,sort Sitecore Databases History
    1. https://github.com/klpatil/HistoryViewer
    2. https://marketplace.sitecore.net/en/Modules/S/Sitecore_History_Viewer.aspx
    3. http://klpatil.github.io/HistoryViewer/
  8. Sitecore Audit Trail : This Tool allows users to see Sitecore Audit Trail — Who did What changes and When?
    1. https://marketplace.sitecore.net/en/Modules/S/Sitecore_Audit_Trail.aspx
    2. https://github.com/klpatil/Sitecore-Audit-Trail
    3. http://klpatil.github.io/Sitecore-Audit-Trail/
  9. SC Log Explorer : The SC Log Explorer tool will give you an easier overview of the Sitecore log files when you need to analyze them. 
    1. https://sitecorebasics.wordpress.com/2012/06/07/sclogexplorer-user-guide/
    2. https://marketplace.sitecore.net/en/Modules/S/SC_Log_Explorer.aspx
  10. Mongo Shell : Sitecore tool to query SQL Data
    1. https://klpatil.github.io/SC-82-Demo/
    2. https://github.com/klpatil/SC-82-Demo
    3. https://marketplace.sitecore.net/en/Modules/M/Mongo_Shelll.aspx
  11. SitecoreBasics.Extensions : Sitecore Extension methods source code, docs and package
    1. https://github.com/klpatil/SitecoreBasics.Extensions
    2. https://sitecorebasics.wordpress.com/2012/07/26/sitecore-extension-methods/

Apart from this, I also contributed to following open source module:

Wow! — 11 Modules in my 10 years of Sitecore experience! To be honest, I had also forgot about few of my modules and this post refreshed my memory. I hope you find this post as helpful as I found it!

October 29, 2019 / kiranpatils

Have you noticed OOTB Sitecore Diagnostics Service (Beta)?

Recently, I was trying to generate Sitecore Support Package using SSPG.

Which looks like following on last step:


Following screen got revealed as soon as I clicked on Done button.

In past, I’ve used SDT (Sitecore Diagnostics Tool) in the past for validating Sitecore instance (I’m sure you as well). But after Azure Web Apps you can’t install SDT. SDS latest version has reesolved this issue, where you can provide SSPG generated package to SDT as input and same analysis will be done by SDT.

As always Sitecore team has gone one step further to SDT as part of Sitecore Support Package generator admin page. When you click on “Agree and Continue”, Tool will upload SSPG Package and provide you analysis report. Which looks like following — Which is Standard SDT report.

Hope this makes your diagnostics faster – Thank to Sitecore!

%d bloggers like this: