Skip to content
December 25, 2018 / kiranpatils

Troubleshoot and fix crashing Sitecore 902 Application on Azure App

Challenge:

Currently we are working for one of our client application, Which is on Sitecore 9.0.2 and deployed on Azure App. It was working fine for sometime and one fine day, post deployment application was not coming up. This was client’s QA environment and in lot of developers and QA team members were keenly looking up to us to resolve this ASAP. So, we can hit our sprint goal.

We tried the lot of options to make it up and running. But we could only see following page (503):

We tried to check logs. But nothing fruitful was found. DevOps team already had Azure ARM Template. So, we thought to install new environment from scratch and redeployed and that worked for a while. But after few days, Same issue happened on QA Environment.

Now, This was the time for us to generate a crash dump and analyze it [You also thought so?]. But then we realized that this is Azure App and DebugDiag won’t work here. Sounds interesting right? [Life is not as easy, as it seems to be]. But each challenge has solution in this world. And that’s true for this challenge as well. If you are also facing similar challenge or you are keen to know how we solved this, then this post is for you!

Solution:

UPDATE : 17-Jan-2019 : Sitecore has official KB article and patch on it :https://kb.sitecore.net/articles/910936 [Thanks Matt for your comment]

  1. Quick google search suggested this post : https://kb.sitecore.net/articles/983839 we checked those things. But with no luck.
  2. Application insights also didn’t help : https://doc.sitecore.com/developers/90/sitecore-experience-management/en/analyze-sitecore-logs-with-application-insights.html
  3. Then we came across, Sitecore’s KB article for Advanced Troubleshooting on Azure App (Was not able to generate dump using it, It might be due to my overlook and few information seems not up-to-date [Azure Portal is always changing]) : https://kb.sitecore.net/articles/111669
  4. Then found this great article, Which gave me some hints : https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/app-service/troubleshoot-http-502-http-503.md

I decided to delve in to Azure Portal and based on what I learnt try something out (I will quickly remind you this is QA environment and roughly ~12 folks are affected by this and overall sprint plan/project plan.)

  • Selected Diagnose and Solve Problems, Which has a lot of interesting options as shown below (Diagnostic Tools is an option, Which I was looking for) – a.k.a. DaaS:
Diagnose and Solve Problems
  • When I clicked on “Memory Dump” [One of the sub option of Diagnostic tools]. It brought few more options:
  • Selected “Collect Memory Dump” option [Make sure you have selected ASP.NET in top]. As soon as you click on that Azure checks whether your tier has this service support or not? [Microsoft — One more opportunity to make money :). But they deserve with what they have done. And when your application is business critical and has impact on revenue, Then no harm in investing in this – it’s all business!]. Based on your tier you will see next screen. This is what I see:

Finally, We have Dump analysis report and this is what we found:

[[HelperMethodFrame_1OBJ] (System.Threading.Monitor.Enter)] System.Threading.Monitor.Enter(System.Object) Sitecore_Kernel!Sitecore.DependencyInjection.LazyResetable`1[[System.__Canon, mscorlib]].get_Value()+5e Sitecore_Kernel!Sitecore.Security.AccessControl.AccessRight.FromName(System.String)+34
Sitecore_Kernel!Sitecore.Security.AccessControl.ItemAccess.CanRead()+1c Sitecore_Kernel!Sitecore.Data.Managers.ItemProvider.ApplySecurity(Sitecore.Data.Items.Item, Sitecore.SecurityModel.SecurityCheck)+51 Sitecore_ContentTesting!Sitecore.ContentTesting.Pipelines.ItemProvider.GetItem.GetItemUnderTestProcessor.Process(Sitecore.Pipelines.ItemProvider.GetItem.GetItemArgs)+b3
Sitecore_ContentTesting!DynamicClass.(System.Object, System.Object)+f Sitecore_Kernel!Sitecore.Pipelines.CorePipeline.Run(Sitecore.Pipelines.PipelineArgs)+1e4 Sitecore_Kernel!Sitecore.Data.Managers.DefaultItemManager.GetItem(Sitecore.Data.ID, Sitecore.Globalization.Language, Sitecore.Data.Version, Sitecore.Data.Database, Sitecore.SecurityModel.SecurityCheck)+131 Sitecore_Kernel!Sitecore.Data.Managers.DefaultItemManager.GetItem(Sitecore.Data.ID, Sitecore.Globalization.Language, Sitecore.Data.Version, Sitecore.Data.Database)+93 Sitecore_Kernel!Sitecore.Data.Managers.ItemManager.GetItem(Sitecore.Data.ID, Sitecore.Globalization.Language, Sitecore.Data.Version, Sitecore.Data.Database)+9a Sitecore_Marketing_xMgmt!Sitecore.Marketing.Definitions.Outcomes.Data.ItemDb.ItemOutcomeDefinitionRepository..ctor(Microsoft.Extensions.Logging.ILogger`1, System.String, Boolean, Sitecore.Marketing.Definitions.Repository.IDefinitionRecordMapper`1)+345 Sitecore_Marketing_xMgmt!Sitecore.Marketing.Definitions.Outcomes.Data.ItemDb.ItemOutcomeDefinitionRepository..ctor(Microsoft.Extensions.Logging.ILogger`1, Sitecore.Marketing.IItemRepositoriesSettings, Sitecore.Marketing.Definitions.Repository.IDefinitionRecordMapper`1)+e4
[[DebuggerU2MCatchHandlerFrame]]


Other threads:
** Microsoft_Extensions_DependencyInjection_Abstractions!Microsoft.Extensions.Internal.ActivatorUtilities.GetServiceOrCreateInstance(System.IServiceProvider, System.Type)+21 Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.CreateFromServiceProvider(System.Xml.XmlNode, Boolean)+1ae Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.CreateObject(System.Xml.XmlNode, System.String[], Boolean, Sitecore.Configuration.IFactoryHelper)+8a Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.CreateObject(System.Xml.XmlNode, System.String[], Boolean)+48 Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.CreateObject(System.String, System.String[], Boolean)+27c Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.CreateFromReference(System.Xml.XmlNode, System.String[], Boolean)+a6 Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.CreateObject(System.Xml.XmlNode, System.String[], Boolean, Sitecore.Configuration.IFactoryHelper)+74 Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.CreateObject(System.Xml.XmlNode, System.String[], Boolean)+48 Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.GetInnerObject(System.Xml.XmlNode, System.String[], Boolean)+6d Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.GetConstructorParameters(System.Xml.XmlNode, System.String[], Boolean)+a1 Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.CreateFromTypeName(System.Xml.XmlNode, System.String[], Boolean)+5e Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.CreateObject(System.Xml.XmlNode, System.String[], Boolean, Sitecore.Configuration.IFactoryHelper)+a3 Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.CreateObject(System.Xml.XmlNode, System.String[], Boolean)+48 Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.CreateObject(System.String, System.String[], Boolean)+27c Sitecore_PathAnalyzer!Sitecore.PathAnalyzer.ApplicationContainer.CreateObject[[System.__Canon, mscorlib]](System.String)+de Sitecore_PathAnalyzer!Sitecore.PathAnalyzer.ApplicationContainer.GetOrCreateProcessingPool(System.Guid ByRef)+29
**
Microsoft_Extensions_DependencyInjection_Abstractions!Microsoft.Extensions.Internal.ActivatorUtilities.GetServiceOrCreateInstance(System.IServiceProvider, System.Type)+21 Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.CreateFromServiceProvider(System.Xml.XmlNode, Boolean)+1ae Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.CreateObject(System.Xml.XmlNode, System.String[], Boolean, Sitecore.Configuration.IFactoryHelper)+8a Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.CreateObject(System.Xml.XmlNode, System.String[], Boolean)+48 Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.CreateObject(System.String, System.String[], Boolean)+27c Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.CreateFromReference(System.Xml.XmlNode, System.String[], Boolean)+a6

After analyzing all stacktraces we concluded that, it is mainly in Sitecore Stack, and then we raised a Sitecore support ticket and shared all relevant information with them.

Sitecore support team were able to rectify the issue in a day and get back to us with following information:

Please use the following hotfix to resolve the issue:
https://dl.sitecore.net/hotfix/SC Hotfix 235278-2 CMS.Core 11.1.2 ContentSearch 3.1.2.zip
To install the hotfix open the archive and follow instructions in the readme.txt file.

The issue is scheduled to be fixed in next Sitecore release.
To track the future status of the issue please use reference numbers 228602, 218852, 208222, 209972

Sitecore has accepted that 9.0.2 has this BUG and it is in Core architecture. That’s why Sitecore.Kernel and Sitecore.ContentSearch DLLs has been updated and shared with us.

Support package has steps to install hotfix using Sitecore Installation Wizard. But as our application was down and we couldn’t see any screen. We extracted Hotfix package and copied files in relevant directory.

It has a step to do Config changes manually. But was able to create patch file out of those steps [It was not straightforward, And I think that’s why SC Support team gave us manual steps]. Sharing Patch file with you.

Hope you found this post helpful!

 

 

Advertisements

2 Comments

Leave a Comment
  1. Matt / Jan 16 2019 8:09 am

    This totally set us on the right track. For reference, there’s a KB article about it: https://kb.sitecore.net/articles/910936. Thanks!

    • kiranpatils / Jan 17 2019 4:09 pm

      Awesome – Thanks for KB link!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: