Challenge:
Currently we are working for one of our client application, Which is on Sitecore 9.0.2 and deployed on Azure App. It was working fine for sometime and one fine day, post deployment application was not coming up. This was client’s QA environment and in lot of developers and QA team members were keenly looking up to us to resolve this ASAP. So, we can hit our sprint goal.
We tried the lot of options to make it up and running. But we could only see following page (503):
We tried to check logs. But nothing fruitful was found. DevOps team already had Azure ARM Template. So, we thought to install new environment from scratch and redeployed and that worked for a while. But after few days, Same issue happened on QA Environment.
Now, This was the time for us to generate a crash dump and analyze it [You also thought so?]. But then we realized that this is Azure App and DebugDiag won’t work here. Sounds interesting right? [Life is not as easy, as it seems to be]. But each challenge has solution in this world. And that’s true for this challenge as well. If you are also facing similar challenge or you are keen to know how we solved this, then this post is for you!
Solution:
UPDATE : 17-Jan-2019 : Sitecore has official KB article and patch on it :https://kb.sitecore.net/articles/910936 [Thanks Matt for your comment]
- Quick google search suggested this post : https://kb.sitecore.net/articles/983839 we checked those things. But with no luck.
- Application insights also didn’t help : https://doc.sitecore.com/developers/90/sitecore-experience-management/en/analyze-sitecore-logs-with-application-insights.html
- Then we came across, Sitecore’s KB article for Advanced Troubleshooting on Azure App (Was not able to generate dump using it, It might be due to my overlook and few information seems not up-to-date [Azure Portal is always changing]) : https://kb.sitecore.net/articles/111669
- Then found this great article, Which gave me some hints : https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/app-service/troubleshoot-http-502-http-503.md
I decided to delve in to Azure Portal and based on what I learnt try something out (I will quickly remind you this is QA environment and roughly ~12 folks are affected by this and overall sprint plan/project plan.)
- Selected Diagnose and Solve Problems, Which has a lot of interesting options as shown below (Diagnostic Tools is an option, Which I was looking for) – a.k.a. DaaS:
- When I clicked on “Memory Dump” [One of the sub option of Diagnostic tools]. It brought few more options:
- Selected “Collect Memory Dump” option [Make sure you have selected ASP.NET in top]. As soon as you click on that Azure checks whether your tier has this service support or not? [Microsoft — One more opportunity to make money :). But they deserve with what they have done. And when your application is business critical and has impact on revenue, Then no harm in investing in this – it’s all business!]. Based on your tier you will see next screen. This is what I see:
- Once you enable this, as soon as application will get crashed. It will generate memory dump, [And the best part] Analyze and give you Analysis Report! You can also Download required files [Dump file etc.]
- Nice to read/watch on DaaS:
Finally, We have Dump analysis report and this is what we found:
[[HelperMethodFrame_1OBJ] (System.Threading.Monitor.Enter)] System.Threading.Monitor.Enter(System.Object) Sitecore_Kernel!Sitecore.DependencyInjection.LazyResetable`1[[System.__Canon, mscorlib]].get_Value()+5e Sitecore_Kernel!Sitecore.Security.AccessControl.AccessRight.FromName(System.String)+34
Sitecore_Kernel!Sitecore.Security.AccessControl.ItemAccess.CanRead()+1c Sitecore_Kernel!Sitecore.Data.Managers.ItemProvider.ApplySecurity(Sitecore.Data.Items.Item, Sitecore.SecurityModel.SecurityCheck)+51 Sitecore_ContentTesting!Sitecore.ContentTesting.Pipelines.ItemProvider.GetItem.GetItemUnderTestProcessor.Process(Sitecore.Pipelines.ItemProvider.GetItem.GetItemArgs)+b3
Sitecore_ContentTesting!DynamicClass.(System.Object, System.Object)+f Sitecore_Kernel!Sitecore.Pipelines.CorePipeline.Run(Sitecore.Pipelines.PipelineArgs)+1e4 Sitecore_Kernel!Sitecore.Data.Managers.DefaultItemManager.GetItem(Sitecore.Data.ID, Sitecore.Globalization.Language, Sitecore.Data.Version, Sitecore.Data.Database, Sitecore.SecurityModel.SecurityCheck)+131 Sitecore_Kernel!Sitecore.Data.Managers.DefaultItemManager.GetItem(Sitecore.Data.ID, Sitecore.Globalization.Language, Sitecore.Data.Version, Sitecore.Data.Database)+93 Sitecore_Kernel!Sitecore.Data.Managers.ItemManager.GetItem(Sitecore.Data.ID, Sitecore.Globalization.Language, Sitecore.Data.Version, Sitecore.Data.Database)+9a Sitecore_Marketing_xMgmt!Sitecore.Marketing.Definitions.Outcomes.Data.ItemDb.ItemOutcomeDefinitionRepository..ctor(Microsoft.Extensions.Logging.ILogger`1, System.String, Boolean, Sitecore.Marketing.Definitions.Repository.IDefinitionRecordMapper`1)+345 Sitecore_Marketing_xMgmt!Sitecore.Marketing.Definitions.Outcomes.Data.ItemDb.ItemOutcomeDefinitionRepository..ctor(Microsoft.Extensions.Logging.ILogger`1, Sitecore.Marketing.IItemRepositoriesSettings, Sitecore.Marketing.Definitions.Repository.IDefinitionRecordMapper`1)+e4
[[DebuggerU2MCatchHandlerFrame]]
Other threads:
** Microsoft_Extensions_DependencyInjection_Abstractions!Microsoft.Extensions.Internal.ActivatorUtilities.GetServiceOrCreateInstance(System.IServiceProvider, System.Type)+21 Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.CreateFromServiceProvider(System.Xml.XmlNode, Boolean)+1ae Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.CreateObject(System.Xml.XmlNode, System.String[], Boolean, Sitecore.Configuration.IFactoryHelper)+8a Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.CreateObject(System.Xml.XmlNode, System.String[], Boolean)+48 Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.CreateObject(System.String, System.String[], Boolean)+27c Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.CreateFromReference(System.Xml.XmlNode, System.String[], Boolean)+a6 Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.CreateObject(System.Xml.XmlNode, System.String[], Boolean, Sitecore.Configuration.IFactoryHelper)+74 Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.CreateObject(System.Xml.XmlNode, System.String[], Boolean)+48 Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.GetInnerObject(System.Xml.XmlNode, System.String[], Boolean)+6d Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.GetConstructorParameters(System.Xml.XmlNode, System.String[], Boolean)+a1 Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.CreateFromTypeName(System.Xml.XmlNode, System.String[], Boolean)+5e Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.CreateObject(System.Xml.XmlNode, System.String[], Boolean, Sitecore.Configuration.IFactoryHelper)+a3 Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.CreateObject(System.Xml.XmlNode, System.String[], Boolean)+48 Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.CreateObject(System.String, System.String[], Boolean)+27c Sitecore_PathAnalyzer!Sitecore.PathAnalyzer.ApplicationContainer.CreateObject[[System.__Canon, mscorlib]](System.String)+de Sitecore_PathAnalyzer!Sitecore.PathAnalyzer.ApplicationContainer.GetOrCreateProcessingPool(System.Guid ByRef)+29
**
Microsoft_Extensions_DependencyInjection_Abstractions!Microsoft.Extensions.Internal.ActivatorUtilities.GetServiceOrCreateInstance(System.IServiceProvider, System.Type)+21 Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.CreateFromServiceProvider(System.Xml.XmlNode, Boolean)+1ae Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.CreateObject(System.Xml.XmlNode, System.String[], Boolean, Sitecore.Configuration.IFactoryHelper)+8a Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.CreateObject(System.Xml.XmlNode, System.String[], Boolean)+48 Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.CreateObject(System.String, System.String[], Boolean)+27c Sitecore_Kernel!Sitecore.Configuration.DefaultFactory.CreateFromReference(System.Xml.XmlNode, System.String[], Boolean)+a6
After analyzing all stacktraces we concluded that, it is mainly in Sitecore Stack, and then we raised a Sitecore support ticket and shared all relevant information with them.
Sitecore support team were able to rectify the issue in a day and get back to us with following information:
Please use the following hotfix to resolve the issue:
https://dl.sitecore.net/hotfix/SC Hotfix 235278-2 CMS.Core 11.1.2 ContentSearch 3.1.2.zip
To install the hotfix open the archive and follow instructions in the readme.txt file.
The issue is scheduled to be fixed in next Sitecore release.
To track the future status of the issue please use reference numbers 228602, 218852, 208222, 209972
Sitecore has accepted that 9.0.2 has this BUG and it is in Core architecture. That’s why Sitecore.Kernel and Sitecore.ContentSearch DLLs has been updated and shared with us.
Support package has steps to install hotfix using Sitecore Installation Wizard. But as our application was down and we couldn’t see any screen. We extracted Hotfix package and copied files in relevant directory.
It has a step to do Config changes manually. But was able to create patch file out of those steps [It was not straightforward, And I think that’s why SC Support team gave us manual steps]. Sharing Patch file with you.
Hope you found this post helpful!
Challenge:
In ~9 years of Sitecore experience have been happy to contribute 8 modules on Sitecore Marketplace (If you are new to Sitecore world and not aware of this, then let me explain you — It is Sitecore’s Play Store [Android]/App Store [iOs] where anyone (Including you) can contribute any Sitecore module ):
Out of these modules, Which modules I really feel proud of and Cache Tuner is one of that. The reason behind it is — It is still widely used by community and lot of community members have already recommended it in community. Thank you folks!
- https://www.cmsbestpractices.com/friday-sitecore-best-practice-sitecore-cache-tuning/
- https://sitecore.stackexchange.com/questions/158/cms-performance-tuning-for-sitecore-8
- https://sitecore.stackexchange.com/questions/8470/optimal-html-cache-size
- https://kverheire.blogspot.com/2014/12/sitecore-hidden-gems-admin-pages-based.html
- https://varunvns.wordpress.com/2012/05/28/is-your-sitecore-application-running-slow-lets-speed-it-up-then/
This was my first shared module : https://sitecorebasics.wordpress.com/2011/04/28/cache-tuner-my-first-shared-source-module/ As you can see it was on old platform trac and If you tried tuning Sitecore cache then you must have come across Cache Tuning guide which has some rules based on which you tune Sitecore caches. CacheTuner helps you automate that, Sounds interesting? Read Cache Tuner guide : https://sitecorebasics.wordpress.com/2011/04/27/cache-tuner-user-guide/
Recently, I was trying to use it for my current project which is on Sitecore 9.0.2 and as soon as I ran it I got error due to Sitecore’s changes in Caching API (Sitecore 8.2).
- https://doc.sitecore.com/developers/82/sitecore-experience-platform/en/cache-api-changes.html
- https://sitecore.stackexchange.com/questions/199/sitecore-caching-customcache-changes-with-8-2
- https://breaksitecoreblog.wordpress.com/2016/09/06/sitecore-8-2-cache-changes/
Changes are mainly done to support DI and that’s why none of the methods are returning concrete classes.
Solution:
I thought to spend time today to make it Sitecore 9 compatible, and you will be happy to know that now CacheTuner supports >=8.2 version. Needless to say, you have to rely on old CacheTuner for earlier versions. [You are still on older version? It’s time to upgrade :-)]
Have also created git repo for updated CacheTuner module : https://github.com/klpatil/CacheTuner
This is my Christmas gift for all of you, Have a Happy Holidays!
Challenge:
We are porting one of our client’s App from Older Sitecore version and infrastructure (IaaS) to Sitecore 9 and Azure Paas Web App.
They have 20+ languages and we learnt that they have few custom languages as well (en-TH, en-VN, en-ae). You might get excited and say, You can use : https://marketplace.sitecore.net/en/Modules/Custom_Language_Registration.aspx
But If you said so, Then you must’ve missed last line, App is on Azure PaaS web App. And Custom Language Tool is Windows desktop application. Which you can’t run on Azure web App. Interesting?
Wondering, How we resolved it? And you are also facing similar challenges? Then this post is for you.
Solution:
First of all, we did quick search and found following links:
- https://sitecore.stackexchange.com/questions/6189/how-to-add-custom-language-to-sitecore-on-azure-paas-web-apps
- https://medium.com/@Vincent_Lui/sitecore-cloud-migration-from-on-premise-to-microsoft-azure-platform-as-a-service-paas-part-2-9e1cdff2fbd2
This link found from first link is really nice read and confirms one thing THAT IT’S NOT POSSIBLE to do and then provokes some really good questions : https://blogs.msdn.microsoft.com/benjaminperkins/2018/06/11/implementing-custom-cultures-cultureinfo-localize-azure-app-service/
You cannot, at this moment, create a custom culture on an Azure App Service. Cultures are part of the standard operating system and require changes to the registry to modify or add them. An Azure App Service runs in a sandbox which does not allow code or script to modify the registry. I do not work on the team responsible for this Windows feature, but I have confidence the team puts lots of diligence into providing the fundamental capabilities to match the requirements of most global cultures.
Second blog gave some good alternative solution using URL Rewrite.
But before we do that, We thought to see what all cultures and languages are available on Azure Web App and was able to get that list. You can also download it from here : https://drive.google.com/open?id=1lmNoZPJRyvkTaAPo8dTMaEaM0VVidhdE
We couldn’t find exact language. But we found alternative languages, Which we could use for our Sitecore solution and after consulting with our client we used it! Reference list is as below:
- en-TH -> th-TH
- en-VN -> vi-VN
- en-ae->ar-ae
Hope you found this helpful!
Challenge
Recently we were facing 503 error on our Sitecore 9.0.2 version Sitecore App deployed on Azure App (Blog on that topic soon)
We reached out to our common friends — Sitecore Support team and they gave me Two Hotfix DLLs :
- Sitecore.Kernel
- Sitecore.ContentSearch
Til this point of time, I got DLLs like Sitecoe.Support.SOMNUMBER.DLL and it was easy to deploy. I can copy in Assemblies/Lib folder in your directory, and reference it from your Project. And it will be deployed on higher environments.
But If you are wondering how to do it for these two DLLs? As they are coming from Sitecore’s Nuget feed in your current solution. Are you? Then this post is for you.
Solution
As you do, I also did a quick search and foud this from Sitecore’s official doc : https://doc.sitecore.com/developers/82/sitecore-experience-platform/en/sitecore-public-nuget-feed-faq.html
That was disappointing 😦 Then thought to reach out to our internal Sitecore experts and following great ideas came up:
- Private Nuget server : This makes sense. But have to reach out to client and ask them to spin up new nuget server and deploy DLL there. But was unable to figure out, how it will resolve from nuget references [Sitecore] and Custom nuget having same DLLs with same name. As I was not clear and had client team’s dependencies thought to look for other solution.
- Local Reference : Another plan was to use old school approach, Where you created Assemblies/Libraries/Libs folder under your Project directory and instead of nuget use local reference in all your projects. But it had two challenges:
- We are using Helix and we have lot of projects, where we need to do these updates ~20. Which I was not sold.
- In future, If we have to do version upgrade. We have to again do these changes manually for all projects. Lot of work Now and in future – not efficient. So, thought to think of another approach
- Post build step : Another idea was to provide post build steps in last project, which copies hotfix’d DLLs. That could have fixed local development environment challenges. But we use Teamcity, Octopack and Octopus deployment and this idea would need additional steps from DevOps team. Which is fine for now. But in future — Same efforts needs to be done to undo it. So, this was also not a good idea.
Then I applied my 99 things learning, Step away from keyboard : https://github.com/97-things/97-things-every-programmer-should-know/blob/master/en/thing_69/README.md
And got a new idea:
By default nugget looks for DLL file from Packages folder and if exist, it doesn’t download and uses it as a reference DLL. For example : Sitecore.Kernel.DLL can be found under <YOURPROJDIR>\packages\Sitecore.Kernel.NoReferences.9.0.180604\lib\NET462
This was Eureka moment. Where I thought to do this:
- Copy new DLL from Sitecore hotfix package and pasted it under
<YOURPROJDIR>\packages\Sitecore.Kernel.NoReferences.9.0.180604\lib\NET462 - Committed : Packages folder and
Sitecore.Kernel.NoReferences.9.0.180604\lib\NET462 folders - That’s it!
- it worked on Local – Teamcity and Octopus!
This helped? You found better approach than this? I’m all ears!
Happy Coding! 🙂
Update:
Somehow, I haven’t done good job sharing context of this blog post. Got some comments and thought to share it here. So, this post’s information is relevant:
@sitecorey @jammykam Thank you everyone for your thoughts - It seems my post was not clear and haven't shared conte… twitter.com/i/web/status/1…
—
Kiran Patil (@kiranpatils) December 06, 2018@sitecorey @jammykam Found this way to resolve it without trying something unknown. Hope this context helps. Also a… twitter.com/i/web/status/1…
—
Kiran Patil (@kiranpatils) December 06, 2018The best solution (In the best case scenario) is to have Filesystem based/Private Nugget server : https://docs.microsoft.com/en-us/nuget/reference/nuget-config-file
Challenge:
We love automation and so as our clients. For almost all our clients, We do automatic deployment. Using Teamcity and Octopus.
As you know Sitecore automatic deployment is not easy, it involves the lot of things like ASP.NET MVC Files, DLLs, Sitecore Items, and Sitecore publishing. For Sitecore publishing we use : http://fortis.ws/fortis-collection/toolcore/publishing/
Which works best. But recently it was failing and we started getting error related to TLS after integrating it with Sitecore 9.0.2 installation deployed on Azure App Service.
You are also facing same error with Toolcore Publishing module? Then this post is for you!
Solution:
We did quick search and came across this SO thread — Which has some pointers for solution : https://sitecore.stackexchange.com/questions/13706/all-sitecore-8-2-x-azure-topologies-seem-to-be-effected-because-of-tls-issues
Microsoft disabled support of TLS 1.0 on App Services recently (https://social.msdn.microsoft.com/Forums/en-US/9dace9cb-a1a8-4f77-af26-324bc2b6740e/tls-plans-for-azure-app-service?forum=windowsazurewebsitespreviewenter link description here)
Technical details can be found here: Transport Layer Security (TLS) best practices with the .NET Framework
https://kb.sitecore.net/articles/688903
Any .Net call relying on TLS 1.0 is leading to authentication issues on Azure App Services:Exception
System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. --->
System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. --->
System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host.
We tried switching our Azure App service to TLS 1.0 and it worked!
I know, You must be thinking this is not the best solution. And so, as we!
You might have noticed from above links that, If any application is making HttpRequest using .NET HttpRequest class and using TLS 1.0 then it won’t work. And Solution is either changing code (ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12) and using TLS 1.2 or compile your application using >=.NET Framework 4.6.0 version and above as it uses TLS 1.2 by default. This makes perfect sense!
But how does it impat Toolcore publishing? After digging bit we understood that FortisCollections.Toolcore.Publish.Service has .asmx web service which is deployed on Sitecore CM Server and FortisCollections.Toolcore.Publish.Runner console application has Proxy which is invoked.
This was Eureka moment!
Thought to check with Jason (repo contributor) via github : https://github.com/Fortis-Collection/Toolcore/issues/5
And parallely thought to fix it by forking it and compiling it with new version — And it worked!
Obviously, So, as you I got busy after that and never got a chance to share it back with larger community. Couple of days back, one of my colleague, reached out to me requesting compiled DLL package and that was a moment, I realized this weekend. Have to share it with larger world!
PR Created : https://github.com/Fortis-Collection/Toolcore/pulls and waiting for Jason to approve. But till then you can use it if you are also facing similar issue!
Hope you go home after fixing this! 🙂
Challenge:
Sitecore 9 comes with ~50 roles. Which is obvious in this Microservices era. Sitecore team has done good job documenting all the roles and scenarios as well providing videos to help us understand.
I wanted to document a quick reference, Which we can refer quickly. It helped me to connect the dots. Thought to share with you as well!
Solution:
Here you go:
While working on this document, Got connected with SUGNCR team. They were hosting their second in-person meetup. So, thought to present this topic there along with my friend – Brijesh Patel
Event was really great and nicely organized — It was really great to meet lot of Sitecore passionate people in-person.
Slides are as below:
Pre-Session survey results are as below:
As you can see ~30 people were there and average rating was 2.4. They had mid level knowledge about Storage Roles, Cloud Services and Scaling.
Our session’s goal was to increase this knowledge and that’s what has been done in above given slides. Due to some reason, we haven’t gathered post-sesion survey. But session feedback, Which we got from organizers is astonishing – Thank you so much audience!
Hope this helps!
Challenge:
Have been crafting on Sitecore since a decade and have came up with lot of unique Sitecore challenges.
Which I’ve been sharing with you via this blog, And have received good feedback from you — And I appreciate you for that. If you haven’t came across it yet. Then you can visit following links:
- https://sitecorebasics.wordpress.com/category/performance/
- https://sitecorebasics.wordpress.com/tag/troubleshooting/
- https://www.scribd.com/document/132658804/Sitecore-Troubleshooting-Path
While learning Debugging/Troubleshooting Tess’s blog helped me a lot. Especially Labs (You as well?!) :
Sine long time, I wanted to build something similar for Sitecore community. But most of the times being super busy, and at times being lazy [Happens with you as well!? ;-)]. But when I thought to present at SUGCON India I thought this is the best time to work on this idea! (If you couldn’t attend SUGCON India then you missed a great event — Everyone did a great job to make this happen first time at India. You can see great pictures here : http://www.sugcon.in/2018-pictures/)
I’m excited to share, What I have done with you. Which I already shared at SUGCON India and SUGUAE events.
Solution:
You can get it from here : https://github.com/klpatil/SCbuggybits (Read Me file should help you to get started) This lab contains 3 Bugs in the form of 3 Labs. It will be fun to find it and fix it!
My Session slides are here : http://www.sugcon.in/wp-content/uploads/2018/05/Troubleshoot-Sitecore-Like-a-Ninja-Kiran-Patil-SUGCON-India-2018.pdf
My session video is here :
Give it a try and let me know what you think? You found new bug in your day to day life, And would like to add it scbuggybits, I encourage you to do so!
Hope this helps you to fix your issues and let’s you go home on time! 🙂
