Skip to content
August 4, 2021 / kiranpatils

Azure PaaS performance challenges due to logging

Challenge

During load testing of a new site before launch, we noticed performance issues, when we dug further we learned that it is happening due to logging. So, thought to share with you.

Solution

The environment on which we were testing was on P2V2 (Scaled to 4 Instances). Before we talk about logging. Let’s first understand how Azure PaaS scaling works.

Disclaimer : Tried to find an official diagram. But couldn’t find anything on MS Doc site. So, this is my understanding based on learnings.

Above diagram depicts Azure PaaS scaling at high level. Your scaled sites are running on different cloud machines (read: VM). So, the worker process is separate. But file system is shared across all the worker processes/websites. That’s why Sitecore creates a folder for each machine name in App_Data/Logs folder. If you would like to identify your machine-specific log file, you can identify using this post.

Each instance also has its own set of directories. Which is also known as Local cache. But we will not discuss that in this post. You can refer : https://docs.microsoft.com/en-us/azure/app-service/overview-local-cache. We had some challenges using it with Sitecore. So, we don’t recommend to use it or If you need to using it for your scenario. Please ensure you verify all caveats.

Now, let’s refer to what we get with P2V2. We get SSD Storage.

Source : https://azure.microsoft.com/en-us/pricing/details/app-service/windows/

To troubleshoot our performance issues, we noticed a lot of INFO Logs (OOTB) were getting logged. We discussed this issue with our internal team and one of the team recently faced performance issues due to heavy logging. We did a quick search and came across the following great posts:

https://briancaos.wordpress.com/2019/12/05/sitecore-and-application-insights-how-to-remove-90-of-all-log-lines-without-sacrificing-traceability/

https://www.sitecorefundamentals.com/how-to-reduce-one-or-many-sitecore-log-entries-from-application-insights

After going through the above posts, we also noticed that Log entries are getting logged not only to Sitecore Physical log files but also into Application Insights.

If you would like to understand more about Azure PaaS logging. Please refer following posts:

  1. https://jeffdarchuk.com/2019/05/03/dude-wherere-my-logs-azure/
  2. https://onelittlespark.bartverdonck.be/sitecore-logging-fundamentals-azure-paas/
  3. https://blog.baslijten.com/improving-the-sitecore-logging-and-diagnostics-experience-part-2-logging-to-application-insights/

Especially, if you are a visual person like me (Most of us are!). The following diagram from post#2 is really helpful

And this as well

Overall we have observed that Azure website File IO is comparatively slower than IaaS (As per various internet articles and personal experience.)

Now, Let’s connect the dots. We have 4 instances writing to the same file system and during load, it will be doing heavy File IO operations. This is going to be slow and in Azure PaaS world, it is comparatively slower.

To resolve this as per earlier post 90% entries are not required for your healthy PROD environment as it’s mostly cluttered.

We had to make minor tweaks for Sitecore 9.3 and created Patch file, Which you can also use in your environment. Feel free to modify logFilterRegExPattern variable as per your need.

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/&quot; xmlns:useApplicationInsights="http://www.sitecore.net/xmlconfig/useApplicationInsights/"&gt;
<sitecore>
<sc.variable name="logFilterRegExPattern" value="^.*(Job started|Job ended|document not found page|Cache created|Solr Query).*$" />
<log4net>
<!–AZ Fallback appender – Logs in log file–>
<!–Delete First–>
<appender name="AzureFallbackAppender" type="Sitecore.ExperienceContentManagement.Administration.Logging.AzureFallbackAppender, Sitecore.ExperienceContentManagement.Administration">
<patch:delete />
</appender>
<!–Add new with filter by SCBasics Patch file–>
<appender name="AzureFallbackAppender" type="Sitecore.ExperienceContentManagement.Administration.Logging.AzureFallbackAppender, Sitecore.ExperienceContentManagement.Administration">
<file value="$(dataFolder)/logs/{machineName}/azure.log.{date}.{time}.txt" />
<appendToFile value="true" />
<rollingStyle value="Size" />
<maxSizeRollBackups value="-1" />
<maximumFileSize value="10MB" />
<immediateFlush value="false" />
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%4t %d{ABSOLUTE} %-5p %m%n" />
</layout>
<encoding value="utf-8" />
<!– Added by SCBasics Patch file –>
<filter type="log4net.Filter.StringMatchFilter">
<regexToMatch value="$(logFilterRegExPattern)" />
<acceptOnMatch value="false" />
</filter>
</appender>
<!–Log4Net appender – Logs in AI–>
<appender name="LogFileAppender" type="log4net.Appender.RollingFileAppender, Sitecore.Logging">
<patch:delete/>
</appender>
<appender name="LogFileAppender" type="log4net.Appender.RollingFileAppender, Sitecore.Logging">
<file value="$(dataFolder)/logs/log.{date}.{time}.txt" />
<appendToFile value="true" />
<rollingStyle value="Size" />
<maxSizeRollBackups value="-1" />
<maximumFileSize value="10MB" />
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%4t %d{ABSOLUTE} %-5p %m%n" />
</layout>
<encoding value="utf-8" />
<!– Added by HZTL Patch file –>
<filter type="log4net.Filter.StringMatchFilter">
<regexToMatch value="$(logFilterRegExPattern)" />
<acceptOnMatch value="false" />
</filter>
<!–Had to add it here as Azure file patch applies first and environent happens later–>
<category useApplicationInsights:require="!false" value="log" />
</appender>
</log4net>
</sitecore>
</configuration>

This improved our load testing results and we also know why 🙂

Hope this helps, One thing I would like to clarify when you have performance issues due to load testing, It can be due to N number of possibilities. So, this could be one of them.

Happy Performance Tuning! 🙂

One Comment

Leave a Comment
  1. Kyle Heon (@kyleheon) / Aug 4 2021 6:33 pm

    Regarding your comment about Azure Local Cache and Sitecore, it can work, but it’s limitations make it tricky. We have some details on how to enable, and the concessions your customer may need to make for it to work outlined here: https://support.sitecore.com/kb?id=kb_article_view&sysparm_article=KB0909298

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: