Skip to content
January 24, 2013 / kiranpatils

Do you really need AccessResultCache on CD servers? If no, then disable it for better performance!

Challenge:

Before few months back. We were working on Performance issue of our CD Servers. It was really a nice challenge!  would like to share findings with you! So, it might be same challenge for your solution! If not, then also knowing that how we reached to it’s root cause is also a good to read!

We were facing following behavior:

1. Sometimes our CD Server’s response times were spiking and we were getting 502-Bad Gateway and 504-gateway timeout from our live servers!, after establishing pattern we found that it was linked with publishing.

Solution:

First we tried to dig in to this issue. But after some point of time. We raised a Sitecore support case. And thanks to Ekaterina for helping us to diagnose such issue.

To diagnose this issue we did following things:

  1. Configured performance counters — http://msdn.microsoft.com/en-us/library/ff647791.aspx [Especially section “Performance Counters Used to Measure ASP.NET Performance” along with contention counters, look for .Net CLR LocksAndThreads] and we found here that  during slowness time Contention rate/sec goes high. [Contention Rate / Sec. Displays the rate at which threads in the runtime attempt to acquire a managed lock unsuccessfully.]
  2. Sitecore log files and found “HttpModule is being initialized” entries. [Just a note : This entries indicates that your server is busy doing something and all current pool of Http Modules are not enough to serve the requests. So, ASP.NET will create new HttpModules to serve things. But if you see lot of this entries it may or may not be the problem. I think this is the whole new topic to blog — will do it sometime later on this. But in the meantime if you have any question on this, please do post a comment here. Will be happy to answer you!]
  3. Checked Diagnostics log — nothing helpful found
  4. We adjusted our EventQueue logging level to “high” on one of our CD Server and then plotted graph based on time taken by item:saved:remote.  And revealed to us that during spike item:saved:remote was taking lot of time.  [Just a note : Sitecore puts a lock during cache clearing and due to that contentions rate/sec may go high]
  5. Verify your GC mode — is it a workstation or server [http://ashishkhandelwal.arkutil.com/csharp/differences-between-server-and-workstation-gc/]  (Just a note: To verify it via code, you can use : System.Runtime.GCSettings.IsServerGC [Created simple aspx page and hosted it in our website])
  6. Generated memory dump using DebugDiag — Good to read Ekaterina’s blog here

out of all above activities, memory dump lead us towards the solution. Eager to know what was that? let’s discuss:

We submitted this memory dump to Sitecore support and from dump we found that our AccessResultCache size was 200 MB on CD server and which was causing this issue.

First, let’s discuss what’s the role of Sitecore cache. In Sitecore caching layers.  Every time when anybody accesses any item in Sitecore result of resolved security right is put to AccessResultCache. It is not related to content editing, but to content accessing at all. Resolving security usually takes much more time than accessing an item in a hashtable, therefore it is not so safe for performance to disable cache completely. So, for example If user A is accessing an item and to decide whether user A has access of that item or not, Sitecore resolves it’s security rights from DB and then puts it in AccessResultCache. So, when next time user A comes back to access the same item Sitecore can save time in resolving security rights again from DB and server it from cache.

But we found from dump is that AccessResultCache was occupying 10 GB. so it’s obvious now why it was so difficult to make cache clear operations on this large hashtable. And yes you must be thinking why it was occupying 10 GB when we restricted it to 200 MB. Exactly, we also had same question and got it answered from SC team that it is not easy to estimate object size accurately.

So, we were thinking why we need AccessResultCache on CD server? Because our content editing and authorization part is on CM server only. To, be more clear. Our server architecture is as follow [Courtesy — http://www.awareweb.com/AwareBlog/ArchConsideration.aspx]:

20110414Sitecore-Isolation-Architecture2

So, as per above diagram, we don’t have any authorization happening on CD Server. What, you also have the same architecture? You also have enabled AccessResultCache on CD server. [Just a note : to check how much space is occupied by each SC cache — CacheTuner module can be your friend — http://marketplace.sitecore.net/en/Modules/Cache_Tuner.aspx]

So, let’s see how to disable AccessResultCache from CD Server.

We were having following options availalbe to do it:

  1. Set AccessResultCache size to 0 MB — But If you disable the cache, then for each item the whole access will be checked up again each time when user accesses the site, and this may decrease performance. So, this is not a good option for you!
  2. Disable Security check completely on CD server — This is the best option. But to achieve this you have other options, out of which you need to decide the best one for you!
    1. Set SecurityEnabled property of corresponding database to false in web.config. [false for node]
    2. Another way is to place SecurityStateSwitcher switcher = new SecurityStateSwitcher(SecurityState.Disabled); in the beginning of httpRequestBegin pipeline.
    3. Rewrite SqlServerAuthorizationProvider

Okay. So, you must be thinking out of above 3 suggestions which you think is good idea to do? As we are striving for performance. Which one will perform high? We did a performance test for all of 3 options and we found that rewriting SqlServerAuthorizationProvider was the fastest option! And it also makes sense because on each item access we no need to do any security checks. So, eager to know how can you rewrite SqlServerAuthorizationProvider? Let’s see how can you do it:

  1. Create a class in your security project [for example — SCBasicsAuthorizationProvider], if not exist it would be great if you create new one. Make sure it’s type is Class Library. All you need to disable security check is to override
    public virtual AccessResult GetAccess(ISecurable entity, Account account, AccessRight accessRight)
    method, so it returns new AccessResult(AccessPermission.Allow,null).  See following sample code:
  2. On your CD server(s) take backup of everything.
  3. Open Web.Config and search for — SqlServerAuthorizationProvider, Sitecore.Kernel” connectionStringName=”web” embedAclInItems=”true” /> comment out this tag replace it with your class name created earlier e.g.     SCBasics.Security.SCBasicsAuthorizationProvider, SCBasics.Security” connectionStringName=”core” embedAclInItems=”true” />

SCBasicsAuthorizationProvider code:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Sitecore.Security.AccessControl;
using Sitecore.Security.Accounts;
using Sitecore.Diagnostics;
using Sitecore.Data.SqlServer;
using System.Collections.Specialized;
using Sitecore.Configuration;

namespace SCBasics.Security
{
public class SCBasicsAuthorizationProvider : SqlServerAuthorizationProvider
{
/// <summary>
/// Override for disabling AccessResultCache
/// on CD servers
/// </summary>
/// <param name="entity">Isecurable</param>
/// <param name="account">Account</param>
/// <param name="accessRight">AccessRight</param>
/// accessResult with Permission
public override AccessResult GetAccess(ISecurable entity, Account account, AccessRight accessRight)
{
AccessExplanation accessExplanation = new AccessExplanation("SCBasics : AccessResultCache disabled on CD Servers.");
AccessResult accessResult = new AccessResult(AccessPermission.Allow, accessExplanation);
return accessResult;
}
}
}

Deploy it and see your Sitecore CD server running faster than your thoughts!

Happy Sitecoring! 🙂

on each item access we no need to do security checks.

Advertisements

11 Comments

Leave a Comment
  1. Thomas / Jan 24 2013 4:40 pm

    Hi

    You suggest to use CacheTuner module to see the size of AccessResultCache. Can one not use /sitecore/admin/cache.aspx for this?
    I an example of a site with 10.000+ items, i have not seen AccessResultCache above 3mb on /sitecore/admin/cache.aspx. Did your site have alot of items?

    • kiranpatils / Jan 25 2013 10:01 pm

      Hello Thomas,

      First of all thanks for reading! Yes, you can use Cache.aspx as well. I do refer CacheTuner because it will highlight your cache if it’s occupying 80% of your max size limit.

      We do have 7Lakh+ items. So, it might be cause of this. But If you also have same architecture as us then you can also try disabling AccessResultCache on CD. Because it is doing unnecessary security checks.

      Keep visiting! Keep reading! Keep Sharing! Happy Sitecoring! 🙂

      Sincerely,
      Kiran Patil

  2. Christopher Giddings / Mar 25 2013 1:57 pm

    great find, what type of load was the server under, eg concurrent users

  3. Christopher Giddings / Mar 25 2013 5:40 pm

    nice find what type of traffic were you experiencing that caused this type of issues, eg: amount of concurrent users

    • kiranpatils / Apr 22 2013 10:19 pm

      Hello Christopher,

      Apologize for delay in reply, Somehow I missed this comment.

      Thank you for reading!

      – Currently our application serves around 35 to 40 requests per page view
      – Currently our application serves around 100-150 requests per second (daily or weekly average suggest around 100 RPS, however during business hours it reaches up to 150 RPS)

      Keep reading, keep sharing!

      Sincerely,
      Kiran Patil

      • Christopher Giddings / Aug 29 2013 6:59 am

        Hi Kiran,

        This doesn’t seem that high, we have servers which are peaking around 2k concurrent views and have the same type of setup. I do agree it is a great fine you have made to disable but i would also ask the question of why is your site not handling the load.

        Try install NewRelic on the servers and also run some performance testing like red-gate, this may fine the issue.

        You can also try caching and or using lucene for some expensive queries. I recently undertook this process with a site which had 3 load balanced servers running 8 cores and had 32gb each. The machines were beasts but this was of the construction of the site. I refactored the systems data paths and this alone improved performance dramatically, to add to this we added a custom Framework which ill be blogging about on my site which introduced Sitecore to distributed cache this pushed the performance through the roof and now each machine can handle around 10k concurrent requests second. Oh yeah and the site has DMS so a little hard work you can get it executing extremely fast.

        Check out my cache util, its a bit old but may help with storing large / expensive data objects in an easy way.

        Kind Regards,
        Chris

      • kiranpatils / Aug 29 2013 7:45 am

        Hello Chris,

        Thank you for sharing your views! Agree. But in our environment we got > 7Lakh+ items, also it’s a multisite solution having 700+ websites served via a Single sitecore solution. Along with our custom templated renderings.

        We do have New Relic installed on our CM and CD server.

        >>I refactored the systems data paths and this alone improved performance dramatically, to add to this we added a custom Framework which ill be blogging about on my site which introduced Sitecore to distributed cache this pushed the performance through the roof and now each machine can handle around 10k concurrent requests second.
        Sounds promising, eager to know more about it.

        We are heavily relying on caching [Mainly HTML and Item]

        >>Check out my cache util, its a bit old but may help with storing large / expensive data objects in an easy way.
        Had a look. But we are not storing anything in cache, and to be honest, was not clear why we need custom cache. Because Sitecore caching and CustomCache support from Sitecore good.

        Happy Sitecoring! 🙂

        Thanks,
        Kiran

  4. Zeba / Dec 11 2014 4:36 am

    Hi Kiran,

    Would you be able to tell more about the “HttpModule being initialized” topic. I was fine tuning our production website and noticed the logs with above entries. The original issue that I was scanning my logs was that our authors/editors complained about “items not been published”. But the case was not such. The items were getting published but writing in the index happened later than expected. (we use OnPublishEndAsync indexing strategy and our IndedUpdateInterval is set to 5 minutes as the fall back) During that timeframe that was reported to me, I checked the logs and found that during that time, there were a couple of entries of the HttpModule getting intialized and later the log entry for the web index being updated. The item eventually appeared into the CD evnronment matching the timeframe reported. I looked further into these log entries and looked up for more information around these log entries. I came across posts that you have mentioned in solution #2. Like you said, its a whole new subject to indulge into, it would be great to know more.

    For now, I have some questions:
    – How many entries are too many entries? For eg. in my log files, there are 5 entries back to back at a time and at times its 1 entry. There is no existing patter that I have been able to establish at this point.

    Looking forward to hear from you.

    Cheers,
    Zeba

Trackbacks

  1. Basic ways of troubleshooting EventQueue challenges « Sitecore basics!
  2. Sitecore 8.2 increase in AccessResultsCache – sitecore saga

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: