Impact of apache-log4j-cve-2021-44228 for your Sitecore Applications

By this time you might have already heard/read about Apache Solr Log4J Security Vulnerability: https://solr.apache.org/security.html#apache-solr-affected-by-apache-log4j-cve-2021-44228

I’ve been also gathering information around it, As I was curious to know more about this and the impact of the same on Sitecore applications. I’ve gathered some basic information. Which thought to share with you as well!

Note: I’m not a security expert and don’t have a full visibility into your environment/implementation/setup. So, please take this advice as a guiding principle. But please work with your internal teams and follow their recommendations

Let’s delve into this further:

What is the apache-log4j-cve-2021-44228 security vulnerability?

The following excerpt from the Official Page describes an issue:

Source: https://solr.apache.org/security.html#apache-solr-affected-by-apache-log4j-cve-2021-44228

Would like to know more about this? Then please use the following image, it explains it nicely and also provides the solution to fix it (Which we will discuss in the next section)

Image Credits: https://www.govcert.ch/blog/zero-day-exploit-targeting-popular-java-library-log4j/

Couldn’t resist sharing the following Image as it really says “1000+ words!” in this situation:

Image Credits: https://xkcd.com/2347 (Similar Image)

Have we been impacted by this Security Vulnerability?

• Versions Affected: 7.4.0 to 7.7.3, 8.0.0 to 8.11.0 : If your Solr version falls in to given list, then you are impacted by this issue. (Applies to : 9.2.0 (Initial Release) +)
• Solr public access :
  • Update#4 : As per recent Twitter comments from Nick Wesselman and Per Manniche Bering, I learned that this issue can also impact your Solr if it’s not public. Solr logs queries, and attackers can try to provide malicious queries via user input.
  • If your Solr is not publicly accessible, Then you won’t be impacted by this issue. (As per my understanding and talking to Solr experts in my network. But if you think otherwise, please comment). This is usually the case with Sitecore applications as internal apps only needs to communicate with Solr and Solr should not be publicly accessible. In case, your team needs to troubleshoot Solr queries via Solr Admin Panel then it should be done using following ways
    • Your Solr VMs should be secured by Azure Bastion : https://azure.microsoft.com/en-us/services/azure-bastion/#overview
    • IP Whitelisting should be used and Solr should be behind login
• What If we are using SearchStax? :
  • I haven’t seen any official information yet from SearchStax. But I will update this post. If I come across any update from them. But you can always reach out to them and find out details.
  • https://www.searchstax.com/blog/how-searchstax-is-handling-cve-2021-44228-log4j-vulnerability – SearchStax has already got you covered, please review this post for more information. [Thanks Sameer for the blog post link!]
  • If you’ve followed https://www.searchstax.com/docs/security/ especially IP Filtering then you should be covered. But as I mentioned earlier, good to double check.

How to fix this for my Sitecore Application?

Update #2: Sitecore KB Article (Covers steps for containerized environments): https://support.sitecore.com/kb?id=kb_article_view&sysparm_article=KB1001391

Update #5: If you are using Cloudflare WAF and you need more time to update your systems then you can also use Cloudflare WAF rules to protect your app from this vulnerability: https://blog.cloudflare.com/log4j-cloudflare-logs-mitigation/ (Note: This is a temporary solution)

Manual

https://solr.apache.org/security.html#apache-solr-affected-by-apache-log4j-cve-2021-44228 – Official page has listed a few options to fix this. As you know in the Sitecore world, You can’t upgrade to Solr if it’s not compatible with Sitecore, and in that case, Option#3 or #4 makes sense! (Highlighted in yellow below):

But if you have multiple Solr running (Solr + Zookeeper in ensemble mode) then it might be monotonous for you to make this change. In this case, the Automated approach is right for you!

Automated

Alex van Wolferen has already done this for you (Thanks!): https://www.alexvanwolferen.nl/sitecore-solr-fix-log4j-cve-2021-44228/

You can take his code and modify it as per your need: https://github.com/avwolferen/Sitecore.Solr-log4j-mitigation

Hope this post helps you answer a lot of questions you’ve about this vulnerability. If you’ve some new learnings, please drop your learnings in a comment, and will update the post!

I will keep this post updated as I learn more about it.

Verion History

1. 13-Dec-2021 1 PM CT : Updated SearchStax Section with blog post link from SearchStax
2. 13-Dec-2021 2.32 PM CT: Added Sitecore KB Article Link
3. 13-Dec-2021 10.27 PM CT : Image Credits added
4. 14-Dec-2021 07.37 AM CT : Update public solr access section as per twitter comments
5. 14-Dec-2021 11.16 AM CT : Added Cloudflare WAF blog link

Invalid Date in Date Math String error with Computed field in Sitecore with Solr

Recently we came across following exception:

Exception: SolrNet.Exceptions.SolrConnectionException
Message: <?xml version=”1.0″ encoding=”UTF-8″?>
<response>
<lst name=”responseHeader”><int name=”status”>400</int><int name=”QTime”>46</int></lst><lst name=”error”><str name=”msg”>Invalid Date in Date Math String:’2017-03-20T13.56.31Z'</str><int name=”code”>400</int></lst>
</response>

Quick search resulted in to this post : https://briancaos.wordpress.com/2017/10/25/sitecore-solr-error-invalid-date-in-date-math-string/

But mentioned Sitecore fix has been already applied for Sitecore 9.3

After spending couple of hours troubleshooting this we noticed that our Computed field’s name (mycustomfield) was same as Sitecore Field name(MyCustomField):

<field fieldName="mycustomfield" returnType="datetime">SCBasics.Feature.PageContent.Index.MyCustomField,SCBasics.Feature.PageContent</field>

To fix this issue, You can either update your field name in Sitecore or change your computed field name in the Config file (We took the route of updating the computed field name).

It might seem like a small issue. But we had to apply various checks before we learned this. As error message was not super clear. Hope this helps someone save a couple of hours from their day. Which can be given to their loved ones!

Happy Sitecore Development! 🙂

SolrCloud with Sitecore troubleshooting

Challenge:

Recently we launched highly scalable Sitecore Projects for few of our Enterprise level Sitecore clients. And during that faced some challenges with SolrCloud. Which I again faced in last week and thought to share it pen down. So, I can share with you and I can also refer, When I face it again.

Solution:

So, If you are new to Sitecore or haven’t worked with Solr or SolrCloud and need some basic knowledge. I will share few great articles already posted by our community members:

• https://pratiksatikunvar.wordpress.com/2019/05/24/solr-solrcloud-and-solr-as-a-service/
• https://sitecore-community.github.io/docs/search/solr/Install-and-configure-SolrCloud/
• https://doc.sitecore.com/developers/90/platform-administration-and-architecture/en/walkthrough–setting-up-solrcloud.html

Above articles are good to get basic knowledge, Once you have that and If you are keen to set it up in ensemble (Dictionary : a group of musicians, actors, or dancers who perform together.)mode. Ensemble mode means — Imagine your Sitecore application is your concert happening on the web and to make sure you have great concert. Three Solr nodes are performing together! For Setup there are few links. But I found following links super helpful:

• https://github.com/ivanbuzyka/Sitecore.SolrCloudConfiguration/wiki – THE BEST SOURCE
• https://sitecorenuke.wordpress.com/2017/08/31/solr-cloud-setup-with-zookeeper-for-sitecore/
• https://varunvns.wordpress.com/2019/05/26/solr-cloud-for-sitecore-step-1-install-and-configure-zookeeper/

Once your setup is done, It will look like this:

Solr cloud setup – Image courtesy : https://github.com/ivanbuzyka/Sitecore.SolrCloudConfiguration/wiki

So far so good. And lot of folks can get setup done. But as soon as you connect with Sitecore. You might come across some challenges in your day to day life working with SolrCloud. If they come they are hard to troubleshoot as they are in JAVA Tech Stack, Which will be tough for any Sitecore Developer – As they are different Tech Stack. Also, at times those errors are hard to decrypt and at times hard to locate as well. Here are my learnings which might help you, When you face challenges with SolrCloud and Sitecore.

Scenario#1

After few days of connecting to SolrCloud and couple of weeks before go-live. Our Sitecore log files were busy with follwoing errors:

2019-03-26T15:58:40 PID[4988] Error ManagedPoolThread #5 15:58:40 ERROR Exception
Exception: System.Reflection.TargetInvocationException
Message: Exception has been thrown by the target of an invocation.
Source: mscorlib
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at Sitecore.Reflection.ReflectionUtil.InvokeMethod(MethodInfo method, Object[] parameters, Object obj)
at Sitecore.Jobs.JobRunner.RunMethod(JobArgs args)
at (Object , Object )
at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain, Boolean failIfNotExists)
at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain)
at Sitecore.Jobs.Job.DoExecute()
at Sitecore.Jobs.Job.ThreadEntry(Object state)

Nested Exception

Exception: SolrNet.Exceptions.SolrConnectionException
Message: The underlying connection was closed: The connection was closed unexpectedly.
Source: SolrNet
at SolrNet.Impl.SolrConnection.PostStream(String relativeUrl, String contentType, Stream content, IEnumerable1 parameters) at SolrNet.Impl.SolrConnection.Post(String relativeUrl, String s) at SolrNet.Impl.SolrBasicServer1.SendAndParseHeader(ISolrCommand cmd)
at Sitecore.ContentSearch.SolrProvider.SolrBatchUpdateContext.AddRange(IEnumerable1 group, Int32 groupSize) at Sitecore.ContentSearch.SolrProvider.SolrBatchUpdateContext.Commit() at Sitecore.ContentSearch.AbstractSearchIndex.PerformUpdate(IEnumerable1 indexableInfo, IndexingOptions indexingOptions)

Nested Exception

Exception: System.Net.WebException
Message: The underlying connection was closed: The connection was closed unexpectedly.
Source: System
at System.Net.HttpWebRequest.GetResponse()
at HttpWebAdapters.Adapters.HttpWebRequestAdapter.GetResponse()
at SolrNet.Impl.SolrConnection.GetResponse(IHttpWebRequest request)

This was happening on each CRUD operation — Which is logical. As you know Sitecore triggers partial index rebuild on each CRUD operation. We invested huge amount of hours on this and then finally, we learnt that Solr and ZK servers were not running on HTTPS. How to find and fix that? Here we go:

1. You go to each of your ZK files and make sure your ZK Configurations are correct – Especially hostnames, and respective entries should be there in host files and make sure ZK is registered to run on HTTPS (See Step #7 – zkcli -zkhost “10.100.1.10:2181,10.100.1.11:2181,10.100.1.12:2181” -cmd clusterprop -name urlScheme -val https) : https://github.com/ivanbuzyka/Sitecore.SolrCloudConfiguration/wiki/3.-Install-Zookeeper-as-a-service
2. Make sure your Solr is configured to run on HTTPS — Two things to check Solr hostname (SOLR_HOST – They should be different for each Solr) and Certificate entries.
3. Make sure ports and firewalls are open : https://github.com/ivanbuzyka/Sitecore.SolrCloudConfiguration/wiki/Ports-and-firewalls-etc
4. Few troubleshooting tips : it is hard to pin point SolrCloud challenges. Following tips might help you in that.
   1. Run each SolrCloud and ZK Instance separately — Which means don’t do ensemble mode. Just have one Solr and ZK entry and run it. Start ZK and Solr using Command prompt. So, you can see what’s going on behind the scenes – Few commands:
      1. solr start -cloud -p 8983 -z “10.100.1.11:2181” -f – To run Single Solr with ZK. Make sure your ZK has only single entry and is up and running. Open individual Solr URL and ensure it works in isolated mode.Do this for each instance and if all good, then and only then run it in isolated mode. When you run in ensemble mode. Look for Leader and follower values in command prompt : http://amrutbudihal.blogspot.com/2012/07/zookeeper-leader-election-and-group.htmlCommon problem – Which I have seen is – ZK is not registered to run in HTTPS mode — (Which happens from this command : zkcli -zkhost “10.100.1.10:2181,10.100.1.11:2181,10.100.1.12:2181” -cmd clusterprop -name urlScheme -val https). How to check ZK is running in HTTPS mode or not?: (Good reference : https://www.searchstax.com/docs/hc/zkcli-zookeeper-searchstax.html)
         1. Execute “zkcli.bat -z “ZKIP:ZKPORT” -cmd list from one of the server – As shown below and look for “urlScheme” value. if your ZK is on HTTPS. Then you should see HTTPS here. Do it for all other ZK. This command also ensures your network and firewall is allowing communication between each ZK nodes.†

Scenario #2

After fixing above error, noticed following error in log file:

2019-03-30T15:16:37 PID[7596] Error 18068 15:16:37 ERROR The remote name could not be resolved: ‘uat1-solrcloud.scbasics.com’
Exception: SolrNet.Exceptions.SolrConnectionException
Message: The remote name could not be resolved: ‘uat1-solrcloud.scbasics.com’
Source: SolrNet
at SolrNet.Impl.SolrConnection.Get(String relativeUrl, IEnumerable1 parameters) at SolrNet.Impl.SolrCoreAdmin.Status(String coreName) at Sitecore.ContentSearch.SolrNetExtension.Cloud.DefaultReplicaStatusProvider.GetReplicaStatus(Replica replica) at Sitecore.ContentSearch.SolrProvider.SolrCloudIndexSummary.GetReplicaIndex(Replica replica) at System.Linq.Enumerable.<>c__DisplayClass7_03.b__0(TSource x)
at System.Linq.Enumerable.WhereSelectEnumerableIterator2.MoveNext() at System.Linq.Enumerable.Sum(IEnumerable1 source)
at Sitecore.ContentSearch.SolrProvider.SolrCloudIndexSummary.get_NumberOfDocuments()
at Sitecore.ContentSearch.Client.Forms.IndexingManagerWizard.BuildIndexCheckbox(String name, String header, ListString selected, ListString indexMap)

Nested Exception

Exception: System.Net.WebException
Message: The remote name could not be resolved: ‘uat1-solrcloud.scbasics.com’

Luckily this one was easy, This is known issue and don’t have any functional impact, It will not show you document count etc. Information in Sitecore indexing manager : https://sitecore.stackexchange.com/questions/8263/object-reference-error-in-sitecore-contentsearch-solrprovider-solrindexsummary-g

In Summary, As all other things ZK and SolrCloud are not rocket science. You just need to know right tools, right commands, right mindset and above all great patience to troubleshoot this. Hope this post helps you go home on time!

Thanks to my colleague Abhishek from whom I learnt few of these things and Thanks to Google as well!