Skip to content
June 23, 2019 / kiranpatils

SolrCloud with Sitecore troubleshooting

Challenge:

Recently we launched highly scalable Sitecore Projects for few of our Enterprise level Sitecore clients. And during that faced some challenges with SolrCloud. Which I again faced in last week and thought to share it pen down. So, I can share with you and I can also refer, When I face it again.

Solution:

So, If you are new to Sitecore or haven’t worked with Solr or SolrCloud and need some basic knowledge. I will share few great articles already posted by our community members:

Above articles are good to get basic knowledge, Once you have that and If you are keen to set it up in ensemble (Dictionary : a group of musicians, actors, or dancers who perform together.)mode. Ensemble mode means — Imagine your Sitecore application is your concert happening on the web and to make sure you have great concert. Three Solr nodes are performing together! For Setup there are few links. But I found following links super helpful:

Once your setup is done, It will look like this:

sitecore solrcloud
Solr cloud setup – Image courtesy : https://github.com/ivanbuzyka/Sitecore.SolrCloudConfiguration/wiki

So far so good. And lot of folks can get setup done. But as soon as you connect with Sitecore. You might come across some challenges in your day to day life working with SolrCloud. If they come they are hard to troubleshoot as they are in JAVA Tech Stack, Which will be tough for any Sitecore Developer – As they are different Tech Stack. Also, at times those errors are hard to decrypt and at times hard to locate as well. Here are my learnings which might help you, When you face challenges with SolrCloud and Sitecore.

Scenario#1

After few days of connecting to SolrCloud and couple of weeks before go-live. Our Sitecore log files were busy with follwoing errors:

2019-03-26T15:58:40 PID[4988] Error ManagedPoolThread #5 15:58:40 ERROR Exception
Exception: System.Reflection.TargetInvocationException
Message: Exception has been thrown by the target of an invocation.
Source: mscorlib
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at Sitecore.Reflection.ReflectionUtil.InvokeMethod(MethodInfo method, Object[] parameters, Object obj)
at Sitecore.Jobs.JobRunner.RunMethod(JobArgs args)
at (Object , Object )
at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain, Boolean failIfNotExists)
at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain)
at Sitecore.Jobs.Job.DoExecute()
at Sitecore.Jobs.Job.ThreadEntry(Object state)

Nested Exception

Exception: SolrNet.Exceptions.SolrConnectionException
Message: The underlying connection was closed: The connection was closed unexpectedly.
Source: SolrNet
at SolrNet.Impl.SolrConnection.PostStream(String relativeUrl, String contentType, Stream content, IEnumerable1 parameters) at SolrNet.Impl.SolrConnection.Post(String relativeUrl, String s) at SolrNet.Impl.SolrBasicServer1.SendAndParseHeader(ISolrCommand cmd)
at Sitecore.ContentSearch.SolrProvider.SolrBatchUpdateContext.AddRange(IEnumerable1 group, Int32 groupSize) at Sitecore.ContentSearch.SolrProvider.SolrBatchUpdateContext.Commit() at Sitecore.ContentSearch.AbstractSearchIndex.PerformUpdate(IEnumerable1 indexableInfo, IndexingOptions indexingOptions)

Nested Exception

Exception: System.Net.WebException
Message: The underlying connection was closed: The connection was closed unexpectedly.
Source: System
at System.Net.HttpWebRequest.GetResponse()
at HttpWebAdapters.Adapters.HttpWebRequestAdapter.GetResponse()
at SolrNet.Impl.SolrConnection.GetResponse(IHttpWebRequest request)

This was happening on each CRUD operation — Which is logical. As you know Sitecore triggers partial index rebuild on each CRUD operation. We invested huge amount of hours on this and then finally, we learnt that Solr and ZK servers were not running on HTTPS. How to find and fix that? Here we go:

  1. You go to each of your ZK files and make sure your ZK Configurations are correct – Especially hostnames, and respective entries should be there in host files and make sure ZK is registered to run on HTTPS (See Step #7 – zkcli -zkhost “10.100.1.10:2181,10.100.1.11:2181,10.100.1.12:2181” -cmd clusterprop -name urlScheme -val https) : https://github.com/ivanbuzyka/Sitecore.SolrCloudConfiguration/wiki/3.-Install-Zookeeper-as-a-service
  2. Make sure your Solr is configured to run on HTTPS — Two things to check Solr hostname (SOLR_HOST – They should be different for each Solr) and Certificate entries.
  3. Make sure ports and firewalls are open : https://github.com/ivanbuzyka/Sitecore.SolrCloudConfiguration/wiki/Ports-and-firewalls-etc
  4. Few troubleshooting tips : it is hard to pin point SolrCloud challenges. Following tips might help you in that.
    1. Run each SolrCloud and ZK Instance separately — Which means don’t do ensemble mode. Just have one Solr and ZK entry and run it. Start ZK and Solr using Command prompt. So, you can see what’s going on behind the scenes – Few commands:
      1. solr start -cloud -p 8983 -z “10.100.1.11:2181” -f – To run Single Solr with ZK. Make sure your ZK has only single entry and is up and running. Open individual Solr URL and ensure it works in isolated mode.Do this for each instance and if all good, then and only then run it in isolated mode. When you run in ensemble mode. Look for Leader and follower values in command prompt : http://amrutbudihal.blogspot.com/2012/07/zookeeper-leader-election-and-group.htmlCommon problem – Which I have seen is – ZK is not registered to run in HTTPS mode — (Which happens from this command : zkcli -zkhost “10.100.1.10:2181,10.100.1.11:2181,10.100.1.12:2181” -cmd clusterprop -name urlScheme -val https). How to check ZK is running in HTTPS mode or not?: (Good reference : https://www.searchstax.com/docs/hc/zkcli-zookeeper-searchstax.html)
        1. Execute “zkcli.bat -z “ZKIP:ZKPORT” -cmd list from one of the server – As shown below and look for “urlScheme” value. if your ZK is on HTTPS. Then you should see HTTPS here. Do it for all other ZK. This command also ensures your network and firewall is allowing communication between each ZK nodes.†

Scenario #2

After fixing above error, noticed following error in log file:

2019-03-30T15:16:37 PID[7596] Error 18068 15:16:37 ERROR The remote name could not be resolved: ‘uat1-solrcloud.scbasics.com’
Exception: SolrNet.Exceptions.SolrConnectionException
Message: The remote name could not be resolved: ‘uat1-solrcloud.scbasics.com’
Source: SolrNet
at SolrNet.Impl.SolrConnection.Get(String relativeUrl, IEnumerable1 parameters) at SolrNet.Impl.SolrCoreAdmin.Status(String coreName) at Sitecore.ContentSearch.SolrNetExtension.Cloud.DefaultReplicaStatusProvider.GetReplicaStatus(Replica replica) at Sitecore.ContentSearch.SolrProvider.SolrCloudIndexSummary.GetReplicaIndex(Replica replica) at System.Linq.Enumerable.<>c__DisplayClass7_03.b__0(TSource x)
at System.Linq.Enumerable.WhereSelectEnumerableIterator2.MoveNext() at System.Linq.Enumerable.Sum(IEnumerable1 source)
at Sitecore.ContentSearch.SolrProvider.SolrCloudIndexSummary.get_NumberOfDocuments()
at Sitecore.ContentSearch.Client.Forms.IndexingManagerWizard.BuildIndexCheckbox(String name, String header, ListString selected, ListString indexMap)

Nested Exception

Exception: System.Net.WebException
Message: The remote name could not be resolved: ‘uat1-solrcloud.scbasics.com’

Luckily this one was easy, This is known issue and don’t have any functional impact, It will not show you document count etc. Information in Sitecore indexing manager : https://sitecore.stackexchange.com/questions/8263/object-reference-error-in-sitecore-contentsearch-solrprovider-solrindexsummary-g

In Summary, As all other things ZK and SolrCloud are not rocket science. You just need to know right tools, right commands, right mindset and above all great patience to troubleshoot this. Hope this post helps you go home on time!

Thanks to my colleague Abhishek from whom I learnt few of these things and Thanks to Google as well!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: