Marc Lognoul's IT Infrastructure Blog

Cloudy with a Chance of On-Prem


SharePoint 2013: AppFabric Caching service crashed. Lease with external store expired

SharePoint 2013

Description of the Problem

You experience poor performances when browsing SharePoint 2013-based sites or when consuming User Profile Service? Take a look at the SharePoint server’s event viewer, they might be full of the error hereunder:

AppFabric Caching service crashed.{Lease with external store expired:
Microsoft.Fabric.Federation.ExternalRingStateStoreException: Lease already expired
at Microsoft.Fabric.Data.ExternalStoreAuthority.UpdateNode(NodeInfo nodeInfo, TimeSpan timeout)
at Microsoft.Fabric.Federation.SiteNode.PerformExternalRingStateStoreOperations(Boolean& canFormRing, Boolean isInsert, Boolean isJoining)}

Similarly to many distributed services and applications, AppFabric (alone or packaged inside SharePoint 2013) heavily depends on perfect time synchronization between servers, therefore, a discrepancy of seconds to a minute may lead to AppFabric service repeated crashes.

Possible Causes

To name but a few possible causes of time discrepancies between Windows hosts:

  • Incorrect Windows Time Configuration on domain Controllers and/or Member Servers
  • Network Connectivity issues between member servers and their authenticating domain controllers
  • External mechanisms interfering with Windows Time such as VMWare, Hyper-V, OS deployment solutions…

Solution (or at least, some guidance)

  1. Fix the external causes preventing correct time sync (Network issues, 3rd party software…). Be particularly careful with virtual machines
  2. Make sure the Domain Controller holding the PDC Emulator role is configured to acquired its time from an authoritative source AND members server are configured to use the Windows domain hierarchy (NOT an authoritative time source). This configuration suits 99% of the implementations.
  3. Once 1 and 2 are fixed, run the command w32tm /resync /rediscover or restart the Windows service Windows Time (w32time). In both case this will force an time sync
  4. Make sure time is also valid on the SQL Server used for SharePoint because some sored procedure requires time accuracy to work properly as welll

Other SharePoint-Related Service Impacted by Windows Time

  • All timer jobs but in particular the one responsible for refreshing configuration. improper time sync may lead to stale timer job cache
  • Same applies to timer jobs responsible for (un)deploying solutions (WSP) in a multi-server farm. Out-of-sync servers may prevent proper WSP handling
  • Incorrect time may also prevent SharePoint Alerts from being sent
  • Customer timer jobs might not be started at the correct time
  • And finally, if you use Kerberos, the “clock skew” issue is one of the most common, preventing pre-authentication

Goodies

Here’s a PowerShell snippet to query every SharePoint server in a farm in order to retrieve their local time and time zone.

$SPServerInFarm = Get-SPServer | Where { $_.Role -eq "Application" }

Foreach ($Server in $SPServerInFarm)

{

    $DateTime = (Get-WmiObject -ComputerName $Server.Name -Query "select LocalDateTime from Win32_OperatingSystem").LocalDateTime

    $DateTimeFormatted = ([wmi]”).ConvertToDateTime($DateTime).tostring("MM/dd/yyyy HH:mm:ss")

    $TimeZoneOffset = (Get-WmiObject -ComputerName $Server.Name -Query "select CurrentTimeZone from Win32_OperatingSystem").CurrentTimeZone.ToString()

   Write-Host $Server.Name $DateTimeFormatted $TimeZoneOffset

}

Note: since servers are note queried exactly at the same time, there might by a small time difference. This is obviously harmless in this case.

Note2: While a difference in the time zone will not influence AppFabric, it is not recommended to operate a SharePoint farm on servers operating across different time zones.

Additional Information’s

Conclusions

SharePoint requires a healthy underlying Windows to run smoothly. Keep your Windows Server and AD in good shape and if you’re not in charge of them, make sure you colleague in charge of them does the job right.

Happy caching!

Marc

Advertisements