DIgest Mismatch ERROR – github ENTERPRISE sso – azure AD

When setting up Github Enterprise Cloud plan single sign on – SSO for Microsoft Azure active directory, you may encounter the ‘FAILED: digest mismatch‘ error when test/saving your SSO configuration.

SSO configuration on github:-

When you hit test SAML configuration before saving button on Github, it will test the SSO configuration to your Azure AD , the Idp provider. You need to input your Sign on URL, unique URL, Certificate (base64) given that you already completed the configuration on Azure AD. You can find the complete Azure AD SSO configuration guide for your specific Github product here from Microsoft, the values are different for Github Organization and Github Enterprises.

You will receive the SAML Signing certificate from Azure by default SHA256 signing algorithm, once saved you can download the Base64 format for input into Github SSO config.

Back to Github SSO config page, once you paste the SHA256 certificate in Base64, you will see the confirmation “Your SAML provider is using the RSA-SHA256 Signature Method and the SHA256 Digest Method”

From there if you try to test/save the configuration it will attempt to test the configuration and after a while returns the error ‘FAILED: digest mismatch

Googling around did not prove very useful however I did find this old github blog – SHA256 support for fingerprints. Debugging my SAML response I noticed the two lines below referencing ‘Digest’ Which tells me that my Digest Method algorithm is that of SHA256. I also read that Github uses OmniAuth

Further digging led me to compare my above SAML response digestMethod algorithm with the github omniauth-saml strategies library found here

This tells me my signing algorithm is SHA256 but Github expects SHA1.

Changing my certificate signing algorithm to SHA1, and providing the SHA1 fingerprint to Github SSO configuration solved the ‘Digest mismatch’ error and my SSO configuration was successful. However this was all in a test environment, but I would not recommend to generate SHA1 certificate in production for use, rather you can generate a SHA256 and then calculate the SHA1 fingerprint using openSSL and use that SHA1 fingerprint on Github SSO config.

This other blog from Gitlab.com SAML related issues says:-

Please note, we currently support SHA-2 as the certificate signature algorithm and we recommend for the certificate to be generated using SHA-2(256) or higher. It is only the certificate fingerprint that requires a SHA-1. The risk on using SHA-1 for the fingerprint is not as great as having the certificate generated using SHA-1.

A little understanding of certificate encryption vs certificate fingerprint:-

The algorithm of the fingerprint/thumbprint is unrelated to the encryption algorithm of the certificate. The fingerprint/thumbprint is a identifier used by some server platforms to locate the certificate in a certificate store.

In conclusion, use a tool such as openSSL, calculate the SHA1 fingerprint from your SHA256 certificate, and use that to complete your Github SSO config. Sample command will be:-

openssl x509 -noout -fingerprint -sha1 -inform pem -in "MyCertificate-domain.com.cer"

Github Enterprise SSO Configuration successful:-

Good luck!

Microsoft.Exchange.MailboxAssistants.Assistants.ELC.ElcEwsException.ArchiveExchangeWebServiceNotAvailable – error after offboarding user mailbox from exchange online to onpremis

On SCOM monitoring server below alert was raised for some user mailboxes on Exchange 2016 on-premise server. In this case, Exchange 2016 is hybrid configured, and some users offboarded from exchange online back to exchange on-premise.

Full alert description from SCOM:-

Probe: {Compliance/ELCComponent_LastSuccessTooLongAgo}
Mailbox guid: {d5xxe174-xxxxxx-b5d8-324xxxx5d93}
In Org: {}
Is archive: {IsArchiveMailbox = False}
With stack trace: {The difference: 7.89530321780208 days between today: 11/26/2021 8:41:11 AM and the date of last successful ELC run: 11/18/2021 11:11:57 AM for mailbox: d5xxe174-xxxxxx-b5d8-324xxxx5d93 is above the threshold: 7; Exception message: Microsoft.Exchange.MailboxAssistants.Assistants.ELC.ElcEwsException.ArchiveExchangeWebServiceNotAvailable.}
Get last ELC exception from Export-MailboxDiagnosticLogs -Component MRM then statistics with -ExtendedProperties and look at all ELC properties, specifically the value of ELCLastSuccessTimestamp. If mailbox is Archive use -Archive.

In my case, a few user mailboxes were offboarded from exchange online to on-premise Exchange 2016 with exchange online archiving still active on those mailboxes and the on-premise exchange servers are in DMZ, not internet facing.

If you run the command suggested by SCOM in the alert details:-

Export-MailboxDiagnosticLogs -Identity Mailboxname -ComponentName MRM

You will see the full error, and look closely at the exception warning:-

ELC EWS failed with error type: ‘ArchiveExchangeWebServiceNotAvailable’. Message: Archive EWS url is unknown.

Next, you can verify the on-premise user mailbox who has exchange online archiving enabled on Exchange online using these commands

Get-Mailbox <on-premises user mailbox> | FL *archive*

Full list of commands:- https://docs.microsoft.com/en-us/exchange/hybrid-deployment/create-cloud-based-archive#step-2-verify-that-the-cloud-based-archive-mailbox-is-created

Take note of the populated mailbox ArchiveGuid and ArchiveName, ArchiveStatus properties

Microsoft says the only supported archive split scenario is a primary mailbox on-premises and an archive mailbox in Exchange Online. (https://docs.microsoft.com/en-us/office365/troubleshoot/archive-mailboxes/archive-mailbox-issues)

For me, we did not need to have exchange online archiving enabled for the on-premise users, we followed the steps Scenerio 5 with provided script to cleanup

Solution(s)

If you have the user primary mailbox seated on-premise and its archive mailbox seated in exchange online and your on-premise exchange servers are on DMZ i.e not internet facing, then:-

  • You will face this error on your dmz exchange if you don’t use a web proxy or proxy firewall is not configured to allow exchange on-premise server to connect to exchange online on port 443, so talk to your network admin, make sure your dmz exchange can reach exchange online.

Lastly, if non of the above scenarios applies to your case, Microsoft provides resolution to several other scenarios, in the link of this documentation ( https://docs.microsoft.com/en-us/office365/troubleshoot/archive-mailboxes/archive-mailbox-issues):-

  • Scenario 1 – Onboarding: You move your on-premises Microsoft Exchange Server mailboxes to Exchange Online.
  • Scenario 2 – Onboarding: Your archive mailbox exists in Exchange Online, and you move your primary mailbox from your on-premises Exchange Server environment to Exchange Online.
  • Scenario 3 – Offboarding: You enable an archive mailbox and then migrate both your primary and archive mailboxes from Exchange Online to your on-premises Exchange Server environment. A similar scenario occurs when your primary mailbox is already on-premises and you decide to offboard your archive mailbox from Exchange Online to your on-premises Exchange Server environment.
  • Scenario 4 – Offboarding: Your primary mailbox does not have an archive mailbox enabled, and you move your primary mailbox from Exchange Online to your on-premises Exchange Server environment.
  • Scenario 5 – Offboarding: Your primary mailbox exists in your on-premises Exchange Server environment, and your archive mailbox exists in Exchange Online. This scenario may occur when you take one of the following actions:
    • You offboard your primary mailbox. However, you leave your archive mailbox in Exchange Online.
    • Both primary and archive mailboxes are located in your on-premises Exchange Server environment. However, you onboard only your archive mailbox.

MailNonUniversalGroup – Distribution List

Happy New year 2020. I wish you more progress this year from where you left off last year. Now let’s dive right to our topic on MailNonUniversalGroup distribution list. So i had a case where users where complaining of not receiving emails sent to a distribution list. After several minutes tracing the messages, it came to a halt with the following interesting details from the trace log:-

——–

Source : ROUTING
EventId : DROP

RecipientStatus : {[{LED=250 2.1.5 RESOLVER.GRP.Expanded; distribution list expanded};{MSG=};{FQDN=};{IP=};{LRT=}]}

——–

Some further digging and here is what i found for the distribution group that was not receiving emails:-

Get-DistributionGroup "nameofmydistributionlist" | fl recipienttypedetails

RecipientTypeDetails : MailNonUniversalGroup

Result shows my distribution group was of type: MailNonUniversalGroup

Further checking on when this group was first created in AD, shows far back as of 2012, and this must have been when it was migrated from previous legacy exchange 2003 to exchange 2010 but was not upgraded to a universal group at that time.

Microsoft recommends to convert all legacy exchange distribution groups to “universal” groups for use on Exchange 2010/2013/2016/2019 especially if you want to have all the features of distribution groups included.

In my case the distribution group members where on office365 and they needed to receive external email sent to the email address of the distribution list.

To solve, change the distribution group from MailNonUniversalGroup to Universal, I did this via exchange powershell:-

Get-DistributionGroup "nameofmydistributionlist" | Set-Group -Universal

wait a few minutes for replication and check again using:-

Get-DistributionGroup "nameofmydistributionlist" | fl recipienttypedetails

RecipientTypeDetails : MailUniversalDistributionGroup

It has now been converted to MailUniversalDistributionGroup

Now we can receive emails sent to the distribution list without any issues.

Tip: To do this in bulk for all your distribution groups that was just migrated over from legacy exchange versions 2003/2007, you can use the following command which will change all mailnonuniversalgroups to universal:-

Get-DistributionGroup -ResultSize unlimited -RecipientTypeDetails mailnonuniversalgroup | Set-Group universal

and then to apply the upgrade :

Get-DistributionGroup -ResultSize unlimited | Set-DistributionGroup -ForceUpgrade

An error occurred trying to connect the WSUS server…

I had this error today, WSUS is installed on Windows server 2016 and used for managing Windows Server updates to several of my Exchange servers. This error appeared today and below is what I did to fix it.

When you click on “reset server node”, nothing happens , the error just reappears. Also you will find event id 7032 windows server update services in the event viewer of your server.

Solution:

  1. Open up IIS – internet information services manager on the affected node

2. locate WSUSpool, by clicking on “Application Pools” after expanding the connections tree from the IIS console

You will see the WsusPool status is “stopped”

3. Start the service

4. Now go back to your WSUS console and click on “reset server node” and after a while it should work and all servers can be found again and you can proceed with your windows updates.

PowerShell Script to easily copy a file and rename in new destination

Sharing with you a simple Powershell script I wrote and use to list all the files names and location information for a given folder and then easily copy that TXT file between other folders in the same server.

E.g:- list all files inside myFolderA and export this info to myFile.txt

This will not copy files between remote servers. That requires a more complex method using Invoke-command and remote powershell authentication

To list content of folder


Get-ChildItem K:\myfolderA\*.*| Set-Content K:\myfolderA\myfile.txt

To copy into myFolderB

$From = "K:\myfolderA\myfile.txt"
Copy-Item -Path $From -Destination "J:\myfolderB\newMyfile.txt" -Recurse -Force

$From = This will be the full path of the file you want to copy

Destination – This will be the destination drive and folder for your copied file

Mail.Que database too large

In this troubleshooting you will learn how to safely delete and recreate the Exchange server transport queue database file “mail.que” and get tips to determine the root cause of your growing mail.que. On deletion of the mail.que file, Exchange will auto create a new mail.que file once you restart the Microsoft Exchange transport service. This applies to Exchange 2010,2013,2016,2019

Below solution will help you avoid messaging downtime, if your mail.que file is getting too large or consuming a lot of space on your disk drive at a critical stage which can cause major impact to mail flow or even Exchange auto shutting down its services, however, be sure to later properly investigate the root cause of the mail.que file growth as it can reoccur. Some known causes of mail.que file growth can be due to organization wide Exchange transport configurations such as the maxdumpstertime(exchange 2010), safetynetholdtime, pipeline tracing value, etc – For me, the safetynetholdtime value on Exchange 2016 was set to 7 days, which resulted in the growth of mail.que as it holds copies of successfully submitted messages for 7 days, another thing was that my day to day mail flow to my Exchange infra had increase from what it used to be several months ago , so i decided to schedule maintenance and expand the disk space where my mail.que resides from 250GB to 500GB , and ever since i no longer have mail.queue space issues. It might also be good to go back to your Exchange Server Role Requirements Calculator to help you determine where you are lacking in terms of disk space requirements, number of inbound messages to your infra etc. and from there you can make adjustments.

By default  your mail.que file location should be at :-  %ExchangeInstallPath%TransportRoles\data\Queue

screenshot-20190116165503-667x215

First, using Exchange powershell we can check the existing size in GB of mail.que, so open up your EMS on the affected Exchange server and run the following:-

Get-ChildItem "D:\Exchange Server\TransportRoles\data\Queue\mail.que" |select name,@{Label="size";Expression={"{0:N0}" -F ($_.Length/1GB)}}

screenshot-20190116161259-906x110

To Solve:-

  1. Put your Exchange server in maintenance mode, if you have SCOM etc, or schedule out-of- office hour maintenance before you proceed to perform these actions.
  2. Suspend Microsoft Exchange Transport service, (NOT STOP). This will drain and allow the current messages in the queue to be processed before it stops accepting new messages to the queue. To do this, on EMS run:- Suspend-service -name "Microsoft Exchange Transport"
  3. Run :- Get-queue  – to check and ensure messages in queues are empty (0).
  4.  Do not worry about shadow redundancy queues, these are fine if it has queues.
  5. Next, stop the MS Exchange transport service, :- Stop-service -name "Microsoft Exchange Transport". Once it has stopped give an extra 5 minutes for everything to settle in.
  6. Open the mail.que location ( %ExchangeInstallPath%TransportRoles\data\Queue) , select all files inside the folder and delete it, optionally you can move it to an external drive with enough space, you can rename it to something like mail.que.old as a backup.
  7. Now, After you have completed above steps. on Exchange management shell enter:- Start-service -name "Microsoft Exchange Transport"
  8. You will see a new mail.que file is auto created by Exchange, and your drive space released and back to normal.
  9. Run on EMS, Get-queue  – to check and monitor and ensure mail flow is back to normal. Above actions will save you from exchange running out of space and shutting down services automatically, and also gives you more time to investigate further on the root-cause of the growth, check event logs, google, technet articles, for more troubleshooting. Good luck.

Tip: use a tool such as Treesize free to get a detailed view of files and the size in your drive. It can come in handy when you want to check the size of files in your exchange server data path.

P.S: Watch out for my next article where I will show you how to change the default directory for your mail.que database, for me I prefer to put it on D:\ or another drive, which is separate from Exchange installation path on C:\ and OS. This is a very good recommendation because you can focus on troubleshooting and increasing the disk space for the mail.que on D:\ without touching your System/Exchange on C:\ drive, also allowing it to be stored on a separate drive from C:\ allows it to make use of the resources on that drive alone.

Add SMTP Server IP to SendConnector – Exchange 2010

First get the existing information of your send connector and any smarthosts IP already configured.

Open exchange management shell and run:-

Get-SendConnector NAMEOFSENDCONNECTOR| fl identity, smarthosts, sourcetransportservers

Result:-

Identity: MYSENDCONNECTOR

SmartHosts : {[192.168.3.4]}

SourceTransportServers : {BB0-MOB4-HUB01, BB0-MOB2-HUB01}


From above result we can see our old smarthost IP that we want to change with the new one and the source transport servers (exchange hub transport servers) which this send connector applies to.

Now set the new IP via exchange management power shell ( this will replace the existing one if it already exists ):-

Set-SendConnector NAMEOFSENDCONNECTOR -SmartHosts “192.168.3.5” -Confirm

Note:-
<smarthosts> parameter
This parameter takes one or more FQDNs, such as server.contoso.com, or one or more IP addresses, separated by commas. If you enter an IP address, you must enter the IP address as a literal as follows, for example: 10.10.1.1. The smart host identity can be the FQDN of a smart host server, a mail exchange (MX) record, or an address (A) record. If you configure an FQDN as the smart host identity, the source server for the Send connector must be able to use DNS name resolution to locate the smart host server.  (https://msdn.microsoft.com/en-us/subscriptions/aa998294(v=exchg.80).aspx)

To do this via Exchange Management Console in Exchange 2010:-   Click on Hub Transport -> Send Connectors -> double click on your send connector -> Network -> under route mail through the following send connector, edit to add the new IP and remove the old one if needed. Ok to accept changes, check that the new IP is added. Test and do a telnet to your new smarthost IP.

Screenshot-20180511114820-439x475