I am not quite sure why this happened but this morning I was greeted with several angry users because the primary Exchange server ran out of virtual memory and stopped the information store and SMTP transport. It was easily resolved and I’m in the process of migrating off that server but I was getting weird bounce backs when I tested using our Gmail account.
The original message was received at Wed, 15 Apr 2009 12:37:14 -0700
from mail-qy0-f122.google.com [209.85.221.122]
----- The following addresses had permanent fatal errors -----
(reason: 550 5.7.1 Requested action not taken: message refused)
----- Transcript of session follows -----
... while talking to [192.168.7.106]:
>>> DATA
<<< 550 5.7.1 Requested action not taken: message refused
554 5.0.0 Service unavailable
Final-Recipient: RFC822; vb@domain.com
Action: failed
Status: 5.7.1
Remote-MTA: DNS; [192.168.7.106]
Diagnostic-Code: SMTP; 550 5.7.1 Requested action not taken: message refused
Last-Attempt-Date: Wed, 15 Apr 2009 12:37:19 -0700
That's weird, I thought to myself. I thought perhaps Untangle may have had a hand in this mess. I run it on our backup server as an inline scanning appliance (Untangle for Windows). I'd kicked off a backup (seeing as the sky was falling) so I was waiting for it to finish. I noticed that although I had been able to connect to the console earlier on, I couldn't anymore so I stopped the services.
Problem solved for now I thought. External e-mail was starting to arrive. One of the users said that they tried sending a message to themselves using their outside e-mail account and it failed.
Your message did not reach some or all of the intended recipients.
Subject: Test
Sent: 4/15/2009 1:40 PM
The following recipient(s) cannot be reached:
User on 4/15/2009 1:40 PM
You do not have permission to send to this recipient. For assistance, contact your system administrator.
< server.domain.local #5.7.1 SMTP; 550 5.7.1 Requested action not taken: message refused>
I thought that was pretty weird. Our mail infrastructure is set up as follows:
MX -> sendmail -> exchange servers {network traffic scanned by Untangle in a VM}
It's strange to get relay denied errors from people who had been able to send to us in the past, especially since we were having other mail-related issues I figured it would be a good test to telnet into our Exchange server and manually send an e-mail to the recipient but strangely I got the same error when trying to send to this recipient. Everything I read on Google was pointing me to a user-level solution (check user hasn't chosen to use server authentication, check under Active Directory Users & Computers on the Exchange General tab that they don't have any delivery restrictions) but after testing sending to a variety of other users, it was apparent that the problem was for all users and not just the one who reported the problem. Turns out that although I had stopped Untangle's services and stopped the VM services, they were still somehow blocking incoming messages. I had tested the failover before Untangle went officially into production and it had worked fine but I guess somehow the SMTP service failure on the Exchange box had affected Untangle and the only solution was to bounce the box. I thought I'd just put this out there in case anyone else ran into the something similar.
Postscript: I set up a backup mail queue and added the MX record for it into DNS, I noticed too that our SPF was way out of date (the old sysadmin wasn't much into housekeeping) so I fixed that too, with no help from the Microsoft Sender ID Framework SPF Record Wizard that I tried to be lazy and use when the lame web host they use pointed me to it. Perhaps it's because I'm using Firefox, stranger things have happened, but pressing the Next button on the third (out of four) page and it does nothing. How useful. I would fire up internet exploder but I've had enough MS-related woes for one day methinks.
