Monday, April 30, 2007

What exactly happens when Repair is clicked on Local Area Connection?

Local Area Connection Repair option produces below results and knowing these are very helpful to deal with

Connectivity problems in windows platform, what exactly happens in windows XP when, repair option is selected on the NIC card Properties Dynamic Host Configuration Protocol (DHCP) lease is renewed: ipconfig /renew

The DHCP lease process has already been explained on one of my old blog, the process involves four primary steps

DHCP discovery

DHCP Lease offer

DHCP lease request

DHCP Lease acknowledgment


 

Client uses 0.0.0.0 as its address and 255.255.255.255 for the server's address. DHCP discover message on UDP port 68 and destination UDP port 67.

Address Resolution Protocol (ARP) cache is flushed: arp -d *

ARP (Address Resolution Protocol) Displays and modifies the IP-to-Physical address translation tables used by

Address resolution protocol (ARP).

ARP –a

192.168.1.2

00 -13 -72 -1d-8d -e8

dynamic

192.168.1.17

00- 02- b3 -a2 -3f- 6a

dynamic

224.0.0.22

01- 00 -5e -00- 00 16

static

224.0.0.252

01 -00 -5e- 00- 00- fc

static

239.255.255.250

01- 00 -5e -7f –ff- fa

static


 

Nbtstat –R Reload of the NetBIOS name cache:

Nbtstat –RR NetBIOS name update is sent

Ipconfig /flushdns Domain Name System (DNS) cache is flushed

Ipconfig /RegisterDNS
DNS name registration


 

Oz Ozugurlu

Baseline Counters Monitoring Exchange Server

You don't need fancy tools, to determine what is going on with exchange server. We will use building Windows performance Monitor to do the job.

Click Start, do to RUN, type, perfmon and hit enter. (Assuming, you are doing this with administrator privileges)Now performance monitor is running in front of you. Let's tune up and discover what is going on with our Exchange server. First on the bottom you will see some default counter defined pages per second, Avg, Disk Queue Length, and % Processor Time. Let's go ahead and delete them, highlight one of them and click delete on top of the window, by clicking delete symbol , and Click on + sign to add the counters below When monitoring Exchange, below counters with a baseline and good to remember or keep it a side.

We can use these counters to maintain our exchange server, or to find out what the problem is.

•Database\Log Record stalls/sec

Average should be below 10 per second and maximum values should not be higher than 100 per second (indicates the number of logs records that cannot be written because the buffers are full

Note that Exchange Server 2000 defaults to 84 buffers whilst Exchange Server 2003 defaults to 512).

•Database\Log Threads Waiting

Average should be below 10 (indicates the number of threads waiting to complete an update to the database by writing their data to the log

if too high, the log may be a bottleneck).

•MSExchangeIS\RPC Requests

Should be below 30 at all times (indicates the number of MAPI requests being serviced by the Microsoft Exchange Information Store service

The default maximum is 100).

•MSExchangeIS\RPC Average Latency

Should be below 50ms at all times and should be in the 10

25ms range on a healthy server (averaged over the last 1024 packets and affects how long it takes for a user's view to change in Outlook).

•MSExchangeIS\RPC Operations/sec

Should rise and fall with MSExchangeIS\RPC Requests (indicates how many RPC operations are being requested and actually responded to).

•MSExchangeIS\Virus Scan Queue Length

If this is consistently high considering a hardware upgrade (indicates the number of outstanding requests queued for virus scanning).

•MSExchangeIS Mailbox\Active Client Logons

This is server Specific but should be baseline and monitored (indicates the number of clients which performed any action within the last 10 minutes).

•Paging File\% Usage

Should remain below 50% high values indicate that the paging file size should be increased or more RAM added to the server (indicates the amount of the paging file used).

•Memory\Available Mbytes (MB)


50Mb available at all times (indicates the amount of physical memory immediately available to a process).

•Memory\Pages/sec


Below 1000 at all times (indicates the rate at which pages are written to disk to resolve hard page faults).

•Memory\Pool Nonpaged Bytes


No more than 100Mb (indicates the amount of memory available for kernel objects which must remain in memory and cannot be written to disk).

•Memory\Pool Paged Bytes

No more than 180Mb, unless a backup or restoration is taking place (indicates the amount of memory available for kernel objects which must remain in memory and can be written to disk).

•Physical Disk\Average Disk Read/sec


average below 20ms and maximum below 100ms for the database volume, average below 5ms and maximum below 50ms for the transaction log volume, average below 10ms and maximum below 50ms for the SMTP queue volume (indicates the average time to read data from the disk).

•Physical Disk\Average Disk Write/sec


average below 20ms and maximum below 100ms for the database volume, average below 10ms and maximum below 50ms for the transaction log volume, average below 10ms and maximum below 50ms for the SMTP queue volume (indicates the average time to read data from the disk).

Thursday, April 26, 2007

Why do we need FSMO ROLES?

Active directory is multi master replication model. Meaning clients can register their records to any available Active directory domain controller and have access to resources within active directory NTDS.DIT database.

In old days where we had single master replication, Primary DNS server had the write copy of DNS data, meaning Client MUST locate the Primary DNS servers, and register their resources so that they can locate all the other resources within active directory infrastructure. The problem with single master model was the single point of failure, if the primary DNS server was not reachable for any reason client could not get register its records to any other domain controller/DNS servers. We have now MultiMate replication model meaning client can register its records to any available Authentication server / DNS servers and can get to the NTDS.DIT database. This is one of the great improvements in Active directory integrated DSN and multi master replication DNS data is being kept in what we call is ZONE. The primary zone is Forward lookup zone in AD.

Reverse lookup zone is highly recommended in almost any size of network

The purpose of having FSMO roles is being cause by Multi master replication model. In this model there has to be a way of preventing the conflict being happened, such as firing up adsiedit.msc and adding to the same object from different locations, which one would win? The NTDS.DIT DataBase would get confuse, Therefore we needed to have schema master so that regardless where you make the changes within the Domain changes gets okay from Schema Master first than, schema master replicates these changes to all other Domain controllers. This is the primary purpose why Microsoft comes up with FSMO roles (Operations Masters)

Knowing these ROLEs and understanding them is Curtail for any Exchange or AD Administrators.

FSMO Roles

In a forest, there are at least five FSMO roles that are assigned to one or more domain controllers. The five FSMO roles are

Schema Master:

The schema master domain controller controls all updates and modifications to the schema. To update the schema of a forest, you must have access to the schema master. There can be only one schema master in the whole forest.

Domain naming master

The domain naming master domain controller controls the addition or removal of domains in the forest. There can be only one domain naming master in the whole forest.

Infrastructure Master:

The infrastructure is responsible for updating references from objects in its domain to objects in other domains. At any one time, there can be only one domain controller acting as the infrastructure master in each domain.

Relative ID (RID) Master:

The RID master is responsible for processing RID pool requests from all domain controllers in a particular domain. At any one time, there can be only one domain controller acting as the RID master in the domain.

PDC Emulator

The PDC emulator is a domain controller that advertises itself as the primary domain controller (PDC) to workstations, member servers, and domain controllers that are running earlier versions of Windows. For example, if the domain contains computers that are not running Microsoft Windows XP Professional or Microsoft Windows 2000 client software, or if it contains Microsoft Windows NT backup domain controllers, the PDC emulator master acts as a Windows NT PDC. It is also the Domain Master Browser, and it handles password discrepancies. At any one time, there can be only one domain controller acting as the PDC emulator master in each domain in the forest


 

How can we see FSMO ROLES?

There are several ways to see FSMO roles the easiest way to see download support tools

Downloads

Go to CMD

:\>netdom query fsmo

Schema owner DC1.smtp25.org

 

Domain role owner VSDC1.smtp25.org

 

PDC role VSDC2.smtp25.org

 

RID pool manager DC1.smtp25.org

 

Infrastructure owner DC1.smtp25.org

 

The command completed successfully.

Symptoms of FSMO Problems

If one or more of your FSMO role holders has problems, bad things can happen. To help you troubleshoot such situations, the table below describes some of the symptoms that can occur when FSMO role holders go missing or don't work properly

Symptom

Possible Role Involved

Reason

Users can't log on.

PDC Emulator

If system clocks become unsynchronized, Kerberos may fail.

Can't change passwords.

PDC Emulator

Password changes need this role holder.

Account lockout not working.

PDC Emulator

Account lockout enforcement needs this role holder.

Can't raise the functional level for a domain.

PDC Emulator

This role holder must be available when the raising the domain functional level.

Can't create new users or groups.

RID Master

RID pool has been depleted.

Problems with universal group memberships.

Infrastructure Master

Cross-domain object references need this role holder.

Can't add or remove a domain.

Domain Naming Master

Changes to the namespace need this role holder.

Can't promote or demote a DC.

Domain Naming Master

Changes to the namespace need this role holder.

Can't modify the schema.

Schema Master

Changes to the schema need this role holder.

Can't raise the functional level for the forest.

Sc

 


 

Some Considerations

The PDC Emulator and RID Master roles should be on the same machine because the PDC Emulator is a large consumer of RIDs.

Tip: Since the PDC Emulator is the role that does the most work by far of any FSMO role, if the machine holding the PDC Emulator role is heavily utilized then move this role and the RID Master role to a different DC, preferable not a global catalog server (GC) since those are often heavily used also.

The Infrastructure Master should not be placed on a GC

Make sure the Infrastructure Master has a GC in the same site as a direct replication partner

It's OK to put the Infrastructure Master on a GC if your forest has only one domain if

It's OK to put the Infrastructure Master on a GC if every DC in your forest has the GC

For simpler management, the Schema Master and Domain Naming Master can be on the same machine, which should also be a GC

Exception: If you've raised your forest functional level to Windows Server 2003, the Domain Naming Master doesn't need to be on a GC, but it should at least be a direct replication partner with a GC in the same site.

Proactively check from time to time to confirm that all FSMO roles are available or write a script to do this automatically.

If any FSMO role holders at a remote site are unavailable, check first to see if your WAN link is do

http://www.windowsdevcenter.com/pub/a/windows/2004/06/15/fsmo.html


 

Best Regards

Oz Ozugurlu

About how long does it take to run ISInteg or ESEutil

Below is some of the good selection of information regarding to ISinteg and ESEutil. The hard ware, CPU and Memory power of course will make the difference in reality.

The repair runs at approximately 4 to 6 gigabytes (GB) per hour

50-GB database requires approximately 8 hours for repair and approximately

8 hours for the ISInteg process, for a total of 16 hours

The defragmentation option makes used storage contiguous, eliminates unused storage, and compacts the database, which reduces the database's size. ESEutil copies database records to a new database. When defragmentation is complete, the original database is deleted or saved to a user-specified location, and the new version is renamed as the original. If the utility encounters a bad record, the utility stops and displays an error message.

Defragmenting a database requires free disk space equal to 110 percent of the size of the database that you want to process. To determine the actual space required, follow these steps

Make sure that the information store service is not running.

At a command prompt, run the following command:

ESEutil /ms "database.edb"

Calculate the free space by multiplying the number of free pages by 4 KB.

Subtract the figure that you obtained in step 3 from the physical size of the database.

The figure that you obtained in step 4 represents the data in the database. Multiply this figure by 110 %. The resulting figure that you obtain is the space that you need to have available to defragment the database.

Divide the figure that you obtained in step 3 by 9 GB per hour. The figure that you obtain is the approximate time that it will take to defragment the database.

Note 9 GB per hour is the speed at which the ESEutil utility runs. This number is only for reference. The exact number depends on your hardware and production environment.


 

    

Full list of ESEutil switches for Windows Exchange

Eseutil /cc Performs a hard recovery after a database restore.

Eseutil /d Performs an offline compaction of a database.

Eseutil /g Verifies the integrity of a database.

Eseutil /k  Verifies the checksums of a database.

Eseutil /m Generates formatted output of various database file types. e.g. /mh

Eseutil /p Repairs a corrupted or damaged database.

Eseutil /r Performs soft recovery to bring a single database into a consistent or clean shutdown state.

Eseutil /y Copies a database, streaming file, or log file.


 

DESCRIPTION: Maintenance utilities for Microsoft(R) Exchange Server databases.

MODES OF OPERATION:

Defragmentation: ESEUTIL /d <database name> [options]

Recovery: ESEUTIL /r <log file base name> [options]

Integrity: ESEUTIL /g <database name> [options]

Checksum: ESEUTIL /k <file name> [options]

Repair: ESEUTIL /p <database name> [options]

File Dump: ESEUTIL /m[mode-modifier] <filename>

Copy File: ESEUTIL /y <source file> [options]

Restore: ESEUTIL /c[mode-modifier] <path name> [options]


 

D=Defragmentation,

R=Recovery,

G=integrity,

K=checksum,

P=repair,

M=file dump,

Y=copy file,

C=restore


 


 

http://support.microsoft.com/kb/192185

Oz Ozugurlu


 


 


 


 

Using RUNAS and Securing Exchange Daily Task

Here in this this article i will write about, one of the most I have needed to work on daily basis, Remote execute program with this tool It is possible to run "CMD" window on the remote server, as long as you have the proper rights and you logged into a domain. Speaking of daily Exchange and AD admin life, I have realized many of the administrators won't work in secure environment, they log into Domain with Domain administrator privileges and they go to internet and perform daily task with that. When it comes to s security, we more complain about windows is not being secure, but I think we need to look at ourselves and use the windows right way so that windows will provide secure environment for work daily routine. I will demonstrate a secure way of working with Windows and getting the entire daily job done without problems

First thing you will need to have two accounts in a Domain, let's say we will create an account named oz

First Account Name

oz

Domain User Mail enabled

Second account

ZZ-oz

Domain administrator, Enterprise administrator No mailbox


Now log into your work station with domain User account, this account is to be logged into system all he times.

We will not log into systems with our ZZ-oz account, we will use RUNAS and get the job done with ZZ account privileges when we need it

After you logged in (remember you are a Domain user now, CANNOT give any damage to anything pretty much, try going to device manager and deleting a device, windows will deny your request.)

Now open a notepad and type


runas /user:archq\zz-oz cmd.exe Change my name into your account name

Click Save, File Name RunAS.bat Save Type as, all files

And save it on your Desktop. Now when you double click on it DOS window will open up and ask you to type your password, ones you successfully type your password ( pay attention this is Domain admin password)

A window will open up with Domain admin privileges.

Now you are still logged in as a domain user, but you have a window in from of you (CMD.EXE) which is running with your domain admin privileges.

So what can you do with this?

Go ahead download,

Windows 2003 Support tools so that you can manage AD with it.

Download Here

When it gets to installation all you need to do, is drag the program into CMD window, and hit enter on the keyboard, the setup installation program will be executed with your Domain administrator privileges

It is kind of cool.


Now after installation if you go to rum command and type

Dsa.msc ADUC snap in wont lunch, will lunch but you won't be able to perform any admin task,

Why because you executed it with your domain user credential so windows know you are a user, and have no business of seeing the ADUC snap in.

However, if you type the same command into CMD window which is running with Domain administrator privileges, ADUC will happily open up, and you can perform any task as you wish as Domain Administrators

Now you got the idea, go ahead and play with other thing,

TIPS: you don't have to remember all the short cut abbreviations, you can simply drag and drop anything into CMD windows running under Domain administrator privileges,( don't forget to press on enter) this will execute the program with domain admin credential.

I open ESM several times just like this, during a working day.

Now you get the idea, working secure and smart is up to you. Making windows and managing exchange is up to you as well.

Now, one of the cools thing Windows Sysinternals (Free) is to get the program called

Psexec

Download the ZIP the suite of the entire tools form my Blog site

http://smtp25.blogspot.com/

What is this Psexec tool? Lets you execute processes on other systems

This is great and always what we wanted to do. Now unzip this and save it to your System32 directory below on your Desktop.


%homeDir%\system32/

Paste all the files (Entire Suite) into this directory

Go back to administrator CMD window. Don't forget you need to be in Domain Environment.

Here is the situation we need want to open Remote CMD window on our exchange server while we are logged into our workstation

Exchange serve name is BIOBR2

So we will type this command into Domain administrator CMD window

Type below command

Psexec \\biobr2 cmd.exe

On the command line if you type hostname, you will noticed you are on BIOBR2 server and If you do IP config you will get the IP configuration of the remote server

Now, you can type there, Services.msc, Compmgmt.msc Notepad You can open internet explorer, remote console user will see internet explorer will open up miserly on the server. There are more cool programs in your system32 directory, along with Psexec.exe which is fun to play with

Special thanks to Ron Buzzon, who is my friend future Exchange and AD MVP candidate

Best Regards,

Oz Ozugurlu

Common NDR Codes, Possible Cause, and Troubleshooting Information

Below is the collection of known NDR codes, the table is handy to get a quick idea of what caused the NDR

Non-delivery reports (NDRs) are usually the first indication of a mail system issue that a sender of an e-mail message will receive. There are many different reasons why a message might not be delivered to the recipient. The following table lists the NDR codes with their respective possible cause and troubleshooting recommendations when available.

Code

Possible Cause – Troubleshooting

4.2.2

The recipient has exceeded their mailbox limit. It could also be that the delivery directory on the Virtual server has exceeded its limit. (Default 22 MB)

4.3.1

Out-of-memory or out-of-disk space condition on the Exchange server. Potentially also means out-of-file handles on IIS.

4.3.2

Message deleted from a queue by the administrator via the Queue Viewer interface in Exchange System Manager.

4.4.1

Host not responding. Check network connectivity. If problem persists, an NDR will be issued.

4.4.2

Connection dropped. Possible temporary network problems.Troubleshooting: This code may be caused by transient network issues or servers that are down. The server tries to deliver the message for a specific time period, and then generates additional status reports.

4.4.6

Maximum hop count for a message has been exceeded. Check the message address, DNS address, and SMTP virtual servers to make sure that nothing is causing the message to loop.

4.4.7

Message expired. Message wait time in queue exceeds limit, potentially due to remote server being unavailable.

4.4.9

A DNS problem. Check your smart host setting on the SMTP connector. For example, check correct SMTP format. Also, use square brackets in the IP address [197.89.1.4] You can get this same NDR error if you have been deleting routing groups.

4.6.5

Multi-language situation. Your server does not have the correct language code page installed

5.1.x

Problem with email address

5.0.0

Generic message for no route is available to deliver a message or failure. If it is an outbound SMTP message, make sure that an address space is available and have proper routing groups listed.

5.1.0

Message categorizer failures. Check the destination addresses and resend the message. Forcing rebuild of Recipient Update Service (RUS) may resolve the issue.

Often seen with contacts. Check the recipient address.

5.1.1

Recipient could not be resolved. Check the destination addresses and resend the message. Potentially e-mail account no longer exists on the destination server.

Another problem with the recipient address. Possibly the user was moved to another server in Active Directory. Maybe an Outlook client replied to a message while offline

5.1.2

SMTP; 550 Host unknown. An error is triggered when the host name can't be found. For example, when trying to send an email to bob@ nonexistantdomain.com.

[Example kindly sent in by Paul T.]

5.1.3

Bad address. Another problem with contacts. Address field maybe empty. Check the address information

5.1.4

Duplicate SMTP address. Use LDIFDE or script to locate duplicate and update as appropriate.

Two objects have the same address, which confuses the categorizer.

Or use Custom search in AD to figured out which object have the same SMTP proxy address

ProxyAddreses=SMTP:oz@smtp25.org (change the SMTP proxy address)

See my blog for saved queries very useful, and handy

http://smtp25.blogspot.com/2007/04/saved-queries-learning-ldap-custom.html

5.2.X

NDR caused by a problem with the large size of the email.

5.2.1

Local mail system rejected message, "over size" message. Check the recipient's limits.

The message is too large. Else it could be a permissions problem. Check the recipient's mailbox

5.2.2

The recipient has exceeded their mailbox limit.

5.2.3

Message too large. Potentially the recipient mailbox is disabled due to exceeding mailbox limit.

5.2.4

Most likely, a distribution list or group is trying to send an email. Check where the expansion server is situated.

5.3.1

Mail system full. Possibly a Standard edition of Exchange reached the 16 GB limit

5.3.2

System not accepting network messages. Look outside Exchange for a connectivity problem

5.3.3

The remote server has run out of disk space to queue messages, possible SMTP protocol error.

Recipient cannot receive messages this big. Server or connector limit exceeded

5.3.4

Message too big. Check limits, System Policy, connector, virtual server

5.3.5

Message loopback detected.
Multiple Virtual Servers are using the same IP address and port. See Microsoft TechNet article: 321721 Sharing SMTP. Email probably looping.

5.4.0

Authoritative host not found. Check message and DNS to ensure proper entry. Potential error in smarthost entry or SMTP name lookup failure.

5.4.1

No answer from host. Not Exchange's fault check connections

5.4.2

Bad connection.

5.4.3

Routing server failure. No available route


5.4.4


No route found to next hop. Make sure connectors are configured correctly and address spaces exist for the message type

5.4.6

Categorizer problems with recipient. Recipient may have alternate recipient specified looping back to self.

5.4.7

Message expired. Message wait time in queue exceeds limit, potentially due to remote server being unavailable

5.4.8

Looping condition detected. Server trying to forward the message to itself. Check smarthost configuration, FQDN name, DNS host and MX records, and recipient policies.

5.5.0

Generic SMTP protocol error.

5.5.2

SMTP protocol error for receiving out of sequence SMTP protocol command verbs. Possible to low disk space/memory of remote server.

5.5.3

Too many recipients in the message. Reduce number of recipients in message and resend.

5.5.5

Wrong protocol version

5.6.3

More than 250 attachments

5.7.1

Access denied. Sender may not have permission to send message to the recipient. Possible unauthorized SMTP relay attempt from SMTP client.

5.7.2

Distribution list cannot expand and so is unable to deliver its messages

5.7.3

Check external IP address of ISA server. Make sure it matches the SMTP publishing rule

5.7.4

Extra security features not supported. Check delivery server settings

5.7.5

Cryptographic failure. Try a plain message with encryption

5.7.6

Certificate problem, encryption level maybe to high

5.7.7

Message integrity problem.


Oz Ozugurlu

Wednesday, April 25, 2007

ESEutil and ISinteg in SMTP25@Nutshell




We were told today our backup team could not backup our Exchange server due to possible corruption on our Corporate Exchange Clustered servers, so I am asked to investigate if the databases for our corporate exchange has some corruption or not. Challenges: Production hours Exchange is heavily utilized, over 7000 mailbox resides on the cluster, so no WAY I get an outage. Even after hours I still cannot get any outage due to politics involved here where I work anyway, Hirrrrrrrrrr


Okay after digging a little bit Google fair enough I bumped into KB 248122 which is what I need exactly
My Guts still won't let me try this on a production server so I decide to try on a LAB.
First let's talk about ESEutil and ISinteg a little bit
ESEutil checks and fixes individual database tables
ISinteg checks and fixes the links between tables
To better understand the difference between ESEutil and ISinteg, let's use a building construction analogy.
Running ESEutil is like opening a web browser, getting the URL for Msexchange911.org and getting on Exchange Forums, start replying all the post and having fun more than going to Bahamas and going bananas, We more care about how accurate and fast we can help Exchange Community without worried about the color of Msexchange911.org Forums website or how annoying all those moving advertisements (-:. Our ultimate goal and focus to help those who is in need of help, and share our knowledge and experience as much as we can.

We focus about giving the right, accurate information and knowledge to the people who need them to get the JOB done.


Running ISinteg is like opening a web browser, getting the URL for Msexchange911.org and getting on Exchange Forums, than worrying about the colors of the page, why there is not enough room for forums, or how many people are online at that time. In this case we don't care about the quality of posts how accurate they are as long as they formatted in nice looking Fonts,

As you can see from the analogy above, I should get a Nobel Prize for this year for Exchange (-:


Both ESEutil and ISinteg are vastly different utilities, but they are complimentary and in some ways dependent upon each other to provide proper Exchange maintenance.

You can use the /ml option of the Eseutil utility to test the transaction integrity of transaction log files.

KB: 248122 (http://support.microsoft.com/kb/248122)


Now we will verify all the transaction logs to see if they have any corruptions or not


Let's get going


E:\Program Files\Exchsrvr\bin>eseutil /ml E:\Logs\SG1-Logs\e00


Knowing ESEutil /ML is great, you can use it against one log or all of the logs as I have done it. When ESEutility was stopped on the Log file which is locked, obviously data is either was written to this particular log or was getting committed to database, I did realized now I know which log files have been committed to a database, just in case backup won't happen logs won't get flushed and I get a call in the night, I cannot mount any of the stores due to no space on t he LUN,


ESEutil /ML E:\Logs\SG1-Logs\e00

Will tell me which logs I can get rid off


Best Regards


Oz Ozugurlu