Marc Lognoul's IT Infrastructure Blog

Cloudy with a Chance of On-Prem


Random Thoughts on Disk Defragmentation

Windows 7

Mostly applicable to Windows 7 and Server 2008 R2 and sometimes to earlier Windows versions…

Defragmentation vs. SSD Drive

Defragmenting is actually useless due to the nature of the SSD technology: there is no mechanical or moving part and the storage geometry is totally different. Therefore, making blocks contiguous does not make a lot of sense. On the other hand, recent SSD drives (as well as Windows) introduced the support for the TRIM command, which is actually responsible for “proactively” cleaning up clutter left by deleted files in order to reduce the effort required at actual file write-time. Look at my blog post over the subject for extra details: Checklist for Getting the TRIM Command to Work with SSD drive on Windows 7.

Some claim out loud that defragmenting an SSD drive is dangerous and may therefore shorten its life: that’s not totally untrue. Depending upon the vendor and underlying SSD technology, the life-cycle of an SSD, which is mostly determined by the number of write access, will vary. since defragmentation is likely to provoke a high amount of writes, therefore it may shorten the SSD’s life. Of course, it also highly depends on the number of years you wish to use the drive: you may not always notice the SSD performance degradation due to excessive writes. By the way, if Windows detects an SSD drive at set-up time, it will automatically disable the scheduled defragmentation.

If you find the time, try to have a look at the WD Whitepaper listed under the section More Information, it contains useful information about estimating the SSD life duration.

Defragmentation vs. SAN-based Storage

There are a lot of theories and claims around defragmenting or not SAN-based storage. For me, it depends on the SAN solution that is actually implemented:

  • Some SAN solution simply behave like any other DAS-based storage. Therefore they will benefit from defragmentation like any other disk
  • Other SAN solutions include built-in storage defragmentation (as well as de-duplication) technology, there is there no need to act at Windows level
  • Finally, some SAN solution are fragmented by design. Attempting to go against is simply counterproductive

You got the point: the only way to know what to do is to ask your SAN vendor (and hope you get smart answers in return).

Defragmentation Efficiency vs. Impact on Usability during Defrag

Whether using defragmentation initiated thanks to the windows Scheduled Tasks or Manually, it will run by default in low-priority therefore reducing the impact on usability. Obviously, the process will take longer. In all case, it will impact usability much less than, for example, a full drive scan by any anti-virus software.

If you wish to force defragmentation in high-priority mode on a Windows 7 system, use the command-line defrag with parameter /H.

Defragmentation vs. Windows Page File

Like any other file and due to its usually large size, the page file(s) may also get fragmented over time. The issue with the page file is that it is difficult to touch since it is required by the OS to work. Although there are various workarounds, they all have pro’s and con’s.

  • If you run a Windows version prior Windows Vista: You’re lucky since  you can still use Sysinternals’ PageDefrag mighty tool which will defrag the page files as well as other system-controlled files during boot time
  • If PageDefrag is not an option and you don’t have a second volume available: Reduce the size of the page file to a minimum (requires a reboot), defrag the whole system volume in order to reclaim contiguous free disk space, then set back the page file to its original size then reboot again. There is no guarantee to drastically reduce the number of fragments since the whole file is not completely relocated
  • If you have a second partition at hand: Reconfigure the page file on that partition and remove the one located on the system volume (requires reboot) then once the page file is located on the second volume, defrag the system volume to reclaim contiguous free space. Once done, reconfigure the page file to use the system volume exclusively (requires reboot as you might guess…). In this case,

Note: to fight against page file fragmentation and if you’ve got a good idea of its ideal size (by experience and measures), you can set up a logical volume dedicated to page file and set it to an appropriate fixed size. Of course, it will not boost page file IO’s like it would do if you placed it on a dedicated physical disk.

Note 2: Configuring Windows to clear the page file at shutdown will not help since this (security not performance) feature will write pages full of zeros and will not modify the file size at all, unlike many still think. All you will get by enabling this is a much slower shutdown (although it improved compared to previous Windows versions).

Defragmentation vs. Disk Reliability

Some are afraid of defragmentation because it my break their file system integrity if a power outage occurs right in the middle of a defragmentation process. This is very unlikely to happen because: 1) When defragmenting, Windows takes care of copying blocks and updating the pointers afterwards 2) NTFS is a transactional file system with automatic recovery capabilities dramatically reducing the like hood of data loss during any type of write operation.

Other fear the fact that performing over a relatively short period a high amount of disk IO (writes in particular) may render their disk unusable by increasing the number of bad blocks: this is only true if the disk is already unreliable or already signs of age. In this case, the only actions you can takes are 1) Backing up everything ASAP hoping all your data are still readable 2) Replace the drive 3)Reinstall/restore

Logical vs. Physical Disks

Although it reorganizes blocks, defragmentation operates up to the boundaries of the logical volumes. Take this into account if you have multiple logical disks on the same physical disk.

Disk vs. File Fragmentation

While most of us focus on disk fragmentation, there a circumstances where file fragmentation matter most. for example, you may use Outlook at OST/PST files massively on the save computer and now your PST files are getting bigger and bigger. It’s likely they are fragmented. In order to save you time, you can can Sysinternal’s Contig command to defrag only selected files

MFT Fragmentation

The NTFS Master File Table (MFT, like any other NTFS metadata file may get fragmented. Similarly to single file defragmentation, Contig is your friend for such tasks:

You can also proactively increase the space reserved for the MFT, preferably at disk initialization stage: About the Master File Table zone reservation in Windows Vista and Windows Server 2008.

Various Claims

  • “I should never care about defragmenting since windows does it for me”: Windows can only defragment files that are not in used by the system or by applications. Therefore, if you wish to efficiently defrag files – and not solely free space – you always should have a minimal set of applications running: ideally only the console or the command-prompt.
  • “With a disk large enough, I will never suffer from fragmentation”: This statement is true of you consider only the fragmentation of free space. For some files and under some circumstances, fragmentation will still occur. Of course, you can decide to defragment only those files instead of the whole disk
  • “My Database is configured for scheduled maintenance, there is therefore no need to defrag at disk-level”: Here there is confusion between defragmenting at database-level (data structure) and at disk-level (file system). In both case, take time to analyze fragmentation and depending upon the results, defrag at one or at both levels.
  • “Fragmentation only occurs on Microsoft Windows, not on %REPLACE_WITH_ANY_OTHER_OS_YOU_MAY_LIKE%”: All file systems are likely to experience fragmentation, more or less. It depends more on the type of workload than on the technology itself. Maybe one difference to mention: NTFS has been including a Defrag API for years and years while some have just recently adopted it….
  • “To defrag (or repair) disks faster, I just unplug them and them plug them as extra disk on another system then run the process ‘offline’: Okay, why not. The advice here is to use strictly identical versions of Windows (OS, service pack and patch levels) in order to avoid adverse effects implied by unequal Windows versions.

A Quick(er) while Not-so-Dirty Whole Data Volume Defrag Method

A straightforward way to defrag a data volume is to 1) Copy all files to another volumes 2) Perform Quick Format on the source volume 3) copy the files back to the original volumes. It often takes less time than performing API-based defrag. Note: do not forget to include permissions and metadata as needed as part of the copy…

Conclusion

Defragmentation is actually rescheduling IO’s at a more appropriate time 🙂

More Information

Marc

Advertisements


Checklist for Getting the TRIM Command to Work with SSD drive on Windows 7

Windows 7

What is the TRIM Command and why you should care

The TRIM command is a way for the operating system to notify the SSD drive that it should perform a garbage collection (a proactive clean-up) on the disk blocks recently affected by file deletion operations. It helps  improving the write performance over time since it reduces the need for cleaning the the blocks right before they are (over)written upon request from the OS.

It may sometimes be confused with disk defragmentation (ensuring blocks of the same files or of free disk space are contiguous) or Vendors’ SSD optimizers (which usually perform bulk clean-up at scheduled time or when the disk is initialized before use).

Check-list to enable TRIM (I mean “for real” :))

The exact procedure depends upon your system’s configuration but to be comprehensive, here is a check-list with what you should consider when trying to enable the TRIM command:

  • Check that you SSD drive itself supports it and update the firmware as necessary. Even recent drives may require firmware update
  • Check that your storage controllers/Chipset (Kudo’s to Loukiana for the reminder) supports it and update the firmware as necessary. Note: carefully check if you use H/W RAID controller, very few so actually support TRIM
  • Check your computer’s BIOS and upgrade as necessary. Also check the computer’s vendor information about the SATA mode in which you should set your BIOS configuration. Usually AHCI will do fine but of course, it will depend on chipset/controller
  • Finally, verify the current Windows configuration by running the command hereunder from an elevated command-prompt:

    fsutil behavior query disabledeletenotify

    DisableDeleteNotify = 0 means TRIM is ENABLED
    DisableDeleteNotify = 1 means TRIM is DISABLED

  • Once all conditions above are met but DisableDeleteNotify = 1 is reported, Open an elevated command prompt and execute the command hereunder. Note: rebooting should not be needed to apply the change however it will not hurt.

    fsutil behavior set disabledeletenotify 0

Some remarks

  • At setup time, Windows 7 will automatically enable the TRIM command if it detects and SSD drive. But in many cases, due to improper configuration, outdated drivers or firmware’s and so on, it may not detected it properly. Therefore you’ll have to enable it manually as explained above…
  • If your system is equipped with a RAID controller adapter, it is likely that it will not support TRIM. Actually, very few H/W RAID do support TRIM at the moment but of course, with time things tend to evolve, like for Intel for example
  • While I am mentioning Windows 7 in the title of this post, the information here above are also applicable to Windows Server 2008 R2 and subsequently WHS 2011 too!
  • Unlike stated in (too) many blog posts or forums, getting zero as output from the command fsutil behavior query disabledeletenotify does not guarantee that the TRIM command is active, it simply reports that is is enabled at operating system level. Windows will continue to check if the storage support it or not…
  • disabledeletenotify flag may also be used by non-SSD storage in order to optimize their background processing of file deletion operations and on the other hand it might also bring adverse effect on some storage technologies. Be careful if you mix SSD and non SSD on the same computer
  • TRIM works with Windows 7 native software RAID. I have never tested on 3rd party software though
  • After enabling the TRIM command, your chance to recover delete files is close to zero. On the other hand, it helps improving privacy a little…

Useful Resources

Marc


Leave a comment

Storage: Whitepaper over NTFS CHKDSK Best Practices and Performance

Windows Server 2008 R2

Microsoft has recently published this very useful white paper bringing best practices and guidance when sizing volumes and CHKDSK execution times: http://www.microsoft.com/downloads/en/details.aspx?FamilyID=35A658CB-5DC7-4C46-B54C-8F3089AC097A

A few facts that caught my attentions:

  • The performance gain from Server 2008 to Server 2008 R2 is huge (I can’t imagine compared to Server 2003)
  • Available memory also greatly improves performance
  • Turning off 8.36 name generation has a positive though limited impact on performance
  • Server 2008 R2 shows more linearity in performance than Server 2008
  • Running CHKDSK in read-only mode is very helpful in predicting the duration of a downtime

Definitely a must ready for Windows Admins and IT Pro’s in charge of designing file services!

Marc


Leave a comment

Windows Server 2008 R2: CIFS/SMB 2.x in Details

While Windows Server 2008 and Vista introduces the version 2.0 of the Server Message Block protocol (aka File and Printer Sharing in humanly readable words), Windows Server 2008-R2 and Seven both bring a refreshed version, the 2.1. Instead of drilling down right-here into the protocol’s details, I found more useful to post links to the most interesting resources on the Web.

Protocol Specification and Details

Tuning and Optimization

Support and Troubleshooting

Extra Goodies: Multi-threaded Robocopy and GUI

Although not directly related to SMB 2.1, Robocopy was also updated with the recent version of Windows. The main improvement is the support for multi-threaded operations, particularly interesting when massive file copy operations must take place against small files over WAN connections. I will cover this in details in a coming post. See http://technet.microsoft.com/en-us/magazine/dd542631.aspx for details. Important to note that this option is NOT compatible with the inter-packet gap option (copy using throttling).

Not so new, for those reluctant to learn and exploit all Robocopy command-lines params, there are cool GUI’s available out there, RichCopy being, like its names says it all, the richest one.

Marc