Messing around in the registry, I noticed the option to "ClearPageFileAtShutdown". It is set by default to "0" (no). What is the purpose of enabling this feature? Are there any performance advantages/disadvantages in doing do?

Security.

The pagefile may contain the data that you were working on. Clearing it a shutdown makes it harder to find the data. I would not enable it. The pagefile will be remade on bootup re-fragmenting your files.

That registry entry is controlled by the Local Security Policy named "Shutdown: Clear virtual memory pagefile". It doesn't remove the pagefile, but merely wipes it. So the pagefile doesn't have to be re-created at boot if the option is turned on. It might be worthwhile option to enable if you keep encrypted data on the system and don't want anyone to be able to snoop the pagefile. If you don't use encrypted data, I'm not sure why anyone would bother to use it. If the pagefile is big, this option can make shutdown take quite a while.

Regards,
Jim

Advantages:

Leaves no trace of what was last worked on in system, this is what is called 'clearing the junk file' on other OS/NOS platforms!

Disadvantages:

SLOW shutdown... really slow! It has to take time to overwrite that file 0,1,0 3x for secure content clearing. ONLY way to securely clean a file on NTFS at least! Odds are, you are running NTFS if you are security conscious on an NT based OS!

* DosFreak's right on its recreation at system restart, IF the file's missing!

You can skate around the fragmentation issues (caused usually by NON-static set ones (meaning not same upper size & lower size which is what I use & 1.5x my RAM set so it should NOT fragment)) with a file from SySInternals called PageDefrag!

(I write about this in my game tuning guide here at NTCompatible in fact...)

That TINY program will defrag your pagefiles, & registry files at each system boot while still in native mode, where chkdsk/f operates from at boottime!

Effectively circumventing ANY fragmentation of them caused by things like using a "open-ended" pagefile/swapfile.

This program defrags it while still in NT based Os' "native" mode (its VMS heritage, & where it uses its TRUE API, that only ntkernel.dll can access & use as far as I know from Win32 GUI mode since it is Ring 0 in nature)! Do look into it!)

http://www.sysinternals.com/ntw2k/freeware/pagedefrag.shtml

*

APK

P.S.=> FREEWARE TOO! Check their site out... many COOL tools! I have corresponded with both Dr. Mark Russinovich & Bryce Cogswell on a couple occasions, they're very sharp & among the BEST out there, I have to give them that! apk

I'm not sure I see the point you're making. The pagefile is NOT deleted by this security setting. The contents are wiped. Fragmentation will not result from the use of this setting. That's all I was saying.

The pagefile and registry hives are defragged by the Sysinternals utility, Pagedefrag. However it doesn't touch the MFT or metadata. O&O makes a decent defragger that performs a defragging operation of all of this stuff at boot time, and in very little more time than it takes for Pagedefrag to run. However, the versions that do boot time defragging are not freeware.

Regards,
Jim

Did I say it was deleted by clearing it above?

It can be deleted, by filesystem corruptions!

(I have had it happen before, or losing a partition!)

I house mine on a second drive, it has happened to me... by losing a stripeset! NOT the one I boot from!

(I gain speed by housing the pagefile/swapfile onto another disk... on EIDE a second one on another EIDE I/O channel, & on ScSi on another drive device on the chain. So, when one drive is seeking/reading/writing for me? The swapfile & temp. operations take place on another.. simultaneously! Makes for good performance sense!)

* Understand now?

APK

P.S.=> You are bringing in the possibility of MFT$ defrags now? Diskeeper from Executive Software also does the same as well... not a freeware one, & not in their LITE versions either! I told folks abotu a FREEBIE they can use for PageFile & Reg file defrags above! apk

"Did I say it was deleted by clearing it above?

It can be deleted, by filesystem corruptions! "

That wasn't the topic under discussion. I was merely trying to be certain that it was understood that the security setting being discussed would NOT delete the pagefile itself, and therefore would not result in a file system fragmentation issue, in and of itself.

"(I gain speed by housing the pagefile/swapfile onto another disk... on EIDE a second one on another EIDE I/O channel, & on ScSi on another drive device on the chain. So, when one drive is seeking/reading/writing for me? The swapfile & temp. operations take place on another.. simultaneously! Makes for good performance sense!)

* Understand now?

APK

P.S.=> You are bringing in the possibility of MFT$ defrags now? Diskeeper from Executive Software also does the same as well... not a freeware one, & not in their LITE versions either! I told folks abotu a FREEBIE they can use for PageFile & Reg file defrags above! apk"

I pointed out the differences in cost in my own post. For the information of anyone who's interested in the differences, the Executive Software product has to be set each time to perform the boot time defrag, whereas the O&O product can be set to perform it automatically at each boot.

As for you, APK, you might want to have that ego checked. Your voluminous posts speak volumes about you but more, I think, about a presumptuous nature than about knowledge.

"
That wasn't the topic under discussion. I was merely trying to be certain that it was understood that the security setting being discussed would NOT delete the pagefile itself, and therefore would not result in a file system fragmentation issue, in and of itself.
"

Really? Is that why you said to me "I don't see your point?" & stated that it does not delete it again?

* After I NEVER SAID it deleted it above, only cleared it out!

(That can be read, plain as day in my first post and your statement in the post right after it!)

"
I pointed out the differences in cost in my own post."

Did I say you didn't?

"For the information of anyone who's interested in the differences, the Executive Software product has to be set each time to perform the boot time defrag, whereas the O&O product can be set to perform it automatically at each boot. "

Not true! It can be set to activate based on HOW MANY FRAGMENTS are present in the pagefile & I believe the MFT$ file as well (I don't have it online to refer to right now, using std. defrag in XP now), correct? Check that yourself...

"
As for you, APK, you might want to have that ego checked. Your voluminous posts speak volumes about you but more, I think, about a presumptuous nature than about knowledge. "

LOL! Maybe you ought to take your own advice? My "voluminous posts" are packed with information & detail, & for GOOD reason!

There are things you have to look out for, & they account for it (extolling things to look out for as well as the good behind doing them). They are not going to be 5 line posts, but are large because of the volume of info. in them. It's not my fault that bothers you.

Do try read them sometime yourself, you might learn something... instead of casting remarks like that at me!

(I don't see any complaints on the content of them, only praise here & elsewhere for the information!)

*

APK

P.S.=> By the way, Get your PhD in psychology before you attempt to psychoanalyze anyone, ok? It might lend credence to your analysis! Because without that degree & making a statement like that about me? Well... who's the presumptious one? And, about knowledge, I can only ask if you have been published in any of this trades' publications like magazine periodicals, or books... has your work appeared in any? apk

I don't really need confirmation of my opnions from others, but I've seen the comments about your "contributions", and they are far from unaniously slanted in your favor. Take a hint. You are presumptuous, and no one needs a doctorate in psychology to see that.

I guess you're at least relatively safe with your puffery online. Hard to get away with it in real life, isn't it?

"
I don't really need confirmation of my opnions from others, but I've seen the comments about your "contributions", and they are far from unaniously slanted in your favor. Take a hint. You are presumptuous, and no one needs a doctorate in psychology to see that.

I guess you're at least relatively safe with your puffery online. Hard to get away with it in real life, isn't it? "

Ok Dr. Freud...

Seems to ME you called me names, like presumptuous, & told me to get an ego check... FIRST!

Understandeable: This is after you got caught screwing up making erroneous statements about Diskeeper!

You need the ego check man. You screwed up, admit it, & be done with it! Don't call me names! Defending yourself is a bit late now isn't it after I caught that in quotes no less on your error regarding it!

(LOL, you don't even know me... and as far as you saying that to me, I'd say that to your face, no problems. First though? Show us all that degree in psychology, before you attempt to berate me.)

In posts here I have made?

I can show you folks here thanking me for the content & getting them out of jams... posts I made in the last few days!

Here is a couple examples:

http://www.ntcompatible.com/vb/showthread.php?threadid=17025

(Look for Nico's post there, he is known here as DremNeonBlack & also TB99's last post there in that thread's second page. 2 folks in one post no less! So eat your words.)

and also here:

http://www.ntcompatible.com/vb/showthread.php?threadid=16939

(Near end of post, eat your words again!)

Once again, you are shown... in error! Need more examples??

http://www.ntcompatible.com/vb/showthread.php?threadid=16899

* Too bad you got caught with your pants down, making mistakes about Diskeeper above... & you call ME presumptuous & make insinuations about me to top it all off??

Give us all a break! Show me your PhD in psychology & I might listen to you calling me names & telling ME to get an ego check when all I did was correct your MISINFORMATION!

APK

P.S.=> You made SUCH a blatant error regarding diskeeper & its ability to detect for fragments at boottime, that even replying to you again would be a waste, I sent this to Phillip the owner here to show him your behaviour... Check YOUR EGO Dr. Freud, before you make analysis like that again about me... & check your defragmenter product knowledge again also! You screwed up! apk

O.K you two, let it go. Besides, you're both talking over my head now anyway. Thanks for initial replies - they're most appreciated. If you're gonna continue the mud slinging, I'm gonna delete this thread.

"
For the information of anyone who's interested in the differences, the Executive Software product has to be set each time to perform the boot time defrag, whereas the O&O product can be set to perform it automatically at each boot.

As for you, APK, you might want to have that ego checked. Your voluminous posts speak volumes about you but more, I think, about a presumptuous nature than about knowledge. "

I agree Ron Jeremy (like your choice of nicks too, lol),

But that statement of his I quote is in MASSIVE error, & for stating it in a quote he called me those names first man! Diskeeper can auto-activate to check for fragments in critical system files at boot, unlike he stated!

(I will defend myself after being called names like he did to me & for the insinuations he made about me, but not long... Phillip has this post sent to him now, he will decide what to do!)

* Glad I could help you though...

APK

P.S.=> I only corrected his error, & got attacked for it... good lord! Egos online amaze me... last I have to say! apk

Ron_Jeremy,

Sorry for the unpleasantness.

Regards,
Jim

"The pagefile and registry hives are defragged by the Sysinternals utility, Pagedefrag. However it doesn't touch the MFT or metadata. O&O makes a decent defragger that performs a defragging operation of all of this stuff at boot time, and in very little more time than it takes for Pagedefrag to run. However, the versions that do boot time defragging are not freeware."

Part of this statement is correct and part is incorrect. The correct part is that Sysinternals doesn't provide a mechanism to defragment the Master File Table ($MFT) or related metadata.

The incorrect part is that O&O's defragger will defragment the MFT and metadata. O&O defragments the $MFT only - it doesn't defragment the $Logfil, $Bitmap, $Upcase, etc... There is only 1 defragger available that will defragment these metadata files - PerfectDisk - it is also the only defragger that tells you how badly fragmented these metadata files are. Defraggers like O&O Defrag only tell you how badly fragmented the $MFT is.

- Greg/Raxco Software

Disclaimer: I work for Raxco Software, the maker of PerfectDisk - a competitor to O&O Defrag, as a systems engineer in the support department.

Well damn ghayes, took ya long enough to get here!

;)

"The incorrect part is that O&O's defragger will defragment the MFT and metadata. O&O defragments the $MFT only - it doesn't defragment the $Logfil, $Bitmap, $Upcase, etc... There is only 1 defragger available that will defragment these metadata files - PerfectDisk - it is also the only defragger that tells you how badly fragmented these metadata files are. Defraggers like O&O Defrag only tell you how badly fragmented the $MFT is.

- Greg/Raxco Software

Disclaimer: I work for Raxco Software, the maker of PerfectDisk - a competitor to O&O Defrag, as a systems engineer in the support department. "

Sorry, I should have been more careful / precise. Have you examined the "Select Additional Files" feature on the Boot Time Defragmentation dialog in O&O? Once you have performed one full defragmentation of a drive, you have the option to add the files that couldn't be defragged with the GUI online by using the Add Exclusive feature. I won't pretend to know whether or not that comprises all the metadata, but that is some or most of it, isn't it? I mentioned it because it's a feature that I've seen many users / evaluaters of O&O overlook. Anyway, once you add the exclusively locked files, they also get defragged at boot time.

In addition to the manual Action | Boot-Time Defragmentation settings, the Executive Software Product does have FragGuard which can be set to run when fragmentation exceeds certain levels on the MFT or registry hives (but without mention of any other items), but I didn't see evidence that it could defrag the "unmovable" files on an NTFS partition.

BTW, I tried out Perfect Disk about a year-and-a-half ago when I was evaluating defraggers for use with Win2K. (I've been using Windows only since a couple of months before the advent of Win2K.) I thought it was generally a good product, but I had some problems with the user interface on a notebook with an ATI graphics subsystem that I couldn't resolve with tech support and had to resort to O&O.

Regards,
Jim

Edit: I asked you if the "additional files" comprised any significant portion of the metadata but didn't tell you what they were. DOH! I'd be glad to PM or e-mail the list to you.

"
"The pagefile and registry hives are defragged by the Sysinternals utility, Pagedefrag. However it doesn't touch the MFT or metadata. O&O makes a decent defragger that performs a defragging operation of all of this stuff at boot time, and in very little more time than it takes for Pagedefrag to run. However, the versions that do boot time defragging are not freeware.""

SysInternals PageDefrag IS freeware... & it only defrags the PageFile & Registry Hives (and backup ones like SYSTEM.ALT). I know, I use it here!

-----------------------------------------------------------------

I never stated PageDefrag did the MFT$, that was not part of the original topic discussion which is about Clearing the PageFile @ Shutdown!

(jwallen brought that up and another defragger PerfectDisk AND O&O defrag...)

So, I mentioned Executive Software's Diskeeper! AND Diskeeper DOES DO MFT$ work at boottime, here is a quote from their products features page on it:

"Frag Guard ® (Windows NT and 2000 only). Online prevention of fragmentation in your most critical NT/2000 system files: the Master File Table (MFT) and Paging Files.

Fragmentation of the MFT can seriously impact performance as the operating system has to go through the MFT to retrieve any file on the disk. If the MFT is already fragmented when Diskeeper is installed, Diskeeper can defragment it with a boot-time defragmentation feature, then maintain this consolidated state.

Frag Guard works much the same way with the Paging File. This is a specialized NT file on the disk which acts as an extension of the computer’s memory. When memory fills up, the system can utilize this file as virtual memory. A fragmented Paging file impacts performance when it reads data back into system memory. The greater the fragmentation, the slower vital computer operations will perform."

http://www.execsoft.com/diskeeper/about/diskeeper.asp

-------------------------------------------------------------------

* AND, note, as I said above? Diskeeper does do it on automatic checks of MFT fragments... if beyond a certain default level or custom one you set? It runs & defrags MFT$, and pagefile too should you set it to run and Frag Guard is an option, giving you control on whether it runs or not: VERY FLEXIBLE PROGRAM!



APK

P.S.=> Norton Speedisk by Symantec? It does pagefile defrags DURING Win32 GUI Operations, only one I know that does! BUT, it has a nasty habit of snapping the MFT$ into 2 parts... always! This is why I keep Diskeeper around additionally, to take care of that! apk

Jim,

"Sorry, I should have been more careful / precise. Have you examined the "Select Additional Files" feature on the Boot Time Defragmentation dialog in O&O? Once you have performed one full defragmentation of a drive, you have the option to add the files that couldn't be defragged with the GUI online by using the Add Exclusive feature. I won't pretend to know whether or not that comprises all the metadata, but that is some or most of it, isn't it? I mentioned it because it's a feature that I've seen many users / evaluaters of O&O overlook. Anyway, once you add the exclusively locked files, they also get defragged at boot time."

I can state with utmost certainty that O&O Defrag does NOT do any of the metatdata besides the $MFT. Even if you go into the Boot Time defrag options and select Additional Files, you are not presented with a way to select any of these other metadata files from their interface (do you see a file called $MFTMir or $Logfile or $Upcase?).


AlecStaar:

Diskeeper also doesn't defragment these other metadata files. The interesting thing about Diskeeper is that even if the $MFT is actually in 1 piece, Diskeeper will always show it as being as in 2 pieces. Why? Because they count the $MFTMirr - one of the metadata files - as a fragment of the $MFT - even though it is a separate file.

This is easier to see on an NT4 NTFS partition.

- Go to a MSDOS prompt and go to the top level of a NTFS partition.
- Issue the following command:
Attrib $MFT
Attrib $MFTMirr
Attrib $Logfile

These are just 3 of the NTFS metadata files.

If you try to find out non-$MFT fragmentation information in any other defrag product, it can not be found.

The reason SpeedDisk can sometimes only get the $MFT down to 2 pieces is that SpeedDisk can't move the 1st records of the $MFT. This means that if the beginning of the $MFT is not at the top of the logical partition, then SpeedDisk has to leave it where it is - but may put the remainder of the $MFT at the top of the logical partition.

Even though I work for a competitor, I do know quite a lot about other defrag products and what they can and cannot do :-)

- Greg/Raxco Software

"

AlecStaar:

Diskeeper also doesn't defragment these other metadata files. The interesting thing about Diskeeper is that even if the $MFT is actually in 1 piece, Diskeeper will always show it as being as in 2 pieces. Why? Because they count the $MFTMirr - one of the metadata files - as a fragment of the $MFT - even though it is a separate file.

This is easier to see on an NT4 NTFS partition.

- Go to a MSDOS prompt and go to the top level of a NTFS partition.
- Issue the following command:
Attrib $MFT
Attrib $MFTMirr
Attrib $Logfile

These are just 3 of the NTFS metadata files.

If you try to find out non-$MFT fragmentation information in any other defrag product, it can not be found.

The reason SpeedDisk can sometimes only get the $MFT down to 2 pieces is that SpeedDisk can't move the 1st records of the $MFT. This means that if the beginning of the $MFT is not at the top of the logical partition, then SpeedDisk has to leave it where it is - but may put the remainder of the $MFT at the top of the logical partition.

Even though I work for a competitor, I do know quite a lot about other defrag products and what they can and cannot do :-)

- Greg/Raxco Software "

Agreed, it does not do the metadata stuff (I never said it did)...

But, it can access (in "real mode" boottime) & defragment the MFT$ nevertheless, & on a 'automatic' setting if set as an option & with FragGuard too.

(From my understanding of "MetaData" (and correct me if I am wrong) it is like a "temp storage" area for files as well. I read once that NTFS actually uses it to store files at times & for extended filesystem data... true? I understand the MFTMirr$ file is also like a second bootrecord is on DOS Fat-16, but NOT the bootrecord (making an analogous comparison of the principle). The Logfile is for rollbacks, correct? Like a TRUE Journalling filesystem does!)

I've done the attrib -r -h -s -a on $MFT, I know that trick to see it... does not seem to work on NTFS5 for me under XP! File not found errors in it, funny... it worked on NT versions!

LOL, with good reason too... you remove that files attribs & burn it? Good bye system!

A good tool to see more about it is sysinternals.com's ntfsinfo.exe!

Speedisk is nice, my fav currently! I used to like Diskeeper the most, but Speedisk can move things around during "Enhanced Mode" GUI operations Diskeeper cannot! NO REBOOT REQUIRED, that's good for System Admins, who have to keep a network running!

If only Norton/Symantec could skate around that issue of snapping the MFT$ into 2 parts! That is WHY I keep Diskeeper around mainly... I like it, but at times, I like Speedisk more!

*

I have not tried PerfectDisk, & always said I'd like to at some point. I have tried the others (O&O Defrag, Diskeeper, Ontrack/Mijenix Fix-It Utilities 3.0 defragger (written I believe by a guy I have worked with Mr. Charles McDevitt) and Speedisk... all nice work, they all have merits (and demerits)!).

On a side note: This is good you know about your competition! That means you are INTO your job, & probably a valued asset to your company... product knowledge? SELLS! Especially SPECIFICS... & details!

APK

P.S.=> Any chance of sending me a FREE copy of it, lol! I have heard alot of good about it! apk

A little bit of info about NTFS metadata...

NTFS is a self-describing file system. This means that all of the information needed to "describe" the file system is contained within the file system itself - in the form of metadata.

The $MFT is where all of the information about files are stored - in the form of file id's. A file ID is comprised of a 64bit number - of which 2/3 is the actual FileID and the remaining 1/3 is a sequence number. When files are deleted from an NTFS partition, the file id isn't immediately re-used. Only after hundreds of thousands of files are created is the sequence number incremented and the "empty" file id re-used. That is why the $MFT continues to grow and grow and grow. It is also why the $MFT Reserved Zone exists - to allow the $MFT to grow "into" it - hopefully in a contiguous fashion. Very small files can be stored "resident" in the $MFT. As much of the $MFT as can fit into memory is loaded when the partition is mounted.

The $MFTMirr is an exact copy of the 16 records of the $MFT. The first 16 records of the $MFT contain files 0 - 15. File 0 is the $MFT. File 1-15 are the remainder of the metadata (not all used btw...). The $MFTMirr is NTFS's "fallback" mechanism in case it can't read the 1st 16 records of the $MFT.

The $Bitmap is exactly that - a file containing a bit for each logical cluster on the partition - with the bit either being set or clear depending if that logical cluster is free or used.

The $Logfile is NTFS's transaction log - all updates to disk first go through the transaction log. This transaction log is what provides for NTFS's recovery (roll back/forward transactions) when the operating system is abnormally shutdown/crashes and provides for enhanced file system integrity.

$Upcase is used for Unicode information (foreign language support, etc...).

These are just a few of the NTFS metadata files and what they are used for. Windows 2000 introduced new metadata files (i.e. $Usnjnl and $Reparse).

Regarding SpeedDisk:

SpeedDisk is the only commercial defragger that does NOT use the defrag APIs provided by Microsoft as part of the NT/2000/XP operating system. These APIs are tightly integrated with the Windows Memory Manager, caching system and file system and take care of all of the low level I/O synchronization that has to occur to allow safe moving of files online - even if the files are in use by other users/processes. The APIs impose some restrictions, however. Pagefiles can't be defragmented online, (nor the hibernate file under Win2k), directories can't be defragmented online under NT4 (FAT and NTFS) and Win2k (FATx). The $MFT and related metadata can't be defragmented online as well. In order to get around these restrictions, SpeedDisk "wrote their own" stuff to move files - it has a filter driver that gets installed/run. This is why SpeedDisk can be service pack/hotfix dependent. Depending on the changes that MS makes to the Memory Manager and file system, SpeedDisk may have to be updated to safely run. That is why (for example), if you have Windows 2000/SP2 installed and run SpeedDisk, it displays a warning message about not being compatible with that version of the operating system and proceed at your own risk...

I know HOW SpeedDisk is doing what they are doing. However, knowing what can happen if they calculate things incorrectly, makes me a bit wary. However, SpeedDisk is alot better product - in terms of actually being able to normal data files - than some of the other defrag products out there.

- Greg/Raxco Software

"
A little bit of info about NTFS metadata...

NTFS is a self-describing file system. This means that all of the information needed to "describe" the file system is contained within the file system itself - in the form of metadata."

Yup, sounds JUST LIKE HPFS for Os/2, ext2 for Linux & previous versions of NTFS as well... Extended atributes data stored for files, like last access time & date stamps for example, as well as NTFSCompression & NTFSEncryption attributes as well as STREAMS data!

(Now this? is interesting stuff... filestreams! Make for one heck of a virus possibility if used for malicious purposes! ALL files on NTFS are really streams! It's the alternate ones to watch for... more on that later!)

As well as Security data (chmod stuff from Unix) & Symbolic Link attribs (sort of like a Unix STUB I would guess)!

PLUS HardLinks (2 files have the same name (another scary one, potentially!) HardLinks are when the same file has two names (some directory entries point to the same MFT record). Say a same file has the names A.txt and B.txt: The user deletes file A, file B still is online. Say the user goes and deletes file B, file A remains STILL. Means both names are completely equal in all aspects at the time of creation onwards. Means the file is physically deleted ONLY when the last name pointing to it gets deleted.)!

"[B]
The $MFT is where all of the information about files are stored - in the form of file id's. A file ID is comprised of a 64bit number - of which 2/3 is the actual FileID and the remaining 1/3 is a sequence number. When files are deleted from an NTFS partition, the file id isn't immediately re-used. Only after hundreds of thousands of files are created is the sequence number incremented and the "empty" file id re-used. That is why the $MFT continues to grow and grow and grow. It is also why the $MFT Reserved Zone exists - to allow the $MFT to grow "into" it - hopefully in a contiguous fashion. Very small files can be stored "resident" in the $MFT. As much of the $MFT as can fit into memory is loaded when the partition is mounted."

Yup, as I heard & mention above, $MFT can store files in its contents...

I think a very dangerous 'bug' exists, because of this & zero-byte files creation: potential for disaster, you may wish to run this by your programmers:

A quote from another developer:

"Each file on NTFS has a rather abstract constitution - it has no data, it has streams. One of the streams has the habitual for us sense - file data. But the majority of file attributes are also streams! Thus we have that the base file nature is only the number in MFT and the rest is optional. The given abstraction can be used for the creation of rather convenient things - for example it is possible to "stick" one more stream to a file, having recorded any data in it - for example information about the author and the file content as it was made in Windows 2000 (the most right bookmark in file properties which is accessible from the explorer). It is interesting that these additional streams are not visible by standard means: the observed file size is only the size of the main stream contains the traditional data. It is possible for example to have a file with a zero length and at its deleting 1 GByte of space is freed just because some program or technology has sticked anadditional stream (alternative data) of gigabyte size on it. But actually at the moment the streams are practically not used, so we might not be afraid of such situations though they are hypothetically possible. Just keep in mind that the file on NTFS is much deeper and more global concept than it is possible to imagine just observing the disk directories. Well and at last: the file name can consist of any characters including the full set of national alphabets as the data is represented in Unicode - 16-bit representation which gives 65535 different characters. The maximum file name length is 255 characters."

type nul > Drive:\Folder\Filename.Extension can create zero byte files, take up no room,right? WRONG!

$MFT knows they're there & creates metadata surrounding them & forces itself to grow, small growth for each one, but growth! Do that long enough?? TROUBLE!

E.G.-> A program creates zero byte files with diff. names on them (1.txt, 2.txt... n.txt) in an endless LOOP? Watch what happens to the $MFT: grows until there is NO MORE ROOM left for anything else! Reservation zone might stop that & disk quotas, but I am not sure! PERSONALLY, I think it'd keep growing & growing until the disk is full... I do not believe the OS enforces Quotas on the $MFT nor does the NTFS drivers!

Disk quotas can be enforced on USERS in Explorer.exe security tab... I don't know if they can be imposed on SYSTEM user or NTFS driver itself!

To change the amount of space NTFS reserves for the MFT:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\FileSystem

Add NtfsMftZoneReservation as a REG_DWORD value & of a range from 1 - 4.

1 is min percentage, 4 is max % used.

* Scary eh? Top that off w/ each of those files possessing a hidden filestream... & you compound the danger! Food for thought for your companies next upgrade... Watch for alternate datastreams in zero byte files & alternatedatastreams period!

"[B]
The $MFTMirr is an exact copy of the 16 records of the $MFT. The first 16 records of the $MFT contain files 0 - 15. File 0 is the $MFT. File 1-15 are the remainder of the metadata (not all used btw...). The $MFTMirr is NTFS's "fallback" mechanism in case it can't read the 1st 16 records of the $MFT."

Yup, again as I stated above via an analogous comparison to FAT using an MBR mirror! Alot like that (but not boot record)... safety & security measure!

"[B]
The $Bitmap is exactly that - a file containing a bit for each logical cluster on the partition - with the bit either being set or clear depending if that logical cluster is free or used."

Yes, it is a bitmapped filesystem that MS uses in any of them they use! Ext2 on Linux is same... bitmapped filesystem, most defraggers & people call it "Volume Bitmap".

"[B]
The $Logfile is NTFS's transaction log - all updates to disk first go through the transaction log. This transaction log is what provides for NTFS's recovery (roll back/forward transactions) when the operating system is abnormally shutdown/crashes and provides for enhanced file system integrity."

Yup, a journalling style filesystem as I mention above!

"[B]
$Upcase is used for Unicode information (foreign language support, etc...)."

More extended attributes data!

"[B]
These are just a few of the NTFS metadata files and what they are used for. Windows 2000 introduced new metadata files (i.e. $Usnjnl and $Reparse)."

I am not aware those currently! I have read about "reparse points" but not about $Usnjnl... is it the 'hidden' folder named "System Volume Information"? I can see it in Explorer.exe but cannot access its contents (must change Explorer's properties to see it)!

"[B]
Regarding SpeedDisk:

SpeedDisk is the only commercial defragger that does NOT use the defrag APIs provided by Microsoft as part of the NT/2000/XP operating system. These APIs are tightly integrated with the Windows Memory Manager, caching system and file system and take care of all of the low level I/O synchronization that has to occur to allow safe moving of files online - even if the files are in use by other users/processes. The APIs impose some restrictions, however. Pagefiles can't be defragmented online, (nor the hibernate file under Win2k), directories can't be defragmented online under NT4 (FAT and NTFS) and Win2k (FATx). The $MFT and related metadata can't be defragmented online as well. In order to get around these restrictions, SpeedDisk "wrote their own" stuff to move files - it has a filter driver that gets installed/run. This is why SpeedDisk can be service pack/hotfix dependent. Depending on the changes that MS makes to the Memory Manager and file system, SpeedDisk may have to be updated to safely run. That is why (for example), if you have Windows 2000/SP2 installed and run SpeedDisk, it displays a warning message about not being compatible with that version of the operating system and proceed at your own risk..."

Works fine on SP2 2k, & previous ones... & that IS how they skate around patches! They do it independent of the MS defrag API calls. Good read on that @ sysinternals.com also! The API's MS uses?? Came from Executive Software code! In NT 3.5x you had to patch the kernel to use Diskeeper... MS licensed that code & integrated it into their kernel, & the native defrag in 2k/XP is a VERY BASIC watered-down Diskeeper. Note, they both run from .msc console shortcuts extensions to Computer Management as well? First time a Symantec Product was NOT the native defrag in a Win32 based OS!

"[B]
I know HOW SpeedDisk is doing what they are doing. However, knowing what can happen if they calculate things incorrectly, makes me a bit wary. However, SpeedDisk is alot better product - in terms of actually being able to normal data files - than some of the other defrag products out there."

It's good stuff, has merits others don't... mainly? System Uptime should appeal to Network Admins! No taking down a server for maintenance when Speedisk works... uptime is assured, defrags can take place & users still access their data!

*

APK

P.S.=> DO RUN THAT EXAMPLE OF ZERO BYTE FILES PAST YOUR CODERS, & THE USE OF ALTERNATE FILESTREAMS AS WELL! I believe this is a 'bug' in NTFS of all types & even setting $MFT Reservation Zone to min & imposing disk quotas users (including SYSTEM) can't stop it! Let me know what they say about that! apk

Nice post Greg.

true, a nice read, but long as h.ell.

Found a selling point for you, for Norton Speedisk... over Diskeeper at least, & I wager Perfect Disk also because it uses the std. Microsoft API purchased from Execsoft & Diskeeper for integration into Kernel level functions!

Read on:

(Speedisk is starting to really show its stuff in many regards, not just uptime in not having to reboot! It's only weakness I can see now is snapping the $MFT into 2 parts... Speedisk can do pagefile defrags NO problem without reboot & also smart file placements by bands of usage PLUS ones YOU can custom specify! ALSO, it allows tuning the memory AND Cpu timeslicing it gets (Diskeeper can do the cpu timeslice part too, but no adjusts on RAM used))

Here is another plus for it...

Diskeeper? It cannot defrag an NTFS partition with clusters of more than 4k size!

I am wagering that PerfectDisk, based on the native NT OS API (really old Diskeeper code bought by Microsoft integrated into their kernel, because old Diskeepers circa model 1.0-1.09 for NT 3.5x?

You needed to patch the NT kernel with a Diskeeper version to use it... and, since PerfectDisk is also based on that std. MS API (Execsoft API) it probably suffers from this 4k limit also)

Speedisk can defrag 64k clustersized disks, I am doing it here now in fact writing this to you, on a RAID & also a non-raid ATA-100 EIDE set of disks!

No problems & on a RAID stripe as well... normal, non-RAID striped as well! I don't know if this would present a hassle with ScSi controllers like the SmartCache by DPT (caching type)... I know it does for Diskeeper or used to!

* Can Raxco PerfectDisk defrag greater than 4k clusters on normal drives & what about RAID ones? What about caching disk controller technology like the DPT SmartCache V for UltraScSi/FastWide etc.?? Does it allow for CPU timeslice allocation via process boosting API calls & also ram used adjustments???

Questions... just curious!

(Thanks! You seem like a guy I want to talk to, I am VERY interested in disk technology... & write code around it here & there too for sharewares I have created!)



APK

P.S.=> I don't know what the FULL capabilities of PerfectDisk are, I don't have it here to try out! apk

Diskeeper 7 can now defraf 4K+ clusters. (Took 'em long enough)