Explorer Eating Memory When Viewing VERY Large Folders

Thanks , that is a really good idea , that could definatly speed things
up with teh amount of files
 >> Stay informed about: Explorer Eating Memory When Viewing VERY Large Folders 

You might try this registry tweak for one thing:
Windows Explorer - File Access and Datestamps.
Whenever a directory on an NTFS drive is accessed a time stamp is added.
Improve the spped of file access and turn this facility off.

a.. Locate the following key in the registry:
'HKEY_LOCAL_MACHINE\System\CurrentControlet\Control\Fileystem'
b.. Create a new DWORD value 'NtfsDisableLastAccesUpdate'
c.. Set the value to








"BIGGERPLUM" <adrian.plumpton DeleteThis @googlemail.com> wrote in message
news:1161890402.610590.19440@i42g2000cwa.googlegroups.com...
> Hi,
>
> I am working with a network moving very large amounts of data round and
> am finding that windows explorer jsut goes nuts on memory. If you try
> viewing the top level of a large folder structure then it jsut hangs
> for ages and makes the machine unusable and from task manager you can
> see explorer is jsut using large amounts of memory (eg . 475MB on a 512
> machine).
>
> Now even after you stop browsing that folder structure periodically
> explorer will just shoot up again and lock the machine, im wondering
> whether it is trying to make an index for this massive folder
> structure.
>
> The folder structure trying to be viewed is 250,000 folders with alot
> of smaller files.
>
> My question is there a fix that allows windows to handle this amount of
> data better? (without having to use unix)
>
> Adrian
>
> PS: This problem occurs on XP pro machines with 2G of memoery, Windows
> Server 2003 with 4G of memory and a Cut down version of 2003 for a
> files server with 512M of memory
>
> Any ideas or comments gratefully recieved
>
 >> Stay informed about: Explorer Eating Memory When Viewing VERY Large Folders 

You might try this registry tweak for one thing:
Windows Explorer - File Access and Datestamps.
Whenever a directory on an NTFS drive is accessed a time stamp is added.
Improve the spped of file access and turn this facility off.

a.. Locate the following key in the registry:
'HKEY_LOCAL_MACHINE\System\CurrentControlet\Control\Fileystem'
b.. Create a new DWORD value 'NtfsDisableLastAccesUpdate'
c.. Set the value to








"BIGGERPLUM" <adrian.plumpton.RemoveThis@googlemail.com> wrote in message
news:1161890402.610590.19440@i42g2000cwa.googlegroups.com...
> Hi,
>
> I am working with a network moving very large amounts of data round and
> am finding that windows explorer jsut goes nuts on memory. If you try
> viewing the top level of a large folder structure then it jsut hangs
> for ages and makes the machine unusable and from task manager you can
> see explorer is jsut using large amounts of memory (eg . 475MB on a 512
> machine).
>
> Now even after you stop browsing that folder structure periodically
> explorer will just shoot up again and lock the machine, im wondering
> whether it is trying to make an index for this massive folder
> structure.
>
> The folder structure trying to be viewed is 250,000 folders with alot
> of smaller files.
>
> My question is there a fix that allows windows to handle this amount of
> data better? (without having to use unix)
>
> Adrian
>
> PS: This problem occurs on XP pro machines with 2G of memoery, Windows
> Server 2003 with 4G of memory and a Cut down version of 2003 for a
> files server with 512M of memory
>
> Any ideas or comments gratefully recieved
>
 >> Stay informed about: Explorer Eating Memory When Viewing VERY Large Folders 

You could look at your problem from the opposite direction! Why do I
need to have so many large files?

Linking files unnecessarily can also be problematic. I encountered this
problem one time using Excel. You can use a macro to update
information thereby avoiding permanent links which load into memory.

--

Hope this helps.

Gerry
~~~~
FCA
Stourport, England

Enquire, plan and execute
~~~~~~~~~~~~~~~~~~~



"BIGGERPLUM" <adrian.plumpton.RemoveThis@googlemail.com> wrote in message
news:1161897372.486456.73640@k70g2000cwa.googlegroups.com...
> Thanks , that is a really good idea , that could definatly speed things
> up with teh amount of files
>
 >> Stay informed about: Explorer Eating Memory When Viewing VERY Large Folders 

"Gerry Cornell" <gcjc.TakeThisOut@tenretnitb.com> wrote in message news:uWwP5RZ%23GHA.4320@TK2MSFTNGP02.phx.gbl...
> You could look at your problem from the opposite direction! Why do I
> need to have so many large files?
>
> Linking files unnecessarily can also be problematic. I encountered this
> problem one time using Excel. You can use a macro to update
> information thereby avoiding permanent links which load into memory.

They didn't say anything about large files, just a large number of files and folders. As to your question, that's just how some
programs are. I've seen lots of big programs (Macromedia apps, Maya, etc.) that come with tens, even hundreds of thousands of tiny
html files in hundreds or thousands of folders. Usually they comprise the extensive documentation scattered amongst thousands of
files (which eat up slack space like crazy!) instead of one large consolidated file (like hlp, chm, pdf, doc, etc.)


--
Alec S.
news/alec->synetech/cjb/net
 >> Stay informed about: Explorer Eating Memory When Viewing VERY Large Folders 

Alec

Files sitting in a folder will not "eat memory" -they will only eat
memory if they are loaded into memory at the same time! If files
are linked, as can be with Excel, then small files when combined
can be a drain on limited available memory!

If a user is viewing files in a folder and does not close a file
after seeing the contents then this will also waste available
memory.

My comments are directed to the original question

"My question is there a fix that allows windows to handle this amount of
data better? (without having to use unix)"

The original question was asking about using existing memory
more efficiently whereas I am suggesting the user makes the
task less demanding of memory.

--

Regards.

Gerry
~~~~
FCA
Stourport, England

Enquire, plan and execute
~~~~~~~~~~~~~~~~~~~

"Alec S." <@> wrote in message
news:eEuvPGh%23GHA.1220@TK2MSFTNGP05.phx.gbl...
> "Gerry Cornell" <gcjc.TakeThisOut@tenretnitb.com> wrote in message
> news:uWwP5RZ%23GHA.4320@TK2MSFTNGP02.phx.gbl...
>> You could look at your problem from the opposite direction! Why do I
>> need to have so many large files?
>>
>> Linking files unnecessarily can also be problematic. I encountered this
>> problem one time using Excel. You can use a macro to update
>> information thereby avoiding permanent links which load into memory.
>
> They didn't say anything about large files, just a large number of files
> and folders. As to your question, that's just how some
> programs are. I've seen lots of big programs (Macromedia apps, Maya,
> etc.) that come with tens, even hundreds of thousands of tiny
> html files in hundreds or thousands of folders. Usually they comprise the
> extensive documentation scattered amongst thousands of
> files (which eat up slack space like crazy!) instead of one large
> consolidated file (like hlp, chm, pdf, doc, etc.)
>
>
> --
> Alec S.
> news/alec->synetech/cjb/net
>
>
 >> Stay informed about: Explorer Eating Memory When Viewing VERY Large Folders 

"Gerry Cornell" <gcjc.DeleteThis@tenretnitb.com> wrote in message news:u32bc5h%23GHA.4464@TK2MSFTNGP02.phx.gbl...
> > Usually they comprise the extensive documentation scattered amongst thousands of
> > files (which eat up slack space like crazy!) instead of one large
> > consolidated file
>
> Files sitting in a folder will not "eat memory" -they will only eat
> memory if they are loaded into memory at the same time! If files
> are linked, as can be with Excel, then small files when combined
> can be a drain on limited available memory!

I meant on the disk. When you've got thousands of small files, the slack space—the unused part of the last cluster of each file—can
add up to quite a bit of wasted space, especially with the larger cluster sizes that are used these days (8KB clusters are the
minimum for partitions>50GB). Of course I'm talking FAT here, but if I remember correctly, NTFS has waste as well.

> If a user is viewing files in a folder and does not close a file
> after seeing the contents then this will also waste available memory.

Explorer is pretty bad in the way it handles files. It leaves open handles, allocated memory, etc. That's why the memory usage of
explorer.exe goes up and up over a session, and files (particularly graphics files) become locked and cannot be deleted.

> My comments are directed to the original question
>
> "My question is there a fix that allows windows to handle this amount of
> data better? (without having to use unix)"
>
> The original question was asking about using existing memory
> more efficiently whereas I am suggesting the user makes the
> task less demanding of memory.

I was referring to your question "You could look at your problem from the opposite direction! Why do I
need to have so many large files?" It may not be up to the user at all; it is not necessarily the design that the OP has created,
but rather the way that a program they use ships, and they are getting peeved by the usage caused by the thousands of files that the
program has.


--
Alec S.
news/alec->synetech/cjb/net
 >> Stay informed about: Explorer Eating Memory When Viewing VERY Large Folders 

Alec S. wrote:

> I meant on the disk. When you've got thousands of small files, the
> slack space-the unused part of the last cluster of each file-can add
> up to quite a bit of wasted space,


Note that, despite what many people think, the size of the file is almost
completely irrelevant in this regard. Unless the file is an exact multiple
of the cluster size, on the average, the last cluster wastes half a cluster.


> especially with the larger cluster
> sizes that are used these days (8KB clusters are the minimum for
> partitions>50GB).


FAT32 clusters larger than 32GB have, by default, 32KB clusters


> Of course I'm talking FAT here, but if I remember
> correctly, NTFS has waste as well.


Yes, that's correct.

Back in the days of small expensive hard drives, the waste due to slack was
an important consideration. But in these days of very large inexpensive hard
drive, the actual value of that waste, in dollars and cents, doesn't amount
to a hill of beans, and can be safely ignored by most people.

--
Ken Blake - Microsoft MVP Windows: Shell/User
Please reply to the newsgroup
 >> Stay informed about: Explorer Eating Memory When Viewing VERY Large Folders 

"Ken Blake, MVP" <kblake.TakeThisOut@this.is.an.invalid.domain> wrote in message news:eOfw%23Ni%23GHA.3256@TK2MSFTNGP02.phx.gbl...
> Alec S. wrote:
>
> > I meant on the disk. When you've got thousands of small files, the
> > slack space-the unused part of the last cluster of each file-can add
> > up to quite a bit of wasted space,
>
>
> Note that, despite what many people think, the size of the file is almost
> completely irrelevant in this regard. Unless the file is an exact multiple
> of the cluster size, on the average, the last cluster wastes half a cluster.

Bingo. Of course that's a universal, global, theoretical really, average. In practice it could be anything since no two systems
are alike.

> > especially with the larger cluster
> > sizes that are used these days (8KB clusters are the minimum for
> > partitions>50GB).
>
>
> FAT32 clusters larger than 32GB have, by default, 32KB clusters

I'm not sure about the default, it depends on the partitioning software (some use one, some use another), but there are definite
minimums depending on the size of the partiton. In fact the size of the FAT itself could (in the past) become and issue. A few
years ago I had to make a partition of 49.99GB (or something like that) to avoid moving up one whole cluster size for just a few
hundred extra meg in the partition. Moving up a size would have been quite determinetal since it was for games which tend to have
thousands of small files. I actually tested it and there was more than ½GB of slack with the larger cluster size; that's a whole
extra game! (I don't remember what the record I've seen is.)


> > Of course I'm talking FAT here, but if I remember
> > correctly, NTFS has waste as well.
>
>
> Yes, that's correct.
>
> Back in the days of small expensive hard drives, the waste due to slack was
> an important consideration. But in these days of very large inexpensive hard
> drive, the actual value of that waste, in dollars and cents, doesn't amount
> to a hill of beans, and can be safely ignored by most people.

It's worth thinking about (considering what will be on it) for a new drive, but for existing ones, the time it takes to change it,
and the risk of data loss usually make it less than desirable. Although, if you are running low on space, those extra couple of
gigabytes being wasted amongst the partitions can look awfully good.


--
Alec S.
news/alec->synetech/cjb/net
 >> Stay informed about: Explorer Eating Memory When Viewing VERY Large Folders