I need to look into this and how I've got things set up. A few years ago, we had an instrument used at several laboratories that created 8000 to 10000 separate files for each "run". It ran daily. Eventually, problems occurred! They wanted to move, combine, or delete files from the SINGLE directory that the machine put the files into by default! But alas, when you clicked on that folder with windows explorer, it appeared that the computer would crash. In actuality, what was happening was that it was trying to prepare the GUI view of the contents of the folder for display. That particular folder had about 2.8 million files in it at the branch were I worked!
I told one guy in one of the other labs what the problem was and he decided to see if windows could choke through it all eventually. He clicked on the folder from hell, and then just let the PC run over a long weekend. Three days later, it was still thrashing with no display, and yet another "power switch reboot" had to be performed.
I could look at the folder in a CMD window where no GUI was in the way, and see what was going on.
I ended up writing a program in VB6 to deal with these folders from Hell. It would work through the files and move them into subdirectories for each day, then month, then year until it had things organized for them. It would do this in just a few hours, usually, because I didn't set it up to display too much as it did its job. I gave the program a name that I felt was appropriate: Die Klistierspritze. Google translate is your friend!
All of the labs used it and liked the results. Then later, we found out that nobody had any use for these files, anyhow. They were keeping them because the EPA told them that they must keep these raw data files in case they ever had to reproduce their results. As it turns out, the whole system was proprietary to the point that even the manufacturer of the instruments couldn't reproduce the raw data for reanalysis even if they had the files. Typical!
