• Home
• About Us
  News Archive Partners Contact Us Acknowledgments
• Systems
  maya Other systems
• Research
  List of Projects Publications Posters Presentation Handout Supporting Materials
• Resources for HPCF Users
  System description Using your account Tutorial: Compiling C programs Scheduling rules Tutorial: Running programs Other Languages and Hardwares Other Packages General Productivity Usage rules Account request form Help request

As you conduct your work on the cluster, you will use more and more of your allocated space, and may not always be aware of how the space is being used. Inevitably, free space will run low and you will need to do some cleaning up. On this page we will provide some help to identify how your space is being used.

One very important place to keep an eye on is your home directory. Many programs use space in your home directory to store preferences, settings, etc. If you don't check occasionally, you may not be aware of how much space is being used this way. Also, if you run out of space in your home, you may begin to get mysterious errors which do not appear to be related to disk space. For example, users often see "X11 connection rejected because of wrong authentication" when trying to run X Windows programs remotely.

Before reading this page, make sure to read Using your account so that you know the location of your storage areas, and how to do basic checks of quotas and partition sizes.

Compression on some filesystems

Some of our filesystems feature automatic compression - namely the attached 160 TB of central storage. Compression of your files on the storage device happens behind the scenes, without any action needed from you. This is very convenient for saving space but can also be slightly confusing when examining the sizes of your files.

For example, suppose we have a large file in the User Workspace portion of our PI's storage.

Notice that output isn't quite consistent. The "ls" command says that our file is 2 GB, but the "total" is 770 MB. Then the "du" command reports 770 MB in use as well. The reason for this discrepancy is that the file "train.dat" has been compressed from its original size (about 2 GB) to 770 MB. Some commands are displaying the actual storage used on the device, and some are counting the number of bytes.

For the purposes of managing disk space, in all likelihood you will want to refer to the actual storage used. The "du" command will usually display this, unless you use certain flags like "--apparent-size" or "-b". See the man page "man du" for details, or stick with "du -h" to be safe, or contact us if you have questions.

"topdu" script

We have provided a commmand called "topdu" which computes the top N largest files / sub-directories in a given directory. This can help you quickly identify where your space is being used the most. If we run the command without arguments, it will give us the top 10 largest files and directories in our current working directory. Notice that the units are in KB. Let's run this in our home It's easy to see that the IDL program is using the most space in home in the output above, at about 24 MB. We can also check other directories besides the current working directory, and also optionally request a different number than top 10. We can also suppress the text at the beginning, to use the output in other scripts (for example). To see all the available options Here's an example using some of the options We can also set up a BASH alias if we'd like, for quick use of a frequently used command