This describes how I am thinking about backup and ftp procedures on 11/22/00.
First off, I should say that I don't like tape backup. The only acceptable backup for me is media that I can address directly just as if it were on my hard drive. So, CD's, Iomega cartridges, floppies, and my laptop computer are all acceptable backup media. My backup includes 18,000 files and 600mb of data, not including anything easily recovered from anybody else's distribution media, and I've never lost anything important, nor have I ever been hurt by work going in a wrong direction, so these procedures are carefully developed.
Backup media is often limited in capacity, so we need a way to draw a circle around what is to be backed up. I do that by giving each backup cartridge (or whatever media) a volume name and then I have a little database (called BKPINST.SF) that delivers any limits I want to impose as to what goes on that volume. Usually that is a list of directories either in root or one level down on my hard drive. This small database is essential if I am to use blank media for backup. But I sure want this database to be as small as possible because it has to be maintained. There is only one BKPINST.SF file and it is located in my normal working directory (quad-CHDIR' ').
Lots of times files & workspaces (apl & other) get created that you really don't want to backup. Sometimes it is because they are temporary, sometimes because you can get them back easily from somebody's distribution disk. One of the clues as to what should be backed up is what is already on the backup media. So, the easiest backup mode for me is to copy from the hard drive to backup media exactly the list of folders and files that are already on the backup media (and that have been changed more recently than the date last changed of the file on the backup media). This is an important feature because I can do backup without thinking. Simply start it, maybe 40 times a day. In this mode the backup functions do not have permission to copy any file or folder that is not already on the backup media, and, of course, the last changed date must be older on the backup media. If the backup media is blank, the functions follow BKPINST.SF, or if there are no limits in BKPINST.SF and the backup media is blank, then everything is copied.
If I allow the backup functions to copy folders or files that are not already on the backup media (provided it is not blank media), then the backup functions must query on every new folder or new file). This is a bit tedious, but I consider it important to keep from proliferating unnecessary files. In this mode, if the backup functions see a file on the backup media that is not on the hard disk, then I am queried as to whether or not I want to erase it from the backup media. This is also important -- to eliminate obsolete files that otherwise will waste my time and create future conflicts.
All queries have defaults that are very carefully developed. Since there will sometimes be many queries, I want to be able to simply click carriage return to accept them. These queries could be handled as a list at the beginning of each folder, but I haven't done that yet. Since the queries have carefully developed defaults, I have the option to "RunFast" so that all queries are answered by the defaults.
I have seen the idea of "synchronize" built into file transfers, so the procedure actually goes both ways. I truly, deeply, strongly would not want this idea to exist in my file transfer procedures because the mindset for transfers in the two directions is quite different. In my world, there is always one computer that is more to be trusted than the other. All files are "owned" by somebody, meaning that person (or computer) is supposed to have the most dependable copy. "Ownership" can frequently change when people are collaborating. But I think it is always important to know who the "owner" is.
With all this description, I now say that I believe this is how my websites should by kept up to date relative to what is on my desktop computer. I presently have about 10,000 files on my server totalling about 120mb. All the problems of backup also exist in maintaining websites. Whereever the words "backup media" exist above, we can substitute the word "server".
Notes on backup for John Gregg: There are three kinds of backup in my view.
I don't know if I made it clear or not, but all my forms of backup provide files that I can access just as if they were on my hard drive. I despise tape backup because you never know whether or not its good til you have to recover it, and then you've got to make sure you don't clobber your live stuff. And, how do you know its any good til you need it.
I can run my systems on or with my backup. So I know darn well that it is good.
I want to be able to retain backups as of various dates. Since I work very aggressively with large databases and utilities used throughout many unrelated systems, it's a comfort to me that I can go back to lots of past dates and reshape the evolution of functions or databases in case I discover much later that I don't like where I wound up. Sometimes I'll change a utility for the system I'm working on this month and many months later discover that I failed to consider its implication in another system.
When I began this process I saved backups on 10mb Bournoulli cartridges which cost $70 each. Everytime I'd archive a backup I tied up $210 worth of media. Then I went to 150mb Iomega-150 units. By that time my backup requirements had grown, so now I would tie up 3 150mb units for a total cost of $150. Later I went to Iomega Zip-100 cartridges costing $13 each and each complete archive would cost $104. Now I use CD's costing 80 cents each and everything fits on one CD. I can have daily backups, if I feel the need, and stack a year's worth in a little box. Ten years worth of daily backups would fit on a shelf in a closet. It would be ridiculous for me to store ten years of daily backups, but this will help you see why I so delighted with my CD-RW and CD-R backups.
My backup procedures are working so well I'm about to enable them to erase files on the backup media. If they aren't there they were either erased or moved and shouldn't clutter up the backup nor should they force me into a tedious set of questions. Just poof them away. I won't do this soon, but when I'm ready, it is clear that the backup procedure is solid enough to justify it.
More interesting is that I now have always-ready-always-on-line backup by means of a HP CD Writer Plus. My backup procedure has criteria stored by the volume name on the backup disk (CD). All my current work in apl and in websites amounts to more than 400mb. If the volume name on the CD is "MyJewels", then the entire 400+mb that I have in the past selected are verified or copied to the CD by the backup procedures that are the result of your and my effort. An example -- the CD is always in the CD-Writer drive -- if I am about to try something experimental, I define "Backup" into the WS and execute it. It backs up anything in the 400+mb that has changed since the last backup (or if I want to save time I can tell it to only check certain folders). Then I try my experimental work. If it fails I can quickly recover whatever files I want to recover.
The HP CD Writer Plus is clearly new technology and has a lot of kinks to work out. But I am getting sufficient performance to justify it.
I'm having a lot of trouble with HP CD Writer Plus 8100i. Documentation of some of this trouble is below. Before you get to that, here's where I think we are. I've had another conversation with HP Tech Support since writing the message below. They told me that they support this unit with NT Workstation, but not with NT Server. The reason is that NT Server is more subject to interrupts which interfere with the functioning of the CD Writer. I want this capability, including the ability to address the CD as a normal drive where space is not lost or wasted when you write to it (referred to below as my on-line backup). Our options appear to be these.
HP appears to be having a great deal of trouble with this product. It appears to be the only HP product that has free 24 hour 7 day support. Also, I picked up from Tech Support I am having far less trouble with it than some other people are.
I think this will be a fantastic product when we get it together right. But we're not there yet. HP is internally confused about whether or not the HP CD Writer Plus 8100 Series is compatible with NT 4.0. I found a web page that says the 8100i is compatible with NT 4.0. The internal software used by the Sales Department called Yoda (probably an intranet) says the 8100i is compatible with NT 4.0. But I went to CompUSA and looked at an 8250 box and it said Windows 95/98 with NT noticeably missing. And the 9110i clearly says Windows 95/98 and also NT 4.0. The tech support staff says unmistakedly that they do not support the 8100i with NT 4.0 and have now told me politely that I should not call them anymore for help on this subject (case #1423676448). I've probably called them a dozen times and some calls were quite long. I'll probably have a very big phone bill this month.
I have been able to get quite a lot of functionality out of it. I really like Direct CD because with it I can operate directly on the CD from APL. The CD sits in the drive and I can make incremental backups anytime I want and they only take a few seconds. The changed file is replaced on the CD and no space is lost; it behaves quite like a normal drive. I'll call this on-line backup.
HP Simple Trax is also very good. It uses the same Direct CD software but allows you to put multiple volumes on the same CD. An example of its usefulness is I can put all my historical backups on a small number of CD's for easy storage in a safe deposit box or anywhere. Easy CD Creator is the way to do this so the archives can be read on any computer. (I think there is an error in my notes here since Simple Trax is built on DirectCD; Easy CD Creator has nothing to do with Simple Trax)
One problem I have with Direct CD is that you cannot copy it unless you have two HP CD Writer Plus units. A CD made with Direct CD is absolutely useless in my traditional CD drive.
Another problem is that it is very hard to write a volume label on the CD. I enter it in the appropriate place, but it usually does not write it on the CD (Direct CD). So I have to keep trying over and over until by accident it appears to write the volume name on the CD. Each attempt takes 5 minutes because the only provision I can find for writing a volume name is in the formatting procedure. HP tech support says it is much better to do the "full" format (60-90 minutes), but because of the problems I have writing the label on the CD I usually use the 5 minute "fast" format.
I've probably spent 50 or 60 hours using it and attempting to use it. I've learned a lot, but I'd say only a third of this time has been productive. So, I guess I've wasted 30 or 40 hours trying to do things that HP tech support says they won't support. Plus I probably have $100 extra on my phone bill.
The 8100i came with Adaptec 3.0 software. It had been modified by HP so that Adaptec does not support it. HP says they do not support Adaptec 4.0. However, 3.0 is very limited and we might prefer to be using 4.0.
Just for the record, here are some notes from my conversations, especially the last.
Never interrupt a formatting process (it ruins "it").
Everytime you format a CD, you should first erase it.
A CD life is limited to 1000 erasures and writes. In truth, it begins to wear out long before that as a shadow of old uses remain and may interfere with later writings.
HP Tech Support says it is better to always do a "full format" because that is a more "pure" way to work. However, since I'm having trouble writing the volume label perhaps it should be my practice to always depend on "fast format".
Closing a session means writing out the table of contents. Nothing can be added after this unless the CD is erased.
DOS is useless with the CD Writer or with any CD created on it (unless special procedures are invoked). I should definately avoid doing DOS copies with the CD Writer even if it seems to be working. Drag & drop works and, assuming we use Direct CD, working under the WIN32 API (as Bill Parke's software does) works. Easy CD Creator cannot be used under WIN32 API program control.
Direct CD uses a UDF format which involves packet writing. Packets are 600kb. The UDF formatting takes up some space so you can only use 490mb or so on the CD. Easy CD Creator lets you use 640mb or so because there is no formatting.
Simple Trax uses Direct CD.
Easy CD Creator should be used to archive files because then they can be read anywhere.
External USB devices don't work with NT.
When in doubt, use Easy CD Creator.
Always uninstall an application before upgrading to a new version.
An HP CD Writer volume label cannot be changed once the CD is written on.
CD Creator creates a disk just like any commercial disk. If CD-R is should work in all CD-Rom drives all computers. If CD-RW, it requires that the remote CD-Rom have multi-reader capability.