Soap making traditionally involves the use of sodium hydroxide (a base, as you note), commonly referred to as lye. This is indeed a caustic base and can burn the skin.

Wikipedia has a pretty good site on the chemistry and use of sodium hydroxide and links out to soap making, which you might want to check out:

The initial steps in making soap involve taking a fat, such as olive oil or a rendered animal fat, and reacting it with lye (the base). This gives a soap that firms up as it cures. It is the "saponification" step that basically uses up the sodium hydroxide - if this is not done properly, then there is excess base remaining and indeed, this would burn your skin. So, soap makers have precise methods for mixing the fat and the base. There are two basic saponification methods, cold and hot, but either way, saponification is the hydrolysis of an ester (the fat, like a triglyceride) under basic conditions (the sodium hydroxide) to form an alcohol and the salt of a carboxylic acid. Soap makers can take the soap through a final step of purification and finishing, removing any excess salts, base or other impurities. By the time you purchase and use the soap, it is no longer a (caustic) base.