Dirt is a non-scientific term for the "rock debris", often mixed with organic matter, that coats much of the land's surface.

Normally it is not used in technical discussion (the term is missing from my geological dictionary). It is dominated by very small particles (clay and silt), but may include larger diameter particles such as sand and even small to medium sized rocks or gravel. All of the inorganic particles, of whatever size, are indeed (as you knew) derived from the physical & chemical break down of pre-existing rock. The degree of their breakdown determines whether the dirt is coarse and gravel-rich, is sandy, or like much dirt in Santa Barbara, is clay-rich.

Dirt that is rich in silica and little else is often fairly sterile (poor substrate for plant growth) but dirt derived from rocks with many mineral types is often quite rich (by example, dirt derived from the erosion of volcanic rocks). The addition of organic matter from previously living plants, earthworms, etc., further enriches the nutritive value of dirt... and often we then address it more respectfully and call it "soil". This my geological dictionary defines as "The Earth material which has been so modified and acted upon by physical, chemical, and biological agents that it will support rooted plants."