Your question gets right to the heart of some interesting issues in optics.
Your second thought comes closer to what is happening to the light in water. Water preferentially absorbs light at longer wavelengths (like red). This means that as white light from the sun travels through the water, the redder components are being absorbed. All wavelengths are absorbed to some degree by water so as water gets deeper, it gets to be a deeper blue until it is essentially black (except for some light that is scattered by plankton and such in the water). This still doesn't explain why we often see water as blue. As an example, think of photographs you may have seen of the Great Barrier Reef around Australia. In these pictures you see large areas of beautiful blue-green water with dark splotches. The brilliant blue-green water areas are actually areas where there isn't much life, only white sand which reflects the light back through the water. The red light is preferentially absorbed so the light that makes it to the bottom and back appears bluish-green. In reef areas, colored corals and kelp absorb blue and green wavelengths while the water absorbs the red wavelengths, leaving very little light to return to your eye so these patches appear dark. You may also notice that as the water gets deeper, the absorption is greater and less light is returned. This is an illustration of the Beer-Bourget-Lambert Law which states that the amount of light received after passing through a medium is equal to the amount of light going into the medium multiplied by e^(-ad) where a is an absorption coefficient related to the absorbing medium and d is the path length through the material. This shows that the thicker your material (e.g. the depth of water multiplied by 2 because you have to go both directions), the more of the original light you will lose. Note that this law depends on wavelength of light so separate calculations must be done for each wavelength you're interested in. As an example, the absorption coefficient of fresh water is 1.709X10^-4 for blue light and 2.45x10^-3 for red light where absorption coefficients are in inverse cm. You might ask your students to calculate the amount by which incoming light is reduced (if they're comfortable with the exponential e^(-ad)) for a white painted swimming pool at the shallow end (1 meter or 100 cm -- you need to keep the units consistent) and the deep end at 6 meters. I hope this is thorough enough for you.

Ok -- I cannot state this with any certainty -- but I think the ocean is blue as a reflection if the sky -- my evidence for this is a couple of observations of when it appeared blue this last week, namely when the sky was largely blue as well. Currently, the sky is overcast, and the ocean is grey, with a hint of green -- obviously from near surface absorption from fine media. (Note, when the media is larger, the ocean can appear brown, black, gray etc, depending on the amount of dispersed particles and their type.)
I know this is not a definitive answer -- but there are known color charts for the deep ocean and I think the model of color dispersion works pretty well here, although it may not work for the sky...

Both the sky and the ocean are blue, but for two very different reasons. The sky is blue because of something called Rayleigh scattering. For very small particles such as individual molecules, the wavelength of light that is reflected is a function of the particle size and type of molecular bonds between atoms. It turns out that air molecules are just the right size and the bonds vibrate in such a way that they scatter blue light the most. This causes the sky to be overwhelmed with blue light. When you watch the sky turn different colors at sunset, you are watching light become increasingly scattered by more atmosphere and larger particles (dust) as the sun descends toward the horizon. On cloudy or foggy days, the water droplets in the atmosphere both absorb the light and scatter all wavelengths equally, causing a grey or white sky.

The ocean is not blue because of scattering. Instead, the ocean is blue (as is pure water) because of selective absorption. Liquid water absorbs preferentially both short wavelength light (UV) and longer wavelength visible and infrared light. If you drop a light sensor that measures the amount of light at different wavelengths into the ocean or into a lake, what you find is that almost all of the UV (280-400nm) and infrared (heat) wavelengths are absorbed. Of the visible light, the longest visible wavelengths are absorbed most (red-orange, 600-700nm), then the medium-length visible wavelengths (yellow-green, 500-600nm) and last the short visible wavelengths (blue-violet, 400-500nm). This explains why very pure water, without many particles, appears deep blue. It also explains why, as you descend with depth in the ocean and the light has to travel further and further though the water, light will weaken and appear more and more blue (you may notice this diving or when looking at underwater
video or pictures).

Particles and dissolved substances, depending on their size and physical properties (including the type of molecular bonds), will scatter or absorb visible light at different wavelengths and actually change the color of the water. Tropical water, which has few
particles and very little color-absorbing dissolved substances, appears deep blue. Water in very productive areas of the ocean with lots of algae will appear green. (James Joyce described the Irish Sea as "snot green" in Ulysses.) Water with a lot of sand or suspended particles will appear brown, and often river water, which
has lots of color-absorbing dissolved compounds and suspended mud, will appear yellow-brown. Scientists actually use this information to study the ocean from space--by measuring the wavelengths of light reflected from the surface of the ocean using satellites. In this way they can measure temperature and estimate the amount and kind of particles suspended in the water.