It appears you have thought this through and have nearly reached the answer on your own! You are correct in that you are mainly burning Mg to form magnesium oxide (MgO). However, as you noted, the atmosphere is 78% nitrogen, so if you burned the Mg in air, there is plenty of nitrogen around. As it turns out, nitrogen will react with Mg if it is hot enough. Mg burns very hot and is more than sufficient to get the reaction with nitrogen going to create some magnesium nitride (Mg₃N₂). When you add water to it, you form MgO and ammonia by the reaction:

Mg₃N₂ + 3 H₂0 --> 3 MgO + 2 NH₃
And that is why you could smell ammonia!

Here is an interesting variation of that experiment that you could try. Mg does not necessarily need O₂ to form MgO. In fact, if you light Mg and then dip it into CO₂, it will continue to "burn" according to the following equation:

2Mg + CO₂ --> 2MgO + C
By immersing it in an atmosphere of CO₂, you can eliminate the reaction with nitrogen so that no Mg₃N₂ is formed (or at least very minimal amounts from the transfer). What you get instead is a mixture of MgO (white powder) and black carbon ash.