A virtual particle is an abstract construct to describe interactions between actual particles, and arises from perturbation theory in quantum field theory. It is not actually a particle at all, but something that pops up in the math. So, what does that all exactly mean?

Perturbation theory is a technique to use known solutions of a simple problem to approximate a more complicated problem that looks like a weakly perturbed version of the simple one. The interaction of actual particles (e.g., electrons, photons) is abstracted to be the exchange of virtual particles. Examples of such interactions include the Coulomb force, magnetic field, and strong nuclear force. You are probably most familiar with the Coulomb force, which in classical mechanics is: Coulomb Force (click here)

It turns out you can also cast the fields from Coulomb interaction in this framework of virtual particles. Virtual particles are abundant in what are known as Feynman diagrams (click here) which are pictorial representations (click here) of the integrals used to calculate these interactions.

This is a pretty good example of a common phenomenon in quantum mechanics and related fields where the English word is much less precise about the mathematical description of some physics. It's important to decouple what you ordinarily associate with words colloquially (i.e., in everyday conversation) from what the physics is communicating to you. If you wanted to attach physical significance to the concept of a virtual particle, it is more useful to think of it not as a particle at all. Rather, you can think of it as a disturbance to a field. For example, in the cased two interacting electrons, (click here) it is said that they "exchange virtual photons." They don't actually exchange any photons; it is a disruption to the electromagnetic field of the electrons (e.g., Coulomb repulsion, magnetic field due to moving charge). It turns out that the mathematical description for this interaction happens to also describe photons as well, hence the name.