OpenVZ is a container based virtualization where your VPS exists in an isolated area of the host node and shares the host's kernel. This provides bare metal performance with low overhead.
KVM is more like what you might think of if you've heard of VirtualBox or VMWare. All of the hardware is virtualized and resources are more dedicated than is typical with OpenVZ. KVM clients do not share a kernel with the host node, but rather maintain their own.