This message can sometimes appear if you uninstalled RMC4 after having installed RMC5 or after updating to Windows 8.1.
Uninstall RMC4, if present, and reboot.
Launch RMC5.  Go to the Library tab.  in the menu, select Library->Settings.
In the Settings window, click the Internet tab and then the Monitor tab below it.
In the bottom section, "Ports to Monitor", click the circular button next to "Use Network Monitor".  Note: if you use VirtualBox, select Use WinPcap instead.  If the button is not active, you will first need to click the installation link to the right and complete that installation.  Click OK to close the Settings window.
Close and restart RMC.  Press the green START button and record something from YouTube as a test.