Your Google devices may have a metaphorical ear to your life even when they're not active.
Google's voice search feature stores a history of all your voice searches, but because of its simplicity (saying Ok Google to activate) it's also storing information gleaned from conversations where you weren't talking to your device at all.
Google is already under the microscope for the vast stores of location data it keeps.
Luckily, this particular data, like your normal search history, is fairly easy to delete. Just go to the voice history page and delete it all through menu commands.
Peace of mind--for a moment anyway.