Pre-amplifer for magnetic phono pick-up.

[Mike Watterson]

So where would you expect the ultrasonic signals to come from?

No idea, but the cartridge might be microphonic and someone might have an "Ultrasonic Pest Repeller", which actually don't repel pests, though may annoy dogs, guinea pigs etc?
I've heard RF pickup, Taxis, Police etc in 1970s on record players.