Once the battery is full (really 80% full), the Prius is just like any other car. Energy of motion and potential energy from the hill are converted to heat the air being pumped through the engine and the brake pads/disks/drums.
Actually, the Prius is a little (a lot !) more subtle than other cars.
I recently experienced what happens when the battery is "very full" thanks to some Pyrenean passes and to the wonderful Prius Mini-Scanner (only available for the "old" one so far).
There are two options :
- if you are in "D" mode, below 80% state of charge (SoC) there are some amperes going to the battery when you're driving downhill, both coasting or using the brake pedal (of course using the brake pedal increases the amperage). The engine can be either at stop (low speed) or spinning at about 1000 rpm at medium speed (don't remember exactly the threshold). When getting to 80% SoC, there is negligible amperage to the battery and the engine is either started (if it was stopped) or accelerated (if it was idling) but with no fuel. My guess is that electricity generated by MG2 (the "big" motor) is then used to spin the engine instead of charging the battery, and consequently this electrical energy is wasted, of course.
- if you are in "B" mode, you can also see a negligible amperage to the battery when reaching 80% SoC, and the engine (which was already spinning) is accelerated to provide even more braking than in the "D" case. Again, energy is wasted because there's nowhere to store it...
All these transitions are seamless ; you would probably ignore them (except if you pay attention to the engine noise) without Mini-Scanner.