There is actually more than that stored in the server's database. Push messaging IDs are there, for example. But ignoring that, let's say the attacker is watching all the Signal server's connections. Who is talking to who can be determined based on size, direction and timing of traffic between clients and the server. The server could also be modified to log it.
The idea with projects such as Vuvuzela is to make metadata less usable.