Yeah, metallic lead is pretty safe to handle overall, as long as you don't inhale the dust. I wouldn't want to go trying to ingest it, but the amount of incidental exposure you'd get from handling, say, a lead brick at room temperature, is not much of a worry. Wash your hands after handling it, wear proper PPE, and you should be fine.
Edit: this isn't counting stuff like using lead solder. In that case, ventilate the crap out of your work area (preferably doing the work under a hood), wear PPE, and clean any equipment used.
Leaded solder is no more dangerous than lead bricks. The lead never vaporizes (look up the boiling point of lead if you don't believe me), so the same hazards of skin contact and lead dust are the relevant ones. And lead dust is less of a problem for solder because of the alloys used being a lot less prone to making dust than lead bricks (source: extensive personal experience with both).
The fluxes used in solder are far more dangerous than any lead.
However, carefully washing one's hands only goes so far. Your work area will still be covered in small amounts of lead residue just from all the transfer via tools and fingertips. It is very reasonable to avoid leaded solder entirely these days in favor of newer alloys like SN100C, which have performance very close to Sn63Pb37 (though not the same).
I’ve switched to lead-free solders for all new work, but still do any repair/rework of old products with leaded solder as I’ve never found a lead-free solder that works well (plus, I’ve now got 5 lifetimes’ worth of leaded solder supplies).
Have you found any that work well in repairs/cross-contamination cases? I suspect a lot of the early complaints about lead-free solder being a dire problem functionally were from people using the wrong temp or cross-contaminated tools. (I use a cheaper segregated station for leaded as I got tired of changing tips on the nicer station.)
...honestly we play pretty fast and loose with mixing alloys around here. We really shouldn't since it can be very bad, as you know.
The three RoHS alloys I have experience with are SAC305 (blech, no idea why people tolerate this stuff), SN100C (seems to be great on new assemblies, never much tried mixing it), and REL22 from AIM (this is on our "high strength" spool holder, next to the Sn62Pb36Ag2 silver solder, so people tend to graduate toward it for no better reason) which seems to do pretty well in typical lab conditions after being used to rework either SAC305 or SnPb joints. We don't do real testing around here.
When I am knowingly mixing alloys I will try to clear or flood the joint or both (remove as much old solder as possible with braid, or add new solder in excess). We are pretty good around here about using Way Too Much flux, which is of course the correct amount. This might help. Either way we seem to do okay for this light use.