People with visual impairments face a variety of obstacles in their daily lives. Recent work has identified specific physical privacy concerns of this population and explored how emerging technology, such as wearable devices, could help. In this study we investigated their physical safety and security concerns and behaviors by conducting interviews (N=19) with participants who have visual impairments in the greater San Francisco metropolitan area. Our participants' detailed accounts shed light on (1) the safety and security concerns of people with visual impairments in urban environments (such as feared and real instances of assault); (2) their behaviors for protecting physical safety (such as avoidance and mitigation strategies); and (3) refined design considerations for future assistive wearable devices that could enhance their awareness of surrounding threats.