I have worn a plain hat for most of my adult life.  It identifies me with other plain Christians.  I never use it as an instrument of belligerence or insubordination; if other men remove (or should remove) their headgear, I remove mine.

I never wear my hat in prayer.  Since meeting for worship is in its entirety an act of prayer for me, I never wear my hat in meeting.

Incidentally, I have saved myself from many cracks on the head (on the farm) by wearing my hat; it often gives a second or two warning when I will be at risk.