Here I am distinguishing "children and other people" from fellow collectors. This blog entry is about mineral-naive people who may interact with your collection. (The classic apocryphal example is the maid who carefully dusts your millerite geode to get rid of those hairs in it.)
Some mineral specimens should not be touched; others should not be picked up. Etiquette for guests is to ask permission first, and to hold the specimen close above a soft surface (such as your free hand) while you examine it. Heavy minerals (including galena and gold nuggets) are easily dropped. Some crystal tips can cleave off, like apophyllite or even topaz.
Attractive specimens sometimes disappear when visitors are allowed free access to a collection. If you like to give samples to encourage children (which is a very good thing to do), do not let them confuse ones to which they are welcome with ones you would rather keep.
Mineral specimens can hurt people also. Okenite specimens are sometimes demonstrated as "pattable" furry minerals; but related species such as pectolite will leave slivers in one's fingers. (And other velvety mineral specimens, such as manganese oxides, are easily crushed by patting them.) It's wise to wash your hands after handling minerals, especially ones you don't know, or ones you do know to be poisonous.
A mineral collector should not expose naive or easily frightened guests to radioactive minerals, asbestos-habit amphiboles, or arsenic ores.