The only time your Phone Number will really be at risk of being Hijacked, is usually if there is some kind of financial gain for the criminal or criminals. For example they can order a new iphone or android phone at your expense, or they can hijack your cell phone plan to eventually get access to your bank information and such. This has happened in the past with very well known and high ranking investors and individuals, who have had their cellphone account hijacked to gain access to their bitcoin account or bank account and had the funds withdrawn. But yes even a small risk like the criminals ordering a new phone is still a risk to you and any method you do to safeguard your account will help ensure it is secure.
One thing to remember, the more information you share with your friends about you, if one of them likely is a bad apple, they can do some no good or give the information to someone else who can be up to no good. Do not share email passwords, do not share voicemail passwords or facebook passwords or banking information or bank pin numbers, etc. Usually its these folks who do these things are the most likely to have their cellphone accounts or other personal accounts hijacked.