You do not have to read them, but losing the book will result in losing the bonus. Reference books provide the bonus by simply being in your inventory. You can purchase them from Book Merchants that are found in taverns. Their primary purpose is to give boosts to your skills or attributes without having to level up. Books can be very useful in the Mount&Blade series.
