Patients such as yourself bring up a good point. What if you would be worse off without this painkiller? What if by not taking it, your pain level goes up while the quality of your life goes down? It's possible the drug is helping you maintain your current level of function.
On the other hand, there is always the concern of addiction and other side effects from the long-term use of this drug. Sometimes a drug holiday is needed to find out where you are in the process and maybe even reset the system. Taking a break from the medication could be a very good idea.
It may be best to follow your doctor's advice. He or she should help you slowly come off the drug to minimize the side effects of withdrawal. Other nonopioid medications for pain control can be used at the same time until you see how much more pain (if any) you are still having without the drug.