The downside of smart phones is their high cost and the high monthly service fees. Can't have a smart phone without a multi-giga-byte data pack, for which they want to charge you a bucket of money. If you just want to use it to control stuff in your house, you can just use wifi and dispense with cell phone service. Of course then you have to pay full price for the smart phone which makes it even more expensive.
Problem with smart phones is they don't have a real keypad, only a virtual one, so you have to look at it to operate it. No being able to locate the mute button by feel. The other problem is they take two hands to operate, one to hold it and one to push the virtual buttons. I could be wrong about this, I don't have a smart phone. Perhaps people who are familiar with them will have memorized the necessary contortions and can operate them without looking.
I was thinking what we need is a device that you could hold in one hand and would have a set of keys/buttons, at least one for each finger. The trick would be to hold onto it while operating the keys, which means at least a couple of keys are always going to have to be depressed. Alternatively, you could put a strap on the back that would go around your hand. Maybe add an automatic tether so that when you drop it, it automatically gets reeled in and held someplace convenient, like your waist, belly or chest, or your back, if you're a Ninja. If you don't need to see the screen you can operate it from it's stored position.
If you wanted to be fancy, you could employ a drone to carry it around for you. Simply raise your hand and snap your fingers and the drone would zip over and deposit your remote control right in your hand. Drop it and the drone would reel it in and hold it in readiness for your next command. Drones are a little noisy, so maybe a blimp drone would work better.