Don't do this. ^^^
If you signed the lease, its a binding contract, unless the lease goes against the local laws. The law only enforces what is REASONABLE. So if it says "the landlord has the right to enter at any given time", they might enforce that. But if it says "All renters must cut off their balls after living in the apartment for a week", the law wouldn't enforce that because its unreasonable.
I would change the locks, its cheap and easy to install. Just let him know how it is when he asks about the lock change.