The Small Key is the key to open the underground door on the two bales of hay in Ye Olde Rumour Mille. Push them all the way to the right and then jump on top and stand between them. You should fall down to the ground. Then push the left bale back to the left side and you will reveal a secret locked trap door in the ground. Use the small key in your backpack to open it and go down.
How to Get the Small Key
When you found the secret door on the Fountain, go down. There are several people here wearing purple robes. Walk to the right and talk to the boy with buck teeth and pimples. He will give you a small key.