Preheat oven to 180°C.
Combine baby potatoes, olive oil, salt, paprika powder, ½ teaspoon crushed peppercorns, garlic, thyme in a bowl and mix well. Set aside for 10 minutes.
Place the baby potatoes mixture on a baking tray and put the tray in the preheated oven. Bake for 15-20 minutes. Remove from heat and set aside.
To make lime mayonnaise, combine mayonnaise, lemon juice, lemon zest, salt and remaining crushed peppercorns in another bowl and mix well. Refrigerate for 10 minutes.
Place the baked potatoes in a serving dish. Garnish with a thyme sprig and serve hot with lime mayonnaise.