Preheat your oven to 375°F (190°C). Lightly butter a 2-quart baking dish and set it aside.
In a mixing bowl, whisk together the milk, heavy cream, salt, pepper, cinnamon, and nutmeg. This will become the creamy layer that brings everything together.
Layer half of the sweet potato slices in the dish, overlapping slightly like you’re making a small rainbow. Top with half of the apple slices. Drizzle a little of the cream mixture over the fruit and potato, then sprinkle half of the brown sugar and half the cheddar.
Repeat with the remaining potatoes and apples. Pour the rest of the cream mixture over the top. Dot with small chunks of butter and finish with the remaining cheddar and a final dusting of cinnamon.
Cover the dish with foil and bake for 25 minutes. Remove the foil and bake another 25–30 minutes, until the potatoes are tender and the top is golden and bubbly.
If you like a deeper, more caramelized top, turn the oven to broil for 1–2 minutes at the end, watching closely so it doesn’t burn.
Let the dish rest for about 5–10 minutes before serving. This helps the flavors settle and makes the slices easier to lift from the dish.