1.Preheat oven to 180°C. Grease a 19cm square cake pan; line base and sides with baking paper.
2.Combine chopped butter and brown sugar in a small saucepan, stir over low heat until smooth; pour into base of cake pan. Place nectarine wedges, cut-side down, over caramel mixture.
3.Beat softened butter, extract and caster sugar in a medium bowl with an electric mixer until light and fluffy. Beat in eggs, one at a time. Stir in sifted flours, and milk, in two batches. Stir in ground almonds.
4.Spread mixture into pan; bake about 50 minutes. Stand cake in pan 15 minutes before turning onto a wire rack to cool. Serve sprinkled with flaked almonds; drizzle with any juices left in pan.