Preheat oven to 350 degrees. Grease 2 9 inch cake pans with butter or oil.
Sift the sugar and lemon zest together in a shallow bowl until combined.
In a large bowl, beat the butter and lemon sugar until fluffy, about 3-4 minutes. Beat in the eggs, one at a time, until fully incorporated. Whip in the vanilla and Greek yogurt. Add the flour, baking powder, baking soda, and salt. With the mixer running, slowly stream in the buttermilk, mixing until just combined.
Divide the batter evenly between the cake pans. Bake 30-35 minutes until the center is set and an inserted toothpick emerges with just a few moist crumbs. Let cool in the pans for 10 minutes, then run a knife around the edges to release the layers and flip onto a wire rack. Cool completely.
When the cakes are cooled, make the frosting. In a food processor or with a mortar/pestle, blitz or crush the sugar and lavender leaves until a powder is formed. It won't be superfine, but the flowers should be less chunky. In a bowl, whip the butter, lavender sugar, powdered sugar, blueberry/blackberry jam, and vanilla on low speed until combined. Add in 1/4 cup heavy cream and mix to combine, then increase the speed to medium high and whip until very light and fluffy. If needed, add more heavy cream 1 tablespoon at a time until the frosting is spreadable and soft.
Place one cake layer on a flat plate or cake stand. Frost the top generously, then add the second layer. Frost all the way around the cake until completely covered. If you have extra frosting, feel free to pipe little decorations onto the cake. Finish with more lavender leaves, as desired.