Heat milk until just simmering. Steep the earl grey tea bags and lavender flowers in the milk for 30 minutes - this creates a very concentrated infusion that flavors the cake. Remove the tea bags and strain out the lavender in a fine mesh sieve. Let cool to room temperature.
Preheat oven to 375 degrees. Grease a 9 x 5 inch loaf pan well with oil or butter.
Toss the lemon zest in the sugar to help the lemon flavor incorporate more evenly into the batter. With a stand or hand mixer, beat in butter until soft and fluffy, about 2 minutes. Add eggs one at a time, whipping until fully combined.
Measure in flour, baking powder, and salt and beat until just mixed.
Pour batter into prepared loaf pan. Bake 35-40 minutes until center is set and a toothpick inserted comes out clean. Remove from oven and cool in the pan for 5 minutes, then carefully flip onto a wire baking rack to cool completely.
When the cake is cooled, prepare the glaze. Whisk together powdered sugar, lemon juice, honey, and vanilla until thick but silky. If the mixture is too thick, add 1 tsp milk or water until smooth and glossy. If too thin, add 1 tbsp powdered sugar at a time. Drizzle over the cooled cake.