My Rav recently spoke about the minhag of wearing a kittel on Rosh Hashana during his dvar Torah/halacha in between mincha and ma'ariv. He mentioned an aspect of the minhag which I was familiar with but did not the reason behind it.
In short, there exists a custom to wear white on Rosh Hashana to symbolize:
1) Purity, like the posuk in Yeshaya 1:18, part of which reads "though your sins be as scarlet, they shall be white as snow; though they be red like crimson, they shall be as wool." Though we know we have been sinful and transgressed HaShem's mitzvos, we are confident that He will purify us from our misdeeds, whitening our neshamos as it were.
2) Burial shrouds, which are also white and very similar to a kittel (for those who have not attended a tahara, I can vouch for the resemblance). The purpose of dressing in a garment that resembles the one that is placed on a corpse for burial is to keep us mindful of the awesome holiness and seriousness of the day - and help us ponder where we might be in the the near future if we are not judged with life for the coming year, lo aleinu.
Women wear white clothing. Some men will wear a white tie or yarmulka, but the prevalent custom for men is to wear a kittel as their white garment.
However, a chosson in his first year of marriage - that's me - does not wear the kittel on Rosh Hashana. I had heard this before, but not the reason my Rav presented: the Torah tells us in last week's parsha of Ki Seitzei that a newlywed man has a special mitzvah to gladden his wife during their first year of marriage, which is why he does not go to the army (Devarim 24:5). Instead, he stays home to be mesameach her.
This is the very reason why the chosson doesn't wear the kittel during shana rishona - because their resemblance to burial shrouds might make his wife start worrying about him too much, and he is under special command to make her happy, not cause her grief!
Sufficed to say, my kittel will be staying right in my closet until Pesach.