Memorial Day (originally called Decoration Day) was first recognized in 1868 to honor the Confederate US Civil War dead. Other veterans have "adopted" the holiday to "be honored" as well, although the true holiday for all veterans is Veterans Day.
Armed Forces Day (sometimes confused with Memorial Day) is also celebrated on the third Saturday in May.
The name Decoration Day comes from the original tradition to decorate Civil War graves with flowers or flags.
It is the politically incorrect view to hold that Memorial Day had anything to do with the Civil War and thus should honor just those that died - both Confederate and Union. Most history books have been updated since historians have chosen to censor history as it relates to Memorial Day because of its Civil War origin.
My Encyclopedia Britannica from 1981 is very clear about the origins and celebration of Memorial Day as honoring the Civil War sacrifices.