No reason was given for the order but the decree said the decision came upon the recommendation of Crown Prince Salman, the defence minister.
Saudi Arabia's King Abdullah sacked the deputy defence minister on Saturday less than two months after he was appointed, the state news agency reported.
The royal decree said the decision to relieve Prince Khaled bin Bandar bin Abdul Aziz of his post came upon the recommendation of Crown Prince Salman, the defence minister. No reason was given for the order.
Saudi Arabia has been on alert following turmoil in Iraq where Sunni Islamist militants have seized control of major cities from the Shi'ite Muslim-led government of Nouri Al Maliki.
Moves in the ruling family are closely watched at home and abroad for clues to who will rule the world's top oil exporter, a country that also has a huge influence over Muslims through its guardianship of Islam's holiest sites.
Prince Khaled was moved to the post of deputy defence minister in May when King Abdullah removed him from his position as governor of Riyadh Province and appointed his son, Prince Turki, in his place.
The move was seen as strengthening the grip of his branch of the ruling dynasty as it approaches a difficult decision over how to transfer power to the next generation.
King Abdullah, who is over 90, has made a series of changes and appointments over the past two years that have consolidated the position of his allies in the family.
The most recent was the appointment of Prince Muqrin, the youngest of King Abdulaziz's sons to survive into adulthood, as deputy crown prince, a newly created position that makes him next in line to rule after King Abdullah and Crown Prince Salman.