Fletcher has been heavily criticised by former England players for persevering with Giles despite his ineffectiveness against the Australians.
The left-arm spinner has managed just three wickets at a cost of 87 runs each in the first two Tests, prompting calls for the promotion of Panesar, who has taken 32 wickets in just 10 Test appearances.
Panesar is regarded as a more attacking bowler but Fletcher said Giles had been picked because his superior batting ability gave the tourists a more balanced lineup and allowed them to play five bowlers.
Giles scored an unbeaten 27 in England's first innings in Adelaide but made a duck in the second and dropped a vital catch off Ricky Ponting as the Australians completed a remarkable win to lead the series 2-0.
"From our point of view, sure there is a lot of anger (at Panesar's non-selection)," Fletcher told reporters in Adelaide on Wednesday.
"We've got five bowlers in so we feel we go in with a well-balanced side."
Fletcher also said the decision to overlook Panesar was not made just by him and had been agreed to by senior players.
"Remember at this stage that I am not the sole selector on this tour...I'm not the one who just says 'right, this is what we are going with'," he said.
"A discussion took place and we, as a selection panel, thought that was the best side to win that Test match."
England's selectors are likely to come under intense pressure to make changes for the third Test starting in Perth on December 14 but Fletcher was reluctant to discuss any possible changes.
Despite their second-innings collapse in Adelaide, England's batsmen all look safe but there are concerns about the bowling attack.
Seamer James Anderson has been heavily criticised after taking just two wickets at 151 and England's main strike bowler Steve Harmison has captured just wicket at 288.
"Of course we stand by those selections, I have got to stand by those selections," Fletcher said.
"But we have got to sit down and say 'what do we think is the best side, with all of the knowledge that we have got around within that group of people, what is the best side to win a Test match?'".
