Brigadier Vasey was the highest ranking Australian officer on the island. There were too many unarmed men on Crete. Both General Freyberg and Brigadier Vasey wanted to transport the unarmed men to Egypt. Thee were too many unorganized unarmed and unequipped men. There seems to have been no interest by higher level authorities in dealing with this issue. The collapse in Greece seems to have created confusion and had left commanders at all levels feeling overwhelmed.
The whole battle for Crete was very damaging. While we tend to think of Crete having been decided by airborne forces it was actually German mountain troops who won the battle. they were well-equipped and trained. The only British troops that approached the capability of German mountain troops were the best of the Australians. They proved themselves in the fighting in Syria in the mountains and fighting for the fortified positions.
The British contingent in Crete was primarily the 14th Brigade. You also had "four weak and improvised battalions from Rangers Northumberland Hussars 7th Medium Regiment and the 106th Royal Horse Artillery. There were also the base units coastal artillery and the anti-aircraft units.
This is based on the account in "Greece Crete and Syria" by Gavin Long.