Are natural building materials  difficult to source?

No - unless you are missing the point - The whole idea is to use as locally sourced natural material as possible...thus only if you are in or near a straw producing area then would it make sense to use straw bale. But if you are surrounded by abundant stone and sand but no clay for instance then one would opt for stone and or sandbag walls, etc. Natural buildings are often a hybrid of naturally occurring locally available materials - preferably from the site itself.