Anything that affects your body that way (alcohol, weed, medicine), will affect you whilst driving. Therefore making it unsafe. People might think they are safe driving whilst high but they really are not. Driving slow doesn't mean driving safe. What if you have to go on the motorway? People who have a bit to drink think they can get away with it if they drive slower.
If it wasn't an issue then their wouldn't be laws and discussions about driving whilst under the influence of a substance or drug.