Building permits are required for the erection of a structure, addition or swimming pool. See permit form for setback requirements.
No permit is required for fences or retaining walls, both can be placed on the property line.
It is the property owner's responsibility to know where the property lines are.