Yes, they have the right to build it. No, they should not.
I am annoyed at their screams of "racist fear" because (a) Islam is not a race and (b) there is no fear involved. What we are talking about here is sensitivity and good taste. IMHO it would be the height of insensitivity and poor taste to build a mosque at the site of the most heinous crime committed by Islamic terrorists.
And now a tidbit for my First Amendment fanatic friends: The Supreme Court has long held that the first amendment right is not absolute. Any expression of that right which constitutes a "clear and present danger" is illegal. It is not difficult to see how innocent people will be hurt during the construction of such a controversial building. It should not be true in a civilized country, but you know it is. And that constitutes a clear and present danger.
I heard a talk show where the hostess read all the Islamic-sounding names of people killed in and around the Twin Towers. I am sorry for their loss, it is a huge tragedy when innocent people are killed for a cause in which they had no part. But it doesn't justify building a mosque there, any more than it justifies a Hindu or Jewish temple, a Christian church or a place of worship for any of the other religions whose followers were killed there.