In majority of cases yes, you do want to visit first. Visiting a Chapter gives you the opportunity to view the process and meet the Members of the Chapter. This also gives the Members and Leadership Team the opportunity to get to know the Visitor prior to approving their Membership Application.
A common situation where you will not visit prior to joining is for a new Chapter. In that situation, you can visit a neighboring Chapter to experience the system if you are not already familiar.