Let me try to answer your question.  There are five boroughs in New York
City.  Manhattan, the Bronx, Brooklyn, Queens and Staten Island.  Brooklyn
and Queens are on Long Island and touch so no special bridges were needed.
Without looking at my notes I think the Brooklyn Bridge linked Manhattan with
Brooklyn in 1886.  Some time later the Manhattan Bridge was built that also
linked Manhattan and Brooklyn but also carried the subway over it.

The Bronx is separated from Manhattan by the Harlem River which is the
smallest of the rivers separating the 5 boroughs.  I think High Bridge was
built linking them in the 1880s or so.  There may have been earlier bridges
across the Harlem River because of it's size this was the easiest river to
span.  But I do not know those dates.

So the only borough that was still isolated was Staen Island and it remained
that way until bridges were built connecting it with New Jersey.  Finally in
the 1960s the Varanzanno Bridge was built connecting Staten Island with
Brooklyn.

As far as ferries Manhattan had excellent ferries connecting it to New Jersey
(at least 3) and (1) Staten Island.  There may have even been one to Queens.
Staten Island had a ferry that also connected it with Brooklyn.

I'm giving you this without checking references.  Others may add more
specifics.

Jim Maguire
[log in to unmask]