When you connect to the switch supervisors at "4" or "5", only the active one will respond. The standby one won't respond. This is normal.
When you connect to the switch supervisors at "4" or "5", you might want to switch to the MSFC on the supervisor using the "switch console" command. Remember to use ^C^C^C to get out of the MSFC when you're done.
If you try to connect from the main menu and you get "connection refused", it's because someone was sloppy and didn't "disconnect" the last time they used the terminal server. Telnet to the terminal server and use "who" and "clear line x" to force-disconnect the active connection.