You don't need internet connection to access your router however you need to be connected to the router (wireless or by Ethernet cable). In your web browser type the default "192.168.1.1". If it asks you to login and you have never accessed your router settings, chances are you need to log in with the defaults. Try:
note: this list is listed from top to bottom with the greatest chance of working to the lowest chance of working. So what I mean is that admin/admin should work. If not, try this site http://www.phenoelit-us.org/dpl/dpl.html. From experience, those who use WEP encryption has never touched their router's settings.
Once you have logged into your router, you MUST change your login and password or else some mischievous devil will hack into your router and screw you over. Now make sure you change some settings on the router:
1. Change your wireless encryption to WPA2 (or at the very least WPA)
2. Change default login of the router
3. Change your SSID to something other than the default and disable the broadcast of SSID
4. Enable MAC address filtering to only allow computers you know to access your router
5. Make sure your routers firewall is on