Posted by Kromey at 6:29pm Sep 12 '11Situation: Last month my ISP slapped me down with $253.80 in overage charges. I have a 100,000MB download limit, and until these insane overages we'd not used more than 86,000MB -- and averaged less than 44,000MB since they implemented customer-facing usage reporting 14 months ago!
Last month was 181,176MB, and this month (well, 8/17-9/16) I've already used 72,860MB as of 9/6!
(To their credit, when they instituted the caps they also instituted a policy that the first overage charge for more than $200 would be reduced to $200. And, when I called them up to complain that their notification e-mail that I was nearing my limit never came last month (but did this month), they credited that remaining $200 to my account in exchange for me agreeing to up my service to a plan with a 200,000MB cap for an extra $50/month. Just as soon as I figure out where all this extra bandwidth is going, though, I'll be dropping back down...)
Where you come in: Help me figure out how to track down where all this data is coming from/going! I'm mostly interested in tools that will help me pin down the particular devices in my network gobbling up massive downloads, and/or particular sources of massive amounts of data (e.g. is it my Netflix usage that's to blame?).
Here's what I've got to work with: My network is a star-star topology with 2 wired switches (not hubs, so a random client simply can't see all network traffic), 1 wireless-G router (running in pseudo-AP mode (it doesn't have a true AP mode, so I just turned off all firewalling and its DHCP server, and put the LAN (which includes the DHCP server (wow, this is the 4th level of parenthetical!!)) on one of its internal Ethernet ports)), and 1 wireless-N AP. (Side note: Fearing someone nearby may have gained access to my WPA2-secured wireless networks, I have changed passwords on both to long, complex pass phrases. Not that they were trivial before, but they're even more "not trivial" now!)
At this moment, the wireless-G router is also serving as the edge router/firewall (okay, so not really in pseudo-AP mode after all), but it's pretty crappy; when I get home from work this evening, I'll be replacing it with the Linux computer I built specially for this very purpose (just haven't actually hooked it all up yet). It will then take over DHCP and DNS duties for my network (using Dnsmasq), in addition to using iptables (configured via Shorewall) for firewall and routing duties. It has two physical Ethernet ports -- "red" will be connected directly to the cable modem and will be the external interface, while "green" will be connected to one of the wired switches and will be the internal interface.
Solutions I've found so far (and why they don't fully do what I want/need):
1) Accounting rules in Shorewall/iptables: Great for what they do, but to get the granularity necessary to identify the heavy downloader(s) in my network, I need a separate accounting rule for each device. And then it completely falls apart once I again open up my wireless networks to let my friends hop on when they're over here. Potential workaround would be to use the wireless-G router to create a sub-LAN inside my LAN, then use an accounting rule targeted at that, but that seems too complex even for my ridiculous over-powered home network for 2 users!
2) tcpdump-based "top talkers": Unless I'm mistaken, this only captures a snapshot at a time, and is based on number of packets, not actual packet size. It might be modifiable to be continuous and account for packet size, but until I can figure out how to do that/find someone who has done that, I don't think it fits my needs (although scripted to run very frequently it could come close).
Oh, and budget's a concern, too -- there's bound to be free stuff out there that can do what I need it to do! (Also, GUI tools aren't an option -- this is a headless server setup, and needs to remain so.) I don't mind doing some extra legwork to cobble together something good and usable from various free tools, either.
So, anyone with any ideas for a Linux-based solution to identify top downloaders on my network, and maybe even to identify where they're downloading stuff from?
Last month was 181,176MB, and this month (well, 8/17-9/16) I've already used 72,860MB as of 9/6!
(To their credit, when they instituted the caps they also instituted a policy that the first overage charge for more than $200 would be reduced to $200. And, when I called them up to complain that their notification e-mail that I was nearing my limit never came last month (but did this month), they credited that remaining $200 to my account in exchange for me agreeing to up my service to a plan with a 200,000MB cap for an extra $50/month. Just as soon as I figure out where all this extra bandwidth is going, though, I'll be dropping back down...)
Where you come in: Help me figure out how to track down where all this data is coming from/going! I'm mostly interested in tools that will help me pin down the particular devices in my network gobbling up massive downloads, and/or particular sources of massive amounts of data (e.g. is it my Netflix usage that's to blame?).
Here's what I've got to work with: My network is a star-star topology with 2 wired switches (not hubs, so a random client simply can't see all network traffic), 1 wireless-G router (running in pseudo-AP mode (it doesn't have a true AP mode, so I just turned off all firewalling and its DHCP server, and put the LAN (which includes the DHCP server (wow, this is the 4th level of parenthetical!!)) on one of its internal Ethernet ports)), and 1 wireless-N AP. (Side note: Fearing someone nearby may have gained access to my WPA2-secured wireless networks, I have changed passwords on both to long, complex pass phrases. Not that they were trivial before, but they're even more "not trivial" now!)
At this moment, the wireless-G router is also serving as the edge router/firewall (okay, so not really in pseudo-AP mode after all), but it's pretty crappy; when I get home from work this evening, I'll be replacing it with the Linux computer I built specially for this very purpose (just haven't actually hooked it all up yet). It will then take over DHCP and DNS duties for my network (using Dnsmasq), in addition to using iptables (configured via Shorewall) for firewall and routing duties. It has two physical Ethernet ports -- "red" will be connected directly to the cable modem and will be the external interface, while "green" will be connected to one of the wired switches and will be the internal interface.
Solutions I've found so far (and why they don't fully do what I want/need):
1) Accounting rules in Shorewall/iptables: Great for what they do, but to get the granularity necessary to identify the heavy downloader(s) in my network, I need a separate accounting rule for each device. And then it completely falls apart once I again open up my wireless networks to let my friends hop on when they're over here. Potential workaround would be to use the wireless-G router to create a sub-LAN inside my LAN, then use an accounting rule targeted at that, but that seems too complex even for my ridiculous over-powered home network for 2 users!
2) tcpdump-based "top talkers": Unless I'm mistaken, this only captures a snapshot at a time, and is based on number of packets, not actual packet size. It might be modifiable to be continuous and account for packet size, but until I can figure out how to do that/find someone who has done that, I don't think it fits my needs (although scripted to run very frequently it could come close).
Oh, and budget's a concern, too -- there's bound to be free stuff out there that can do what I need it to do! (Also, GUI tools aren't an option -- this is a headless server setup, and needs to remain so.) I don't mind doing some extra legwork to cobble together something good and usable from various free tools, either.
So, anyone with any ideas for a Linux-based solution to identify top downloaders on my network, and maybe even to identify where they're downloading stuff from?
Calling all computer nerds!! - Kromey