Linux伊甸园论坛 » Shell编程 » 求助啦 呵呵 如何统计文件中重复信息？

liu_yc 发表于 2007-2-26 16:55

求助啦 呵呵 如何统计文件中重复信息？

想统计一下文件中重复的IP，如果在文件中出现一次，则不显示，如果该IP重复出现则屏幕显示该IP

liu_yc 发表于 2007-2-26 17:05

文件如下
10.9.230.86.1
10.9.238.94.1
10.9.211.105.1
10.9.237.72.1
10.9.230.92.1
10.9.235.202.1
10.9.216.87.1
10.9.225.117.1
10.9.245.105.1
10.9.243.103.1
10.9.228.102.1
10.8.112.30.1
10.8.113.79.1
10.8.113.74.1
10.9.228.120.1
10.9.222.95.1
10.6.109.3.1
10.9.201.76.1
10.9.201.65.1
10.9.210.105.1
10.7.200.133.1
10.9.198.121.1
10.9.230.86.1
10.9.217.125.1

dearvoid 发表于 2007-2-26 17:05

[code]
[color=blue]-(dearvoid@LinuxEden:tty3)-(~/tmp)-
[765 0][/color] [color=red]; cat file
[/color]IP1
IP2
IP3
IP2
IP3
IP3
[color=blue]-(dearvoid@LinuxEden:tty3)-(~/tmp)-
[765 0][/color] [color=red]; sort file | uniq -d
[/color]IP2
IP3
[color=blue]-(dearvoid@LinuxEden:tty3)-(~/tmp)-
[765 0][/color] [color=red]; o
[/color][/code]

liu_yc 发表于 2007-2-26 17:11

收到 谢谢啦 很好用  linux命令太多了，越看越多 呵呵

dearvoid 发表于 2007-2-26 17:18

[code]
[color=blue]-(dearvoid@LinuxEden:tty3)-(~/tmp)-
[764 0][/color] [color=red]; cat file
[/color]IP1
IP2
IP3
IP2
IP3
IP3
[color=blue]-(dearvoid@LinuxEden:tty3)-(~/tmp)-
[764 0][/color] [color=red]; awk '{ ++a[$1] } END { for (i in a) { if (a[i] > 1) { print i }}}' file
[/color]IP2
IP3
[color=blue]-(dearvoid@LinuxEden:tty3)-(~/tmp)-
[764 0][/color] [color=red]; o
[/color][/code]

dearvoid 发表于 2007-2-26 17:36

用 awk 那个效率高些:
[code]
[color=blue]-(dearvoid@LinuxEden:tty3)-(~/tmp)-
[764 0][/color] [color=red]; cat file
[/color]10.9.230.86.1
10.9.238.94.1
10.9.211.105.1
10.9.237.72.1
10.9.230.92.1
10.9.235.202.1
10.9.216.87.1
10.9.225.117.1
10.9.245.105.1
10.9.243.103.1
10.9.228.102.1
10.8.112.30.1
10.8.113.79.1
10.8.113.74.1
10.9.228.120.1
10.9.222.95.1
10.6.109.3.1
10.9.201.76.1
10.9.201.65.1
10.9.210.105.1
10.7.200.133.1
10.9.198.121.1
10.9.230.86.1
10.9.217.125.1
[color=blue]-(dearvoid@LinuxEden:tty3)-(~/tmp)-
[764 0][/color] [color=red]; time sort file | uniq -d
[/color]10.9.230.86.1

real    0m0.021s
user    0m0.012s
sys     0m0.008s
[color=blue]-(dearvoid@LinuxEden:tty3)-(~/tmp)-
[764 0][/color] [color=red]; time awk '{ ++a[$1] } END { for (i in a) { if (a[i] - 1) { print i }}}' file
[/color]10.9.230.86.1

real    0m0.011s
user    0m0.004s
sys     0m0.008s
[color=blue]-(dearvoid@LinuxEden:tty3)-(~/tmp)-
[764 0][/color] [color=red]; o
[/color][/code]

查看完整版本: 求助啦 呵呵 如何统计文件中重复信息？