Discussion:
[Snort-devel] Snort-devel Digest, Vol 12, Issue 9
Anuj Patel via Snort-devel
2018-05-29 18:00:51 UTC
Permalink
This is the patch for the memory leak with zlib 1.2.11. It can be applied
with: patch -p1 < 001-fix-zlib-1211-memleak.patch.

*Summary of issue:*
zlib 1.11 adds a check to verify that zlib's z_stream's state has a back
pointer to the original stream.

See: https://fossies.org/diffs/zlib/1.2.8_vs_1.2.9/inflate.c-diff.html
if state->strm != strm in inflateStateCheck, inflateEnd will not free the
z_stream's state.

This additional contstraint exposed a problem with the way snort was using
z_stream in uncompress gzip.
Here is a summary of the problem:
1. uncompress gzip declares z_stream struct on the stack
2. uncompress gzip passes the z_stream by reference to inflateInit2_
(inflateInit2_(&stream))
3. inflateInit2_ allocates memory for its state
4. inflateInit2_ stores a *back pointer* to the passed in strm from
step2 (state->strm = strm).
5. uncompress_gzip returns and the address passed in step2 goes out of
scope, but is unfortunately
stored within z_streams state.
(http_session_data->decomp_state->d_stream->state)
6. FreeHttpSessionData later attempts to free the d_stream handle but
because the back pointer
(d_stream->state->strm) in d_stream's state is invalid the
inflateStateCheck fails causing state free to fail.
7. Since FreeHttpSessionData does not check for Z_STREAM_ERROR so the
leak occurs silently.

The fix is to use the address &sd->decomp_state->d_stream. This structure
is allocated within the gzip mempool and won't go out of scope until the
entire HttpSessionData is freed.


Please reach out to me if there are questions.

Thanks,
Anuj Patel
Send Snort-devel mailing list submissions to
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.snort.org/mailman/listinfo/snort-devel
or, via email, send a message with subject or body 'help' to
You can reach the person managing the list at
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Snort-devel digest..."
1. Re: Critical: Memory leak in snort 2.9 and FreeBSD >= 10.4
(Lokesh Bevinamarad (lbevinam))
----------------------------------------------------------------------
Message: 1
Date: Mon, 28 May 2018 09:09:56 +0000
Subject: Re: [Snort-devel] Critical: Memory leak in snort 2.9 and
FreeBSD >= 10.4
Content-Type: text/plain; charset="utf-8"
Hi Elof,
Thanks for reporting this issue. We will look into it and get back to you
with our findings
We might contact you for help in reproduction of the problem and other
observations if needed.
Thanks
-Lokesh
-----Original Message-----
Sent: Sunday, May 27, 2018 12:50 AM
Subject: Re: [Snort-devel] Critical: Memory leak in snort 2.9 and FreeBSD
= 10.4
Hi Carter!
No, I use no extra plugins.
I'm trying to generate a snort.conf as identical as possible to my
original, but disabling all http_inspect decompression.
On the first test-sensor I remove "inspect_gzip" and
"unlimited_decompress" from the "preprocessor http_inspect_server: server
default" profile.
GLOBAL CONFIG
Detect Proxy Usage: NO
IIS Unicode Map Filename: /usr/local/etc/snort/unicode.map
IIS Unicode Map Codepage: 1252
Memcap used for logging URI and Hostname: 603979776
Max Gzip Memory: 10000000
Max Gzip Sessions: 19230
Gzip Compress Depth: 32767
Gzip Decompress Depth: 65535
Server profile: All
Ports (PAF): 80 81 311 383 591 593 901 1220 1414 1741 1830 2301 2381
2809 3037 3128 3702 4343 4848 5250 6988 7000 7001 7144 7145 7510 7777 7779
8000 8008 8014 8028 8080 8085 8088 8090 8118 8123 8180 8181 8243 8280 8300
8800 8888 8899 9000 9060 9080 9090 9091 9443 9999 10080 11371 34443 34444
41080 50002 55555
Server Flow Depth: 800
Client Flow Depth: 600
Max Chunk Length: 500000
Small Chunk Length Evasion: chunk size <= 10, threshold >= 5 times
Max Header Field Length: 750
Max Number Header Fields: 100
Max Number of WhiteSpaces allowed with header folding: 200
Inspect Pipeline Requests: YES
URI Discovery Strict Mode: NO
Allow Proxy Usage: NO
Disable Alerting: YES
Oversize Dir Length: 500
Only inspect URI: NO
Normalize HTTP Headers: NO
Inspect HTTP Cookies: YES
Inspect HTTP Responses: YES
Extract Gzip from responses: NO
Unlimited decompression of gzip data from responses: NO
Normalize Javascripts in HTTP Responses: YES
Max Number of WhiteSpaces allowed with Javascript Obfuscation in
HTTP responses: 200
Normalize HTTP Cookies: NO
Enable XFF and True Client IP: NO
Log HTTP URI data: NO
Log HTTP Hostname data: NO
Extended ASCII code support in URI: NO
Ascii: YES alert: NO
Double Decoding: YES alert: NO
%U Encoding: YES alert: YES
Bare Byte: YES alert: NO
UTF 8: YES alert: NO
IIS Unicode: YES alert: NO
Multiple Slash: YES alert: NO
IIS Backslash: YES alert: NO
Directory Traversal: YES alert: NO
Web Root Traversal: YES alert: NO
Apache WhiteSpace: YES alert: NO
IIS Delimiter: YES alert: NO
IIS Unicode Map: GLOBAL IIS UNICODE MAP CONFIG
Non-RFC Compliant Characters: 0x00 0x01 0x02 0x03 0x04 0x05 0x06
0x07
Whitespace Characters: 0x09 0x0b 0x0c 0x0d
Legacy mode: NO
On the second test-sensor I disable http_inspect completely.
In both tests there are no memory leak. :-)
Before, snort grew to use 100% of the RAM in 20-30 minutes.
15:55 process start
PID USERNAME THR PRI NICE SIZE RES STATE C TIME
WCPU COMMAND
15:55 : 32032 snort 2 20 0 2193M 211M nanslp 5 0:05
5.76% snort
16:05 : 32032 snort 2 20 0 2193M 218M nanslp 3 0:42
6.30% snort
16:15 : 32032 snort 2 20 0 2201M 288M nanslp 5 1:43
12.26% snort
16:34 : 32032 snort 2 20 0 2209M 296M nanslp 6 3:49
10.06% snort
16:46 : 32032 snort 2 20 0 2213M 300M nanslp 6 4:59
8.06% snort
17:04 : 32032 snort 2 20 0 2213M 301M nanslp 3 6:12
6.05% snort
18:02 : 32032 snort 2 20 0 2245M 329M nanslp 5 9:55
5.47% snort
21:02 : 32032 snort 2 20 0 2249M 335M nanslp 5 21:56
6.88% snort
14:58 process start
PID USERNAME THR PRI NICE SIZE RES STATE C TIME
WCPU COMMAND
15:07 : 60477 snort 2 20 0 833M 360M nanslp 4 4:10
46.97% snort
15:26 : 60477 snort 2 20 0 909M 438M nanslp 1 12:42
45.07% snort
15:36 : 60477 snort 2 20 0 953M 478M nanslp 0 17:07
39.99% snort
16:05 : 60477 snort 2 20 0 969M 495M nanslp 5 30:02
41.70% snort
16:34 : 60477 snort 2 20 0 973M 523M nanslp 2 43:51
43.55% snort
16:46 : 60477 snort 2 20 0 973M 523M nanslp 1 49:26
44.09% snort
17:04 : 60477 snort 2 20 0 973M 523M nanslp 7 58:14
43.16% snort
18:02 : 60477 snort 2 20 0 973M 525M nanslp 7 85:43
46.68% snort
21:02 : 60477 snort 2 20 0 981M 651M nanslp 6 172:05
45.75% snort
My conclusion: libz is ruled in.
I hope you can investigate and bugfix this.
(As I said earlier, I'm no programmer, but I can assist in compiling and
debugging)
/Elof
Hi Elof,
I want to rule in or out the interaction with zlib. It?s is only used in
http_inspect, so if you disable decompression, swf decompression, and
pdf_decompression, do you still see a memory leak? Also, are you running
with any extra / custom plugins that might use zlib?
Thanks,
Carter
Hi list (and Zi from FreeBSD ports)!
I sent the below question to snort-users in March but got no response.
Now I have upgraded a couple of more systems. This time from FreeBSD
10.3 to 10.4 (not from 10.3 to 11.1 as before) and snort start leaking
memory on all of them, just as it did in FreeBSD 11.1!
FreeBSD 10.3: snort --version
,,_ -*> Snort! <*-
o" )~ Version 2.9.11.1 (Build 268)
http://www.snort.org/contact#team
Copyright (C) 2014-2017 Cisco and/or its affiliates. All
rights
reserved.
Copyright (C) 1998-2013 Sourcefire, Inc., et al.
Using libpcap version 1.8.1
Using PCRE version: 8.40 2017-01-11
Using ZLIB version: 1.2.8
Snort is working fine. No memory leak. :-)
FreeBSD 10.4: snort --version
,,_ -*> Snort! <*-
o" )~ Version 2.9.11.1 (Build 268)
http://www.snort.org/contact#team
Copyright (C) 2014-2017 Cisco and/or its affiliates. All
rights
reserved.
Copyright (C) 1998-2013 Sourcefire, Inc., et al.
Using libpcap version 1.8.1
Using PCRE version: 8.42 2018-03-20
Using ZLIB version: 1.2.11
Snort has a memory leak. :-(
FreeBSD 11.1: snort --version
,,_ -*> Snort! <*-
o" )~ Version 2.9.11.1 (Build 268)
http://www.snort.org/contact#team
Copyright (C) 2014-2017 Cisco and/or its affiliates. All
rights
reserved.
Copyright (C) 1998-2013 Sourcefire, Inc., et al.
Using libpcap version 1.8.1
Using PCRE version: 8.40 2017-01-11
Using ZLIB version: 1.2.11
Snort has a memory leak. :-(
The snort version+build is exactly the same in all three OS versions.
Libpcap is the same.
PCRE is the same between 10.3 and the test on 11.1.
ZLIB has changed from v1.2.8 to v1.2.11 in both cases.
So in 10.3 everything is working fine.
In 10.4 and 11.1, with ZLIB 1.2.11, there is a memory leak.
I suspect there's a problem in snort together with ZLIB 1.2.11.
On a sensor which see lots of traffic, all of its 16 GB RAM is
consumed by
snort in roughly 30 minutes. (swap get full and things crash)
This problem is reproduceable all the time, on all upgraded boxes.
Can you developers please take a look at this?
Let me know if you need more information/testing from me.
I'm no programmer, but I can compile snort with debugging symbols and
I
can run gdb commands if you provide them.
Arch: amd64
RAM: 16 GB
I'm building snort from FreeBSD ports, using poudriere.
---Begin OPTIONS List---
===> The following configuration options are available for
APPID=off: Build with application id support (EXPERIMENTAL)
DOCS=on: Build and/or install documentation
FILEINSPECT=off: Build with extended file inspection features
(EXPERIMENTAL)
GRE=off: GRE support
HA=off: Enable high-availability state sharing (EXPERIMENTAL)
IPV6=off: IPv6 in snort.conf
LRGPCAP=off: Pcaps larger than 2GB
NONETHER=off: Non-Ethernet Decoders
NORMALIZER=on: Normalizer
PERFPROFILE=on: Performance profiling
SOURCEFIRE=on: Sourcefire recommended build options
====> Depend on 3rd party addons
BARNYARD=off: Depend on barnyard2 (supports also snortsam)
PULLEDPORK=off: Depend on pulledpork
====> Developer options
DBGSNORT=off: Enable debugging symbols+core dumps
===> Use 'make config' to modify these settings
---End OPTIONS List---
Snort daq is running in pcap mode.
Snort is running in passive mode.
Snort is using search-method ac-split.
preprocessor http_inspect: global iis_unicode_map
/usr/local/etc/snort/unicode.map 1252 compress_depth 65535
decompress_depth 65535 max_gzip_mem 1000000 memcap 603979776
http_inspect: unlimited_decompress is enabled
http_inspect: inspect_gzip is enabled
libz is part of the FreeBSD base system, it is not a port, so I can't
hold
it back when upgrading the OS.
/Elof
---------- Forwarded message ----------
Date: Fri, 2 Mar 2018 16:35:24 +0100 (CET)
Subject: Memory leak in snort 2.9 and FreeBSD 11?
Critical issue.
After I upgraded a few FreeBSD 10.3 machines to 11.1, snort has begun
eating
memory until it crashes.
This seem to be happening on all upgraded machines, all the time.
I suspect there's a memory leak somewhere.
while true
do
ps faxuw | egrep "^USER|/[s]nort "
echo "---"
top | grep -B3 ^Swap
echo "---"
sleep 120
done
####
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME
COMMAND
snort 7337 98.4 1.4 356096 232376 - Rs 14:40 0:01.35
snort
---
Mem: 550M Active, 174M Inact, 1585M Wired, 13G Free
ARC: 711M Total, 153M MFU, 545M MRU, 1600K Anon, 4623K Header, 7465K
Other
593M Compressed, 1647M Uncompressed, 2.78:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME
COMMAND
snort 7337 54.3 7.2 3002112 1199900 - Rs 14:40 1:04.85
snort
---
Mem: 1499M Active, 191M Inact, 1670M Wired, 12G Free
ARC: 763M Total, 178M MFU, 572M MRU, 1308K Anon, 4860K Header, 7441K
Other
646M Compressed, 1779M Uncompressed, 2.75:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME
COMMAND
snort 7337 54.3 12.9 5644032 2155388 - Ss 14:40 2:07.16
snort
---
Mem: 2427M Active, 191M Inact, 1682M Wired, 11G Free
ARC: 777M Total, 178M MFU, 585M MRU, 1344K Anon, 4935K Header, 7513K
Other
661M Compressed, 1815M Uncompressed, 2.75:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME
COMMAND
snort 7337 54.5 18.7 8275712 3114844 - Rs 14:40 3:09.86
snort
---
Mem: 3357M Active, 192M Inact, 1768M Wired, 10G Free
ARC: 821M Total, 194M MFU, 614M MRU, 556K Anon, 5195K Header, 7513K
Other
711M Compressed, 1942M Uncompressed, 2.73:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME
COMMAND
snort 7337 54.2 24.3 10862336 4053456 - Rs 14:40 4:11.28
snort
---
Mem: 4270M Active, 194M Inact, 1778M Wired, 9646M Free
ARC: 890M Total, 299M MFU, 578M MRU, 400K Anon, 5243K Header, 7442K
Other
726M Compressed, 1978M Uncompressed, 2.73:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME
COMMAND
snort 7337 56.2 29.9 13461248 4998904 - Ss 14:40 5:13.96
snort
---
Mem: 5188M Active, 195M Inact, 1798M Wired, 8708M Free
ARC: 826M Total, 261M MFU, 551M MRU, 528K Anon, 5300K Header, 7410K
Other
741M Compressed, 2015M Uncompressed, 2.72:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME
COMMAND
snort 7337 53.1 35.5 16033536 5929068 - Rs 14:40 6:15.56
snort
---
Mem: 6091M Active, 195M Inact, 1823M Wired, 7779M Free
ARC: 870M Total, 255M MFU, 602M MRU, 276K Anon, 5391K Header, 7521K
Other
755M Compressed, 2051M Uncompressed, 2.72:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME
COMMAND
snort 7337 51.2 41.1 18605824 6867124 - Ss 14:40 7:16.95
snort
---
Mem: 7002M Active, 195M Inact, 1848M Wired, 6843M Free
ARC: 885M Total, 221M MFU, 651M MRU, 288K Anon, 5454K Header, 7515K
Other
769M Compressed, 2087M Uncompressed, 2.71:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME
COMMAND
snort 7337 54.0 46.8 21212928 7810464 - Rs 14:40 8:19.24
snort
---
Mem: 7924M Active, 195M Inact, 1943M Wired, 5826M Free
ARC: 936M Total, 218M MFU, 703M MRU, 952K Anon, 5766K Header, 7829K
Other
824M Compressed, 2222M Uncompressed, 2.70:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME
COMMAND
snort 7337 53.0 52.5 23834368 8762624 - Ss 14:40 9:21.14
snort
---
Mem: 8849M Active, 195M Inact, 1954M Wired, 4891M Free
ARC: 951M Total, 218M MFU, 718M MRU, 920K Anon, 5827K Header, 7814K
Other
838M Compressed, 2259M Uncompressed, 2.69:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME
COMMAND
snort 7337 50.9 58.2 26472192 9721948 - Rs 14:40 10:23.33
snort
---
Mem: 9782M Active, 195M Inact, 1971M Wired, 3941M Free
ARC: 965M Total, 220M MFU, 731M MRU, 920K Anon, 5882K Header, 7822K
Other
853M Compressed, 2295M Uncompressed, 2.69:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED
TIME COMMAND
snort 7337 50.3 63.9 29105920 10676928 - Rs 14:40
11:26.69 snort
---
Mem: 10G Active, 204M Inact, 2116M Wired, 2852M Free
ARC: 1062M Total, 274M MFU, 774M MRU, 1052K Anon, 6252K Header, 7728K
Other
939M Compressed, 2510M Uncompressed, 2.67:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED
TIME COMMAND
snort 7337 55.5 69.6 31735552 11617828 - Rs 14:40
12:28.81 snort
---
Mem: 11G Active, 224M Inact, 2132M Wired, 1921M Free
ARC: 1092M Total, 300M MFU, 777M MRU, 1072K Anon, 6285K Header, 7591K
Other
954M Compressed, 2547M Uncompressed, 2.67:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED
TIME COMMAND
snort 7337 55.1 75.1 34324224 12545340 - Rs 14:40
13:30.78 snort
---
Mem: 12G Active, 230M Inact, 2222M Wired, 931M Free
ARC: 1117M Total, 325M MFU, 777M MRU, 1204K Anon, 6542K Header, 7580K
Other
1010M Compressed, 2686M Uncompressed, 2.66:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED
TIME COMMAND
snort 7337 55.2 80.4 36824832 13427160 - Ss 14:40
14:32.93 snort
---
Mem: 12G Active, 263M Inact, 1428M Laundry, 1929M Wired, 367M Free
ARC: 1156M Total, 325M MFU, 816M MRU, 1068K Anon, 6614K Header, 7613K
Other
1025M Compressed, 2722M Uncompressed, 2.66:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED
TIME COMMAND
snort 7337 54.6 82.6 39397120 13800500 - Rs 14:40
15:35.14 snort
---
Mem: 12G Active, 110M Inact, 1784M Laundry, 1937M Wired, 191M Free
ARC: 1159M Total, 323M MFU, 822M MRU, 936K Anon, 6648K Header, 7488K
Other
1039M Compressed, 2758M Uncompressed, 2.65:1 Ratio
Swap: 4096M Total, 685M Used, 3411M Free, 16% Inuse
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED
TIME COMMAND
snort 7337 53.8 82.6 41801472 13800476 - Ss 14:40
16:35.38 snort
---
Mem: 11G Active, 107M Inact, 1846M Laundry, 1990M Wired, 182M Free
ARC: 1208M Total, 338M MFU, 854M MRU, 1552K Anon, 6866K Header, 7564K
Other
1087M Compressed, 2878M Uncompressed, 2.65:1 Ratio
Swap: 4096M Total, 1508M Used, 2587M Free, 36% Inuse
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED
TIME COMMAND
snort 7337 54.5 82.2 44414720 13735076 - Rs 14:40
17:38.62 snort
---
Mem: 12G Active, 34M Inact, 1766M Laundry, 2014M Wired, 292M Free
ARC: 1221M Total, 337M MFU, 868M MRU, 1436K Anon, 6944K Header, 7616K
Other
1102M Compressed, 2914M Uncompressed, 2.64:1 Ratio
Swap: 4096M Total, 2551M Used, 1544M Free, 62% Inuse
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED
TIME COMMAND
snort 7337 54.6 82.1 46968576 13714884 - Ss 14:40
18:42.19 snort
---
Mem: 11G Active, 64M Inact, 1884M Laundry, 2058M Wired, 259M Free
ARC: 1252M Total, 352M MFU, 885M MRU, 672K Anon, 7150K Header, 7525K
Other
1153M Compressed, 3043M Uncompressed, 2.64:1 Ratio
Swap: 4096M Total, 3457M Used, 639M Free, 84% Inuse
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME
COMMAND
snort 7337 2.2 0.0 0 16 - R<Es 14:40 19:49.65
snort
---
Mem: 12G Active, 5252K Inact, 786M Laundry, 1937M Wired, 461M Free
ARC: 1319M Total, 384M MFU, 920M MRU, 967K Anon, 7275K Header, 7534K
Other
1181M Compressed, 3112M Uncompressed, 2.63:1 Ratio
Swap: 4096M Total, 4096M Used, K Free, 100% Inuse
---
After maxing out at 82.2% or RAM for a copuple of minutes, the
process is
Mar 2 15:17:48 chobetsu-10 kernel: swap_pager: out of swap space
failed
Mar 2 15:20:18 chobetsu-10 kernel: pid 7337 (snort), uid 100, was
killed: out
of swap space
...the while-loop continues...
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME
COMMAND
<no snort process started>
---
Mem: 272M Active, 12M Inact, 2079M Wired, 13G Free
ARC: 1376M Total, 516M MFU, 844M MRU, 1552K Anon, 7347K Header, 7524K
Other
1197M Compressed, 3150M Uncompressed, 2.63:1 Ratio
Swap: 4096M Total, 225M Used, 3871M Free, 5% Inuse
---
I startup snort again.
...the while-loop continues...
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME
COMMAND
snort 14277 57.3 5.1 2055936 856740 - Rs 15:23 0:43.21
snort
---
Mem: 1090M Active, 56M Inact, 2136M Wired, 12G Free
ARC: 1330M Total, 435M MFU, 879M MRU, 1432K Anon, 7487K Header, 7824K
Other
1213M Compressed, 3193M Uncompressed, 2.63:1 Ratio
Swap: 4096M Total, 190M Used, 3906M Free, 4% Inuse
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME
COMMAND
snort 14277 57.9 11.0 4714240 1829656 - Ss 15:23 1:49.20
snort
---
Mem: 2046M Active, 97M Inact, 2366M Wired, 11G Free
ARC: 1401M Total, 438M MFU, 945M MRU, 1296K Anon, 8041K Header, 8700K
Other
1314M Compressed, 3438M Uncompressed, 2.62:1 Ratio
Swap: 4096M Total, 172M Used, 3924M Free, 4% Inuse
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME
COMMAND
snort 14277 54.3 16.8 7362304 2798096 - Ss 15:23 2:54.43
snort
---
Mem: 2985M Active, 98M Inact, 2378M Wired, 10G Free
ARC: 1450M Total, 438M MFU, 994M MRU, 1468K Anon, 8120K Header, 8754K
Other
1329M Compressed, 3475M Uncompressed, 2.62:1 Ratio
Swap: 4096M Total, 172M Used, 3924M Free, 4% Inuse
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME
COMMAND
snort 14277 60.4 22.6 10053376 3776916 - Ss 15:23 4:00.53
snort
...and so on until pid 14277 gets out of swap.
FreeBSD 11.1-RELEASE-p4 amd64
Snort Version 2.9.11.1 (Build 268)
Using libpcap version 1.8.1
Using PCRE version: 8.40 2017-01-11
Using ZLIB version: 1.2.11
Has anyone else observed this?
Any tips on how I can help debug this further?
A SIGHUP don't reveal anything about what subsystem is eating memory.
I've tried setting two memcaps to a really low value, to see if the
process
preprocessor stream5_global: ......... memcap 128257751
preprocessor http_inspect: global .... memcap 85505167
No luck. The snort process grows to >80% of system RAM and then dies.
Running in IDS mode
Detection: Search-Method = AC-Full-Q
Split Any/Any group = enabled
Search-Method-Optimizations = enabled
Maximum pattern length = 20
+-----------------------[detection-filter-config]------
------------------------
memory-cap : 1048576 bytes
+-----------------------[rate-filter-config]----------------
-------------------
memory-cap : 1048576 bytes
+-----------------------[event-filter-config]----------
------------------------
memory-cap : 1048576 bytes
pass->activation->dynamic->drop->alert->log->sdrop->reject
pcap DAQ configured to passive.
chroot
Set gid to 100
Set uid to 100
The same snort version was running just fine on FreeBSD 10.3 before
the
upgrade.
/Elof
_______________________________________________
Snort-devel mailing list
https://lists.snort.org/mailman/listinfo/snort-devel
Please visit http://blog.snort.org for the latest news about Snort!
_______________________________________________
Snort-devel mailing list
https://lists.snort.org/mailman/listinfo/snort-devel
Please visit http://blog.snort.org for the latest news about Snort!
------------------------------
Subject: Digest Footer
_______________________________________________
Snort-devel mailing list
https://lists.snort.org/mailman/listinfo/snort-devel
------------------------------
End of Snort-devel Digest, Vol 12, Issue 9
******************************************
Bhargava Jandhyala (bjandhya) via Snort-devel
2018-05-31 07:13:19 UTC
Permalink
Thanks four your inputs. we will take care of this.

Thanks,
Bhargava

On 29-May-2018, at 11:30 PM, Anuj Patel via Snort-devel <snort-***@lists.snort.org<mailto:snort-***@lists.snort.org>> wrote:

This is the patch for the memory leak with zlib 1.2.11. It can be applied with: patch -p1 < 001-fix-zlib-1211-memleak.patch.

Summary of issue:
zlib 1.11 adds a check to verify that zlib's z_stream's state has a back pointer to the original stream.

See: https://fossies.org/diffs/zlib/1.2.8_vs_1.2.9/inflate.c-diff.html
if state->strm != strm in inflateStateCheck, inflateEnd will not free the z_stream's state.

This additional contstraint exposed a problem with the way snort was using z_stream in uncompress gzip.
Here is a summary of the problem:
1. uncompress gzip declares z_stream struct on the stack
2. uncompress gzip passes the z_stream by reference to inflateInit2_ (inflateInit2_(&stream))
3. inflateInit2_ allocates memory for its state
4. inflateInit2_ stores a *back pointer* to the passed in strm from step2 (state->strm = strm).
5. uncompress_gzip returns and the address passed in step2 goes out of scope, but is unfortunately
stored within z_streams state. (http_session_data->decomp_state->d_stream->state)
6. FreeHttpSessionData later attempts to free the d_stream handle but because the back pointer
(d_stream->state->strm) in d_stream's state is invalid the inflateStateCheck fails causing state free to fail.
7. Since FreeHttpSessionData does not check for Z_STREAM_ERROR so the leak occurs silently.

The fix is to use the address &sd->decomp_state->d_stream. This structure is allocated within the gzip mempool and won't go out of scope until the entire HttpSessionData is freed.


Please reach out to me if there are questions.

Thanks,
Anuj Patel



On Mon, May 28, 2018 at 2:09 AM, <snort-devel-***@lists.snort.org<mailto:snort-devel-***@lists.snort.org>> wrote:
Send Snort-devel mailing list submissions to
snort-***@lists.snort.org<mailto:snort-***@lists.snort.org>

To subscribe or unsubscribe via the World Wide Web, visit
https://lists.snort.org/mailman/listinfo/snort-devel
or, via email, send a message with subject or body 'help' to
snort-devel-***@lists.snort.org<mailto:snort-devel-***@lists.snort.org>

You can reach the person managing the list at
snort-devel-***@lists.snort.org<mailto:snort-devel-***@lists.snort.org>

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Snort-devel digest..."


Today's Topics:

1. Re: Critical: Memory leak in snort 2.9 and FreeBSD >= 10.4
(Lokesh Bevinamarad (lbevinam))


----------------------------------------------------------------------

Message: 1
Date: Mon, 28 May 2018 09:09:56 +0000
From: "Lokesh Bevinamarad (lbevinam)" <***@cisco.com<mailto:***@cisco.com>>
To: "***@sentor.se<mailto:***@sentor.se>" <***@sentor.se<mailto:***@sentor.se>>, "Carter Waxman (cwaxman)"
<***@cisco.com<mailto:***@cisco.com>>
Cc: "***@FreeBSD.org<mailto:***@FreeBSD.org>" <***@FreeBSD.org<mailto:***@FreeBSD.org>>, snort-devel mailinglist
<snort-***@lists.snort.org<mailto:snort-***@lists.snort.org>>
Subject: Re: [Snort-devel] Critical: Memory leak in snort 2.9 and
FreeBSD >= 10.4
Message-ID: <***@XCH-ALN-002.cisco.com<mailto:***@XCH-ALN-002.cisco.com>>
Content-Type: text/plain; charset="utf-8"

Hi Elof,

Thanks for reporting this issue. We will look into it and get back to you with our findings
We might contact you for help in reproduction of the problem and other observations if needed.

Thanks
-Lokesh

-----Original Message-----
From: Snort-devel <snort-devel-***@lists.snort.org<mailto:snort-devel-***@lists.snort.org>> On Behalf Of ***@sentor.se<mailto:***@sentor.se>
Sent: Sunday, May 27, 2018 12:50 AM
To: Carter Waxman (cwaxman) <***@cisco.com<mailto:***@cisco.com>>
Cc: ***@FreeBSD.org<mailto:***@FreeBSD.org>; snort-devel mailinglist <snort-***@lists.snort.org<mailto:snort-***@lists.snort.org>>
Subject: Re: [Snort-devel] Critical: Memory leak in snort 2.9 and FreeBSD >= 10.4


Hi Carter!

No, I use no extra plugins.

I'm trying to generate a snort.conf as identical as possible to my original, but disabling all http_inspect decompression.
On the first test-sensor I remove "inspect_gzip" and "unlimited_decompress" from the "preprocessor http_inspect_server: server default" profile.

HttpInspect Config:
GLOBAL CONFIG
Detect Proxy Usage: NO
IIS Unicode Map Filename: /usr/local/etc/snort/unicode.map
IIS Unicode Map Codepage: 1252
Memcap used for logging URI and Hostname: 603979776
Max Gzip Memory: 10000000
Max Gzip Sessions: 19230
Gzip Compress Depth: 32767
Gzip Decompress Depth: 65535
DEFAULT SERVER CONFIG:
Server profile: All
Ports (PAF): 80 81 311 383 591 593 901 1220 1414 1741 1830 2301 2381
2809 3037 3128 3702 4343 4848 5250 6988 7000 7001 7144 7145 7510 7777 7779
8000 8008 8014 8028 8080 8085 8088 8090 8118 8123 8180 8181 8243 8280 8300
8800 8888 8899 9000 9060 9080 9090 9091 9443 9999 10080 11371 34443 34444
41080 50002 55555
Server Flow Depth: 800
Client Flow Depth: 600
Max Chunk Length: 500000
Small Chunk Length Evasion: chunk size <= 10, threshold >= 5 times
Max Header Field Length: 750
Max Number Header Fields: 100
Max Number of WhiteSpaces allowed with header folding: 200
Inspect Pipeline Requests: YES
URI Discovery Strict Mode: NO
Allow Proxy Usage: NO
Disable Alerting: YES
Oversize Dir Length: 500
Only inspect URI: NO
Normalize HTTP Headers: NO
Inspect HTTP Cookies: YES
Inspect HTTP Responses: YES
Extract Gzip from responses: NO
Decompress response files:
Unlimited decompression of gzip data from responses: NO
Normalize Javascripts in HTTP Responses: YES
Max Number of WhiteSpaces allowed with Javascript Obfuscation in HTTP responses: 200
Normalize HTTP Cookies: NO
Enable XFF and True Client IP: NO
Log HTTP URI data: NO
Log HTTP Hostname data: NO
Extended ASCII code support in URI: NO
Ascii: YES alert: NO
Double Decoding: YES alert: NO
%U Encoding: YES alert: YES
Bare Byte: YES alert: NO
UTF 8: YES alert: NO
IIS Unicode: YES alert: NO
Multiple Slash: YES alert: NO
IIS Backslash: YES alert: NO
Directory Traversal: YES alert: NO
Web Root Traversal: YES alert: NO
Apache WhiteSpace: YES alert: NO
IIS Delimiter: YES alert: NO
IIS Unicode Map: GLOBAL IIS UNICODE MAP CONFIG
Non-RFC Compliant Characters: 0x00 0x01 0x02 0x03 0x04 0x05 0x06
0x07
Whitespace Characters: 0x09 0x0b 0x0c 0x0d
Legacy mode: NO


On the second test-sensor I disable http_inspect completely.



In both tests there are no memory leak. :-)

Before, snort grew to use 100% of the RAM in 20-30 minutes.
Now, the first sensor has not grown much after 5 hours:

15:55 process start
PID USERNAME THR PRI NICE SIZE RES STATE C TIME
WCPU COMMAND
15:55 : 32032 snort 2 20 0 2193M 211M nanslp 5 0:05
5.76% snort
16:05 : 32032 snort 2 20 0 2193M 218M nanslp 3 0:42
6.30% snort
16:15 : 32032 snort 2 20 0 2201M 288M nanslp 5 1:43
12.26% snort
16:34 : 32032 snort 2 20 0 2209M 296M nanslp 6 3:49
10.06% snort
16:46 : 32032 snort 2 20 0 2213M 300M nanslp 6 4:59
8.06% snort
17:04 : 32032 snort 2 20 0 2213M 301M nanslp 3 6:12
6.05% snort
18:02 : 32032 snort 2 20 0 2245M 329M nanslp 5 9:55
5.47% snort
21:02 : 32032 snort 2 20 0 2249M 335M nanslp 5 21:56
6.88% snort


...and neither has the second sensor after 6 hours:
14:58 process start
PID USERNAME THR PRI NICE SIZE RES STATE C TIME
WCPU COMMAND
15:07 : 60477 snort 2 20 0 833M 360M nanslp 4 4:10
46.97% snort
15:26 : 60477 snort 2 20 0 909M 438M nanslp 1 12:42
45.07% snort
15:36 : 60477 snort 2 20 0 953M 478M nanslp 0 17:07
39.99% snort
16:05 : 60477 snort 2 20 0 969M 495M nanslp 5 30:02
41.70% snort
16:34 : 60477 snort 2 20 0 973M 523M nanslp 2 43:51
43.55% snort
16:46 : 60477 snort 2 20 0 973M 523M nanslp 1 49:26
44.09% snort
17:04 : 60477 snort 2 20 0 973M 523M nanslp 7 58:14
43.16% snort
18:02 : 60477 snort 2 20 0 973M 525M nanslp 7 85:43
46.68% snort
21:02 : 60477 snort 2 20 0 981M 651M nanslp 6 172:05
45.75% snort


My conclusion: libz is ruled in.


I hope you can investigate and bugfix this.
(As I said earlier, I'm no programmer, but I can assist in compiling and
debugging)

/Elof
Hi Elof,
I want to rule in or out the interaction with zlib. It?s is only used in http_inspect, so if you disable decompression, swf decompression, and pdf_decompression, do you still see a memory leak? Also, are you running with any extra / custom plugins that might use zlib?
Thanks,
Carter
Hi list (and Zi from FreeBSD ports)!
I sent the below question to snort-users in March but got no response.
Now I have upgraded a couple of more systems. This time from FreeBSD
10.3 to 10.4 (not from 10.3 to 11.1 as before) and snort start leaking
memory on all of them, just as it did in FreeBSD 11.1!
FreeBSD 10.3: snort --version
,,_ -*> Snort! <*-
o" )~ Version 2.9.11.1 (Build 268)
http://www.snort.org/contact#team
Copyright (C) 2014-2017 Cisco and/or its affiliates. All rights
reserved.
Copyright (C) 1998-2013 Sourcefire, Inc., et al.
Using libpcap version 1.8.1
Using PCRE version: 8.40 2017-01-11
Using ZLIB version: 1.2.8
Snort is working fine. No memory leak. :-)
FreeBSD 10.4: snort --version
,,_ -*> Snort! <*-
o" )~ Version 2.9.11.1 (Build 268)
http://www.snort.org/contact#team
Copyright (C) 2014-2017 Cisco and/or its affiliates. All rights
reserved.
Copyright (C) 1998-2013 Sourcefire, Inc., et al.
Using libpcap version 1.8.1
Using PCRE version: 8.42 2018-03-20
Using ZLIB version: 1.2.11
Snort has a memory leak. :-(
FreeBSD 11.1: snort --version
,,_ -*> Snort! <*-
o" )~ Version 2.9.11.1 (Build 268)
http://www.snort.org/contact#team
Copyright (C) 2014-2017 Cisco and/or its affiliates. All rights
reserved.
Copyright (C) 1998-2013 Sourcefire, Inc., et al.
Using libpcap version 1.8.1
Using PCRE version: 8.40 2017-01-11
Using ZLIB version: 1.2.11
Snort has a memory leak. :-(
The snort version+build is exactly the same in all three OS versions.
Libpcap is the same.
PCRE is the same between 10.3 and the test on 11.1.
ZLIB has changed from v1.2.8 to v1.2.11 in both cases.
So in 10.3 everything is working fine.
In 10.4 and 11.1, with ZLIB 1.2.11, there is a memory leak.
I suspect there's a problem in snort together with ZLIB 1.2.11.
On a sensor which see lots of traffic, all of its 16 GB RAM is consumed by
snort in roughly 30 minutes. (swap get full and things crash)
This problem is reproduceable all the time, on all upgraded boxes.
Can you developers please take a look at this?
Let me know if you need more information/testing from me.
I'm no programmer, but I can compile snort with debugging symbols and I
can run gdb commands if you provide them.
Arch: amd64
RAM: 16 GB
I'm building snort from FreeBSD ports, using poudriere.
---Begin OPTIONS List---
===> The following configuration options are available for
APPID=off: Build with application id support (EXPERIMENTAL)
DOCS=on: Build and/or install documentation
FILEINSPECT=off: Build with extended file inspection features
(EXPERIMENTAL)
GRE=off: GRE support
HA=off: Enable high-availability state sharing (EXPERIMENTAL)
IPV6=off: IPv6 in snort.conf
LRGPCAP=off: Pcaps larger than 2GB
NONETHER=off: Non-Ethernet Decoders
NORMALIZER=on: Normalizer
PERFPROFILE=on: Performance profiling
SOURCEFIRE=on: Sourcefire recommended build options
====> Depend on 3rd party addons
BARNYARD=off: Depend on barnyard2 (supports also snortsam)
PULLEDPORK=off: Depend on pulledpork
====> Developer options
DBGSNORT=off: Enable debugging symbols+core dumps
===> Use 'make config' to modify these settings
---End OPTIONS List---
Snort daq is running in pcap mode.
Snort is running in passive mode.
Snort is using search-method ac-split.
preprocessor http_inspect: global iis_unicode_map
/usr/local/etc/snort/unicode.map 1252 compress_depth 65535
decompress_depth 65535 max_gzip_mem 1000000 memcap 603979776
http_inspect: unlimited_decompress is enabled
http_inspect: inspect_gzip is enabled
libz is part of the FreeBSD base system, it is not a port, so I can't hold
it back when upgrading the OS.
/Elof
---------- Forwarded message ----------
Date: Fri, 2 Mar 2018 16:35:24 +0100 (CET)
Subject: Memory leak in snort 2.9 and FreeBSD 11?
Critical issue.
After I upgraded a few FreeBSD 10.3 machines to 11.1, snort has begun eating
memory until it crashes.
This seem to be happening on all upgraded machines, all the time.
I suspect there's a memory leak somewhere.
while true
do
ps faxuw | egrep "^USER|/[s]nort "
echo "---"
top | grep -B3 ^Swap
echo "---"
sleep 120
done
####
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
snort 7337 98.4 1.4 356096 232376 - Rs 14:40 0:01.35 snort
---
Mem: 550M Active, 174M Inact, 1585M Wired, 13G Free
ARC: 711M Total, 153M MFU, 545M MRU, 1600K Anon, 4623K Header, 7465K Other
593M Compressed, 1647M Uncompressed, 2.78:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
snort 7337 54.3 7.2 3002112 1199900 - Rs 14:40 1:04.85 snort
---
Mem: 1499M Active, 191M Inact, 1670M Wired, 12G Free
ARC: 763M Total, 178M MFU, 572M MRU, 1308K Anon, 4860K Header, 7441K Other
646M Compressed, 1779M Uncompressed, 2.75:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
snort 7337 54.3 12.9 5644032 2155388 - Ss 14:40 2:07.16 snort
---
Mem: 2427M Active, 191M Inact, 1682M Wired, 11G Free
ARC: 777M Total, 178M MFU, 585M MRU, 1344K Anon, 4935K Header, 7513K Other
661M Compressed, 1815M Uncompressed, 2.75:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
snort 7337 54.5 18.7 8275712 3114844 - Rs 14:40 3:09.86 snort
---
Mem: 3357M Active, 192M Inact, 1768M Wired, 10G Free
ARC: 821M Total, 194M MFU, 614M MRU, 556K Anon, 5195K Header, 7513K Other
711M Compressed, 1942M Uncompressed, 2.73:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
snort 7337 54.2 24.3 10862336 4053456 - Rs 14:40 4:11.28 snort
---
Mem: 4270M Active, 194M Inact, 1778M Wired, 9646M Free
ARC: 890M Total, 299M MFU, 578M MRU, 400K Anon, 5243K Header, 7442K Other
726M Compressed, 1978M Uncompressed, 2.73:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
snort 7337 56.2 29.9 13461248 4998904 - Ss 14:40 5:13.96 snort
---
Mem: 5188M Active, 195M Inact, 1798M Wired, 8708M Free
ARC: 826M Total, 261M MFU, 551M MRU, 528K Anon, 5300K Header, 7410K Other
741M Compressed, 2015M Uncompressed, 2.72:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
snort 7337 53.1 35.5 16033536 5929068 - Rs 14:40 6:15.56 snort
---
Mem: 6091M Active, 195M Inact, 1823M Wired, 7779M Free
ARC: 870M Total, 255M MFU, 602M MRU, 276K Anon, 5391K Header, 7521K Other
755M Compressed, 2051M Uncompressed, 2.72:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
snort 7337 51.2 41.1 18605824 6867124 - Ss 14:40 7:16.95 snort
---
Mem: 7002M Active, 195M Inact, 1848M Wired, 6843M Free
ARC: 885M Total, 221M MFU, 651M MRU, 288K Anon, 5454K Header, 7515K Other
769M Compressed, 2087M Uncompressed, 2.71:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
snort 7337 54.0 46.8 21212928 7810464 - Rs 14:40 8:19.24 snort
---
Mem: 7924M Active, 195M Inact, 1943M Wired, 5826M Free
ARC: 936M Total, 218M MFU, 703M MRU, 952K Anon, 5766K Header, 7829K Other
824M Compressed, 2222M Uncompressed, 2.70:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
snort 7337 53.0 52.5 23834368 8762624 - Ss 14:40 9:21.14 snort
---
Mem: 8849M Active, 195M Inact, 1954M Wired, 4891M Free
ARC: 951M Total, 218M MFU, 718M MRU, 920K Anon, 5827K Header, 7814K Other
838M Compressed, 2259M Uncompressed, 2.69:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
snort 7337 50.9 58.2 26472192 9721948 - Rs 14:40 10:23.33 snort
---
Mem: 9782M Active, 195M Inact, 1971M Wired, 3941M Free
ARC: 965M Total, 220M MFU, 731M MRU, 920K Anon, 5882K Header, 7822K Other
853M Compressed, 2295M Uncompressed, 2.69:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
snort 7337 50.3 63.9 29105920 10676928 - Rs 14:40 11:26.69 snort
---
Mem: 10G Active, 204M Inact, 2116M Wired, 2852M Free
ARC: 1062M Total, 274M MFU, 774M MRU, 1052K Anon, 6252K Header, 7728K Other
939M Compressed, 2510M Uncompressed, 2.67:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
snort 7337 55.5 69.6 31735552 11617828 - Rs 14:40 12:28.81 snort
---
Mem: 11G Active, 224M Inact, 2132M Wired, 1921M Free
ARC: 1092M Total, 300M MFU, 777M MRU, 1072K Anon, 6285K Header, 7591K Other
954M Compressed, 2547M Uncompressed, 2.67:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
snort 7337 55.1 75.1 34324224 12545340 - Rs 14:40 13:30.78 snort
---
Mem: 12G Active, 230M Inact, 2222M Wired, 931M Free
ARC: 1117M Total, 325M MFU, 777M MRU, 1204K Anon, 6542K Header, 7580K Other
1010M Compressed, 2686M Uncompressed, 2.66:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
snort 7337 55.2 80.4 36824832 13427160 - Ss 14:40 14:32.93 snort
---
Mem: 12G Active, 263M Inact, 1428M Laundry, 1929M Wired, 367M Free
ARC: 1156M Total, 325M MFU, 816M MRU, 1068K Anon, 6614K Header, 7613K Other
1025M Compressed, 2722M Uncompressed, 2.66:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
snort 7337 54.6 82.6 39397120 13800500 - Rs 14:40 15:35.14 snort
---
Mem: 12G Active, 110M Inact, 1784M Laundry, 1937M Wired, 191M Free
ARC: 1159M Total, 323M MFU, 822M MRU, 936K Anon, 6648K Header, 7488K Other
1039M Compressed, 2758M Uncompressed, 2.65:1 Ratio
Swap: 4096M Total, 685M Used, 3411M Free, 16% Inuse
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
snort 7337 53.8 82.6 41801472 13800476 - Ss 14:40 16:35.38 snort
---
Mem: 11G Active, 107M Inact, 1846M Laundry, 1990M Wired, 182M Free
ARC: 1208M Total, 338M MFU, 854M MRU, 1552K Anon, 6866K Header, 7564K Other
1087M Compressed, 2878M Uncompressed, 2.65:1 Ratio
Swap: 4096M Total, 1508M Used, 2587M Free, 36% Inuse
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
snort 7337 54.5 82.2 44414720 13735076 - Rs 14:40 17:38.62 snort
---
Mem: 12G Active, 34M Inact, 1766M Laundry, 2014M Wired, 292M Free
ARC: 1221M Total, 337M MFU, 868M MRU, 1436K Anon, 6944K Header, 7616K Other
1102M Compressed, 2914M Uncompressed, 2.64:1 Ratio
Swap: 4096M Total, 2551M Used, 1544M Free, 62% Inuse
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
snort 7337 54.6 82.1 46968576 13714884 - Ss 14:40 18:42.19 snort
---
Mem: 11G Active, 64M Inact, 1884M Laundry, 2058M Wired, 259M Free
ARC: 1252M Total, 352M MFU, 885M MRU, 672K Anon, 7150K Header, 7525K Other
1153M Compressed, 3043M Uncompressed, 2.64:1 Ratio
Swap: 4096M Total, 3457M Used, 639M Free, 84% Inuse
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
snort 7337 2.2 0.0 0 16 - R<Es 14:40 19:49.65 snort
---
Mem: 12G Active, 5252K Inact, 786M Laundry, 1937M Wired, 461M Free
ARC: 1319M Total, 384M MFU, 920M MRU, 967K Anon, 7275K Header, 7534K Other
1181M Compressed, 3112M Uncompressed, 2.63:1 Ratio
Swap: 4096M Total, 4096M Used, K Free, 100% Inuse
---
After maxing out at 82.2% or RAM for a copuple of minutes, the process is
Mar 2 15:17:48 chobetsu-10 kernel: swap_pager: out of swap space
Mar 2 15:17:48 chobetsu-10 kernel: swap_pager_getswapspace(11): failed
Mar 2 15:20:18 chobetsu-10 kernel: pid 7337 (snort), uid 100, was killed: out
of swap space
...the while-loop continues...
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
<no snort process started>
---
Mem: 272M Active, 12M Inact, 2079M Wired, 13G Free
ARC: 1376M Total, 516M MFU, 844M MRU, 1552K Anon, 7347K Header, 7524K Other
1197M Compressed, 3150M Uncompressed, 2.63:1 Ratio
Swap: 4096M Total, 225M Used, 3871M Free, 5% Inuse
---
I startup snort again.
...the while-loop continues...
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
snort 14277 57.3 5.1 2055936 856740 - Rs 15:23 0:43.21 snort
---
Mem: 1090M Active, 56M Inact, 2136M Wired, 12G Free
ARC: 1330M Total, 435M MFU, 879M MRU, 1432K Anon, 7487K Header, 7824K Other
1213M Compressed, 3193M Uncompressed, 2.63:1 Ratio
Swap: 4096M Total, 190M Used, 3906M Free, 4% Inuse
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
snort 14277 57.9 11.0 4714240 1829656 - Ss 15:23 1:49.20 snort
---
Mem: 2046M Active, 97M Inact, 2366M Wired, 11G Free
ARC: 1401M Total, 438M MFU, 945M MRU, 1296K Anon, 8041K Header, 8700K Other
1314M Compressed, 3438M Uncompressed, 2.62:1 Ratio
Swap: 4096M Total, 172M Used, 3924M Free, 4% Inuse
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
snort 14277 54.3 16.8 7362304 2798096 - Ss 15:23 2:54.43 snort
---
Mem: 2985M Active, 98M Inact, 2378M Wired, 10G Free
ARC: 1450M Total, 438M MFU, 994M MRU, 1468K Anon, 8120K Header, 8754K Other
1329M Compressed, 3475M Uncompressed, 2.62:1 Ratio
Swap: 4096M Total, 172M Used, 3924M Free, 4% Inuse
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
snort 14277 60.4 22.6 10053376 3776916 - Ss 15:23 4:00.53 snort
...and so on until pid 14277 gets out of swap.
FreeBSD 11.1-RELEASE-p4 amd64
Snort Version 2.9.11.1 (Build 268)
Using libpcap version 1.8.1
Using PCRE version: 8.40 2017-01-11
Using ZLIB version: 1.2.11
Has anyone else observed this?
Any tips on how I can help debug this further?
A SIGHUP don't reveal anything about what subsystem is eating memory.
I've tried setting two memcaps to a really low value, to see if the process
preprocessor stream5_global: ......... memcap 128257751
preprocessor http_inspect: global .... memcap 85505167
No luck. The snort process grows to >80% of system RAM and then dies.
Running in IDS mode
Detection: Search-Method = AC-Full-Q
Split Any/Any group = enabled
Search-Method-Optimizations = enabled
Maximum pattern length = 20
+-----------------------[detection-filter-config]------------------------------
memory-cap : 1048576 bytes
+-----------------------[rate-filter-config]-----------------------------------
memory-cap : 1048576 bytes
+-----------------------[event-filter-config]----------------------------------
memory-cap : 1048576 bytes
pass->activation->dynamic->drop->alert->log->sdrop->reject
pcap DAQ configured to passive.
chroot
Set gid to 100
Set uid to 100
The same snort version was running just fine on FreeBSD 10.3 before the
upgrade.
/Elof
_______________________________________________
Snort-devel mailing list
https://lists.snort.org/mailman/listinfo/snort-devel
Please visit http://blog.snort.org<http://blog.snort.org/> for the latest news about Snort!
_______________________________________________
Snort-devel mailing list
Snort-***@lists.snort.org<mailto:Snort-***@lists.snort.org>
https://lists.snort.org/mailman/listinfo/snort-devel

Please visit http://blog.snort.org<http://blog.snort.org/> for the latest news about Snort!

------------------------------

Subject: Digest Footer

_______________________________________________
Snort-devel mailing list
Snort-***@lists.snort.org<mailto:Snort-***@lists.snort.org>
https://lists.snort.org/mailman/listinfo/snort-devel


------------------------------

End of Snort-devel Digest, Vol 12, Issue 9
******************************************

<001-fix-zlib-1211-memleak.patch>_______________________________________________
Snort-devel mailing list
Snort-***@lists.snort.org<mailto:Snort-***@lists.snort.org>
https://lists.snort.org/mailman/listinfo/snort-devel

Please visit http://blog.snort.org for the latest news about Snort!
Loading...