Terabyte Systems

Network Security Monitoring & Threat Detection

By Tera • • 11 min read
security networking threat-detection monitoring cybersecurity

Modern networks face a constantly evolving threat landscape where sophisticated attackers employ advanced techniques to breach defenses. According to recent research, the average time to detect a network breach is 207 days[1], giving adversaries ample opportunity to establish persistence, escalate privileges, and exfiltrate sensitive data. Network Security Monitoring (NSM) provides the visibility and detection capabilities necessary to identify threats before they cause significant damage.

Effective network security monitoring goes beyond simply deploying sensors and collecting logs. It requires a comprehensive strategy encompassing traffic analysis, behavioral detection, threat intelligence integration, and rapid incident response. This guide explores the technologies, methodologies, and best practices for implementing robust network security monitoring that can detect even the most sophisticated threats.

Understanding Network Security Monitoring

Network Security Monitoring is the practice of collecting, analyzing, and escalating indications of compromises on enterprise networks. Unlike traditional perimeter security that focuses on prevention, NSM emphasizes detection and response, operating under the assumption that breaches will occur[2].

Core NSM Principles

Defense in Depth

Layered security controls provide multiple opportunities to detect threats:

  • Perimeter monitoring (firewall logs, IDS/IPS)
  • Internal network monitoring (NetFlow, packet capture)
  • Endpoint monitoring (EDR, host logs)
  • Application monitoring (web server logs, database audit)

Full Packet Capture

While resource-intensive, full packet capture provides complete forensic evidence:

  • Reconstruct complete attack sequences
  • Analyze encrypted handshakes
  • Identify data exfiltration
  • Support incident investigation

Flow Data Analysis

NetFlow and similar protocols provide metadata about network connections:

  • Source and destination IPs
  • Ports and protocols
  • Byte and packet counts
  • Connection timestamps

Statistical Analysis

Baseline normal network behavior to detect anomalies:

  • Traffic volume patterns
  • Protocol distribution
  • Geolocation anomalies
  • Connection frequency

Network security operations center monitoring
Security operations center with network monitoring dashboards

Network Monitoring Architecture

Effective NSM requires strategic sensor placement and data collection infrastructure.

Sensor Placement Strategies

Perimeter Monitoring

Deploy sensors at network boundaries:

Internet
   │
   ├─── Firewall
   │      │
   │      ├─── IDS/IPS Sensor (DMZ Traffic)
   │      │
   │      └─── DMZ Servers
   │
   └─── Border Router
          │
          └─── Network TAP/SPAN
                 │
                 └─── NSM Sensor

Internal Network Monitoring

Monitor east-west traffic between internal segments:

  • Data center core switches
  • Inter-VLAN traffic
  • Critical server segments
  • Remote office connections

Critical Asset Monitoring

Dedicated monitoring for high-value systems:

  • Database servers
  • Domain controllers
  • Payment processing systems
  • Intellectual property repositories

Data Collection Methods

Network TAPs

TAPs (Test Access Points) provide passive network monitoring without impacting production traffic:

# Configure monitor interface (Linux)
ip link set eth0 promisc on
ethtool -K eth0 gro off
ethtool -K eth0 lro off

Advantages:

  • No performance impact
  • Invisible to attackers
  • Captures all traffic including errors
  • No single point of failure

SPAN/Mirror Ports

Switch Port Analyzer (SPAN) copies traffic to monitoring port:

# Cisco switch SPAN configuration
Switch(config)# monitor session 1 source interface Gi0/1 - 24
Switch(config)# monitor session 1 destination interface Gi0/48
Switch(config)# monitor session 1 filter vlan 100, 200

Considerations:

  • May drop packets under load
  • Can impact switch performance
  • Single point of failure
  • Visible in switch configuration

Intrusion Detection and Prevention Systems

IDS/IPS systems form the backbone of network threat detection.

Snort: Network Intrusion Detection

Snort is an open-source network intrusion detection system capable of real-time traffic analysis and packet logging[3].

Installation and Configuration

# Install Snort (Ubuntu/Debian)
apt-get update
apt-get install snort

# Configure network interface
vi /etc/snort/snort.conf

# Set HOME_NET to your network range
ipvar HOME_NET 10.0.0.0/8

# Set EXTERNAL_NET
ipvar EXTERNAL_NET !$HOME_NET

# Configure rule paths
var RULE_PATH /etc/snort/rules
var SO_RULE_PATH /etc/snort/so_rules
var PREPROC_RULE_PATH /etc/snort/preproc_rules

Writing Snort Rules

# Rule structure
# action protocol src_ip src_port direction dst_ip dst_port (rule options)

# Detect SQL injection attempts
alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (
    msg:"SQL Injection Attempt";
    flow:to_server,established;
    content:"union"; nocase;
    content:"select"; nocase; distance:0;
    pcre:"/union.+select/i";
    classtype:web-application-attack;
    sid:1000001;
    rev:1;
)

# Detect SSH brute force
alert tcp $EXTERNAL_NET any -> $HOME_NET 22 (
    msg:"SSH Brute Force Attempt";
    flow:to_server;
    flags:S;
    threshold:type threshold, track by_src, count 5, seconds 60;
    classtype:attempted-admin;
    sid:1000002;
    rev:1;
)

# Detect suspicious DNS tunneling
alert udp $HOME_NET any -> any 53 (
    msg:"Possible DNS Tunneling - Long Query";
    content:"|00 01 00 00 00 00 00|";
    dsize:>100;
    threshold:type both, track by_src, count 10, seconds 60;
    classtype:policy-violation;
    sid:1000003;
    rev:1;
)

# Detect ICMP tunneling
alert icmp $HOME_NET any -> $EXTERNAL_NET any (
    msg:"ICMP Tunnel Detected";
    icode:0;
    itype:8;
    dsize:>64;
    threshold:type both, track by_src, count 20, seconds 60;
    classtype:misc-activity;
    sid:1000004;
    rev:1;
)

Performance Tuning

# Enable multi-threading
./snort --daq-dir /usr/lib/daq \
    -c /etc/snort/snort.conf \
    --daq afpacket \
    --daq-var fanout_type=hash \
    --daq-var fanout_flag=rollover \
    -z 4  # 4 threads

Suricata: Next-Generation IDS/IPS

Suricata provides multi-threaded performance and advanced protocol detection.

Suricata Configuration

# /etc/suricata/suricata.yaml
vars:
  address-groups:
    HOME_NET: "[10.0.0.0/8]"
    EXTERNAL_NET: "!$HOME_NET"
    
  port-groups:
    HTTP_PORTS: "80"
    HTTPS_PORTS: "443"
    SSH_PORTS: "22"

# Threading configuration
threading:
  set-cpu-affinity: yes
  cpu-affinity:
    - management-cpu-set:
        cpu: [ 0 ]
    - receive-cpu-set:
        cpu: [ 1,2,3,4 ]
    - worker-cpu-set:
        cpu: [ 5,6,7,8 ]

# Output configuration
outputs:
  - fast:
      enabled: yes
      filename: fast.log
  - eve-log:
      enabled: yes
      filetype: regular
      filename: eve.json
      types:
        - alert
        - http
        - dns
        - tls
        - files
        - ssh

Advanced Suricata Rules

# HTTP-specific detection
alert http $EXTERNAL_NET any -> $HOME_NET any (
    msg:"Malicious User-Agent Detected";
    http.user_agent; content:"malware";
    classtype:trojan-activity;
    sid:2000001;
)

# TLS certificate validation
alert tls $EXTERNAL_NET any -> $HOME_NET any (
    msg:"Self-Signed Certificate Detected";
    tls.cert_subject; content:"CN=localhost";
    classtype:policy-violation;
    sid:2000002;
)

# File extraction and analysis
alert http $EXTERNAL_NET any -> $HOME_NET any (
    msg:"Executable Downloaded via HTTP";
    flow:established,to_client;
    http.content_type; content:"application/octet-stream";
    fileext:"exe";
    filestore;
    classtype:policy-violation;
    sid:2000003;
)

Network traffic analysis and packet inspection
Deep packet inspection and traffic analysis

Security Information and Event Management (SIEM)

SIEM platforms aggregate, correlate, and analyze security events from across the enterprise.

ELK Stack for Security Monitoring

The ELK Stack (Elasticsearch, Logstash, Kibana) provides powerful, open-source SIEM capabilities.

Logstash Configuration

# /etc/logstash/conf.d/network-security.conf
input {
  # Firewall logs
  udp {
    port => 5514
    type => "firewall"
  }
  
  # Snort alerts
  file {
    path => "/var/log/snort/alert"
    type => "ids"
    codec => plain
  }
  
  # Zeek (Bro) logs
  file {
    path => "/opt/zeek/logs/current/*.log"
    type => "zeek"
    codec => json
  }
}

filter {
  if [type] == "firewall" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} %{GREEDYDATA:message}" }
    }
    
    # Extract IP addresses
    grok {
      match => { "message" => "SRC=%{IP:src_ip} DST=%{IP:dst_ip}" }
    }
    
    # GeoIP lookup
    geoip {
      source => "src_ip"
      target => "geoip"
    }
  }
  
  if [type] == "ids" {
    # Parse Snort alerts
    grok {
      match => { "message" => "\[%{INT:priority}\] \{%{WORD:protocol}\} %{IP:src_ip}:%{INT:src_port} -> %{IP:dst_ip}:%{INT:dst_port}" }
    }
  }
  
  # Threat intelligence enrichment
  translate {
    field => "src_ip"
    destination => "threat_intel"
    dictionary_path => "/etc/logstash/threat_intel.yml"
    fallback => "unknown"
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "network-security-%{+YYYY.MM.dd}"
  }
  
  # Alert on critical events
  if [priority] <= 2 {
    email {
      to => "[email protected]"
      subject => "Critical Security Alert: %{message}"
      body => "Alert details: %{message}"
    }
  }
}

Splunk Enterprise Security

For enterprise environments, Splunk provides comprehensive SIEM capabilities with advanced correlation and investigation tools.

Splunk Search Queries

# Detect potential data exfiltration
index=network sourcetype=firewall action=allowed
| stats sum(bytes_out) as total_bytes by src_ip, dest_ip
| where total_bytes > 1073741824  # 1GB threshold
| sort -total_bytes
| table src_ip, dest_ip, total_bytes

# Identify lateral movement
index=windows EventCode=4624 Logon_Type=3
| stats count by src_ip, dest_ip, user
| where count > 20
| sort -count

# Detect DNS tunneling
index=dns
| stats count, avg(query_length) as avg_len, max(query_length) as max_len by src_ip
| where avg_len > 50 OR max_len > 100
| sort -avg_len

# Correlation search for multi-stage attack
index=network (sourcetype=ids OR sourcetype=firewall)
| transaction src_ip maxspan=1h
| where eventcount > 3
| table _time, src_ip, dest_ip, signature, action

Network Traffic Analysis Tools

Specialized tools provide deep insight into network communications.

Zeek (formerly Bro)

Zeek is a powerful network analysis framework that provides protocol-specific logging and detection capabilities[4].

Zeek Installation and Configuration

# Install Zeek
apt-get install cmake make gcc g++ flex bison libpcap-dev libssl-dev python3 python3-dev swig zlib1g-dev
git clone --recursive https://github.com/zeek/zeek
cd zeek && ./configure && make && make install

# Configure Zeek
cat > /opt/zeek/etc/node.cfg << EOF
[zeek]
type=standalone
host=localhost
interface=eth0
EOF

# Start Zeek
zeekctl deploy
zeekctl status

Custom Zeek Scripts

# Detect suspicious HTTP user agents
@load base/protocols/http

event http_header(c: connection, is_orig: bool, name: string, value: string) &priority=5
{
    if ( is_orig && name == "USER-AGENT" )
    {
        local suspicious_agents = set("sqlmap", "nikto", "nmap", "masscan");
        
        for ( agent in suspicious_agents )
        {
            if ( agent in value )
            {
                NOTICE([$note=HTTP::Suspicious_User_Agent,
                       $msg=fmt("Suspicious user agent: %s", value),
                       $conn=c,
                       $identifier=cat(c$id$orig_h, value)]);
            }
        }
    }
}

# Detect SMB brute force
@load base/protocols/smb

global smb_failures: table[addr] of count &create_expire=5min &default=0;

event smb_login_failed(c: connection, user: string)
{
    local src = c$id$orig_h;
    ++smb_failures[src];
    
    if ( smb_failures[src] >= 5 )
    {
        NOTICE([$note=SMB::Brute_Force,
               $msg=fmt("SMB brute force from %s (%d failures)", src, smb_failures[src]),
               $conn=c,
               $identifier=cat(src)]);
    }
}

Wireshark/TShark for Deep Analysis

Wireshark provides unmatched protocol analysis capabilities for investigation.

TShark Command-Line Analysis

# Capture and analyze suspicious traffic
tshark -i eth0 -f "tcp port 443" -w capture.pcap

# Extract HTTP objects
tshark -r capture.pcap --export-objects http,./extracted_files/

# Analyze SSL/TLS connections
tshark -r capture.pcap -Y "ssl.handshake.type == 1" -T fields \
    -e ip.src -e ip.dst -e ssl.handshake.ciphersuite

# Detect beaconing behavior
tshark -r capture.pcap -Y "ip.dst == suspicious_ip" -T fields \
    -e frame.time_relative | awk '{print $1}' | \
    awk 'NR>1 {print $1 - prev} {prev=$1}' | \
    datamash mean 1 sstdev 1

# Find large file transfers
tshark -r capture.pcap -q -z conv,tcp | \
    awk '$7 > 1000000 {print $1, $3, $7}'

Threat Intelligence Integration

Integrating threat intelligence feeds enhances detection capabilities by providing context about known threats.

MISP Integration

MISP (Malware Information Sharing Platform) facilitates threat intelligence sharing.

import pymisp
import requests

def check_ioc_against_misp(ioc, misp_url, misp_key):
    """Check indicator against MISP threat intelligence"""
    misp = pymisp.PyMISP(misp_url, misp_key, False)
    
    # Search for IOC
    result = misp.search('attributes', value=ioc)
    
    if result['response']:
        for attribute in result['response']['Attribute']:
            print(f"Threat found: {attribute['Event']['info']}")
            print(f"Date: {attribute['Event']['date']}")
            print(f"Threat level: {attribute['Event']['threat_level_id']}")
            return True
    
    return False

def enrich_logs_with_threat_intel(log_entry):
    """Enrich network logs with threat intelligence"""
    src_ip = log_entry.get('src_ip')
    dst_ip = log_entry.get('dst_ip')
    domain = log_entry.get('domain')
    
    threats_found = []
    
    # Check IPs against threat feeds
    if check_ioc_against_misp(src_ip, MISP_URL, MISP_KEY):
        threats_found.append(f"Source IP {src_ip} is known malicious")
    
    if check_ioc_against_misp(dst_ip, MISP_URL, MISP_KEY):
        threats_found.append(f"Destination IP {dst_ip} is known malicious")
    
    # Check domain reputation
    if domain and check_ioc_against_misp(domain, MISP_URL, MISP_KEY):
        threats_found.append(f"Domain {domain} is known malicious")
    
    log_entry['threats'] = threats_found
    return log_entry

Commercial Threat Intelligence

Integration with commercial feeds (Recorded Future, ThreatConnect, etc.):

def query_threat_intelligence(indicator, feed_api_key):
    """Query multiple threat intelligence sources"""
    
    sources = {
        'virustotal': f'https://www.virustotal.com/api/v3/ip_addresses/{indicator}',
        'abuseipdb': f'https://api.abuseipdb.com/api/v2/check?ipAddress={indicator}',
        'shodan': f'https://api.shodan.io/shodan/host/{indicator}'
    }
    
    threat_data = {}
    
    for source, url in sources.items():
        try:
            headers = {'x-apikey': feed_api_key}
            response = requests.get(url, headers=headers, timeout=10)
            
            if response.status_code == 200:
                threat_data[source] = response.json()
        except Exception as e:
            print(f"Error querying {source}: {e}")
    
    return threat_data

Incident Response Integration

NSM must feed directly into incident response processes.

Automated Response Playbooks

def automated_threat_response(alert):
    """Automated response to security alerts"""
    
    severity = alert.get('severity')
    indicator = alert.get('indicator')
    indicator_type = alert.get('type')
    
    actions_taken = []
    
    # High-severity threats
    if severity == 'critical':
        # Block at firewall
        block_ip_at_firewall(indicator)
        actions_taken.append('Blocked IP at firewall')
        
        # Isolate affected host
        if alert.get('dest_ip'):
            isolate_host(alert['dest_ip'])
            actions_taken.append('Isolated affected host')
        
        # Create high-priority ticket
        create_incident_ticket(alert, priority='P1')
        actions_taken.append('Created P1 incident ticket')
        
        # Alert on-call team
        send_alert_to_oncall(alert)
        actions_taken.append('Notified on-call team')
    
    # Medium-severity threats
    elif severity == 'high':
        # Rate limit suspicious IP
        rate_limit_ip(indicator)
        actions_taken.append('Applied rate limiting')
        
        # Create ticket for investigation
        create_incident_ticket(alert, priority='P2')
        actions_taken.append('Created P2 incident ticket')
    
    # Low-severity threats
    else:
        # Log for analysis
        log_threat_for_analysis(alert)
        actions_taken.append('Logged for analysis')
    
    # Update threat intelligence
    add_to_threat_intel(indicator, indicator_type)
    
    return actions_taken

Performance Optimization

High-performance NSM requires careful optimization.

Hardware Acceleration

# Enable offload features
ethtool -K eth0 rx off
ethtool -K eth0 tx off
ethtool -K eth0 sg off
ethtool -K eth0 tso off
ethtool -K eth0 gso off
ethtool -K eth0 gro off
ethtool -K eth0 lro off

# Increase ring buffer size
ethtool -G eth0 rx 4096 tx 4096

# Bind to specific CPUs
taskset -c 0,1,2,3 /usr/bin/snort -c /etc/snort/snort.conf -i eth0

Distributed Architecture

For large networks, deploy distributed monitoring:

┌──────────────┐
│ Collection   │
│   Sensors    │
│   (Edge)     │
└──────┬───────┘
       │
       ▼
┌──────────────┐
│ Aggregation  │
│   Layer      │
└──────┬───────┘
       │
       ▼
┌──────────────┐
│   Analysis   │
│   Platform   │
│    (SIEM)    │
└──────────────┘

Conclusion

Network Security Monitoring is not a single tool or technology but a comprehensive discipline that combines visibility, detection, analysis, and response. Success requires the right combination of technologies, skilled analysts, well-defined processes, and continuous improvement based on threat intelligence and incident lessons learned.

The most effective NSM programs embrace the reality that prevention alone is insufficient. By assuming breach and implementing robust detection and response capabilities, organizations can significantly reduce the impact of successful attacks. The key lies in achieving comprehensive visibility across the network, deploying effective detection mechanisms, maintaining skilled security operations teams, and ensuring rapid incident response when threats are identified.

As networks grow more complex and threats become more sophisticated, NSM capabilities must evolve accordingly. Investment in automation, threat intelligence, and advanced analytics empowers security teams to detect and respond to threats faster than ever before. Organizations that prioritize NSM and continuously refine their capabilities will be best positioned to defend against today’s advanced threats.

References

[1] IBM Security. (2024). Cost of a Data Breach Report 2024. Available at: https://www.ibm.com/security/data-breach (Accessed: November 2025)

[2] Bejtlich, R. (2013). The Practice of Network Security Monitoring. No Starch Press. ISBN: 978-1593275334

[3] Snort. (2024). Snort Users Manual. Available at: https://www.snort.org/documents (Accessed: November 2025)

[4] Zeek. (2024). The Zeek Network Security Monitor Documentation. Available at: https://docs.zeek.org/ (Accessed: November 2025)

[5] SANS Institute. (2024). Network Security Monitoring and Analysis. Available at: https://www.sans.org/cyber-security-courses/network-monitoring-threat-detection/ (Accessed: November 2025)

Thank you for reading! If you have any feedback or comments, please send them to [email protected].