TCP/IP Fundamentals & the OSI Model

The OSI Model, Layer by Layer

The Open Systems Interconnection (OSI) model is a seven-layer reference framework published by the ISO in 1984 to describe how network communication should be broken into independent, replaceable functions. No production network implements OSI directly, but it remains the shared vocabulary the entire industry uses to talk about where a problem or a protocol lives. When someone says "that's a layer 3 issue" or "this is a layer 7 attack," they are pointing at a specific, well-defined slice of the stack, and the tools and fixes differ completely between layers.

Layers 1-3: Getting the Bits There

Layer 1, Physical, covers the actual transmission of raw bits as electrical signals, light pulses, or radio waves. This is cabling, connectors, NICs, and hubs. A cut fiber line or a failed transceiver is a layer 1 problem, and no amount of routing configuration will fix it.

Layer 2, Data Link, handles node-to-node delivery across a single local segment using physical addressing. This is where Ethernet frames and MAC addresses live, along with switches that forward frames based on MAC address tables and protocols like ARP that resolve an IP address to a MAC address on the local segment.

Layer 3, Network, is responsible for logical addressing and routing packets across multiple networks. IP addresses, routers, and routing protocols like OSPF and BGP operate here, deciding the best path for a packet to reach a destination that may be many hops away.

Layer 4: Reliability

Layer 4, Transport, provides end-to-end communication with the reliability and flow control characteristics an application needs. TCP and UDP live here, along with the concept of port numbers that identify which application on a host should receive the data.

Layers 5-7: Application Concerns

Layer 5, Session, manages the establishment, maintenance, and termination of a communication session between two hosts, tracking which requests belong to the same logical conversation. Layer 6, Presentation, translates data between the application format and the network format, handling tasks like encryption, compression, and character encoding, such as TLS negotiation. Layer 7, Application, is the layer closest to the user, where protocols like HTTP, DNS, and SMTP define how specific applications structure their requests and responses.

In practice, layers 5 through 7 blur together in most modern implementations, since a single library often handles session state, encryption, and application formatting in one pass. The separation still matters analytically: a DNS tunneling detection is a layer 7 problem, a VLAN hopping attack is a layer 2 problem, a BGP hijack is a layer 3 problem. Framing an incident by its correct layer tells you immediately which logs, which tools, and which team should be involved.

The TCP/IP Model and How It Maps to OSI

While OSI is the teaching model, the TCP/IP model is the one actually running on every network you will ever work on. It predates OSI, having grown directly out of ARPANET research in the 1970s, and it collapses the seven-layer theoretical structure into four practical layers: Link, Internet, Transport, and Application.

Why TCP/IP Won

Where OSI was designed top-down as a specification, TCP/IP was designed bottom-up from working protocols, which is a large part of why it won in the real world. It described what already worked rather than what should theoretically work.

The Link layer (sometimes called Network Access) absorbs OSI's Physical and Data Link layers into one, covering everything needed to move a frame across a single local link: Ethernet, Wi-Fi, and ARP. The Internet layer maps directly to OSI Layer 3, handling logical addressing and routing through IP, ICMP, and related protocols. The Transport layer likewise maps directly to OSI Layer 4, with TCP and UDP as its two dominant protocols.

The Application Layer Collapse

The Application layer is where the real collapsing happens. TCP/IP merges OSI's Session, Presentation, and Application layers into a single layer, because in practice a single protocol implementation, like an HTTP library, typically owns session tracking, TLS handling, and message formatting together rather than as three cleanly separable modules. This is not a shortcut, it reflects how software engineers actually built the internet: application developers do not usually reason about "session" and "presentation" as distinct concerns, they reason about "the HTTP request" as one unit of work.

Translating Between the Two Models

You will hear both models used interchangeably in security work, and you need to translate fluently between them. A vendor advisory describing a "layer 7 DDoS attack" is using OSI numbering to describe an attack against the TCP/IP Application layer. A firewall rule set filtering by port number is operating at what OSI calls Layer 4 and TCP/IP calls the Transport layer.

Neither model is wrong, they are just different levels of granularity for describing the same running system, and analysts move between them constantly without thinking about it.

Encapsulation: How Data Actually Moves

Encapsulation is the process by which data gets wrapped in successive layers of header information as it moves down the stack from application to physical medium, and unwrapped in reverse order as it moves up the stack on the receiving end. Every layer adds its own header (and occasionally a trailer) in front of whatever it received from the layer above, treating that entire chunk as an opaque payload it does not need to understand. This is what makes the layered model actually work in practice: each layer only has to know how to talk to the layer directly above and below it.

Going Down the Stack

1. Application generates data. An application produces the actual payload, say an HTTP request.
2. Transport layer wraps it. A TCP header containing source and destination ports, sequence numbers, and flags is added, producing a segment.
3. Internet layer wraps it again. An IP header containing source and destination IP addresses is added, producing a packet.
4. Link layer wraps it once more. An Ethernet header and trailer containing source and destination MAC addresses and a frame check sequence is added, producing a frame.
5. Physical layer transmits it. The frame is converted into raw bits and sent across the medium as electrical signals, light, or radio waves.

Each stage has specific terminology for its Protocol Data Unit, and using the right term precisely tells other analysts exactly which layer you are talking about. Saying "I captured the packet" when you actually mean the full Ethernet frame is a small imprecision that adds up in incident reports and can genuinely confuse someone reading a Wireshark capture description.

Going Back Up the Stack: De-encapsulation

On the receiving end, the process runs in reverse. Each layer strips off the header it understands, checks the relevant address or port, and passes the remaining payload up to the next layer.

1. NIC receives bits. The network interface reconstructs a frame from the incoming signal.
2. Link layer strips its header. It checks the destination MAC address and passes the remaining payload up as a packet.
3. Internet layer strips its header. It checks the destination IP address and passes the payload up as a segment.
4. Transport layer strips its header. It uses the destination port to determine which application should receive the data, then passes the payload up.
5. Application receives the data. The original HTTP request arrives unchanged by the trip.

This matters directly for packet analysis. When you open a capture in Wireshark, what you are looking at is a stack of nested headers: an Ethernet frame containing an IP packet containing a TCP segment containing an HTTP request. Malware analysis and network forensics both depend on being able to peel that stack apart layer by layer and understand what each header is actually telling you.

IP Addressing and Subnetting

An IPv4 address is a 32-bit number, conventionally written as four decimal octets separated by dots, such as 192.168.1.10. Each octet represents 8 bits and ranges from 0 to 255. The address is split conceptually into a network portion and a host portion, and it is the subnet mask that determines exactly where that split happens. A subnet mask like 255.255.255.0 says the first 24 bits identify the network and the remaining 8 bits identify the host within that network.

CIDR Notation

CIDR notation expresses that same split more compactly by appending a slash and the number of network bits directly to the address, so 192.168.1.10/24 means the same thing as pairing that address with a 255.255.255.0 mask. CIDR replaced the older classful addressing system (Class A, B, C) in the 1990s because classful boundaries wasted enormous numbers of addresses; CIDR lets the network boundary fall anywhere, not just on octet lines.

Worked Example: Splitting a /24

Subnetting is the practice of dividing a larger network into smaller ones by borrowing bits from the host portion and giving them to the network portion. Here is how that plays out on a real block.

1. Start with the allocated block. 192.168.1.0/24 has 24 network bits and 8 host bits, giving 256 total addresses and 254 usable host addresses (the first address is the network address, the last is the broadcast address).
2. Borrow host bits. Taking 2 bits from the host portion turns the /24 into a /26.
3. Recalculate the split. A /26 leaves only 6 host bits, producing 4 subnets of 64 addresses each, 62 of them usable per subnet.
4. List the resulting subnets. 192.168.1.0/26, 192.168.1.64/26, 192.168.1.128/26, and 192.168.1.192/26.

This is exactly how organizations carve one allocated block into separate subnets for different VLANs, departments, or security zones.

RFC 1918: Private Address Space

RFC 1918 reserves three blocks of IPv4 address space for private use that is never routed on the public internet:

• 10.0.0.0/8: the largest private block, common in enterprise networks.
• 172.16.0.0/12: a mid-sized private block.
• 192.168.0.0/16: the block most home routers use by default.

Every home router, corporate LAN, and cloud VPC uses addresses from one of these ranges internally, translated to a public address at the edge through NAT. Recognizing these ranges instantly is a basic triage skill: traffic sourced from a 10.x address inside your environment is expected, the same address showing up as a source in an external-facing log is a red flag worth investigating.

IPv6

IPv6 exists because IPv4's 32-bit address space, roughly 4.3 billion addresses, was exhausted by the growth of the internet and especially by mobile and IoT devices. IPv6 uses 128-bit addresses, written as eight groups of hexadecimal digits separated by colons, providing an address space large enough that exhaustion is not a practical concern for the foreseeable future.

IPv6 also removes NAT as a practical necessity, since every device can have a globally routable address, and it changes some fundamentals like using Neighbor Discovery Protocol instead of ARP. Adoption remains uneven, and most enterprise networks run IPv4 and IPv6 side by side, which analysts need to account for when writing detections that assume IPv4 formatting.

The TCP Three-Way Handshake

TCP is connection-oriented, meaning two hosts must formally establish a session before exchanging application data. That session is established through a three-step exchange known as the three-way handshake.

The Three-Way Handshake

1. SYN, client to server. The client initiates by sending a segment with the SYN flag set, along with an initial sequence number it has chosen.
2. SYN-ACK, server to client. The server responds with a segment that has both the SYN and ACK flags set, acknowledging the client's sequence number and supplying its own initial sequence number.
3. ACK, client to server. The client completes the exchange by sending a segment with the ACK flag set, acknowledging the server's sequence number.

Once all three segments have been exchanged, the connection is established and either side can begin sending data.

Ports and the Four-Tuple

Port numbers are what let a single IP address support many simultaneous, independent conversations. Every TCP segment carries a source port and a destination port, and the combination of source IP, source port, destination IP, and destination port (the four-tuple) uniquely identifies one specific connection. A destination port of 443 tells the receiving host to hand the data to whatever process is listening for HTTPS traffic, typically a web server.

Well-known ports below 1024 are conventionally reserved for standard services. Ephemeral ports in the higher ranges are assigned dynamically by the client's operating system for the lifetime of a single connection.

Sequence Numbers and Reliability

Sequence numbers are what make TCP reliable. Every byte of data sent gets a sequence number, and the receiving side acknowledges receipt by sending back the sequence number it expects next. If a segment is lost or arrives out of order, the receiver's acknowledgments make that gap visible, and the sender retransmits the missing data.

This is the mechanism that lets TCP guarantee in-order, complete delivery even over a network that can drop, duplicate, or reorder packets, at the cost of additional overhead and latency compared to a protocol that does not bother with any of this.

TCP vs UDP

UDP takes the opposite approach: it is connectionless, with no handshake, no guaranteed delivery, and no built-in ordering. A UDP datagram is simply sent, and it either arrives or it does not, with no automatic retransmission. Understanding which protocol a given service uses is a direct clue to how it will fail and how it should be monitored.

Connection Teardown: FIN vs RST

Connection teardown normally happens through a four-step exchange of FIN segments: each side sends a FIN when it has finished sending data, and the other side acknowledges it, so a graceful close involves both directions independently signaling they are done.

An RST (reset) segment, by contrast, is an abrupt, immediate termination, sent when a host receives data for a connection it does not recognize, encounters an unrecoverable error, or wants to reject a connection outright. Seeing an unusually high volume of RST segments in a capture is often a sign of a scan, a blocked connection attempt, or a service actively refusing traffic, and it is one of the first things to check when triaging unexpected connection failures.

What This Module Covers

Eight chapters take you from foundational packet mechanics to full traffic analysis. Chapter 1 (this chapter) covers the OSI and TCP/IP models, encapsulation, IP addressing, subnetting, and the TCP handshake, the vocabulary and mechanics everything else in this module depends on.

• Chapter 2, Routing, Switching & VLANs: how routers select paths, how switches learn and forward on MAC addresses, and how VLANs segment a physical network into isolated logical ones.
• Chapter 3, DNS: internals and resolution, from the recursive resolver chain down to record types and cache poisoning.
• Chapter 4, TLS in Practice: the handshake, certificate validation, and where encryption starts and stops mattering for visibility.
• Chapter 5, Packet Analysis with Wireshark: filters, following streams, and reading a capture the way an analyst actually needs to.
• Chapter 6, Protocols Attackers Abuse: SMB, RDP, and DNS tunneling, and why these specific protocols keep showing up in intrusions.
• Chapter 7, Network Detection: NetFlow, Zeek, and IDS signatures, connecting protocol knowledge directly to detection engineering.
• Chapter 8, Advanced Topics: VPNs, proxies, NAT traversal, and full-PCAP analysis workflows for incident response, tying every earlier chapter together into a single investigative skill set.