File Transfer Applications: Protocols and Go Implementation

"Sending a chat message is like passing a note across the room. Sending a file is like passing an entire box of documents — you need to say how big the box is, make sure nothing falls out along the way, and confirm the recipient got every last page. That's file transfer."

From Messages to Files

In the chat application chapter, every message was small and self-contained: a line of text, sent and displayed almost instantly. File transfer introduces a new set of problems that small text messages never expose:

• Size: A file can be a few bytes or several gigabytes. You cannot always assume it fits in one Read call, or even in memory.
• Framing: TCP gives you a stream of bytes with no built-in concept of "where one file ends." Your protocol has to say so explicitly.
• Integrity: A dropped or corrupted byte in a chat message is a minor annoyance. A dropped byte in an executable or a database backup can be catastrophic.
• Progress and resumption: Large transfers take time, and users expect to see progress — and ideally resume after a failure instead of starting over.

None of this requires a new protocol from scratch. It requires being deliberate about framing (how the receiver knows how much to read) and reusing tools Go already gives you: io.Copy, io.Reader/io.Writer, and the net package you've been using since the TCP chapter.

A Look at Real-World File Transfer Protocols

• FTP (File Transfer Protocol): One of the oldest internet protocols, using separate control and data connections. Still around, but its lack of built-in encryption and awkward active/passive modes have pushed most systems toward alternatives.
• SFTP: File transfer over SSH — encrypted, authenticated, and a single connection. Common in ops tooling.
• HTTP uploads/downloads: Because HTTP already solved framing (Content-Length, chunked transfer encoding) and is universally supported, plain HTTP PUT/POST and multipart form uploads have become the de facto way most applications move files today.
• Custom TCP protocols: For internal tools, peer-to-peer sync, or anything performance-sensitive, teams often roll a small custom framing protocol on top of raw TCP — exactly what we already have the tools to build.

We'll implement two of these approaches: a minimal custom TCP protocol, and an HTTP-based transfer using the standard library's net/http.

Designing a Simple Framing Protocol

The core idea of framing: before sending the file bytes, send a small, fixed-format header describing what's coming. A workable minimal header is:

[8 bytes: file size, big-endian uint64][file name length: 2 bytes]
[file name][file bytes...]

Once the receiver reads the header, it knows exactly how many bytes to expect next, and can stop reading (or write to disk) at precisely the right point — no ambiguity about where the file ends.

Go Implementation: A Custom TCP File Transfer Protocol

Server (accepts a connection, reads the header, then streams the file body to disk):

package main

import (
	"encoding/binary"
	"fmt"
	"io"
	"net"
	"os"
)

func handleConn(conn net.Conn) {
	defer conn.Close()

	// 1. Read the 8-byte size prefix.
	var size uint64
	if err := binary.Read(conn, binary.BigEndian, &size); err != nil {
		fmt.Println("failed to read size:", err)
		return
	}

	// 2. Read the 2-byte name length, then the name itself.
	var nameLen uint16
	if err := binary.Read(conn, binary.BigEndian, &nameLen); err != nil {
		fmt.Println("failed to read name length:", err)
		return
	}
	nameBuf := make([]byte, nameLen)
	if _, err := io.ReadFull(conn, nameBuf); err != nil {
		fmt.Println("failed to read name:", err)
		return
	}
	name := string(nameBuf)

	// 3. Create the destination file and copy exactly `size` bytes into it.
	out, err := os.Create("received_" + name)
	if err != nil {
		fmt.Println("failed to create file:", err)
		return
	}
	defer out.Close()

	// io.CopyN stops after exactly `size` bytes, protecting us from a
	// misbehaving or malicious client sending more than it declared.
	written, err := io.CopyN(out, conn, int64(size))
	if err != nil && err != io.EOF {
		fmt.Println("transfer error:", err)
		return
	}
	fmt.Printf("received %q (%d bytes)\n", name, written)
}

func main() {
	ln, err := net.Listen("tcp", ":9100")
	if err != nil {
		panic(err)
	}
	fmt.Println("file server listening on :9100")
	for {
		conn, err := ln.Accept()
		if err != nil {
			continue
		}
		go handleConn(conn)
	}
}

Exercise: TCP File Transfer Server

Client (sends a local file with the matching header):

package main

import (
	"encoding/binary"
	"fmt"
	"io"
	"net"
	"os"
	"path/filepath"
	"time"
)

// dialWithRetry attempts to connect up to attempts times, backing off
// between failures. A bare net.Dial in production code is often too
// fragile: a server still starting up, a brief network blip, or a load
// balancer mid-failover can all cause a single dial to fail even though
// the destination is reachable moments later.
func dialWithRetry(
	addr string, attempts int, backoff time.Duration,
) (net.Conn, error) {
	dialer := net.Dialer{Timeout: 5 * time.Second}

	var lastErr error
	for i := 0; i < attempts; i++ {
		conn, err := dialer.Dial("tcp", addr)
		if err == nil {
			return conn, nil
		}
		lastErr = err
		time.Sleep(backoff * time.Duration(i+1))
	}
	return nil, fmt.Errorf("after %d attempts: %w", attempts, lastErr)
}

func sendFile(path, addr string) error {
	f, err := os.Open(path)
	if err != nil {
		return err
	}
	defer f.Close()

	info, err := f.Stat()
	if err != nil {
		return err
	}

	conn, err := dialWithRetry(addr, 3, 500*time.Millisecond)
	if err != nil {
		return err
	}
	defer conn.Close()

	name := filepath.Base(path)

	// Write the header: size, then name length, then name.
	size := uint64(info.Size())
	if err := binary.Write(conn, binary.BigEndian, size); err != nil {
		return err
	}
	nameLen := uint16(len(name))
	if err := binary.Write(conn, binary.BigEndian, nameLen); err != nil {
		return err
	}
	if _, err := conn.Write([]byte(name)); err != nil {
		return err
	}

	// io.Copy streams the file in chunks — it never loads the whole
	// file into memory, so this works the same for a 1 KB file or a 10 GB one.
	sent, err := io.Copy(conn, f)
	if err != nil {
		return err
	}
	fmt.Printf("sent %d bytes\n", sent)
	return nil
}

func main() {
	if err := sendFile(os.Args[1], "localhost:9100"); err != nil {
		fmt.Println("error:", err)
		os.Exit(1)
	}
}

Exercise: TCP File Transfer Client

Framing is a promise about how much, not when — the receiver never has to guess where a file ends if the sender says so up front.

Go Implementation: File Transfer over HTTP

Since HTTP already handles framing, uploads and downloads often need almost no custom protocol work — just the standard net/http server and multipart form parsing you saw when working with JSON and XML payloads.

Upload handler:

func uploadHandler(w http.ResponseWriter, r *http.Request) {
	// 32 MB is the memory ceiling for parsed form parts; larger files
	// spill over to temporary disk files automatically.
	if err := r.ParseMultipartForm(32 << 20); err != nil {
		http.Error(w, "bad upload", http.StatusBadRequest)
		return
	}

	file, header, err := r.FormFile("file")
	if err != nil {
		http.Error(w, "missing file field", http.StatusBadRequest)
		return
	}
	defer file.Close()

	dst, err := os.Create("uploads/" + filepath.Base(header.Filename))
	if err != nil {
		http.Error(w, "server error", http.StatusInternalServerError)
		return
	}
	defer dst.Close()

	written, err := io.Copy(dst, file)
	if err != nil {
		http.Error(w, "write failed", http.StatusInternalServerError)
		return
	}
	fmt.Fprintf(w, "uploaded %d bytes as %s\n", written, header.Filename)
}

Download handler, using http.ServeFile, which sets Content-Length, honors Range requests for resumable downloads, and streams straight from disk:

func downloadHandler(w http.ResponseWriter, r *http.Request) {
	name := filepath.Base(r.URL.Query().Get("name"))
	http.ServeFile(w, r, filepath.Join("uploads", name))
}

Exercise: HTTP File Upload/Download

http.ServeFile is worth pausing on: it already implements conditional requests (If-Modified-Since) and byte-range requests, which is exactly the mechanism resumable downloads rely on — a client that got disconnected at byte 4,000,000 can ask for Range: bytes=4000000- and continue rather than restarting.

Progress Reporting and Integrity

For a progress bar, wrap the underlying reader with a small type that counts bytes as they pass through — io.Copy doesn't care what kind of io.Writer or io.Reader you give it, as long as it satisfies the interface:

type progressWriter struct {
	total, written int64
}

func (p *progressWriter) Write(b []byte) (int, error) {
	p.written += int64(len(b))
	fmt.Printf("\rprogress: %d/%d bytes", p.written, p.total)
	return len(b), nil
}

Pass io.MultiWriter(dst, &progressWriter{total: size}) to io.Copy and every chunk written to disk also updates the progress counter.

For integrity, compute a checksum while copying using io.MultiWriter with a crypto/sha256 hash, then compare the hex digest the sender computed against what the receiver ends up with — a cheap way to detect corruption without a heavier protocol.

Performance: Why io.Copy Can Be Faster Than It Looks

io.Copy checks whether its source implements io.WriterTo or its destination implements io.ReaderFrom before falling back to the generic 32 KB buffer loop. *os.File implements WriteTo, and on Linux *net.TCPConn implements ReadFrom by calling the sendfile(2) syscall under the hood. When both ends line up — copying straight from a file to a raw TCP connection, as the client in this chapter does — the kernel can move bytes from the file's page cache directly to the socket buffer without ever copying them into your process's user-space memory. That's the same optimization behind http.ServeFile's efficiency for large downloads.

Try It Yourself: Resuming an Interrupted Transfer

The custom TCP protocol in this chapter always starts a transfer from byte zero, even if a previous attempt already wrote most of the file to disk. Extend it to support resuming:

1. Add a third field to the header — an 8-byte offset — sent by the client to tell the server "start writing (or reading, for downloads) at this byte position" instead of always zero.
2. On the client, before dialing, os.Stat any partially-received file left over from a prior attempt and use its size as the offset.
3. On the server, open the destination file with os.OpenFile using os.O_WRONLY|os.O_CREATE (not os.Create, which truncates), then Seek to the offset before calling io.CopyN for the remaining bytes (size - offset).
4. Have the client seek its local file reader (f.Seek(offset, io.SeekStart)) past the bytes already sent, so it only streams the remainder.

This is the same idea behind HTTP's Range header and resumable downloads via http.ServeFile — you're just implementing it by hand for the custom protocol instead of getting it for free from net/http.

Frequently Asked Questions

Why not just read from the socket until it hits EOF instead of building a header? That only works if you close the connection right after each file, which throws away connection reuse and makes it impossible to send more than one file over the same socket. Sending an explicit size up front — the same principle behind HTTP's Content-Length — lets the receiver know exactly where the file ends, so the connection can stay open for the next transfer instead of relying on a close to signal "done."

My custom TCP file server slows to a crawl or crashes under a burst of connections — why? The for { conn, _ := ln.Accept(); go handleConn(conn) } pattern shown in this chapter spawns one goroutine per connection with no ceiling, and each one holds a file descriptor and a stack. A burst of connections, accidental or malicious, can exhaust memory or file descriptors long before any single transfer misbehaves. The fix is to bound concurrency explicitly, for example with a buffered channel used as a semaphore (sem <- struct{}{} before handling, <-sem when done).

Why did wrapping my copy in a progress bar make the transfer noticeably slower? Because it silently disables the zero-copy fast path. io.Copy normally detects that *net.TCPConn implements ReaderFrom and hands the work to the kernel's sendfile(2) syscall, moving bytes straight from the file's page cache to the socket buffer without ever touching user space. The moment you wrap the destination in io.MultiWriter(dst, &progressWriter{...}) or a crypto/sha256 hash, that destination is just a plain io.Writer again, so every byte has to pass through your process to be counted or hashed. It's a real tradeoff: raw speed for trusted transfers, or the copy cost when you need live progress or integrity checks.

Should I trust the file size and name a client sends in the header? No — treat both as hostile input. io.CopyN bounds how many bytes the server reads to the declared size, but a client can still lie about that size or send a name containing ../ to attempt path traversal outside your intended upload directory. Production code needs an explicit maximum file size check and must sanitize the filename (as the HTTP handlers in this chapter do with filepath.Base) before ever touching the filesystem.

How is resuming an interrupted transfer in the custom protocol related to HTTP's Range header? They're the same idea implemented at different layers. http.ServeFile gets resumable downloads for free because it already honors Range: bytes=4000000- requests and Content-Length. The custom TCP protocol in this chapter has no such built-in mechanism, so the "Try It Yourself" exercise walks through adding a manual offset field to the header and seeking both the client's file reader and the server's destination file to that position — essentially hand-rolling what net/http already gives you.

Key Takeaways

• File transfer is a framing problem: the receiver must always know exactly how many bytes belong to the file.
• io.Copy, io.CopyN, and io.ReadFull are the workhorses — they stream data without loading entire files into memory.
• HTTP already solves most of this for you: multipart uploads for sending, http.ServeFile for sending back (including range requests for resumable downloads).
• Always validate declared sizes and file names before trusting them — a file transfer server is a prime target for path traversal and resource-exhaustion attacks.