Request Line Review

Congratulations! Parsing data as it comes into a program is much harder than parsing an entire message... as you've seen. Ah, the things we do for efficiency!

I want to just point out a few things that I did in my code, and why I did them.

const crlf = "\r\n"
const bufferSize = 8

func RequestFromReader(reader io.Reader) (*Request, error) {
	buf := make([]byte, bufferSize, bufferSize)
	readToIndex := 0
	req := &Request{
		state: requestStateInitialized,
	}
	for req.state != requestStateDone {
	// ...

Our buffer size here is teensy tiny. If you look at our tests, you'll also recall that we added some test cases where only 1 or 2 bytes are read at a time. We want to test at these small buffer sizes to ensure that our parser can handle the edge cases where even something as small as the request line is split across multiple reads.

That said, in production, that's probably a ridiculous buffer size. You'd want to optimize for performance, and my guess is you'd likely start with a buffer size of 1024 or 4096 or something... I don't know I just work here. The point is, we haven't chosen an optimal number, we've chosen a small number for testing purposes.

Next, I just want to point out the difference between reading and parsing.

for req.state != requestStateDone {
    // ...

    // read into the buffer
		numBytesRead, err := reader.Read(buf[readToIndex:])
		if err != nil {
			if errors.Is(err, io.EOF) {
				req.state = requestStateDone
				break
			}
			return nil, err
		}
		readToIndex += numBytesRead

    // parse from the buffer
		numBytesParsed, err := req.parse(buf[:readToIndex])
		if err != nil {
			return nil, err
		}

		// ...
	}

This can be confusing, but it's important to understand the difference. When we read, all we're doing is moving the data from the reader (which in the case of HTTP is a network connection, but it could be a file as well, our code is agnostic) into our program. When we parse, we're taking that data and interpreting it (moving it from a []byte to a RequestLine struct). Once its parsed, we can discard it from the buffer to save memory.