Persistent Storage with JSON

Using JSON for storage

If you are a keen tester you will perhaps have noticed that when goto is started two times, the 2^nd time it has the short URLs and works perfectly. However, from the 3^rd start onwards, we get the error: Error loading URLStore: extra data in the buffer. This is because gob is a stream-based protocol that doesn’t support restarting. We will remedy this situation, here, by using JSON as a storage protocol, which stores the data as plain text, so it can also be read by processes written in other languages than Go. This also shows how easy it is to change to a different persistent protocol because the code dealing with the storage is cleanly isolated in 2 methods, namely load and saveLoop.

Start by creating a new empty file store.json, and change the line in main.go where the variable for the file is declared:

var dataFile = flag.String("file", "store.json", "data store file name")

In store.go, import json instead of gob. Then, in saveLoop, the only line that has to be changed is: e := gob.NewEncoder(f). We change it in: e := json.NewEncoder(f). Similarly, in the load method, the line d := gob.NewDecoder(f) is changed to d := json.NewDecoder(f). This is everything we need to change! Compile, start and test; you will see that the previous error does not occur anymore.

As an exercise, write a distributed version of the program, as described in the following section.

Multiprocessing on many machines

Until now, goto runs as a single process, but even using goroutines, a single process that runs on one machine can only serve so many concurrent requests. A URL shortener typically serves many more Redirects (reads, using Get()) than it does Adds (writes, using Put()). So, it is better to create an arbitrary number of read-only slaves that serve and cache Get requests, and pass Puts through to the master, like in the following schema: