Idiot's guide to cleanly migrating to persistent volume claims

Alot of apps start out pretending to be stateless.... This is usually because

- State is necessary, but it is recoverable, so your not 'really' stateless unless performance is important (i.e. im avoiding premature optimization ).
- Its easier to start a project by pretending that state doesnt matter (i.e. I'm just prototyping)
- The critical state can exist outside of your cloud native environment, (i.e. throw it over the wall state).

So, how do you move to statefull cloud native apps ? 

There are 3 patterns:

1) Create a  "state" holding microservice that is durable.  For example, zookeeper / etcd that is distributed, with volume mounts.

2) Store state in pod annotations (this is a crazy idea - but it works, because of ETCD).

3) Do it one pod at a time.

For this post, lets try 3: Its docker compatible.

This is pretty easy, and to avoid screwing it up, I'll give you explicit directions that are impossible to fail:

- convert every 'important directory' to an emptyDir volume mount.
- make sure your app still deploys.
- replace every emptyDir with a persistentVolumeClaim, which doesnt exist yet.
- kubectl get events -n -o go-template='{{range .items}} {{.message}} + {{end}}' | tr "+" "\n"  | grep pers

This will print out all your pvcs, now create them (if you have dynamic storage) like so:


echo "apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name:
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi" | kubectl create -f -

Issues you might run into:

- permission denied ~ when moving to real storage, there are real permissions.  IF your container doesnt work properly due to permissions, check (1) who the container is running as and (2) what the directory permissions are on the storage.  The simplest way to do (2) is to execute the container with an emptyDir, and run `whoami` in the shell.  

You can fix this with an init pod, like so (using postgres as an example, where, user 26 is the user that winds up trying to write to directories - in filesystems like GKE, where only root owns things by default, you need to bootstrap permissions like so):

        - name: chmod

          command: ["chown", "-R", "26:26", "/var/lib/pgsql/data/" ]

          image: alpine

          volumeMounts:

          - name: postgres-persistent-vol

            mountPath: /var/lib/pgsql/data

            subPath: blackduck

            readOnly: false

          securityContext:

            runAsUser: 0