Using ASF BigTop's BigPetStore to test kubernetes

The BigPetStore application in ASF BigTop generates fake pet store transactions for you.  It can generate around 20,000 or more a minute pretty easily, and each one has 1-2 kb of text data in json format.  The transactions exhibit interesting local and global patterns, as well as correlations, making it great for building realistic, high-load applications in for testing bigdata or transaction oriented workflows.

Using the bigpetstore data generator, we are able to generate a realistic, real time flow of data into a SOA application.  This allows us to recreate scaling bottlenecks, memory hungry loads, and so on as well as a easily understandable, domain driven workload which is univeral, and animal-frendly.  The JSON dashboard at the top continually updates with the latest 3 transactions, so that users can confirm indeed that the data flowing into the system is heterogenous and dense enough  to mimic a real world workload (i.e. we're not just reinserting the same string or an easily indexed simple string and incrementing a counter).

Using the BigPetStore generator for testing kubernetes.


We decided to take guestbook and boot it on booster rockets, using the k8petstore load generator.  We did this in 3 steps.

1. Modify ASF BigTop to support a transaction queue.  This is a java application which queues builds petstore transactions and then pumps them into either a file endpoint, or a rest endpoint.
2. Modfiy GuestBook to support
1. Store all transactions as a list, rather than as a key/value pair.  This way we can count, isolate, and parallelize writes to redis.
2. Graphical interface which visualizes how many transactions are stored.  This way, by launching k8petstore, you can immediately watch and see how your cluster scales.

 The Transaction Queue.


In the transaction-queue, we continually write new customers until a rotating queue is full.  Then, we let the rotating queue is full.  Once the queue is full, we return to the calling function.  Actually, the  calling function is an infinite loop, so the generator continues writing transactions to an endpoint as long as its alive...

Using BPS-LoadGen in kubernetes.

The K8PetStore application borrows from Brendan Burn's redis based Guestbook application, but changes the Go REST API so that it supports a few extra operations, and changes the data model to that of a list with many values (as opposed to a list with a single value).  This allows us to test redis and kubernetes ability to handle transaction load by scalaing the number of generators.

Now we use a Go webserver as a REST gateway, and we allow it to read from a /push/ API endpoint.  The push endpoint accepts a JSON payload, and pushes it into the redis keyvalue list.  This list can then be queried for how many transactions were inserted a minute, and so on.

Results

Using the datagenerator at large scale, we were able to do some interesting things in redis.

1) Breaking containers with bad memory settings.

Because the data generator continues to generate lots of java objects in memory,if a container is running in a underprovisioned VM, we found that sometimes it would "just die".


2) Watching kubernetes plateau

As a load test, we found that overtime kubernetes could plateau for one of two reasons, either the redis-master might slow down, or else, the ability to scale up the number of generator containers would slow down.   In the end, we found that a long running kubernetes application would eventually plateau when we run k8petstore for sufficiently long.  This is likely do to the simple fact that the Redis master slows down once it hits a certain critical size....For the most part, handling 1000 transactions per second is not a problem in redis, as shown in the screenshot above.