Introduction
In this article, you'll learn about pickle, Redis, why you would want to save python objects in Redis, and how you would be able to achieve that with the pros, cons, and alternatives of doing so.
What is Redis?
Redis (which stands for REmote DIctionary Server) is an open-source in-memory data structure store that is commonly used to implement non-relational key-value databases and caches. It is used as a caching database where the data is saved in key-value pairs. You can check this out to learn more.
What are Python Objects?
Python has been an object-oriented language since it existed. Unlike procedure-oriented programming, where the main emphasis is on functions, object-oriented programming stresses on objects.
An object is simply a collection of data (variables) and methods (functions) that act on those data. Similarly, a class is a blueprint for that object.
We can think of a class as a sketch (prototype) of a house. It contains all the details about the floors, doors, windows, etc. Based on these descriptions we build the house. House is the object.
As many houses can be made from a house's blueprint, we can create many objects from a class. An object is also called an instance of a class and the process of creating this object is called instantiation.
Click here if you want to learn more.
Why would you want to save Python Objects in Redis?
Redis being a simple caching server, it doesn't come with many of the data types available in relational databases such as Postgres. Due to the lack of variety in data types, we may sometimes need to save Python Objects in the Redis server itself.
You may sometimes want to save nested and complex python objects in the cache, which may be hard to serialize and deserialize manually into default data types provided by Redis. That is the perfect use case for directly saving python objects in redis cache.
How to save Python Objects in Redis?
Redis stores everything as a string, hence we can save the python object as a binary string. One of the easiest ways to dump and load binary in Python for Python Objects is using the python module pickle.
What is Pickle?
Python pickle module is used for serializing and de-serializing python object structures. The process to convert any kind of python object (list, dict, etc.) into byte streams (0s and 1s) is called pickling or serialization or flattening or marshaling. We can convert the byte stream (generated through pickling) back into python objects by a process called unpickling.
How to use pickle?
import pickle
class Person:
def __init__(self, name: str, age: int) -> None:
self.name = name
self.age = age
def __str__(self) -> str:
return f'{self.name} {self.age}'
def main() -> None:
person_a = Person('John', 20)
with open('test.pkl', 'wb+') as f:
pickle.dump(person_a, f)
with open('test.pkl', 'rb') as f:
person_b = pickle.load(f)
print(str(person_a), str(person_b), sep='\n')
if __name__ == '__main__':
main()
Explanation of above code
- Pickle is imported using
import pickle
- A class named
Person is created.
person_a is created as an instance of the class Person.- 'test.pkl' file is opened with
write-binary(wb) mode.
person_a object is dumped into 'test.pkl' after pickling.- 'test.pkl' file is opened with
read-binary(rb) mode.
- Data
unpickled from the file is loaded into the variable person_b.
person_a and person_b variables are printed as strings.
Warning: The pickle module is not secure. Only unpickle data you trust.
How to use Pickle with Redis?
We're going to use a python module called aioredis as our redis client. You can install it using pip install aioredis.
Pure Redis
import asyncio
import aioredis
async def main():
redis = aioredis.from_url("redis://localhost")
await redis.set("my-key", "value")
value = await redis.get("my-key")
print(value)
if __name__ == "__main__":
asyncio.run(main())
Explanation of Above Code
- Redis client is initialized by connecting to the
redis server after providing the connection URI.
- Value of "my-key" is set as "value" in the
redis server.
- Value of the key "my-key" is retrived from the
redis server.
Redis with Pickle
import asyncio
import pickle
import aioredis
class Person:
def __init__(self, name: str, age: int) -> None:
self.name = name
self.age = age
def __str__(self) -> str:
return f'{self.name} {self.age}'
async def main():
person_a = Person('John', 20)
pickled_person_a = pickle.dumps(person_a)
redis = aioredis.from_url("redis://localhost")
await redis.set("person-a", pickled_person_a)
value = await redis.get("person-a")
unpickled_person_a = pickle.loads(value)
print(str(person_a), str(unpickled_person_a), sep='\n')
if __name__ == "__main__":
asyncio.run(main())
Key Points
- The above code is
asynchronous if you don't know about asynchronous programming. Please check this out.
- Don't set the
decode_responses keyword-argument True while initializing aioredis client. I will write a detailed blog explaining why you shouldn't do this soon enough.
- In the Redis example, we're not saving the
pickled objects in file, rather we're saving it in the redis-server so we don't have to open some file and write/ read from them.
Alternatives to Pickle
If you can cache your object in other ways while implementing the data structures and types available in Redis, you should use it instead of pickle.
One alternative is to convert the python object into JSON and save it. It can be done by using the json module. You'd have to convert the python object into a dictionary using the vars built-in method. After that you can use the json.loads and json.dumps functions to dump and load the dictionary. Click here to learn more.
You can check this thread on various alternatives for storing python objects in Redis.
Vulnerabilities that come with using Pickle and Redis
What is actually happening behind the scenes while unpickling is that the byte-stream created by pickle.dumps contains opcodes that are then one-by-one executed as soon as we load the pickle back in. For this reason, it's really easy to create a vulnerable app. unpickling is almost equivalent to using eval() so if your Redis server is compromised, your whole app would be compromised. You can check this out to learn more.
Credits: Regmi C. Mahesh
Conclusion
pickle is a great tool for storing python objects in Redis, but it comes with its own drawbacks. Pickle would be perfect if you're only looking to store python objects in Redis quick and easy. You're recommended to not use it in a production environment unless you have checked out other alternatives and weighed the pros and cons of using pickle. In conclusion, I would recommend you check out other alternatives and decide which would be perfect for your use case.
