In our previous lessons, we built a basic key-value store using a simple Python dictionary. The concept of key-value store is simple but highly efficient and essential in many areas of software engineering, including finance and AI. For example, we can consider a scenario where an AI model in finance uses a key-value storage system to keep track of different types of stocks and their corresponding values. The unique key is the stock symbol, and the value is the current price of the stock. Using this elementary key-value store, we can rapidly fetch the price for any stock symbol - exactly how we fetch the data in the main memory.

Let's remind ourselves how we can store and fetch data from our key-value store. Suppose we have a key 'foo' with the value 'bar'. The Python code to implement this operation would be as follows:

1kv_store = {}
2kv_store['foo'] = 'bar'
3print(kv_store.get('foo'))

This code simply creates a kv_store dictionary and then assigns the key 'foo' the value 'bar'. The get method is then used to retrieve the value associated with the key 'foo', which should output 'bar'. While this is a simplified example, key-value stores can scale to handle large amounts of data, highlighting their importance in areas like AI and finance.

xxxxxxxxxx

if __name__ == "__main__":

  # Python logic here

  kv_store = {}

  kv_store['foo'] = 'bar'

  print(kv_store.get('foo'))

In the search for more efficient, scalable, and feature-rich key-value stores, we turn our attention to Redis. Going far beyond a simple dictionary-based key-value store, Redis shines with its rich set of data types and high-performance data handling capabilities. Alias for 'Remote Dictionary Service', Redis is an in-memory, distributed database that provides a high-performance datastore offering various data structures.

As a noSQL database, Redis is a perfect fit for a scenario where data is unstructured and the relationships between entities are not as important as speed and scalability. This feature fits beautifully in our example where a basket of stocks and their values, which are quite volatile, might be monitored.

One of the key advantages of Redis over simple key-value stores is the support for different data types like strings, lists, maps, sets, sorted sets, HyperLogLogs, bitmaps, streams, and spatial indexes. Redis also supports persistent storage, an important feature that allows data to survive server restarts. To understand it better, consider the following Python logic. Here we have an imaginary scenario, where using Redis we are denoting the AAPL (Apple Inc.) stock price in our in-memory store.

This is just scratching the surface. In the following tutorials, we will explore advanced features of Redis like data replication, automatic partitioning, and expiration times for keys, all with an eye toward integrating these features into our custom key-value store.

xxxxxxxxxx

if __name__ == "__main__":

    # Quoting items on the stock market using Redis

    import redis

    r = redis.StrictRedis(host='localhost', port=6379, db=0)

    r.set('AAPL', 262.16)

    print(float(r.get('AAPL')))

Having the ability to set expiration times for keys can be especially useful in scenarios where your data's relevance diminishes over time, like in stock market data, weather forecasts, or caching scenarios. This approach is often found in databases like Redis, and we'll be incorporating it into our own store.

Take for example, monitoring the stock price of a company: if you're tracking Apple Inc. (AAPL) and set a 10-second expiration on the data, a subsequent retrieval after this duration would indicate the data has already expired, necessitating a fresh fetch of the current stock price. This ensures that you're working with the most recent and relevant data and not stale information

In Python, we can extend the built-in dict type to develop a dictionary which allows us to set an expiry time for keys. Whenever we attempt to retrieve a key, we check if the current time is later than the expiry time for that key. If it is, we delete the key and raise a KeyError, otherwise we return the key's value. Thus, the basic logic to set the value with an expiration time includes additional steps.

Here's Python logic of how we might implement this expiration feature. The code demonstrates how if we try to access the 'AAPL' key in our ExpiringDict after its expiry time of 10 seconds, we get a 'Key expired' message.

Let's run this code to simulate key expiration in our key-value store.

xxxxxxxxxx

if __name__ == '__main__':

    import time

​

    class ExpiringDict(dict):

        def __init__(self, *args, **kwargs):

            super().__init__(*args, **kwargs)

            self.expiry_times = {}

​

        def __setitem__(self, key, value, expiry=None):

            super().__setitem__(key, value)

            if expiry:

                self.expiry_times[key] = time.time() + expiry

​

        def __getitem__(self, key):

            if key in self.expiry_times and time.time() > self.expiry_times[key]:

                self.__delitem__(key)

                raise KeyError

            else:

                return super().__getitem__(key)

​

    cache = ExpiringDict()

​

    cache['AAPL'] = 150, 10

    time.sleep(11)

    try:

        print(cache['AAPL'])

    except KeyError:

        print('Key expired')

Least Recently Used (LRU) caching is a cache replacement policy that removes the least recently used items first. This algorithm is often used for memory management or Disk I/O, similar to how an OS operates. The purpose is to maximize data that is hot in cache and reduce expensive fetch operations.

Consider a finance application where frequent stock market updates occur. Some stocks (e.g., AAPL, TSLA) might be accessed more frequently than others (e.g., XYZ). In this case, the LRU algorithm can keep frequently accessed data 'hot' in the cache. When cache becomes full, the algorithm will eliminate the 'coldest' data or the data that hasn't been accessed in a while.

To implement an LRU cache in Python, one can use an OrderedDict. This data structure maintains insertion order, which allows us to easily decide which item to remove when cache is full, i.e., the item at the top (first inserted) should be removed first. To make an item hot, we can move it to the end of the OrderedDict when accessed.

Let's look at some basic code that demonstrates how this might be done.

xxxxxxxxxx

if __name__ == '__main__':

  from collections import OrderedDict

​

  class LRUCache(OrderedDict):

    def __init__(self, capacity):

      self.capacity = capacity

​

    def get(self, key):

      if key not in self:

        return -1

      self.move_to_end(key)

      return self[key]

​

    def put(self, key, value):

      if key in self:

        self.move_to_end(key)

      self[key] = value

      if len(self) > self.capacity:

        self.popitem(last=False)

​

  cache = LRUCache(2)

  cache.put('AAPL', 120)

  cache.put('TSLA', 600)

  print(cache.get('AAPL')) # output: 120

  cache.put('XYZ', 20) # this will remove 'TSLA' from the cache as it's the least recently used.

  print(cache.get('TSLA')) # output: -1

In the context of our key-value store, let's add an LRU cache to handle scenarios where memory capacity may not be sufficient to hold all keys. Check out the following Python code block which defines a basic implementation of an LRU cache. The cache is created using Python's OrderedDict which maintains order of insertion. This will allow us to remove the least recently used item when the cache reaches its capacity.

Two main methods are defined in our implementation:

1. get(key: int): This method returns the value of the key if it exists in the cache and moves the key to the end to mark it as recently used. If the key doesn't exist in the cache, it returns -1.

2. put(key: int, value: int): This method adds a particular key-value pair to the cache. If the key already exists, it updates the value and moves the key to the end to denote recent usage. If the cache is full, it removes the least recently used item before adding the new key-value pair.

Think of a practical use for this algorithm - it's similar to how memory is managed in cloud-based AI applications that require huge data processing. Data that is accessed often (hot data) stays in the cache and data that is rarely accessed (cold data) is kicked out as the cache fills up. This optimizes memory usage, providing quick access to frequently used data.

xxxxxxxxxx

    print(cache.get(4))        # returns 4

from collections import OrderedDict

​

class LRUCache:

    def __init__(self, capacity: int):

        self.cache = OrderedDict()

        self.capacity = capacity

​

    def get(self, key: int) -> int:

        if key not in self.cache:

            return -1

        else:

            self.cache.move_to_end(key)

            return self.cache[key]

​

    def put(self, key: int, value: int) -> None:

        self.cache[key] = value

        self.cache.move_to_end(key)

        if len(self.cache) > self.capacity:

            self.cache.popitem(last=False)

​

if __name__ == '__main__':

    cache = LRUCache(2)

​

    cache.put(1, 1)

    print(cache.get(1))        # returns 1

    cache.put(3, 3)            # evicts key 2

    print(cache.get(2))        # returns -1 (not found)

    cache.put(4, 4)            # evicts key 1

    print(cache.get(1))        # returns -1 (not found)

Congratulations, you made it! We've been on a journey to expand our simple key-value store and have added integral features such as expiration times and LRU caching, learning from the popular data store, Redis.

Our key-value store has been revved up. It can now hold values for a specified time and uses LRU caching for handling keys when memory capacity becomes a potential concern while also preserving performance. We mimic Redis in two main aspects: fast access to stored data and optimized memory use.

The expanded key-value store can now be seen as a rudimentary version of Redis, the ultra-fast, in-memory data store with rich data structure support. Does it mean our key-value store is as powerful as Redis? Not entirely, because Redis has much more complex features (such as data replication, persistence options, various types of data structures, built-in Lua scripting, etc.). However, we've gained a glimpse into the inner workings of Redis and built something similar. This can be a great confidence booster, ever boosting your understanding of computer science and software development.

Here's the final snapshot of our key-value store implemented in Python:

xxxxxxxxxx

if __name__ == '__main__':

    from collections import OrderedDict

    import time

​

    class LRU_Cache:

        def __init__(self, capacity):

            self.capacity = capacity

            self.cache = OrderedDict()

​

        def get(self, key):

            try:

                value = self.cache.pop(key)

                self.cache[key] = value

                return value

            except KeyError:

                return -1

​

        def put(self, key, value):

            try:

                self.cache.pop(key)

            except KeyError:

                if len(self.cache) >= self.capacity:

                    self.cache.popitem(last=False)

            self.cache[key] = value

​

    key_value_store = LRU_Cache(3)

    key_value_store.put(1, 'value1')

    key_value_store.put(2, 'value2')

    print(key_value_store.get(1))  # Output: 'value1'

By the end of this expansive lesson, we have greatly leveled up our basic key-value store. We found inspiration in Redis, a powerhouse in-memory data store, to implement features such as expiration times for keys and LRU caching, which are vital for optimizing memory usage and performance.

We explored deep into computer science concepts, gaining invaluable insights into how Redis, a software staple in various industries including AI and finance, operates under the hood. This surely expanded our dimension of knowledge and set a strong foundation for us to further delve into software development.

Remember, our improved key-value store is still a basic version when compared to Redis. Redis supports complex functionality like data replication, various data structures, Lua scripting, to name a few. Nonetheless, our expanded key-value store is a strong and confident stride into the world of building datastores from scratch, giving us a much comprehensive understanding of their working.

We encourage further exploration and experimentation with the key-value store, maybe adding enhancements or other features from Redis. The journey of learning and discovery continues, carrying forward our growth in the realm of software development and computer science.

xxxxxxxxxx

if __name__ == "__main__":

  # Final version of our Key-Value store

  store = KeyValueStore()

  store.set('AI', 'Artificial Intelligence')

  store.set('ML', 'Machine Learning')

  print(store.get('AI'))

  print(store.get('ML'))

  # Demonstrate LRU eviction

  for i in range(store.limit):

    store.set(f'key{i}', f'value{i}')

  print('Store state after LRU eviction', store.get_store())

​

  # Demonstrate expiration

  import time

  store.set('exp_key', 'exp_value', 5)

  time.sleep(5)

  print('Exp_key after expiration', store.get('exp_key'))