2

I have 2 different cache type as follow: (I am using MySQL connector because the client wants it like this.)

import os,sys
from pathlib import Path

sys.path.append(os.path.dirname(os.path.abspath('.')))
os.environ['DJANGO_SETTINGS_MODULE'] = 'api.settings'
import django
django.setup()


import mysql.connector

from django.core.cache import cache,caches


db_crawl = mysql.connector.connect(
        host="xx.xx.xx.xx",
        user="xxx",
        password="xxxx",
        database="xxx"
)

cursor_crawl = db_crawl.cursor(dictionary=True)

def marketprices():
    query = "some query here"
    cursor_crawl.execute(query)
    result = cursor_crawl.fetchall()
    m_cache = caches['marketprice']
    m_cache.set('marketprices', result,1024 * 1024 * 1024)

The query output has about 18 MB cache and I want to hold the cache 1024 * 1024 * 1024 times.

My cache settings:

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.filebased.FileBasedCache',
        'LOCATION': BASE_DIR.joinpath('caches'),
        'TIMEOUT':None,
        'OPTIONS': {
            'server_max_value_length': 1024 * 1024 * 1024,
        }
    },
    'marketprice': {
        'BACKEND': 'django.core.cache.backends.memcached.PyMemcacheCache',
        'LOCATION': '127.0.0.1:11211',
        'TIMEOUT':None,
    },
} 

I set and start the Memcached in my server and everything works fine. The service Memcached status is:

● memcached.service - memcached daemon
     Loaded: loaded (/lib/systemd/system/memcached.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2021-07-19 22:33:49 +03; 1h 10min ago
       Docs: man:memcached(1)
   Main PID: 2126782 (memcached)
      Tasks: 10 (limit: 72283)
     Memory: 1.6M
     CGroup: /system.slice/memcached.service
             └─2126782 /usr/bin/memcached -I 256M -m 8192 -p 11211 -u memcache -l 127.0.0.1 -P /var/run/memcached/memcached.pid

Jul 19 22:33:49 vmi552735.contaboserver.net systemd[1]: Started memcached daemon.

However when I try in python manage.py shell the following code every works fine too.

>> m_cache = caches['marketprice']
>> m_cache.set("foo","bar")
>> m_cache.get('foo')
>> bar

After running the marketprices function it takes about 2 minutes to finish and there is no error. I also checked the query there is no error at the cursor but when I want to display the output of the following view function:

m_cache = caches['marketprice']
datas = m_cache.get('marketprices')
print(datas[0])

I am getting the error: 'NoneType' object is not iterable

I also checked the m_cache.get('marketprices') in manage.py shell nothing return.

I think Memcached doesn't hold the data in memory what it cause the problem and what is your solution?

Murat Demir
  • 716
  • 7
  • 26

0 Answers0