We are using Twemproxy to replace Memcached::addServers recently for faster processing speed, especially for Memcached::Get. After some benchmarks, we found that Memcache(PHP) is, however, 5-10% faster than Memcached(PHP). So we rushed up replacing the whole program by Memcache(PHP) + Twemproxy. But then problem came, the Memcache::Delete failed with unknown reason. This note shows how I found out the problem and solve it.
1. After testings with tcpdump, we confirmed that there are packages sending from Twemproxy to Memcached daemon, which means it really works.
2. Open the Twemproxy’s log with highest level: -v 11
3. Runs Memcache::Delete(PHP) again and catched the following logs:
[codesyntax lang=”bash” lines=”no”]
[Thu Dec 6 19:16:49 2012] nc_memcache.c:712 parsed bad req 114 res 1 type 3 state 17 00000000 64 65 6c 65 74 65 20 74 65 73 74 74 65 73 74 32 |delete testtest2| 00000010 32 20 30 0d 0a |2 0..| [Thu Dec 6 19:16:49 2012] nc_core.c:168 recv on c 16 failed: Invalid argument
[/codesyntax]
4. Runs Memcached::Delete(PHP) again instead:
[codesyntax lang=”bash” lines=”no”]
[Thu Dec 6 19:33:51 2012] nc_memcache.c:702 parsed req 130 res 0 type 3 state 0 rpos 19 of 19
00000000 64 65 6c 65 74 65 20 74 65 73 74 74 65 73 74 32 |delete testtest2|
00000010 32 0d 0a |2..|
[Thu Dec 6 19:33:51 2012] nc_server.c:640 key ‘testtest22’ on dist 0 maps to server ‘10.11.80.47:22223:1’
[/codesyntax]
5. Based on the source code of Memcached(C), “Invalid argument” is thrown when sending incorrect command.
6. We go on digging the source code of Twemproxy. Error “nc_memcache.c:712 parsed bad req 114 res 1 type 3 state 17” means below:
req 114 : error!
res 1 : MSG_PARSE_ERROR
type 3 : msg_type.MSG_REQ_MC_DELETE
state 17 : SW_RUNTO_CRLF
We could, now 99% confirm that Memcache(PHP) did send a WRONG command.
7. Comparing logs in 3 and 4, it’s obviously that there are two more characters: “0x20 0x30 -> 0” within Memcache(PHP)’s log. The ASCII shows us they are “space” and “\0”.
8. How about the source code of Memcache(PHP)…? It’s a great pity that the deprecated “time” is STILL inside the stable releases!
[codesyntax lang=”C” lines=”no”]
command_len = spprintf(&command, 0, “delete %s %d”, key, time);
command[command_len] = ‘\0’;
[/codesyntax]
9. So, fixing it is just as easy as removing the “time”! Then recompile/install.
[codesyntax lang=”C” lines=”no”]
command_len = spprintf(&command, 0, “delete %s”, key);
[/codesyntax]
10. The conclusion goes as follow:
(1) Using Twemproxy + Memcache(self-compiling version with bug fixed)
(2) Using Memcached instead, which is 5-10% slower but nothing deprecated.
11. We finally go with 2nd choice.