python - python subprocess.Popen"OSError: [Errno 12] Cannot allocate memory"

python脚本使用sched模块,每60秒运行一组类函数:


# sc is a sched.scheduler instance
sc.enter(60, 1, self.doChecks, (sc, False))

在doChecks中使用subprocess模块来调用系统函数,以获取系统统计信息:


ps = subprocess.Popen(['ps', 'aux'], stdout=subprocess.PIPE).communicate()[0]

在整个脚本崩溃以下错误之前,这段时间运行得很好:


File"/home/admin/sd-agent/checks.py", line 436, in getProcesses
File"/usr/lib/python2.4/subprocess.py", line 533, in __init__
File"/usr/lib/python2.4/subprocess.py", line 835, in _get_handles
OSError: [Errno 12] Cannot allocate memory

脚本崩溃后,服务器上free -m的输出为:


$ free -m
 total used free shared buffers cached
Mem: 894 345 549 0 0 0
-/+ buffers/cache: 345 549
Swap: 0 0 0

服务器正在运行CentOS 5.3 ,

我尝试了许多调试此问题的方法,如原始问题中所示:

  1. 在Popen调用之前和之后记录free -m的输出,换句话说,内存使用没有显著变化,因为脚本运行时内存不会逐渐被使用。

  2. 我在Popen调用中添加了close_fds=True,但是这没有什么不同—脚本仍然因相同的错误而崩溃,

  3. 我检查了rlimits显示RLIMIT_DATA和RLIMIT_AS上的(-1,-1 ),如下所示这里

脚本在崩溃之前与下面的输出一起运行:


recv(4,"Total Accesses: 516662nTotal kBy"..., 234, 0) = 234
gettimeofday({1250893252, 887805}, NULL) = 0
write(3,"2009-08-21 17:20:52,887 - checks"..., 91) = 91
gettimeofday({1250893252, 888362}, NULL) = 0
write(3,"2009-08-21 17:20:52,888 - checks"..., 74) = 74
gettimeofday({1250893252, 888897}, NULL) = 0
write(3,"2009-08-21 17:20:52,888 - checks"..., 67) = 67
gettimeofday({1250893252, 889184}, NULL) = 0
write(3,"2009-08-21 17:20:52,889 - checks"..., 81) = 81
close(4) = 0
gettimeofday({1250893252, 889591}, NULL) = 0
write(3,"2009-08-21 17:20:52,889 - checks"..., 63) = 63
pipe([4, 5]) = 0
pipe([6, 7]) = 0
fcntl64(7, F_GETFD) = 0
fcntl64(7, F_SETFD, FD_CLOEXEC) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb7f12708) = -1 ENOMEM (Cannot allocate memory)
write(2,"Traceback (most recent call last"..., 35) = 35
open("/usr/bin/sd-agent/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/bin/sd-agent/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python24.zip/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/plat-linux2/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python2.4/lib-tk/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/lib-dynload/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/site-packages/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
write(2," File"/usr/bin/sd-agent/agent."..., 52) = 52
open("/home/admin/sd-agent/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/bin/sd-agent/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python24.zip/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/plat-linux2/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python2.4/lib-tk/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/lib-dynload/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/site-packages/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
write(2," File"/home/admin/sd-agent/dae"..., 60) = 60
open("/usr/bin/sd-agent/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/bin/sd-agent/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python24.zip/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/plat-linux2/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python2.4/lib-tk/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/lib-dynload/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/site-packages/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
write(2," File"/usr/bin/sd-agent/agent."..., 54) = 54
open("/usr/lib/python2.4/sched.py", O_RDONLY|O_LARGEFILE) = 8
write(2," File"/usr/lib/python2.4/sched"..., 55) = 55
fstat64(8, {st_mode=S_IFREG|0644, st_size=4054, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7d28000
read(8,""""A generally useful event sche"..., 4096) = 4054
write(2,"", 4) = 4
write(2,"void = action(*argument)n", 25) = 25
close(8) = 0
munmap(0xb7d28000, 4096) = 0
open("/usr/bin/sd-agent/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/bin/sd-agent/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python24.zip/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/plat-linux2/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python2.4/lib-tk/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/lib-dynload/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/site-packages/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
write(2," File"/usr/bin/sd-agent/checks"..., 60) = 60
open("/usr/bin/sd-agent/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/bin/sd-agent/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python24.zip/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/plat-linux2/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python2.4/lib-tk/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/lib-dynload/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/site-packages/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
write(2," File"/usr/bin/sd-agent/checks"..., 64) = 64
open("/usr/lib/python2.4/subprocess.py", O_RDONLY|O_LARGEFILE) = 8
write(2," File"/usr/lib/python2.4/subpr"..., 65) = 65
fstat64(8, {st_mode=S_IFREG|0644, st_size=39931, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7d28000
read(8,"# subprocess - Subprocesses with"..., 4096) = 4096
read(8,"lso, the newlines attribute of t"..., 4096) = 4096
read(8,"code < 0:n print >>sys.st"..., 4096) = 4096
read(8,"alse does not exist on 2.2.0ntry"..., 4096) = 4096
read(8," p2creadn # c2pread <-"..., 4096) = 4096
write(2,"", 4) = 4
write(2,"errread, errwrite)n", 19) = 19
close(8) = 0
munmap(0xb7d28000, 4096) = 0
open("/usr/lib/python2.4/subprocess.py", O_RDONLY|O_LARGEFILE) = 8
write(2," File"/usr/lib/python2.4/subpr"..., 71) = 71
fstat64(8, {st_mode=S_IFREG|0644, st_size=39931, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7d28000
read(8,"# subprocess - Subprocesses with"..., 4096) = 4096
read(8,"lso, the newlines attribute of t"..., 4096) = 4096
read(8,"code < 0:n print >>sys.st"..., 4096) = 4096
read(8,"alse does not exist on 2.2.0ntry"..., 4096) = 4096
read(8," p2creadn # c2pread <-"..., 4096) = 4096
read(8,"table(self, handle):n"..., 4096) = 4096
read(8,"rrno using _sys_errlist (or siml"..., 4096) = 4096
read(8," p2cwrite = None, Nonen"..., 4096) = 4096
write(2,"", 4) = 4
write(2,"self.pid = os.fork()n", 21) = 21
close(8) = 0
munmap(0xb7d28000, 4096) = 0
write(2,"OSError", 7) = 7
write(2,":", 2) = 2
write(2,"[Errno 12] Cannot allocate memor"..., 33) = 33
write(2,"n", 1) = 1
unlink("/var/run/sd-agent.pid") = 0
close(3) = 0
munmap(0xb7e0d000, 4096) = 0
rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x589978}, {0xb89a60, [], SA_RESTORER, 0x589978}, 8) = 0
brk(0xa022000) = 0xa022000
exit_group(1) = ?

时间:

你是否尝试过:


(status,output) = commands.getstatusoutput("ps aux")

这已经为我解决了同样的问题,

经过一些测试之后,我发现这只发生在旧版本的python上: 它发生在2.6.5,但不在2.7.2


watch"ls /proc/$PYTHONPID/fd | wc -l"

munmap(0xb7d28000,4096)= 0
write(2,“ OSError”,7)= 7

我看到的代码看起来像这样:


serrno = errno;
some_Syscall(...)
if (serrno != errno)
/* sound alarm: CATROSTOPHIC ERROR !!! */

你应该检查一下python代码中是否发生了这种情况,只有在执行系统调用失败时,Errno才有效。

  • 分支过程
  • 未使用的数据结构
  • 共享函数库
  • 内存映射文件

使用 GDB跟踪进一步挖掘 clone() 调用和/或者后续调用中发生的事情。 因为everthe是 操作系统,你很快就会到达 root,或者以后没有垫子。

看着 free -m的输出,我觉得你实际上没有交换内存。 我不确定在Linux中,交换是否总是可以根据需要自动使用,但我有同样的问题,没有任何答案真正帮助我。 但是,添加一些交换内存,将问题固定在我的案例中,因为这可能会帮助其他人遇到同样的问题,我将回答如何添加 1 GB交换( 在 Ubuntu 12.04上,但它应该类似于其他发行版)

你可以首先检查是否启用了交换内存。

 
$sudo swapon -s

 

如果为空,表示你没有启用任何交换。 要添加 1 GB的交换:


$sudo dd if=/dev/zero of=/swapfile bs=1024 count=1024k
$sudo mkswap/swapfile
$sudo swapon/swapfile

将以下行添加到 fstab 以使交换永久化。


$sudo vim/etc/fstab

/swapfile none swap sw 0 0 

源和更多信息可以在这里找到

...