Bref using all space in /tmp folder

[atrope]

Hi,

I was having some issues with our caching files (it writes to /tmp) , telling me that /tmp was full.

Debugging it with i found out that it was core dump logs that were taking all space:

Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/xvda1     ext4       30G  5.4G   24G  19% /
/dev/loop1     ext4      526M  515M     0 100% /tmp
/dev/loop7     squashfs   43M   43M     0 100% /opt
total 514M
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.106
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.108
-rw------- 1 sbx_user1075 471 143M Mar 15 03:21 core.php-fpm.11
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.110
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.116
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.118
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.119
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.120
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.121
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.122
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.125
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.126
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.127
-rw------- 1 sbx_user1075 471 142M Mar 15 03:21 core.php-fpm.13
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.130
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.132
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.133
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.134
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.135
-rw------- 1 sbx_user1075 471 150M Mar 15 03:21 core.php-fpm.14
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.141
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.142
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.144
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.145
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.146
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.147
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.148
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.151
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.152
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.154
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.155
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.156
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.157
-rw------- 1 sbx_user1075 471  93M Mar 15 03:21 core.php-fpm.16
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.163
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.165
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.166
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.167
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.168
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.169
-rw------- 1 sbx_user1075 471    0 Mar 15 03:21 core.php-fpm.17
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.177
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.178
-rw------- 1 sbx_user1075 471    0 Mar 15 03:21 core.php-fpm.18
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.185
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.186
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.188
-rw------- 1 sbx_user1075 471    0 Mar 15 03:21 core.php-fpm.20
-rw------- 1 sbx_user1075 471    0 Mar 15 03:21 core.php-fpm.21
-rw------- 1 sbx_user1075 471    0 Mar 15 03:21 core.php-fpm.22
-rw------- 1 sbx_user1075 471    0 Mar 15 03:21 core.php-fpm.26
-rw------- 1 sbx_user1075 471    0 Mar 15 03:21 core.php-fpm.32
-rw------- 1 sbx_user1075 471    0 Mar 15 03:21 core.php-fpm.35
-rw------- 1 sbx_user1075 471    0 Mar 15 03:21 core.php-fpm.36
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.38
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.42
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.43
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.44
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.45
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.46
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.47
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.48
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.49
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.51
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.54
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.55
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.56
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.58
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.59
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.60
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.61
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.62
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.65
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.73
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.74
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.75
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.77
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.79
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.80
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.85
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.87
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.89
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.90
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.92
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.93
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.95
-rw------- 1 sbx_user1075 471    0 Mar 15 03:22 core.php-fpm.96

I Managed to mitigate it by adding rlimit_core = 1 in php-fpm.conf
It did solved the problem, but i don't know if we need this for other things so i am holding on PR for this.

Anyone has any comment if we should or shouldn't add this as default value?

Thanks

[mnapoli]

Thank you for the info!

@bubba-h57 might know a bit more about FPM as well.

Limiting could make sense, but what are those logs? FPM logs should end up in stderr (per the config, and that's what we want because those end up on CloudWatch). Are those memory dumps like when a segfault happens?

Could you check in cloudwatch if there are lines like exited on signal 11 or core dumped?

[atrope]

Are those memory dumps like when a segfault happens?

Yes it exited with core dumped.

[bubba-h57]

If we want to ignore the core dumps altogether, then adding rlimit_core = 1 in php-fpm.conf is the way to go.

However, I would argue that is probably not what we want to do, because then folks will have coredumps and be ignorant of it.

Rather, I suggest that we modify the bootstrap so that after every event, when we check that FPM is still running and restart if it is not, we simultaneously check for any core dumps. If they are present, we read/echo them stdout so that they show up in the cloudwatch logs ... then clean up the mess by deleting core.php-fpm* prior to handling the next event. I believe that gives us the best of all worlds.

Thoughts?

[mnapoli]

Sounds really good! Aren't those dump huge memory dumps? Or are they just stack traces? (as you can probably tell I am ignorant on that topic ^^)

[bubba-h57]

It is a file containing a process's address space (memory) when the process terminates unexpectedly ... you can indeed get a stack trace from it, though it would not be what typical PHP Exception stacktrace would look like.

As far as size, the majority of the ones cleaned up would be no size at all, just a zero byte file taking up an inode. For PHP in Lambda ... I suspect a big one to be around 200MB and average around 100mb. But the technical term for that estimation is wild arse guess. :-)

But, if I am smellin' what yer steppin' in regarding the size ... it is probably a good idea to default to just cleaning up the '/tmp' dir of any core dumps there and if the developer likes, allow for an environment variable to be set that, if set, will write to CloudWatch. Or maybe even be more clever and if the environment variable is set to an S3_bucket we upload the dump to S3?

More configuration options are always complicated ... I mean ... Good. :-)

[bubba-h57]

@mnapoli In retrospect, I have changed my opinion. I think we should just add rlimit_core = 1 in php-fpm.conf ... if the developer really wants the core dumps they can modify the setting themselves. Whoever wants to look at coredumps would necessarily need to know how to use the tools required to interpret it, which means they are likely comfortable with Assembly/Assemblers/Disassemblers. I suspect that those individuals would find it no problem to manage to sort out how to modify the php-fpm.conf if they so desire.

[mnapoli]

That sounds good!

[atrope]

@mnapoli So should we add this to bref?
It's a simple line addition in php-fpm.conf

[mnapoli]

@atrope yes let's do this 👍

[mnapoli]

#321 should fix this, this is merged and I built new runtimes: https://runtimes.bref.sh/

Thank you both!

[atrope]

@mnapoli we should have changelogs for the layers no?

[mnapoli]

Ideally yes, I haven't taken the time to set it up yet. My goal would be fix layer versions with Bref versions (tags), #320 might make that possible. That way layers changelog can be in Bref releases.