docker空间爆满导致的进入容器失败的解决方案

由于问题发生的比较突然，业务催的比较急，所以没来得及截图案发现场，所以现场截图为后期正常的环境。

周一上班，照例进入服务器，进入docker容器

>>> docker exec -i -t xxx /bin/bashDocker error : no space left on device没空间了？

立马想到查看系统的空间

>>> df -h

Filesystem Size Used Avail Use% Mounted ondevtmpfs 3.9G 0 3.9G 0% /devtmpfs 3.9G 0 3.9G 0% /dev/shmtmpfs 3.9G 984K 3.9G 1% /runtmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup/dev/vda1 50G 50G 0G 100% /overlay 50G 5.9G 41G 13% /var/lib/docker/overlay2/4d0941b78fa413f3b77111735e06045b41351748bcea7964205bcfbf9d4ec0b6/mergedoverlay 50G 5.9G 41G 13% /var/lib/docker/overlay2/54a7a7b66d9c12d7e42158d177a6b67321f7da1f223b301e838e9bc109a2bda4/mergedshm 64M 0 64M 0% /var/lib/docker/containers/712f65baea85d898f6c948e7149f84f2f5eaf3b1934540603f32ab278f9acec4/mounts/shmshm 64M 0 64M 0% /var/lib/docker/containers/a5fa673086c8f46ba98fc4425b353ed2e12de3277a5fe2dc5b8b7affa63b4518/mounts/shmoverlay 50G 5.9G 41G 13% /var/lib/docker/overlay2/7fbbc8a29119a1eaa1f212c50b75405a1f16fd68e3ae3949cc0c963d0727a9ab/mergedshm 64M 0 64M 0% /var/lib/docker/containers/011a83deceacecbacb4ef7eb06eb5b812babf9e83914a4fb33d4925cc1ad375b/mounts/shmtmpfs 783M 0 783M 0% /run/user/0原来是根目录满了

进一步查找

>>> cd />>> du -h --max-depth=1

984K ./run16K ./opt13M ./root4.0K ./mediadu: cannot access ‘./proc/4382/task/4382/fd/4’: No such file or directorydu: cannot access ‘./proc/4382/task/4382/fdinfo/4’: No such file or directorydu: cannot access ‘./proc/4382/fd/3’: No such file or directorydu: cannot access ‘./proc/4382/fdinfo/3’: No such file or directory0 ./proc204M ./boot12K ./redis39M ./etc16K ./lost+found4.0K ./srv0 ./sys47G ./var2.8G ./usr4.0K ./mnt36K ./tmp0 ./dev4.0K ./home50G .

>>> cd var>>> du -h --max-depth=1

116M ./cache8.0K ./empty4.0K ./games4.0K ./opt24K ./db46G ./lib4.0K ./gopher4.0K ./adm4.0K ./crash12K ./kerberos4.0K ./preserve4.0K ./nis16K ./tmp4.0K ./yp4.0K ./local104K ./spool374M ./log47G .

>>> cd lib>>> du -h --max-depth=1

76K ./systemd24K ./NetworkManager4.0K ./tuned4.0K ./games248K ./cloud215M ./rpm8.0K ./plymouth46G ./docker248K ./containerd4.0K ./dbus4.0K ./initramfs4.0K ./os-prober8.0K ./rsyslog24K ./alternatives8.0K ./authconfig12K ./stateless4.0K ./misc4.0K ./ntp8.0K ./dhclient4.0K ./selinux8.0K ./chrony4.0K ./rpm-state12M ./yum8.0K ./postfix28K ./polkit-14.0K ./machines8.0K ./logrotate46G .

>>> cd docker>>> du -h --max-depth=1

84K ./network108K ./buildkit4.0K ./trust4.0K ./runtimes5.7M ./image42G ./volumes24K ./plugins4.0K ./tmp20K ./builder180K ./containers3.3G ./overlay24.0K ./swarm46G .

>>> cd volume>>> du -h --max-depth=1

172K ./kudu_to_jdy_kudu_to_jdy42G ./jdy_extensions_logs748K ./bot_etl_bot_etl42G .

原来是jdy_extensions_logs这个volume爆满导致的，后面就是找代码bug的过程了，略

其实docker提供了很多命令来对容器镜像进行管理，其中的docker system df就是一个比较有用的命令

# 显示docker文件系统使用情况>>> docker system df

TYPE TOTAL ACTIVE SIZE RECLAIMABLEImages 7 3 2.146GB 693.2MB (32%)Containers 3 3 127.5kB 0B (0%)Local Volumes 3 3 1.511MB 0B (0%)Build Cache 0 0 0B 0B

# 显示空间使用情况>>> docker system df -v

Images space usage:REPOSITORY TAG IMAGE ID CREATED SIZE SHARED SIZE UNIQUE SIZE CONTAINERSjdy_extension 1.7 e3be3be9664a 15 hours ago 627.6MB 73.86MB 553.7MB 1<none> <none> 4b5825747ae9 19 hours ago 74.02MB 73.86MB 158.8kB 0redis 6.2.1 f877e80bb9ef 2 weeks ago 105.3MB 0B 105.3MB 0kudu_to_jdy 2.9 888b72288bca 2 weeks ago 538.1MB 73.86MB 464.3MB 1jdy_to_db 1.9 c345c4e15c1a 7 months ago 587.7MB 73.86MB 513.9MB 0bot_etl 2.3 020d41691ec7 7 months ago 508.6MB 73.86MB 434.8MB 1ubuntu 20.04 adafef2e596e 8 months ago 73.86MB 73.86MB 0B 0Containers space usage:CONTAINER ID IMAGE COMMAND LOCAL VOLUMES SIZE CREATED STATUS NAMES011a83deceac jdy_extension:1.7 'supervisord -n -c /…' 1 17.8kB 15 hours ago Up 15 hours jdy_extension_1.7712f65baea85 kudu_to_jdy:2.9 'supervisord -n -c /…' 1 36.9kB 2 weeks ago Up 45 hours kudu_to_jdy_2.9a5fa673086c8 bot_etl:2.3 'supervisord -n -c /…' 1 72.8kB 7 months ago Up 45 hours bot_etl_2.3Local Volumes space usage:VOLUME NAME LINKS SIZEjdy_extensions_logs 1 658.5kBkudu_to_jdy_kudu_to_jdy 1 128kBbot_etl_bot_etl 1 724kBBuild cache usage: 0BCACHE ID CACHE TYPE SIZE CREATED LAST USED USAGE SHARED

# 删除所有没有在用的volume>>> docker volume rm $(docker volume ls -q)

补充：记一次docker磁盘占用100%情况

在执行git pull origin master的时候，抛出如下错误：

error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500

fatal: the remote end hung up unexpectedly

百度之后好多说是nginx文件大小限制，但是查看配置文件，大小限制已经设置到了500M，应该不是该原因造成的：

server{ listen 80; server_name localhost; client_max_body_size 500M; location / { proxy_redirect off; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_pass http://localhost:180; }}

然后使用df -h查看磁盘情况，发现data2磁盘已经100%（docker存储在这里）：

使用docker system prune -a释放了很多空间，问题解决。

注意：使用上述命令会删除一下内容

WARNING! This will remove: - all stopped containers - all networks not used by at least one container - all images without at least one container associated to them - all build cacheAre you sure you want to continue? [y/N] y

以上为个人经验，希望能给大家一个参考，也希望大家多多支持好吧啦网。如有错误或未考虑完全的地方，望不吝赐教。