How to link docker images to their composing layers on the disk? -

since docker v1.10, introduction of content addressable storage, docker has changed way image data handled on disk. understand layers , images separated. layers merely become collections of files , directories have no notion of images , can freely shared across images. see update , blog better explanation.

during docker push , docker pull, via stdout can seen layers transported, though resulting sha hashes regenerated on destination.

with locally built image ubuntu:14.04 base, when use docker history command, can see chain of intermediary images used during build process, , disk space usage contributed.

root@ruifeng-virtualbox:/var/lib/docker/aufs/diff# docker history image_size image               created             created                                      size                comment 9ae1f372d83c        11 weeks ago        /bin/sh -c #(nop)  cmd ["/bin/sh" "-c" "/bin/   0 b                  aaf66e9fa85b        11 weeks ago        /bin/sh -c chown -r martian /home/martian       6.299 mb             9568768134c1        11 weeks ago        /bin/sh -c rm -rf /home/martian/potatoes        0 b                  2f40f3f58306        11 weeks ago        /bin/sh -c mv /home/martian/water_tanks /home   6.289 mb             062e2702ffa2        11 weeks ago        /bin/sh -c mv /home/martian/potatoes /home/ma   5.394 kb             7b2d8b4c1dd0        11 weeks ago        /bin/sh -c chown -r martian /home/martian       6.299 mb             8fd47fed98d6        11 weeks ago        /bin/sh -c #(nop) copy dir:421da6c71a1f252881   6.289 mb             ... 

and can use docker inspect command underlying layers.

root@ruifeng-virtualbox:/var/lib/docker/aufs/diff# docker inspect image_size | jq -r '.[].rootfs' {   "layers": [     "sha256:a85f35566a268e6f4411c5157ffcffe4f5918b068b04d79fdd80003901ca39da",     "sha256:eaaf7298332642da0f8190fa4b96ad46c04b9c1d1682bc3a35d77bded2b1e0a9",     "sha256:33a212e8aa5642d3a2ddead146e85912407fc5bbb2a896dab11fcf329177a999",     "sha256:f1f25d8c6e56dc4891df147a77f57e756873b57f33ce95e6a0acbe47117c0c8a",     "sha256:67852b7d2cf5f0885293fa9df91ebfd8ef0c42ba11a5155f94806f3a96c5e916",     "sha256:480d48b7e2864a44c1b2fca0c7e32fbab505f7526ccb25bbfed191c04a9bb7b0",     "sha256:18d270fe64aa423e0ffdf24faf0103432027da3d5c12f4505e7daedad9fe2195",     "sha256:a73c3f5eb83790bc6d03381a43a20aef7d0d9d97de0cff4b040e8e4c01a3aee5",     "sha256:e8d1b67ace73cb92cc00725354e84024153bedae4280149c03fcb52f34d83757",     "sha256:19a4b80afc677825fec94adf8b6a45a866f42a38675f87f86e50171ff5e0a280",     "sha256:77d412270fbdd9baba1fe73028b786c3a1709feefa9b03be74b8e9f9ce148635",     "sha256:2ad21e37389addd577161c981d0c69ab60aa47945172f41f9ec71ada1c1dd4ee",     "sha256:771d1e47ca8d8dcf55069786e4c499894fba86f704c808413df00f4f980564e1",     "sha256:f9c02c6fa436213c0f220d49c4ee1b913372081010d4506757ec75d3e788847c"   ],   "type": "layers" } 

my question is, how link these layers marked sha hashes images listed in image column of previous command output? , there way find out actual location , size of these layers on disk?

if not wrong, layers should kept @ /var/lib/docker/aufs/diff if storage driver selection aufs. contents in folder named randomly generated ids not match of layer literally. seems match kept within docker engine security concerns.

it seems match kept within docker engine security concerns.

well, has stored somewhere on disk because information needs persistent. i'm using overlay2 driver rather aufs, i'm guessing layout relatively similar. let's start image have locally:

# docker images | grep alpine alpine                                              latest              baa5d63471ea        5 months ago        4.8 mb 

which has following layers:

# docker inspect alpine | jq '.[0].rootfs' {   "type": "layers",   "layers": [     "sha256:011b303988d241a4ae28a6b82b0d8262751ef02910f0ae2265cb637504b72e36"   ] } 

let's in /var/lib/docker matching image id prefix:

# cd /var/lib/docker # find . -name 'baa5d63471ea*' -print ./image/overlay2/imagedb/content/sha256/baa5d63471ead618ff91ddfacf1e2c81bf0612bfeb1daf00eb0843a41fbfade3 

that json file containing bunch of data, including looks relevant:

  "rootfs": {     "type": "layers",     "diff_ids": [       "sha256:011b303988d241a4ae28a6b82b0d8262751ef02910f0ae2265cb637504b72e36"     ]   } 

great! using information, should able take layer id , find images use it. example, have locally built image looks like:

# docker inspect larsks/qgroundcontrol | jq '.[0].rootfs' {   "type": "layers",   "layers": [     "sha256:c854e44a1a5a22c9344c90f68ef6e07fd479e8ce1030fe141e3236887f1a4920",     "sha256:8ba4b4ea187c5ea58c11ee99bbc159b88b303c290b18c2220c9b477f4427bb9e",     "sha256:46c98490f5756634de1b1b9ed02a9fae2732984049f4f8fa182959fea924a45c",     "sha256:1633f88f8c9fa73c5c0c24f314e81b10dda6c310d41fb87eba02421e1652f6dc",     "sha256:0e20f4f8a593705219d1b3c5b1d2f7b8664eb04d706e99add87adbdcceea4a9f",     "sha256:cb16829cadf4f4320799bdf23f7400816f1552a011f3e30c2c929382896c3f6f",     "sha256:5e6951567308b8aacd8f6bded126ab33a72e7aa584d012a8d0d6283c29d32995",     "sha256:66a1378b08992e4043cf4e391d5b7f52f0d8c4b825dc62a2d87c23ba6ea1dd35",     "sha256:d397a7c12cc95021d41059a44c137000dfcbf12e6ba295ccb647c075e368e39c",     "sha256:8a2c46060eadf56c93467f9445cc49a715a935b0e3b4b439ae8c00fcf3a2157b",     "sha256:70a195ccb5fc7423cc15dd55fb446a19bfd2e1d1a4e5132b79f9433b7d7df750",     "sha256:349fbf13a3797683fe9a2c8355df2a272da391efab8e11c9e083e3c95c094859"   ] } 

let's find list of other images include same base layer (sha256:c854e44a1a5a22c9344c90f68ef6e07fd479e8ce1030fe141e3236887f1a4920):

# cd /var/lib/docker # grep -rl sha256:c854e44a1a5a22c9344c90f68ef6e07fd479e8ce1030fe141e3236887f1a4920 image/overlay2/imagedb/ 

which return like:

image/overlay2/imagedb/content/sha256/d404c11f391c3588ad665fa9ad3f779eb56efc1abbed3cf309b834c824d3c93f image/overlay2/imagedb/content/sha256/dc3313d83519292279466fb5ee7913350d49b8d82f85d537b713ca83d75049e7 image/overlay2/imagedb/content/sha256/dda2981c2844dd1c4a5e004d8bc14633b445f61d23312abba8468251389ed0bc image/overlay2/imagedb/content/sha256/e865d00f6e1e56e7efcfcaf111c52064fc732e68de3eace195492ebf66c7bc74 image/overlay2/imagedb/content/sha256/ea697b65eff199541ec38bbf6ee28085463f0679c9aec3867834f0c14d29d6f4 

that list of image ids include same layer. if want map ids names, need consult image/overlay2/repositories.json, maps names layers, or need parse output of docker images. maybe like:

grep -rl sha256:c854e44a1a5a22c9344c90f68ef6e07fd479e8ce1030fe141e3236887f1a4920 image/overlay2/imagedb/  | while read path;    id=${path##*/}   docker images --no-trunc | grep $id | awk '{print $1, $3}' done 

which on system output:

larsks/mavproxy sha256:0af8d29ecea9dc870ba0a7740d9f23a55aad8d9edacf4f89f6d6b239b58c7829 larsks/apmplanner sha256:5e715eb065698db5444af5ff341d30007d0b67507885f8aab89701ec2c4731fe larsks/qgroundcontrol sha256:2a6265c23c52d1842ac38ea78fde670910dd40d15a8f0f62f60646ad9b209542 sitl sha256:7420866bd587f7b76fbd23b1c15d0a2b9ca5a04fd2d6e442c62a6b25a195b378 cmd/mavproxy sha256:d00e9707a3d8b1cae319ec88b4ccb26f111bb979ec1978cd32147274ab1704e4 cmd/apmplanner sha256:d17cae44602ad335f518276dfcc8a27a251b619f3f9037c55c278eb49d83d74b cmd/qgroundcontrol sha256:d404c11f391c3588ad665fa9ad3f779eb56efc1abbed3cf309b834c824d3c93f mavproxy sha256:dda2981c2844dd1c4a5e004d8bc14633b445f61d23312abba8468251389ed0bc ubuntu sha256:f753707788c5c100f194ce0a73058faae1a457774efcda6c1469544a114f8644 

...which seems reasonable.