The 2DFS container layer

2dfs.field is a new layer type that we place on top of a regular OCI container image. The field is composed of allotments, each positioned in a specific row and column.

Each allotment can contain one or more files or, for instance, a split of a neural network, a binary file, a driver, or any other large data file.

Instead of creating a new container layer to place a file using the ADD primitive of the Dockerfile, we create a descriptor called 2dfs.json that places each file (or a set of files) into a row and a column of the field.

For example, this 2dfs.json descriptor file:

{
  "allotments": [
 {
      "src": "./file1.txt",
      "dst": "/file1.txt",
      "row": 0,
      "col": 0
 },
 {
      "src": "./file2.txt",
      "dst": "/file2.txt",
      "row": 0,
      "col": 1
 },
 {
      "src": ["./file3.txt", "./file4.txt"],
      "dst": ["/file3.txt", "/file4.txt"],
      "row": 1,
      "col": 0
 },
 {
      "src": "./file5.txt",
      "dst": "/file5.txt",
      "row": 1,
      "col": 1
 },
 {
      "src": "./file6.txt",
      "dst": "/file6.txt",
      "row": 2,
      "col": 1
 }
 ]
}

Will create a field like this:

The benefits?

• If I update file1.txt, the cache will preserve all the other allotemtns, boosting the image build. No cache invalidation happening.
• If I want to retrieve only file1.txt, I can only pull column 0 of row 0, and I will get only that file.
• We can create partitions by drawing a rectangle on the field, and retrieve only the files we need. For example, if I want to retrieve file1.txt and file2.txt, I can pull the rectangle defined from row 0 and column 0 to row 0 and column 1, and I will get only those two files.
• Since the field is a sparse matrix, we can have empty rows and columns which don’t decrease the performance of the image build and retrieval, and don’t consume space in the image.