Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Based on what?

Every CI system I've ever seem has pulled dependencies in from the network.



   git clone --recursive --branch "$commitid" "$repourl" "$repodir"
   img="$(docker build --network=none -f "$dockerfile" "$repodir")"
   docker run --rm -ti --network=none "$img"
Sure, CI pulls in from the network... but execution occurs without network.


Now try this with eg. OpenCV, or ONNX, or tflite, or a million other packages that try to download additional information at compile time.


Well there's you're problem; you're using packages written and maintained in an unsustainably insecure way.

It's been my experience that usually pull requests which make a library capable of being built offline are welcome. Usually. So get off your duff and get to fixing the problems in the free packages you use.


Can you explain wtf is happening here?


After the CI system (with network access) pulls down the code, including submodules, this code is then placed into a container with no network access to perform the actual build.


Is such a clean separation possible? I’ve seen some crazy things…


For well-designed projects it's not only possible but required.


I'd assumed some familiarity with common CI systems (and assumed the commands would be tailored to your use case). Let me walk it into some more depth.

First:

> git clone --recursive --branch "$commitid" "$repourl" "$repodir"

The `git clone` will take your git repository URL as the $repourl variable. It will also take your commit id (commit hash or tagged version which a pull request points to) as the $commitid variable (`--branch $commitid`). It will also take a $repodir variable which points to the directory that will contain the contents of the cloned git repository and already checked out at the commit id specified. It will do so recursively (`--recursive`): if there are submodules then they will also automatically be cloned.

This of course assumes that you're cloning a public repository and/or that any credentials required have already been set up (see `man git-config`).

Then:

> img="$(docker build --network=none -f "$dockerfile" "$repodir")"

Okay so this is sort've broken: you'd need a few more parameters to `docker build` to get it to work "right". But as-is, `docker build` usually has network access so `--network=none` will specify that the build process will not have access to the network. I hope your build system doesn't automatically download dependencies because that will fail (and also suggests that the build system may be susceptible to attack). You specify a dockerfile to build using `-f "$dockerfile"`. Finally, you specify the build context using "$repodir" -- and that assumes that your whole git repository should be available to the dockerfile.

However, `docker build` will write a lot more than just the image name to standard output and so this is where some customization would need to occur. Suffice to say that you can use `--quiet` if that's all you want; I do prefer to see the output because it normally contains intermediate image names useful for debugging the dockerfile.

Finally:

> docker run --rm -ti --network=none "$img"

Finally, it runs the built image in a new container with an auto-generated name. `-ti` here is wrong: it will attach a standard input/output terminal and so if it drops you into an interactive program (such as bash) then it could hang the CI process. But you can remove that. It also assumes that your dockerfile correctly specifies ENTRYPOINT and/or CMD. When the container has exited then the container will automatically be removed (--rm) -- usually they linger around and pollute your docker host. Finally, the --network=none also ensures that your container does not have network access so your unit tests should also be capable of running without the network or else they will fail. You could use `--volume` to specify a volume with data files if you need them. You might also want to look at `--user` if you don't want your container to have root privileges...

And of course if you want integration tests with other containers then you should create a dedicated docker network and specify its alias with `--network`: see `man docker-network-create`; you can use `docker network create -d internal` to create a network which shouldn't let containers out.

Does that answer your question?


Yes, thank you for the detailed explanation.


Oy, when I was starting we had it rough. Nexus filtering on every dependency repo, and that was if the proxy was up!

Luxury! When I were a youngun our devnet was airgapped and we had to sneakernet dependencies to it. Using vhs tape on a 9track spool. Because real tape was too expensive.

More seriously I've seen jenkins and gitlab pipelines with only access to sneakernet maintained mirrors.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: