That seems to be a packaging issue with MacPorts. It looks like it's including build dependencies, too. It's pulling TeX Live and the Sphinx Python bindings for formatting documentation. I'm not sure what Perl is for, or why it needs 3 different versions of Python.
The Debian package doesn't require any of those. The Homebrew package has no dependencies at all (and only "python" and "sphinx-doc" as build dependencies).
On Arch Pandoc “only” requires a few dozen or so Haskell packages. Installing it does kind of mess up pacman’s output when updating etc, but that’s a very minor annoyance for a pretty useful tool to me.
The Debian package doesn't require any of those. The Homebrew package has no dependencies at all (and only "python" and "sphinx-doc" as build dependencies).