Stack install - Symbolic link dest not found...

[Cmdv]

Trying to use tensorflow as part of a project but getting stumped when using stack.

stack 2.3.1 x86_64 hpack-0.34.2
ghc 8.8.3
MacOS 10.15.x (Catalina)

running stack install I get the following error:

Cloning 8cde4d6a277f188ca495d29096a62b36e8534a3f from https://github.com/tensorflow/haskell.git
Unsupported tarball from /private/var/folders/gp/s6jjtfpd2lb5fdnq4b6vqq8m0000gn/T/with-repo-archive67964/foo.tar: Symbolic link dest not found from tensorflow-opgen/third_party to ../third_party/tensorflow, looking for third_party/tensorflow.
This may indicate that the source is a git archive which uses git-annex.
See https://github.com/commercialhaskell/stack/issues/4579 for further information.

here is my setup

stack.yaml:

resolver: lts-15.11
packages:
- .
extra-deps:
  - git: https://github.com/tensorflow/haskell.git
    commit: 8cde4d6a277f188ca495d29096a62b36e8534a3f
    subdirs:
    - tensorflow
    - tensorflow-core-ops
    - tensorflow-logging
    - tensorflow-opgen
    - tensorflow-ops
    - tensorflow-proto
    - tensorflow-mnist
    - tensorflow-mnist-input-data
    - tensorflow-records
    - tensorflow-records-conduit
    - tensorflow-test
  - proto-lens-protobuf-types-0.2.1.0
  - snappy-0.2.0.2
  - snappy-framing-0.1.2
  - tensorflow-logging-0.1.0.0
  - tensorflow-records-0.1.0.0
  - tensorflow-records-conduit-0.1.0.0

extra-lib-dirs:
    - /usr/local/lib
extra-include-dirs:
    - /usr/local/include

package.yaml:

name:                Haskell-image-recognition
version:             0.1.0.0
github:              "githubuser/Haskell-image-recognition"
license:             BSD3
author:              "Author name here"
maintainer:          "example@example.com"
copyright:           "2020 Author name here"

extra-source-files:
- README.md
- ChangeLog.md

description:         Please see the README on GitHub at <https://github.com/githubuser/Haskell-image-recognition#readme>

dependencies:
- base >= 4.7 && < 5
- tensorflow
- tensorflow-opgen
- tensorflow-ops
- tensorflow-logging
- tensorflow-mnist
- proto-lens

library:
  source-dirs: src

executables:
  Haskell-image-recognition-exe:
    main:                Main.hs
    source-dirs:         app
    ghc-options:
    - -threaded
    - -rtsopts
    - -with-rtsopts=-N
    dependencies:
    - Haskell-image-recognition

tests:
  Haskell-image-recognition-test:
    main:                Spec.hs
    source-dirs:         test
    ghc-options:
    - -threaded
    - -rtsopts
    - -with-rtsopts=-N
    dependencies:
    - Haskell-image-recognition

I tried taking a look at https://github.com/commercialhaskell/stack/issues/4579 but couldn't work out how it relates or can be fixed. :)

If more information is required please let me know.

[Cmdv]

also this is the verbose output:

[debug] Running hpack on /Users/cmdv/Code/Haskell-OSS/Haskel-image-recognition/package.yaml
[debug] hpack output unchanged in /Users/cmdv/Code/Haskell-OSS/Haskel-image-recognition/Haskell-image-recognition.cabal
[debug] Loading package from third-party: Repo from https://github.com/tensorflow/haskell.git, commit 8cde4d6a277f188ca495d29096a62b36e8534a3f in subdir tensorflow
[info] Cloning 8cde4d6a277f188ca495d29096a62b36e8534a3f from https://github.com/tensorflow/haskell.git
[debug] Run process: /Users/cmdv/.nix-profile/bin/git clone https://github.com/tensorflow/haskell.git /private/var/folders/gp/s6jjtfpd2lb5fdnq4b6vqq8m0000gn/T/with-repo68160/cloned
[debug] Process finished in 2177ms: /Users/cmdv/.nix-profile/bin/git clone https://github.com/tensorflow/haskell.git /private/var/folders/gp/s6jjtfpd2lb5fdnq4b6vqq8m0000gn/T/with-repo68160/cloned
[debug] Run process within /private/var/folders/gp/s6jjtfpd2lb5fdnq4b6vqq8m0000gn/T/with-repo68160/cloned: /Users/cmdv/.nix-profile/bin/git reset --hard 8cde4d6a277f188ca495d29096a62b36e8534a3f
[debug] Process finished in 252ms: /Users/cmdv/.nix-profile/bin/git reset --hard 8cde4d6a277f188ca495d29096a62b36e8534a3f
[debug] Run process within /private/var/folders/gp/s6jjtfpd2lb5fdnq4b6vqq8m0000gn/T/with-repo68160/cloned: /Users/cmdv/.nix-profile/bin/git submodule update --init --recursive
[debug] Process finished in 63463ms: /Users/cmdv/.nix-profile/bin/git submodule update --init --recursive
[debug] Run process within /private/var/folders/gp/s6jjtfpd2lb5fdnq4b6vqq8m0000gn/T/with-repo68160/cloned: /Users/cmdv/.nix-profile/bin/git -c core.autocrlf=false archive -o /private/var/folders/gp/s6jjtfpd2lb5fdnq4b6vqq8m0000gn/T/with-repo-archive68160/foo.tar HEAD
[debug] Process finished in 109ms: /Users/cmdv/.nix-profile/bin/git -c core.autocrlf=false archive -o /private/var/folders/gp/s6jjtfpd2lb5fdnq4b6vqq8m0000gn/T/with-repo-archive68160/foo.tar HEAD
[debug] Run process within /private/var/folders/gp/s6jjtfpd2lb5fdnq4b6vqq8m0000gn/T/with-repo68160/cloned: /usr/bin/tar --version
[debug] Process finished in 2ms: /usr/bin/tar --version
[debug] Run process within /private/var/folders/gp/s6jjtfpd2lb5fdnq4b6vqq8m0000gn/T/with-repo68160/cloned: /Users/cmdv/.nix-profile/bin/git submodule foreach --recursive "git -c core.autocrlf=false archive --prefix=$displaypath/ -o bar.tar HEAD; rm -rf temp; mkdir temp; mv bar.tar temp/; tar  -C temp -xf temp/bar.tar; rm temp/bar.tar; tar  -C temp -rf /private/var/folders/gp/s6jjtfpd2lb5fdnq4b6vqq8m0000gn/T/with-repo-archive68160/foo.tar . ;"
[debug] Process finished in 4263ms: /Users/cmdv/.nix-profile/bin/git submodule foreach --recursive "git -c core.autocrlf=false archive --prefix=$displaypath/ -o bar.tar HEAD; rm -rf temp; mkdir temp; mv bar.tar temp/; tar  -C temp -xf temp/bar.tar; rm temp/bar.tar; tar  -C temp -rf /private/var/folders/gp/s6jjtfpd2lb5fdnq4b6vqq8m0000gn/T/with-repo-archive68160/foo.tar . ;"
[debug] parseArchive of GZIP-ed tar file: ZlibException (-3)
[error] Unsupported tarball from /private/var/folders/gp/s6jjtfpd2lb5fdnq4b6vqq8m0000gn/T/with-repo-archive68160/foo.tar: Symbolic link dest not found from tensorflow-opgen/third_party to ../third_party/tensorflow, looking for third_party/tensorflow.
This may indicate that the source is a git archive which uses git-annex.

[JamesCarl]

@Cmdv here is a workaround I used in order to get things working.

https://github.com/JamesCarl/deep-learning

Adding the repo as a submodule and removing the symbolic links and replacing it with the files prevented this error.
If you find a simpler fix I'd definitely be interested in knowing what the appropriate method is to deal with issue.

Hope this helps!

[Cmdv]

@JamesCarl ah yes that's a good idea will try that out. It's a shame it has to be bought in like that but I'm by no means a stack pro so couldn't say why this is happening.

I am on macos, wonder if that has something to do with it?!

[Cmdv]

@JamesCarl I added a git submodule by:

> git submodule add https://github.com/tensorflow/haskell.git tensorflow

then pull down all of it's submodules

> git submodule update --init --recursive
> stack install

and it seems I still get the same error 😞

Cloning 568c9b6f03e5d66a25685a776386e2ff50b61aa9 from ./tensorflow
Unsupported tarball from /private/var/folders/gp/s6jjtfpd2lb5fdnq4b6vqq8m0000gn/T/with-repo-archive10543/foo.tar: Symbolic link dest not found from tensorflow-opgen/third_party to ../third_party/tensorflow, looking for third_party/tensorflow.
This may indicate that the source is a git archive which uses git-annex.
See https://github.com/commercialhaskell/stack/issues/4579 for further information.

I'll try pulling down your repo see if I have something wrong else where.

edit: looks like now it's failing on snappy-0.2.0.2 with:

While building package snappy-0.2.0.2 using:
      /Users/cmdv/.stack/setup-exe-cache/x86_64-osx/Cabal-simple_mPHDZzAJ_3.0.1.0_ghc-8.8.3 --builddir=.stack-work/dist/x86_64-osx/Cabal-3.0.1.0 build --ghc-options " -fdiagnostics-color=always"
    Process exited with code: ExitFailure 1

so having to look into that

Had to point to a fork of snappy which fixed the macos build failure.

extra-deps:
  - git: git@github.com:factisresearch/snappy.git
    commit: 91b088b687bad08d358d377aa90dda11a2641d6f

[JamesCarl]

@Cmdv the tensorflow submodule still has symbolic links in each of the folders the fork (https://github.com/JamesCarl/haskell.git) in the repo above has them removed.
You could potentially just add the fork as a submodule to your project.

[Cmdv]

I managed to get your repo to build 🎉 turns out my nix home manager had made all the simlinks etc but it hadn't run this line:

haskell/tools/install_macos_dependencies.sh

Line 39 in 568c9b6

sudo install_name_tool -id libtensorflow.dylib /usr/local/lib/libtensorflow.dylib

so ran it manually and it now seems to install/build fine.

It works, will leave this open as it would be nice to not have to do all this. Unless it's there isn't a way to get around it then feel free to close. 👍