37

I am trying to install the ernie and tokenizers on Windows, but for both cases it is showing the following error:

ERROR: Command errored out with exit status 1:
   command: 'c:\users\my pc\appdata\local\programs\python\python39\python.exe' 'c:\users\my pc\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py' build_wheel 'C:\Users\MYPC~1\AppData\Local\Temp\tmpyfqm86h3'
       cwd: C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7
  Complete output (38 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build\lib
  creating build\lib\tokenizers
  copying tokenizers\__init__.py -> build\lib\tokenizers
  running build_ext
  running build_rust
      Updating crates.io index
  cargo rustc --lib --manifest-path Cargo.toml --features pyo3/extension-module --target x86_64-pc-windows-msvc --release --verbose -- --crate-type cdylib
  warning: unused manifest key: target.x86_64-apple-darwin.rustflags
     Compiling proc-macro2 v1.0.30
     Compiling unicode-xid v0.2.2
     Compiling syn v1.0.80
     Compiling autocfg v1.0.1
       Running `rustc --crate-name build_script_build --edition=2018 "C:\Users\my pc\.cargo\registry\src\github.com-1ecc6299db9ec823\proc-macro2-1.0.30\build.rs" --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"proc-macro\"" -C metadata=0b72d9facdf9b740 -C extra-filename=-0b72d9facdf9b740 --out-dir "C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\build\proc-macro2-0b72d9facdf9b740" -L "dependency=C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\deps" --cap-lints allow`
       Running `rustc --crate-name unicode_xid "C:\Users\my pc\.cargo\registry\src\github.com-1ecc6299db9ec823\unicode-xid-0.2.2\src\lib.rs" --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" -C metadata=c5404e573edb1c5f -C extra-filename=-c5404e573edb1c5f --out-dir "C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\deps" -L "dependency=C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\deps" --cap-lints allow`
       Running `rustc --crate-name build_script_build --edition=2018 "C:\Users\my pc\.cargo\registry\src\github.com-1ecc6299db9ec823\syn-1.0.80\build.rs" --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"clone-impls\"" --cfg "feature=\"default\"" --cfg "feature=\"derive\"" --cfg "feature=\"extra-traits\"" --cfg "feature=\"full\"" --cfg "feature=\"parsing\"" --cfg "feature=\"printing\"" --cfg "feature=\"proc-macro\"" --cfg "feature=\"quote\"" -C metadata=b3126876d28730d4 -C extra-filename=-b3126876d28730d4 --out-dir "C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\build\syn-b3126876d28730d4" -L "dependency=C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\deps" --cap-lints allow`
       Running `rustc --crate-name autocfg "C:\Users\my pc\.cargo\registry\src\github.com-1ecc6299db9ec823\autocfg-1.0.1\src\lib.rs" --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off -C metadata=077a4bc4a690ef7e -C extra-filename=-077a4bc4a690ef7e --out-dir "C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\deps" -L "dependency=C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\deps" --cap-lints allow`
     Compiling memchr v2.4.1
       Running `rustc --crate-name build_script_build --edition=2018 "C:\Users\my pc\.cargo\registry\src\github.com-1ecc6299db9ec823\memchr-2.4.1\build.rs" --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=420338dd5ea363a5 -C extra-filename=-420338dd5ea363a5 --out-dir "C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\build\memchr-420338dd5ea363a5" -L "dependency=C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\deps" --cap-lints allow`
  error: linker `link.exe` not found
    |
    = note: The system cannot find the file specified. (os error 2)

  note: the msvc targets depend on the msvc linker but `link.exe` was not found

  note: please ensure that VS 2013, VS 2015, VS 2017 or VS 2019 was installed with the Visual C++ option

  error: could not compile `syn` due to previous error

  Caused by:
    process didn't exit successfully: `rustc --crate-name build_script_build --edition=2018 "C:\Users\my pc\.cargo\registry\src\github.com-1ecc6299db9ec823\syn-1.0.80\build.rs" --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"clone-impls\"" --cfg "feature=\"default\"" --cfg "feature=\"derive\"" --cfg "feature=\"extra-traits\"" --cfg "feature=\"full\"" --cfg "feature=\"parsing\"" --cfg "feature=\"printing\"" --cfg "feature=\"proc-macro\"" --cfg "feature=\"quote\"" -C metadata=b3126876d28730d4 -C extra-filename=-b3126876d28730d4 --out-dir "C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\build\syn-b3126876d28730d4" -L "dependency=C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\deps" --cap-lints allow` (exit code: 1)
  warning: build failed, waiting for other jobs to finish...
  error: build failed
  error: cargo failed with code: 101

  ----------------------------------------
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects
Improve this question
3
  • Now that you've included the relevant information, the cause of the error is pretty clear, right? error: linker link.exe not found And the solution is also mentioned in the error message: please ensure that VS 2013, VS 2015, VS 2017 or VS 2019 was installed with the Visual C++ option Commented Oct 19, 2021 at 11:03
  • Sorry the solution was not clear, could you please elaborate. I am using the visual studio code and i installed an extension called visual c++ option in visual studio code but what is vs 2013 and how do i ensure it. Commented Oct 20, 2021 at 16:14
  • You need Visual Studio (the compiler) version 2013 or higher to build this Python library. Visual Studio Code is an IDE. Although the name is similar it's something completely different! You can download Visual Studio here. I recently did it to install another Python library. This answer might be helpful as well. Commented Oct 20, 2021 at 16:20

8 Answers 8

Reset to default
49

It need to install rust compiler in your environment, you can use the command:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

and then you can do it successfully.

Improve this answer
Sign up to request clarification or add additional context in comments.

7 Comments

this almost worked for me except that I had to take the extra step of executing source $HOME/.carge/env after the rust compiler was installed.
There's a small typo -- it should be source $HOME/.cargo/env but yes, this worked for me too.
where do we run this command?
@FurkanGözükara in the terminal once the Rust install is complete. It should tell you to do it as soon as it's done. It even gives you the command. Restarting the shell has the same effect.
It works! But you need to log out and log in back then reactivate the virtualenv.
|
5

After exploring rust, cargo, the wheels, and a lot of stuff I realized that my python version (3.10) was the problem. I created another environment with the 3.8 version while this rust bug is fixed.

Improve this answer

Comments

4

So what worked for me is twofold: -1. Install Rust -2. Install Visual Studio (latest version, 2022) and make sure to install BOTH Python workloads and Desktop Development C++ Workloads.

Improve this answer

Comments

4

It could be because of the version of python. I got the same error when I tried to install the transformers library and I was using python 3.6 but when I upgraded it to python 3.8, the library got installed successfully.

Improve this answer

Comments

3

If you are install Transformer in docker like me.

Try install the transformers in Docker file via this way

Use --no-dependencies tag, it will install without tokenizers as we have a soft dependency on tokenizers

RUN pip install --no-dependencies transformers
Improve this answer

Comments

2

I solved this problem by installing Visual Studio 2019 Community Edition with Python development workloads. I am not sure whether the Python development workloads were essential. Rust compiler has already been installed. (rustup-init.exe from www.rust-lang.org)

Edit: When I uninstalled Python 3.10, Visual Studio, and Rust completely and tried again with only Python 3.10 and Visual Studio Community edition with Python development workloads, it didn't work. But it succeeded after I installed Desktop Development with C++ workloads.

Improve this answer

Comments

1

I have face same issue so I have uninstall my python version 3.13 then install python version 3.12 which it's resolved my issue

Improve this answer

2 Comments

This solved my issue as well. python 3.13 deprecates some functions that older versions of the transformers library is dependent upon. this is a valid, easy step to take.
The comment of @ThomasN. should be included in the answer (if it's correct, which I don't know / can't verify), because simply "install Python 3.12" without any explanation is not a good answer IMHO. But with a proper reason it would be a good answer.
0

I was getting the same error while installing The transformers Library From hugginface, Specifically while installing flax and tensor flow

i Found a Solution To my problem, Leaving it here in case Anyone Else gets Stuck on this.

pip install 'transformers[tf-cpu]'

pip install 'transformers[flax]' (these commands are given on hugginface transformers installation page)

i tried multiple things but nothing i saw on the internet Worked.

then out of Frustration i installed Flax and TF(Tensor-Flow) like normal commands inside a virtual python Env,

pip install tf

pip install flax

and they Worked.

maybe this will help someone.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.