transformers-stream-generator

0.0.5last stable release 10 months ago
Complexity Score
Low
Open Issues
N/A
Dependent Projects
17
Weekly Downloadsglobal
41,778

License

  • MIT
    • Yesattribution
    • Permissivelinking
    • Permissivedistribution
    • Permissivemodification
    • Nopatent grant
    • Yesprivate use
    • Permissivesublicensing
    • Notrademark grant

Downloads

Readme

transformers-stream-generator

Description

This is a text generation method which returns a generator, streaming out each token in real-time during inference, based on Huggingface/Transformers.

Web Demo

  • original
  • stream

Installation

pip install transformers-stream-generator

Usage

  1. just add two lines of code before your original code
from transformers_stream_generator import init_stream_support
init_stream_support()
  1. add do_stream=True in model.generate function and keep do_sample=True, then you can get a generator
generator = model.generate(input_ids, do_stream=True, do_sample=True)
for token in generator:
    word = tokenizer.decode(token)
    print(word)

Example

  • run python script example by gpt2
  • run web example by gpt2 and test in client example

Dependencies

No runtime dependency information found for this package.

CVE IssuesActive
0
Scorecards Score
No Data
Test Coverage
No Data
Follows Semver
Yes
Github Stars
96
Dependenciestotal
0
DependenciesOutdated
0
DependenciesDeprecated
0
Threat Modelling
No Data
Repo Audits
No Data

Learn how to distribute transformers-stream-generator in your own private PyPI registry

pip install transformers-stream-generator
Processing...
Done

5 Releases

PyPI on Cloudsmith

Getting started with PyPI on Cloudsmith is fast and easy.