magic-animate、cutie、AnimateAnyone试用

1 技术选型

MagicAnimate: 字节跳动推出AI视频新模型，Magic Animation 卷起来，支持动作捕捉，图片转视频，开源免费，含本地运行

Animate Anyone: 是一个由阿里巴巴智能计算研究院开发的软件，它可以从静态图像生成动态视频，可将任意图像角色动画化。Animate Anyone有以下几个特点。一致性:它可以保持图像中角色的外观和特征的一致性，不会出现失真或闪烁的问题。可控性:它可以让用户提供任何类型的动作或姿势序列，来控制角色的动画效果。通用性:它可以适用于人类角色、动漫角色、卡通角色等，具有广泛的应用范围。

Animate Anyone > MagicAnimate

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
https://www.bilibili.com/video/BV1Ti4y1v7Vx/   多种方法介绍 sd  剪映 无界ai  wink  达摩院视频漫画风格 【 去重 防侵权换脸 换背景。新颖】

[视频抠图换背景Cutie全面演示 ](https://www.bilibili.com/video/BV1wC4y1D7QU/)    [抠图换背景软件CUTIE新增一键绿幕制作功 ](https://www.bilibili.com/video/BV1PW4y1P7uB/)

https://www.bilibili.com/video/BV1si421d7Hc/ 美女舞蹈
https://www.bilibili.com/video/BV1Ti4y1v7Vx/ stablediffusion Mov2Mov插件
https://www.bilibili.com/video/BV1ZZ421e71C/  真人  漫画 2.5d风格
https://www.bilibili.com/video/BV1Dz421f7Ux/  美女帧数变化


magicanimate  图片 结合骨骼动作  生成真人舞蹈视频


Magic-Animate整合包 内含Magic-Animate 与 Video2DensePose

 此软件为二合一软件（内含Magic-Animate 与 Video2DensePose）
Magic-Animate 就是合成数字人动作的主程序。但是需要先生成动作视频，可以通过Video2DensePose

Video2DensePose就是第二个软件，用以生成动作视频


MagicAnimate

magic-animate一键整合包  

2 magic-animate安装

prerequisites: python>=3.8, CUDA>=11.3, and ffmpeg. 建议安装python3.10. cuda11.3 如果windows系统需要注释nvidia-nccl-cu11==2.14.3这行。

下面是官方安装：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
pushd  D:\Software\AI\magic-animate
git clone https://github.com/magic-research/magic-animate.git  #https://github.com/sdbds/magic-animate-for-windows
cd magic-animate



conda env create -f environment.yaml  报错 

或者
conda create -n magic-animate python=3.10  【 python=3.8   python=3.11 ；pip install -r requirements.txt都报错】
conda activate magic-animate
pip install -r requirements.txt

报错：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
Downloading https://developer.download.nvidia.cn/compute/redist/nvidia-nccl-cu11/nvidia-nccl-cu11-2022.5.19.tar.gz (16 kB)
  Preparing metadata (setup.py) ... done
Collecting nvidia-nccl-cu116 (from nvidia-nccl-cu11)
  Downloading nvidia-nccl-cu116-0.0.1.dev5.tar.gz (4.4 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [17 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "C:\Users\Administrator\AppData\Local\Temp\pip-install-5gneo80v\nvidia-nccl-cu116_a62d407656474a539bd65aae9c4eaba8\setup.py", line 152, in <module>
          raise RuntimeError(open("ERROR.txt", "r").read())
      RuntimeError:

###########################################################################################
      The package you are trying to install is only a placeholder project on PyPI.org repository.
      This package is hosted on NVIDIA Python Package Index.

      This package can be installed as:

      $ pip install nvidia-pyindex
      $ pip install nvidia-nccl-cu116
 
      ###########################################################################################
    
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

1
2
3
4
5
6
conda activate magic-animate  能在这个环境中安装CUDA 11.3。只供这个环境使用它。不影响系统原来的cuda环境 

#  在激活的环境中安装 CUDA 11.3

conda install -c anaconda cudatoolkit=11.3
pip install -r requirements.txt

2.1 问题

1 ：nvidia-cudnn-cu11报错

1
2
3
4
ERROR: Ignored the following yanked versions: 8.9.4.19
ERROR: Ignored the following versions that require a different python version: 1.21.2 Requires-Python >=3.7,<3.11; 1.21.3 Requires-Python >=3.7,<3.11; 1.21.4 Requires-Python >=3.7,<3.11; 1.21.5 Requires-Python >=3.7,<3.11; 1.21.6 Requires-Python >=3.7,<3.11
ERROR: Could not find a version that satisfies the requirement nvidia-cudnn-cu11==8.5.0.96 (from versions: 0.0.1.dev5, 8.9.4.25, 8.9.5.29, 9.0.0.312)
ERROR: No matching distribution found for nvidia-cudnn-cu11==8.5.0.96

1
2
3
4
5
6
7
8
9
nvidia-cudnn-cu11==8.5.0.96  8.9.5.29
nvidia-nccl-cu11==2.14.3  #这行注释掉  #linux下安装  nvidia_nccl_cu11-2.21.5   2.21.5

pip install yolk3k
yolk -V nvidia-nccl-cu11

#triton==2.0.0   # 这行注释掉  手动安装 pip install triton-2.0.0-cp310-cp310-win_amd64.whl

另外需要把cuda 12.1 降级到11.3 

安装正常安装完成。

1
2
3
4
5
>pip install triton-2.0.0-cp310-cp310-win_amd64.whl
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
ERROR: triton-2.0.0-cp310-cp310-win_amd64.whl is not a supported wheel on this platform.
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError(SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1135)'))) - skipping
Could not fetch URL https://pypi.ngc.nvidia.com/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.ngc.nvidia.com', port=443): Max retries exceeded with url: /pip/ (Caused by SSLError(SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1135)'))) - skipping

设置临时代理

set http_proxy=http://127.0.0.1:10819 解决

启动：

python -m demo.gradio_animate

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
    PyTorch 2.0.1+cu118 with CUDA 1108 (you have 2.0.1+cpu)
    Python  3.10.11 (you have 3.10.14)
  Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
  Memory-efficient attention, SwiGLU, sparse and more won't be available.
  Set XFORMERS_MORE_DETAILS=1 for more details
D:\Software\AI\magic-animate\magicanimate\pipelines\pipeline_animation.py:43: FutureWarning: Importing `DiffusionPipeline` or `ImagePipelineOutput` from diffusers.pipeline_utils is deprecated. Please import from diffusers.pipelines.pipeline_utils instead.
  from diffusers.pipeline_utils import DiffusionPipeline
Initializing MagicAnimate Pipeline...
Traceback (most recent call last):
  File "D:\Software\miniconda3\envs\magic-animate\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "D:\Software\miniconda3\envs\magic-animate\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "D:\Software\AI\magic-animate\demo\gradio_animate.py", line 19, in <module>
    animator = MagicAnimate()
  File "D:\Software\AI\magic-animate\demo\animate.py", line 58, in __init__
    tokenizer = CLIPTokenizer.from_pretrained(config.pretrained_model_path, subfolder="tokenizer")
  File "D:\Software\miniconda3\envs\magic-animate\lib\site-packages\transformers\tokenization_utils_base.py", line 1838, in from_pretrained
    raise EnvironmentError(
OSError: Can't load tokenizer for 'pretrained_models/stable-diffusion-v1-5'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'pretrained_models/stable-diffusion-v1-5' is the correct path to a directory containing all relevant files for a CLIPTokenizer tokenizer.

torch 报错

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
    PyTorch 2.0.1+cu118 with CUDA 1108 (you have 2.0.1+cpu)
    Python  3.10.11 (you have 3.10.14)
>pip list |findstr torch
pytorch-lightning         2.0.7
torch                     2.0.1
torchmetrics              1.1.0
torchvision               0.15.2
pip uninstall torch  torchvision
nvcc --version cuda用的是12.1版本 python是 3.10.11
pip install torch==2.0.1 -f https://download.pytorch.org/whl/cu118/torch_stable.html
pip install torchvision==0.15.2 -f https://download.pytorch.org/whl/cu118/torch_stable.html

其它：

conda install ffmpeg # 还需单独下载 download ffmpeg-static and export to FFMPEG_PATH https://github.com/BtbN/FFmpeg-Builds/releases pip install ffmpeg-python

conda deactivate

conda env remove –name magic-animate

3 cutie 安装

Python 3.8+
PyTorch 1.12+ and corresponding torchvision

conda create -n cutie python=3.9 conda activate cutie

Clone our repository:

1
git clone https://github.com/hkchengrex/Cutie.git

Install with pip:

1
2
cd Cutie
pip install -e .

3.1 问题

报错：

1
error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/

问题解决。

Download the pretrained models:

1
python scripts/download_models.py

tart the interactive demo with:

1
python interactive_demo.py --video ./examples/example.mp4 --num_objects 1

cutie 使用的是cpu 没用gpu处理。怎样改为使用gpu 处理:

1
2
3
4
5
6
7
8

pip list |findstr  torch
torch                     2.2.2
torchvision               0.17.2
pip uninstall torch  torchvision
nvcc --version cuda用的是12.1版本
pip install torch==2.2.2 -f https://download.pytorch.org/whl/cu121/torch_stable.html
pip install torchvision==0.17.2 -f https://download.pytorch.org/whl/cu121/torch_stable.html

解决。

4 Moore-AnimateAnyone 安装

此存储库再现 AnimateAnyone。为了与原始论文所展示的结果保持一致，我们采用了各种方法和技巧，这些方法和技巧可能与论文和另一种实现有所不同。

值得注意的是，这是一个非常初步的版本，旨在接近 AnimateAnyone 中显示的性能（在我们的测试中约为 80%）

4：troche gpu报错

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
set FFMPEG_PATH=D:\Software\AI\magic-animate\ffmpeg-n6.1\bin

(magic-animate) D:\Software\AI\magic-animate>python scripts/inference.py --inference_config configs/inference/test.yaml
add ffmpeg to path
Loads checkpoint by local backend from path: ./models/dwpose/dw-ll_ucoco_384.pth
Traceback (most recent call last):
  File "D:\Software\AI\magic-animate\scripts\inference.py", line 14, in <module>
    from magic-animate.utils.blending import get_image
  File "D:\Software\AI\magic-animate\magic-animate\utils\blending.py", line 6, in <module>
    fp = FaceParsing()
         ^^^^^^^^^^^^^
  File "D:\Software\AI\magic-animate\magic-animate/utils\face_parsing\__init__.py", line 12, in __init__
    self.net = self.model_init()
               ^^^^^^^^^^^^^^^^^
  File "D:\Software\AI\magic-animate\magic-animate/utils\face_parsing\__init__.py", line 19, in model_init
    net.cuda()
  File "D:\Software\miniconda3\envs\magic-animate\Lib\site-packages\torch\nn\modules\module.py", line 905, in cuda
    return self._apply(lambda t: t.cuda(device))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Software\miniconda3\envs\magic-animate\Lib\site-packages\torch\nn\modules\module.py", line 797, in _apply
    module._apply(fn)
  File "D:\Software\miniconda3\envs\magic-animate\Lib\site-packages\torch\nn\modules\module.py", line 797, in _apply
    module._apply(fn)
  File "D:\Software\miniconda3\envs\magic-animate\Lib\site-packages\torch\nn\modules\module.py", line 797, in _apply
    module._apply(fn)
  File "D:\Software\miniconda3\envs\magic-animate\Lib\site-packages\torch\nn\modules\module.py", line 820, in _apply
    param_applied = fn(param)
                    ^^^^^^^^^
  File "D:\Software\miniconda3\envs\magic-animate\Lib\site-packages\torch\nn\modules\module.py", line 905, in <lambda>
    return self._apply(lambda t: t.cuda(device))
                                 ^^^^^^^^^^^^^^
  File "D:\Software\miniconda3\envs\magic-animate\Lib\site-packages\torch\cuda\__init__.py", line 239, in _lazy_init
    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

(magic-animate) D:\Software\AI\magic-animate>pip list |grep  torch
'grep' 不是内部或外部命令，也不是可运行的程序
或批处理文件。

(magic-animate) D:\Software\AI\magic-animate>pip list | findstr  torch
torch                        2.0.1
torchaudio                   2.0.2
torchvision                  0.15.2

(magic-animate) D:\Software\AI\magic-animate>nvidia-smi
Sat Apr  6 19:54:21 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 546.65                 Driver Version: 546.65       CUDA Version: 12.3     |

卸载 torch 重新安装cuda

卸载当前的torch，torchvision，和torchaudio库：

1
pip uninstall torch torchvision torchaudio

然后，使用如下命令重新安装带有CUDA支持的torch库：

1
2
3
pip install torch==2.0.1 -f https://download.pytorch.org/whl/cu118/torch_stable.html
pip install torchvision==0.15.2 -f https://download.pytorch.org/whl/cu118/torch_stable.html
pip install torchaudio==2.0.2 -f https://download.pytorch.org/whl/cu118/torch_stable.html

最后，你可以再次运行以下Python代码来检查torch库是否安装正确：

1
2
3
4
import torch
  
print(torch.__version__)
print(torch.cuda.is_available())

问题解决.

原因是pip install -r requirements.txt 后，安装mmlab packages 后面的包把原tourch-gpu版本重新安装为了 cpu版本导致cuda报错。

5 整合包优化

整合包下载： magic-animate一键整合包下载。

加入高清codeformer合成。