[PR #3519] [MERGED] ci: reduce build time #5751

New issue

Closed

opened 2026-03-23 22:20:09 +00:00 by mirror · 0 comments

mirror commented

2026-03-23 22:20:09 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/AppFlowy-IO/AppFlowy/pull/3519
Author: @zoli
Created: 9/24/2023
Status: ✅ Merged
Merged: 10/23/2023
Merged by: @LucasXu0

Base: main ← Head: ci/reduce-build-time

📝 Commits (10+)

98ae504 ci: reduce rust-lib build time
51c9c6d ci: updated super_clipboard version to use precompiled library
fe5181b ci: add environment variables to test task
e90a3ca build: merging integration test to flutter ci
e6de7bb ci: using artifacts to cache builded files
741c691 ci: fix workflow file indent problem
023af3d ci: zip the artifact before upload
edfd432 ci: fix unzip path
2960b68 ci: limiting workflows for the sake of test
2c985ac ci: removing other workflows

📊 Changes

21 files changed (+426 additions, -100 deletions)

View changed files

📝 .github/workflows/flutter_ci.yaml (+279 -16)
➕ frontend/appflowy_flutter/cargokit_options.yaml (+1 -0)
📝 frontend/appflowy_flutter/packages/appflowy_backend/lib/ffi.dart (+2 -1)
📝 frontend/appflowy_flutter/pubspec.lock (+8 -8)
📝 frontend/appflowy_flutter/pubspec.yaml (+1 -1)
📝 frontend/rust-lib/Cargo.lock (+2 -2)
📝 frontend/rust-lib/flowy-server/Cargo.toml (+1 -1)
📝 frontend/rust-lib/flowy-sqlite/Cargo.toml (+0 -1)
➖ frontend/scripts/code_generation/env/generate_env.cmd (+0 -17)
➖ frontend/scripts/code_generation/env/generate_env.sh (+0 -19)
📝 frontend/scripts/code_generation/flowy_icons/generate_flowy_icons.cmd (+1 -1)
📝 frontend/scripts/code_generation/flowy_icons/generate_flowy_icons.sh (+11 -1)
📝 frontend/scripts/code_generation/freezed/generate_freezed.cmd (+3 -2)
📝 frontend/scripts/code_generation/freezed/generate_freezed.sh (+18 -5)
📝 frontend/scripts/code_generation/generate.cmd (+0 -9)
📝 frontend/scripts/code_generation/generate.sh (+0 -9)
📝 frontend/scripts/code_generation/language_files/generate_language_files.cmd (+1 -1)
📝 frontend/scripts/code_generation/language_files/generate_language_files.sh (+11 -1)
📝 frontend/scripts/docker-buildfiles/Dockerfile (+5 -3)
📝 frontend/scripts/makefile/flutter.toml (+11 -1)

...and 1 more files

📄 Description

This started to reduce Appflowy desktop app build time but in the middle, It became more about enhancements on CI. I have some ideas about reducing user build time but first, it should be decided what is the best option overall.

Changes I made

Common

Flutter build

Building "lib_super_native_extension.so" static library which is used for "super_clipboard" dependency took noticeably time during the flutter build. In the new version, there is an option available for downloading a precompiled static library. I updated to new version and added "cargokit_options.yaml" file to make downloading precompiled libraries as default option.
Time-saving: this saves about 1:20

Code generation

First I removed generate_env script because it was doing the same as generate freezed script (am I right?).
Second in bash scripts, I made a flag available for not running flutter packages pub get each time. Only run it once before running "generate.sh". For "generate.cmd" I didn't know how to make a flag available so I just commented out the flutter packages pub get.
Time-saving: this saves about 1:50

Flutter CI

The changes I'll mention made it possible to merge the whole flutter ci into one workflow. And run flutter build, unit tests, integration tests job parallel.

Cache rust build and code generation

Created an artifact from libdart_ffi and code generations so to eliminate the need for running "appflowy-core-dev" and code generation task for each of flutter unit tests and integration tests.
I added "dart_unit_test_no_build" task to make it possible just copy the built dart ffi static library to ".sandbox" directory which is used for unit tests.
Note, I couldn't do this for mac os unit tests because somehow it wasn't possible to make macos use ffi ".a" file it had to be ".dylib" file. So for macos unit tests, cargo make runs two times.
Also added "appflowy-make-product-dev" task to enable running the flutter build part separate from the "appflowy-core-dev" and code generation task.

Use prebuilt dockscript binary

Used "taiki-e/install-action" github action to install duckscipt_cli and avoid compiling it each time.
Time-saving: this saves about 2:20 for each job.

Docker CI

Use precompiled rocksdb, zstd static libraries

I will discuss this in more details later. But as Appflowy dockerfile is based on Arch Linux image. I changed it so it wouldn't compile librocksdb, libzstd. Couldn't make it to use ssl precompiled static library because we directly ask it to use vendored one as a cargo feature.
Time-saving: docker ci workflow is about 19 minutes faster now

What are other options to reduce build time, especially for first-time users?

Building from scratch still takes a lot of time. Just take a look Docker CI workflow it takes about 43 minutes on average and 35 minutes of it is "appflowy-linux" task. I think the dev version even takes a bit more.
Here is how much time each of "appflowy-linux" sub tasks takes (these are before my changes):

24 minutes "appflowy-core-release" which is cargo build
7.5 minutes "code_generation"
3 minutes flutter build

Cargo build

If you run cargo build with timing flag (cargo build —timings) you will see that about 65 percent (14-15 minutes) of the 24 minutes is spend on building librocksdb, libzstd, libssl. Providing them as pre-compiled libraries in the repo or in a web server and downloading them would do the job. Or at least putting it in documentation like here which has done it. They are available as package in Arch Linux and probably Arch based distros. But the version on Ubuntu packages doesn't match what Appflowy requires (Openssl 3.1.2, ZSTD 1.5.5, Rocksdb 8.1.1). I don't know how is the case for Windows and MacOS.
Nathan suggested creating a precompiled library for collab-* which also Rocksdb and ZSTD are from there. Cargo doesn't support precompiled dependencies as far as I know (look here). I think the only available option for this is creating a static library and maintaining some c code to define specified functionalities.

Code generation

Removing generate_env script and redundant flutter packages pub get saved about 2 minutes. But I suggest another option for better performance.
Why not commit generated freezed files? Also ".g.dart" files. This saves another 3-4 minutes. And in CI we can check if there is a not committed change in generated files. We just need to run "code_generation" task and check it with git status that the code does not get dirty. So if the CI passes we will be sure the generated files are rightly generated from the latest changes.

Flutter build

I don't see any other enhancement than I made for using the precompiled library for super_clipboard dependency which reduced the time by 1:20.

PR Checklist

My code adheres to the AppFlowy Style Guide
I've listed at least one issue that this PR fixes in the description above.
I've added a test(s) to validate changes in this PR, or this PR only contains semantic changes.
All existing tests are passing.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/AppFlowy-IO/AppFlowy/pull/3519 **Author:** [@zoli](https://github.com/zoli) **Created:** 9/24/2023 **Status:** ✅ Merged **Merged:** 10/23/2023 **Merged by:** [@LucasXu0](https://github.com/LucasXu0) **Base:** `main` ← **Head:** `ci/reduce-build-time` --- ### 📝 Commits (10+) - [`98ae504`](https://github.com/AppFlowy-IO/AppFlowy/commit/98ae504b90d91644cafc4524a49f3a22e298f101) ci: reduce rust-lib build time - [`51c9c6d`](https://github.com/AppFlowy-IO/AppFlowy/commit/51c9c6d7f6ef204843c0900c9c0d82978439ce6f) ci: updated super_clipboard version to use precompiled library - [`fe5181b`](https://github.com/AppFlowy-IO/AppFlowy/commit/fe5181b650203899183bcfbb9d810a43406f6366) ci: add environment variables to test task - [`e90a3ca`](https://github.com/AppFlowy-IO/AppFlowy/commit/e90a3ca8ecfe62c383b614fbacb158b94f3b079b) build: merging integration test to flutter ci - [`e6de7bb`](https://github.com/AppFlowy-IO/AppFlowy/commit/e6de7bb91e15b26edc8b2ddce804a7ffc11db634) ci: using artifacts to cache builded files - [`741c691`](https://github.com/AppFlowy-IO/AppFlowy/commit/741c691b155ca666cad82574e200792b6673059d) ci: fix workflow file indent problem - [`023af3d`](https://github.com/AppFlowy-IO/AppFlowy/commit/023af3d9b3fc2ae93d90415714c2c10af7bdad22) ci: zip the artifact before upload - [`edfd432`](https://github.com/AppFlowy-IO/AppFlowy/commit/edfd4322ed28632460100c0dfe0bc5634b01f962) ci: fix unzip path - [`2960b68`](https://github.com/AppFlowy-IO/AppFlowy/commit/2960b682bd0a87d5341dc8542a5361cd646e0f1c) ci: limiting workflows for the sake of test - [`2c985ac`](https://github.com/AppFlowy-IO/AppFlowy/commit/2c985acec5aac65fd5e26e24bcba898437b75ea3) ci: removing other workflows ### 📊 Changes **21 files changed** (+426 additions, -100 deletions) <details> <summary>View changed files</summary> 📝 `.github/workflows/flutter_ci.yaml` (+279 -16) ➕ `frontend/appflowy_flutter/cargokit_options.yaml` (+1 -0) 📝 `frontend/appflowy_flutter/packages/appflowy_backend/lib/ffi.dart` (+2 -1) 📝 `frontend/appflowy_flutter/pubspec.lock` (+8 -8) 📝 `frontend/appflowy_flutter/pubspec.yaml` (+1 -1) 📝 `frontend/rust-lib/Cargo.lock` (+2 -2) 📝 `frontend/rust-lib/flowy-server/Cargo.toml` (+1 -1) 📝 `frontend/rust-lib/flowy-sqlite/Cargo.toml` (+0 -1) ➖ `frontend/scripts/code_generation/env/generate_env.cmd` (+0 -17) ➖ `frontend/scripts/code_generation/env/generate_env.sh` (+0 -19) 📝 `frontend/scripts/code_generation/flowy_icons/generate_flowy_icons.cmd` (+1 -1) 📝 `frontend/scripts/code_generation/flowy_icons/generate_flowy_icons.sh` (+11 -1) 📝 `frontend/scripts/code_generation/freezed/generate_freezed.cmd` (+3 -2) 📝 `frontend/scripts/code_generation/freezed/generate_freezed.sh` (+18 -5) 📝 `frontend/scripts/code_generation/generate.cmd` (+0 -9) 📝 `frontend/scripts/code_generation/generate.sh` (+0 -9) 📝 `frontend/scripts/code_generation/language_files/generate_language_files.cmd` (+1 -1) 📝 `frontend/scripts/code_generation/language_files/generate_language_files.sh` (+11 -1) 📝 `frontend/scripts/docker-buildfiles/Dockerfile` (+5 -3) 📝 `frontend/scripts/makefile/flutter.toml` (+11 -1) _...and 1 more files_ </details> ### 📄 Description This started to reduce Appflowy desktop app build time but in the middle, It became more about enhancements on CI. I have some ideas about reducing user build time but first, it should be decided what is the best option overall. ## Changes I made ### Common #### Flutter build Building "lib_super_native_extension.so" static library which is used for "super_clipboard" dependency took noticeably time during the flutter build. In the new version, there is an option available for downloading a precompiled static library. I updated to new version and added "cargokit_options.yaml" file to make downloading precompiled libraries as default option. Time-saving: this saves about 1:20 #### Code generation First I removed generate_env script because it was doing the same as generate freezed script (am I right?). Second in bash scripts, I made a flag available for not running flutter packages pub get each time. Only run it once before running "generate.sh". For "generate.cmd" I didn't know how to make a flag available so I just commented out the flutter packages pub get. Time-saving: this saves about 1:50 ### Flutter CI The changes I'll mention made it possible to merge the whole flutter ci into one workflow. And run flutter build, unit tests, integration tests job parallel. #### Cache rust build and code generation Created an artifact from libdart_ffi and code generations so to eliminate the need for running "appflowy-core-dev" and code generation task for each of flutter unit tests and integration tests. I added "dart_unit_test_no_build" task to make it possible just copy the built dart ffi static library to ".sandbox" directory which is used for unit tests. Note, I couldn't do this for mac os unit tests because somehow it wasn't possible to make macos use ffi ".a" file it had to be ".dylib" file. So for macos unit tests, cargo make runs two times. Also added "appflowy-make-product-dev" task to enable running the flutter build part separate from the "appflowy-core-dev" and code generation task. #### Use prebuilt dockscript binary Used "taiki-e/install-action" github action to install duckscipt_cli and avoid compiling it each time. Time-saving: this saves about 2:20 for each job. ### Docker CI #### Use precompiled rocksdb, zstd static libraries I will discuss this in more details later. But as Appflowy dockerfile is based on Arch Linux image. I changed it so it wouldn't compile librocksdb, libzstd. Couldn't make it to use ssl precompiled static library because we directly ask it to use vendored one as a cargo feature. Time-saving: docker ci workflow is about 19 minutes faster now ## What are other options to reduce build time, especially for first-time users? Building from scratch still takes a lot of time. Just take a look [Docker CI](https://github.com/AppFlowy-IO/AppFlowy/actions/workflows/docker_ci.yml?query=branch%3Amain+is%3Asuccess+event%3Apush) workflow it takes about 43 minutes on average and 35 minutes of it is "appflowy-linux" task. I think the dev version even takes a bit more. Here is how much time each of "appflowy-linux" sub tasks takes (these are before my changes): 1. 24 minutes "appflowy-core-release" which is cargo build 2. 7.5 minutes "code_generation" 3. 3 minutes flutter build ### Cargo build If you run cargo build with timing flag (cargo build —timings) you will see that about 65 percent (14-15 minutes) of the 24 minutes is spend on building librocksdb, libzstd, libssl. Providing them as pre-compiled libraries in the repo or in a web server and downloading them would do the job. Or at least putting it in documentation like [here](https://near.github.io/nearcore/practices/fast_builds.html#prebuilt-rocksdb) which has done it. They are available as package in Arch Linux and probably Arch based distros. But the version on Ubuntu packages doesn't match what Appflowy requires (Openssl 3.1.2, ZSTD 1.5.5, Rocksdb 8.1.1). I don't know how is the case for Windows and MacOS. Nathan suggested creating a precompiled library for collab-* which also Rocksdb and ZSTD are from there. Cargo doesn't support precompiled dependencies as far as I know (look [here](https://github.com/rust-lang/cargo/issues/1139)). I think the only available option for this is creating a static library and maintaining some c code to define specified functionalities. ### Code generation Removing generate_env script and redundant flutter packages pub get saved about 2 minutes. But I suggest another option for better performance. Why not commit generated freezed files? Also ".g.dart" files. This saves another 3-4 minutes. And in CI we can check if there is a not committed change in generated files. We just need to run "code_generation" task and check it with git status that the code does not get dirty. So if the CI passes we will be sure the generated files are rightly generated from the latest changes. ### Flutter build I don't see any other enhancement than I made for using the precompiled library for super_clipboard dependency which reduced the time by 1:20. #### PR Checklist - [x] My code adheres to the [AppFlowy Style Guide](https://appflowy.gitbook.io/docs/essential-documentation/contribute-to-appflowy/software-contributions/submitting-code/style-guides) - [ ] I've listed at least one issue that this PR fixes in the description above. - [ ] I've added a test(s) to validate changes in this PR, or this PR only contains semantic changes. - [x] All existing tests are passing. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>