[SR-1140] [llbuild] Optimize storage of dependency keys #811
Assignees
Labels
Comments
|
Here is a concrete example from llbuild's build of itself. First, notice how much the dependencies cost in size: $ sqlite3_analyzer build/build.db | grep "Page counts.*sepa" -A 7
*** Page counts for all tables and indices separately *************************
RULE_DEPENDENCIES................................. 290 89.0%
RULE_RESULTS...................................... 22 6.7%
RULE_RESULTS_IDX.................................. 12 3.7%
INFO.............................................. 1 0.31%
SQLITE_MASTER..................................... 1 0.31% And a concrete example of the bloat: $ sqlite3 build/build.db .dump | grep DataTypes.h | head -10
INSERT INTO "rule_results" VALUES(210,'../include/llvm/Support/DataTypes.h',X'00000000010000000A000001000000008EC50B0500000000A481000000000000810D000000000000AD804B560000000000000000000000000000000000000000',2,2);
INSERT INTO "rule_dependencies" VALUES(204,'../include/llvm/Support/DataTypes.h');
INSERT INTO "rule_dependencies" VALUES(214,'../include/llvm/Support/DataTypes.h');
INSERT INTO "rule_dependencies" VALUES(227,'../include/llvm/Support/DataTypes.h');
INSERT INTO "rule_dependencies" VALUES(239,'../include/llvm/Support/DataTypes.h');
INSERT INTO "rule_dependencies" VALUES(244,'../include/llvm/Support/DataTypes.h');
INSERT INTO "rule_dependencies" VALUES(248,'../include/llvm/Support/DataTypes.h');
INSERT INTO "rule_dependencies" VALUES(253,'../include/llvm/Support/DataTypes.h');
INSERT INTO "rule_dependencies" VALUES(256,'../include/llvm/Support/DataTypes.h');
... |
|
Done in : #15 Followup: https://bugs.swift.org/browse/SR-1244 |
This issue was closed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Additional Detail from JIRA
md5: d6933d9eaccb9e5c54c85a636c38c3db
Issue Description:
Currently, llbuild stores the full key for all dependencies in the database. On a large project like LLVM, this leads to a lot of redundancy because individual dependencies (e.g., a header file) are repeated a large number of times.
This is particularly redundant because almost all keys have a corresponding rule which embeds the name as well.
We should move to storing the dependency indirectly via an ID to the rule. I forget the exact numbers, but this reduces the dependency database for LLVM by a very large mount (IIRC, around 80%).
The text was updated successfully, but these errors were encountered: