|
Ada 4.1.1
Fast spec-compliant URL parser
|
Functions | |
| dict[str, Any] | _parse_blob_meta (str text) |
| dict[str, list[int]] | load_blob_sections (Path path=BLOB_PATH) |
| None | write_blob (dict[str, Any] sections, Path path=BLOB_PATH) |
| None | write_mapping_constants (*, str version, int block_bits, int block_size, int block_mask, int sentinel_valid, int sentinel_disallowed, int ignored_idx, int bool_flag, int low_range_end, int high_ignored_start, int high_ignored_end, list[int] stage1, list[int] mixed_data, list[int] bool_words, list[int] utf8_table, Path path=MAPPING_CPP) |
| None | write_id_tables_stub (*, str version, list[list[int]] id_continue, list[list[int]] id_start, Path path=ID_CPP) |
| None | update_mapping_in_blob (list[int] stage1, list[int] mixed_data, list[int] bool_words, list[int] utf8_table) |
| None | update_id_in_blob (list[list[int]] id_continue, list[list[int]] id_start) |
| int | main (list[str]|None argv=None) |
Variables | |
| ROOT = Path(__file__).resolve().parents[1] | |
| str | BLOB_PATH = ROOT / "src" / "table_blob.inc" |
| str | MAPPING_CPP = ROOT / "src" / "mapping_tables.cpp" |
| str | ID_CPP = ROOT / "src" / "id_tables.cpp" |
| list | SECTION_ORDER |
| dict | ALIGN = {"u8": 1, "u16": 2, "u32": 4, "u64": 8} |
| dict | PACK = {"u8": "B", "u16": "H", "u32": "I", "u64": "Q"} |
| dict | WIDTH = {"u8": 1, "u16": 2, "u32": 4, "u64": 8} |
Pack IDNA/Unicode tables into a raw-DEFLATE blob (src/table_blob.inc).
Runtime tables are stored compressed and expanded once by ensure_tables()
(see src/table_store.hpp). This script is the only writer of table_blob.inc.
Typical workflows
-----------------
Regenerate mapping tables from Unicode (IdnaMappingTable.txt) and repack:
python3 scripts/idna_table.py --write
Regenerate identifier tables from DerivedCoreProperties.txt and repack:
python3 scripts/derived_table.py --write
Regenerate normalization and validity (bidi/combining/joining) tables from
UnicodeData.txt, CompositionExclusions.txt and DerivedJoiningType.txt and
repack:
python3 scripts/unicode_data_table.py --write
Keep every generator on the same Unicode version as IdnaMappingTable.txt.
Repack using whatever is already in the blob (no-op rebuild / verify):
python3 scripts/pack_tables.py
Update only some sections programmatically:
from pack_tables import load_blob_sections, write_blob, write_mapping_constants
sections = load_blob_sections()
sections['idna_stage1'] = (...) # list of ints
...
write_blob(sections)
write_mapping_constants(...)
Section layout (all present in the blob)
----------------------------------------
Mapping: idna_stage1, idna_stage2, idna_bool_blocks, idna_utf8_mappings
Norm: decomposition_*, ccc_*, composition_*
Identifier: id_continue_flat, id_start_flat
Validity: dir_start, dir_final, dir_value, combining_flat,
joining_start, joining_final, joining_value
load_blob_sections() skips sections that the current blob does not have yet,
so a generator can add a new section; write_blob() requires every section.
|
protected |
Definition at line 93 of file pack_tables.py.
Referenced by load_blob_sections().
| dict[str, list[int]] load_blob_sections | ( | Path | path = BLOB_PATH | ) |
Load every table section from the current compressed blob.
Definition at line 121 of file pack_tables.py.
References _parse_blob_meta().
Referenced by main(), update_id_in_blob(), unicode_data_table.update_in_blob(), and update_mapping_in_blob().
| int main | ( | list[str] | None | argv = None | ) |
Definition at line 366 of file pack_tables.py.
References load_blob_sections(), main(), and write_blob().
Referenced by main().
| None update_id_in_blob | ( | list[list[int]] | id_continue, |
| list[list[int]] | id_start ) |
Definition at line 351 of file pack_tables.py.
References load_blob_sections(), and write_blob().
Referenced by derived_table.write_derived().
| None update_mapping_in_blob | ( | list[int] | stage1, |
| list[int] | mixed_data, | ||
| list[int] | bool_words, | ||
| list[int] | utf8_table ) |
Definition at line 337 of file pack_tables.py.
References load_blob_sections(), and write_blob().
Referenced by idna_table.write_idna().
| None write_blob | ( | dict[str, Any] | sections, |
| Path | path = BLOB_PATH ) |
Write sections (name -> list[int]) as src/table_blob.inc.
Definition at line 144 of file pack_tables.py.
Referenced by main(), update_id_in_blob(), unicode_data_table.update_in_blob(), and update_mapping_in_blob().
| None write_id_tables_stub | ( | * | , |
| str | version, | ||
| list[list[int]] | id_continue, | ||
| list[list[int]] | id_start, | ||
| Path | path = ID_CPP ) |
Write slim src/id_tables.cpp (arrays live in the blob).
Definition at line 307 of file pack_tables.py.
Referenced by derived_table.write_derived().
| None write_mapping_constants | ( | * | , |
| str | version, | ||
| int | block_bits, | ||
| int | block_size, | ||
| int | block_mask, | ||
| int | sentinel_valid, | ||
| int | sentinel_disallowed, | ||
| int | ignored_idx, | ||
| int | bool_flag, | ||
| int | low_range_end, | ||
| int | high_ignored_start, | ||
| int | high_ignored_end, | ||
| list[int] | stage1, | ||
| list[int] | mixed_data, | ||
| list[int] | bool_words, | ||
| list[int] | utf8_table, | ||
| Path | path = MAPPING_CPP ) |
Write constants-only src/mapping_tables.cpp (arrays live in the blob).
Definition at line 233 of file pack_tables.py.
Referenced by idna_table.write_idna().
| dict pack_tables.ALIGN = {"u8": 1, "u16": 2, "u32": 4, "u64": 8} |
Definition at line 88 of file pack_tables.py.
| str pack_tables.BLOB_PATH = ROOT / "src" / "table_blob.inc" |
Definition at line 59 of file pack_tables.py.
| str pack_tables.ID_CPP = ROOT / "src" / "id_tables.cpp" |
Definition at line 61 of file pack_tables.py.
| str pack_tables.MAPPING_CPP = ROOT / "src" / "mapping_tables.cpp" |
Definition at line 60 of file pack_tables.py.
| dict pack_tables.PACK = {"u8": "B", "u16": "H", "u32": "I", "u64": "Q"} |
Definition at line 89 of file pack_tables.py.
| pack_tables.ROOT = Path(__file__).resolve().parents[1] |
Definition at line 58 of file pack_tables.py.
| list pack_tables.SECTION_ORDER |
Definition at line 64 of file pack_tables.py.
| dict pack_tables.WIDTH = {"u8": 1, "u16": 2, "u32": 4, "u64": 8} |
Definition at line 90 of file pack_tables.py.